Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models

Daniela Occhipinti, Michele Marchi, Irene Mondella, Huiyuan Lai, Felice Dell'Orletta, Malvina Nissim, Marco Guerini

OnderzoeksoutputAcademicpeer review

37 Downloads (Pure)

Samenvatting

Automatic methods for generating and gathering linguistic data have proven effective for fine-tuning Language Models (LMs) in languages less resourced than English. Still, while there has been emphasis on data quantity, less attention has been given to its quality. In this work, we investigate the impact of human intervention on machine-generated data when fine-tuning dialogical models. In particular, we study (1) whether post-edited dialogues exhibit higher perceived quality compared to the originals that were automatically generated; (2) whether fine-tuning with post-edited dialogues results in noticeable differences in the generated outputs; and (3) whether post-edited dialogues influence the outcomes when considering the parameter size of the LMs. To this end we created HED-IT, a large-scale dataset where machine-generated dialogues are paired with the version post-edited by humans. Using both the edited and unedited portions of HED-IT, we fine-tuned three different sizes of an LM. Results from both human and automatic evaluation show that the different quality of training data is clearly perceived and it has an impact also on the models trained on such data. Additionally, our findings indicate that larger models are less sensitive to data quality, whereas this has a crucial impact on smaller models. These results enhance our comprehension of the impact of human intervention on training data in the development of high-quality LMs.

Originele taal-2English
TitelFindings of the Association for Computational Linguistics ACL 2024
RedacteurenLun-Wei Ku, Andre Martins, Vivek Srikumar
UitgeverijAssociation for Computational Linguistics, ACL Anthology
Pagina's11892-11907
Aantal pagina's16
ISBN van elektronische versie9798891760998
DOI's
StatusPublished - 2024
EvenementFindings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Hybrid, Bangkok, Thailand
Duur: 11-aug.-202416-aug.-2024

Publicatie series

NaamProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN van geprinte versie0736-587X

Conference

ConferenceFindings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024
Land/RegioThailand
StadHybrid, Bangkok
Periode11/08/202416/08/2024

Vingerafdruk

Duik in de onderzoeksthema's van 'Fine-tuning with HED-IT: The impact of human post-editing for dialogical language models'. Samen vormen ze een unieke vingerafdruk.

Citeer dit