UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations

Gabriele Sarti

UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

This work describes a self-supervised data augmentation approach used to improve learning models' performances when only a moderate amount of labeled data is available. Multiple copies of the original model are initially trained on the downstream task. Their predictions are then used to annotate a large set of unlabeled examples. Finally, multi-task training is performed on the parallel annotations of the resulting training set, and final scores are obtained by averaging annotator-specific head predictions. Neural language models are fine-tuned using this procedure in the context of the AcCompl-it shared task at EVALITA 2020, obtaining considerable improvements in prediction quality.

Original language	English
Title of host publication	Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020)
Editors	Valerio Basile, Danilo Croce, Maria Di Maro, Lucia Passaro
Place of Publication	Online
Publisher	CEUR Workshop Proceedings (CEUR-WS.org)
Publication status	Published - 17-Dec-2020
Externally published	Yes
Event	Evaluation Campaign of Natural Language Processing and Speech Tools for Italian - Online Duration: 17-Dec-2020 → … Conference number: 7

Workshop

Workshop	Evaluation Campaign of Natural Language Processing and Speech Tools for Italian
Abbreviated title	EVALITA 2020
Period	17/12/2020 → …

Keywords

natural language processing
deep learning
self-training
neural language models
multi-task learning
linguistic complexity
linguistic acceptability

Access to Document

http://ceur-ws.org/Vol-2765/paper92.pdfLicence: CC BY

Handle.net

Persistent link

Cite this

Sarti, G. (2020). UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations. In V. Basile, D. Croce, M. Di Maro, & L. Passaro (Eds.), Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020) CEUR Workshop Proceedings (CEUR-WS.org). http://ceur-ws.org/Vol-2765/paper92.pdf

Sarti, Gabriele. / UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations. Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020). editor / Valerio Basile ; Danilo Croce ; Maria Di Maro ; Lucia Passaro. Online : CEUR Workshop Proceedings (CEUR-WS.org), 2020.

@inproceedings{97eb182b08d84a5684400027ddd3cd79,

title = "UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations",

abstract = "This work describes a self-supervised data augmentation approach used to improve learning models' performances when only a moderate amount of labeled data is available. Multiple copies of the original model are initially trained on the downstream task. Their predictions are then used to annotate a large set of unlabeled examples. Finally, multi-task training is performed on the parallel annotations of the resulting training set, and final scores are obtained by averaging annotator-specific head predictions. Neural language models are fine-tuned using this procedure in the context of the AcCompl-it shared task at EVALITA 2020, obtaining considerable improvements in prediction quality.",

keywords = "natural language processing, deep learning, self-training, neural language models, multi-task learning, linguistic complexity, linguistic acceptability",

author = "Gabriele Sarti",

year = "2020",

month = dec,

day = "17",

language = "English",

editor = "Valerio Basile and Danilo Croce and {Di Maro}, Maria and Lucia Passaro",

booktitle = "Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020)",

publisher = "CEUR Workshop Proceedings (CEUR-WS.org)",

note = "Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, EVALITA 2020 ; Conference date: 17-12-2020",

}

Sarti, G 2020, UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations. in V Basile, D Croce, M Di Maro & L Passaro (eds), Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020). CEUR Workshop Proceedings (CEUR-WS.org), Online, Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, 17/12/2020. <http://ceur-ws.org/Vol-2765/paper92.pdf>

UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations. / Sarti, Gabriele.
Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020). ed. / Valerio Basile; Danilo Croce; Maria Di Maro; Lucia Passaro. Online: CEUR Workshop Proceedings (CEUR-WS.org), 2020.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations

AU - Sarti, Gabriele

N1 - Conference code: 7

PY - 2020/12/17

Y1 - 2020/12/17

N2 - This work describes a self-supervised data augmentation approach used to improve learning models' performances when only a moderate amount of labeled data is available. Multiple copies of the original model are initially trained on the downstream task. Their predictions are then used to annotate a large set of unlabeled examples. Finally, multi-task training is performed on the parallel annotations of the resulting training set, and final scores are obtained by averaging annotator-specific head predictions. Neural language models are fine-tuned using this procedure in the context of the AcCompl-it shared task at EVALITA 2020, obtaining considerable improvements in prediction quality.

AB - This work describes a self-supervised data augmentation approach used to improve learning models' performances when only a moderate amount of labeled data is available. Multiple copies of the original model are initially trained on the downstream task. Their predictions are then used to annotate a large set of unlabeled examples. Finally, multi-task training is performed on the parallel annotations of the resulting training set, and final scores are obtained by averaging annotator-specific head predictions. Neural language models are fine-tuned using this procedure in the context of the AcCompl-it shared task at EVALITA 2020, obtaining considerable improvements in prediction quality.

KW - natural language processing

KW - deep learning

KW - self-training

KW - neural language models

KW - multi-task learning

KW - linguistic complexity

KW - linguistic acceptability

UR - http://ceur-ws.org/Vol-2765/paper92.pdf

M3 - Conference contribution

BT - Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020)

A2 - Basile, Valerio

A2 - Croce, Danilo

A2 - Di Maro, Maria

A2 - Passaro, Lucia

PB - CEUR Workshop Proceedings (CEUR-WS.org)

CY - Online

T2 - Evaluation Campaign of Natural Language Processing and Speech Tools for Italian

Y2 - 17 December 2020

ER -

Sarti G. UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations. In Basile V, Croce D, Di Maro M, Passaro L, editors, Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020). Online: CEUR Workshop Proceedings (CEUR-WS.org). 2020

UmBERTo-MTSA@ AcCompl-It: Improving Complexity and Acceptability Prediction with Multi-task Learning on Self-Supervised Annotations

Abstract

Workshop

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this