The Importance of Context in Very Low Resource Language Modeling

Lukas Edman; Antonio Toral Ruiz; Gertjan van Noord

The Importance of Context in Very Low Resource Language Modeling

Lukas Edman, Antonio Toral Ruiz, Gertjan van Noord

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

46 Downloads (Pure)

Abstract

This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model’s performance in the low-resource setting, finding that limiting the model’s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.

Original language	English
Title of host publication	Proceedings of the 18th International Conference on Natural Language Processing (ICON)
Editors	Sivaji Bandyopadhyay, Sobha Lalitha Devi, Pushpak Bhattacharyya
Publisher	Association for Computational Linguistics (ACL)
Pages	86-92
Number of pages	7
Publication status	Published - Dec-2021

Access to Document

The Importance of Context in Very Low Resource Language ModelingFinal publisher's version, 220 KBLicence: CC BY

Handle.net

Persistent link

Cite this

@inproceedings{f6ed73848c3c4914a771ae3a5ad6dfd5,

title = "The Importance of Context in Very Low Resource Language Modeling",

abstract = "This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model{\textquoteright}s performance in the low-resource setting, finding that limiting the model{\textquoteright}s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.",

author = "Lukas Edman and {Toral Ruiz}, Antonio and {van Noord}, Gertjan",

note = "The proceedings of the conference will be made available soon in the ACL Anthology, https://aclanthology.org/",

year = "2021",

month = dec,

language = "English",

pages = "86--92",

editor = "Sivaji Bandyopadhyay and Devi, {Sobha Lalitha} and Pushpak Bhattacharyya",

booktitle = "Proceedings of the 18th International Conference on Natural Language Processing (ICON)",

publisher = "Association for Computational Linguistics (ACL)",

}

The Importance of Context in Very Low Resource Language Modeling. / Edman, Lukas; Toral Ruiz, Antonio; van Noord, Gertjan.
Proceedings of the 18th International Conference on Natural Language Processing (ICON). ed. / Sivaji Bandyopadhyay; Sobha Lalitha Devi; Pushpak Bhattacharyya. Association for Computational Linguistics (ACL), 2021. p. 86-92.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - The Importance of Context in Very Low Resource Language Modeling

AU - Edman, Lukas

AU - Toral Ruiz, Antonio

AU - van Noord, Gertjan

N1 - The proceedings of the conference will be made available soon in the ACL Anthology, https://aclanthology.org/

PY - 2021/12

Y1 - 2021/12

N2 - This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model’s performance in the low-resource setting, finding that limiting the model’s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.

AB - This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model’s performance in the low-resource setting, finding that limiting the model’s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.

M3 - Conference contribution

SP - 86

EP - 92

BT - Proceedings of the 18th International Conference on Natural Language Processing (ICON)

A2 - Bandyopadhyay, Sivaji

A2 - Devi, Sobha Lalitha

A2 - Bhattacharyya, Pushpak

PB - Association for Computational Linguistics (ACL)

ER -

The Importance of Context in Very Low Resource Language Modeling

Abstract

Access to Document

Handle.net

Fingerprint

Cite this