The Importance of Context in Very Low Resource Language Modeling

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    30 Downloads (Pure)


    This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model’s performance in the low-resource setting, finding that limiting the model’s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.
    Original languageEnglish
    Title of host publicationProceedings of the 18th International Conference on Natural Language Processing (ICON)
    EditorsSivaji Bandyopadhyay, Sobha Lalitha Devi, Pushpak Bhattacharyya
    PublisherAssociation for Computational Linguistics (ACL)
    Number of pages7
    Publication statusPublished - Dec-2021


    Dive into the research topics of 'The Importance of Context in Very Low Resource Language Modeling'. Together they form a unique fingerprint.

    Cite this