Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution

Lukas Edman; Antonio Toral Ruiz; Gertjan Noord, van

Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution

Lukas Edman^*, Antonio Toral Ruiz, Gertjan Noord, van

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

6 Citations (Scopus)

2002 Downloads (Pure)

Abstract

Unsupervised Machine Translation hasbeen advancing our ability to translatewithout parallel data, but state-of-the-artmethods assume an abundance of mono-lingual data. This paper investigates thescenario where monolingual data is lim-ited as well, finding that current unsuper-vised methods suffer in performance un-der this stricter setting. We find that theperformance loss originates from the poorquality of the pretrained monolingual em-beddings, and we propose using linguis-tic information in the embedding train-ing scheme. To support this, we look attwo linguistic features that may help im-prove alignment quality: dependency in-formation and sub-word information. Us-ing dependency-based embeddings resultsin a complementary word representationwhich offers a boost in performance ofaround 1.5 BLEU points compared to stan-dardWORD2VECwhen monolingual datais limited to 1 million sentences per lan-guage. We also find that the inclusion ofsub-word information is crucial to improv-ing the quality of the embeddings

Original language	English
Title of host publication	Proceedings of the 22nd Annual Conference of the European Association for Machine Translation
Editors	André Martins, Helena Moniz, Sara Fumega, Bruno Martins, Fernando Batista, Luisa Coheur, Carla Parra, Isabel Trancoso, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof, Mary Nurminen, Lena Marg, Mikel L. Forcada
Place of Publication	Lisbon
Pages	81-90
Number of pages	10
ISBN (Electronic)	978-989-33-0589-8
Publication status	Published - 2020
Event	22nd Annual Conference of the European Association for Machine Translation - Online Conference Duration: 3-Nov-2020 → 5-Nov-2020 https://eamt2020.inesc-id.pt/

Conference

Conference	22nd Annual Conference of the European Association for Machine Translation
Abbreviated title	EAMT 2020
Period	03/11/2020 → 05/11/2020
Internet address	https://eamt2020.inesc-id.pt/

Access to Document

proceedings-eamt2020Final publisher's version, 18.2 MBLicence: CC BY-NC-ND

Handle.net

Persistent link

Cite this

Edman, L., Toral Ruiz, A., & Noord, van, G. (2020). Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution. In A. Martins, H. Moniz, S. Fumega, B. Martins, F. Batista, L. Coheur, C. Parra, I. Trancoso, M. Turchi, A. Bisazza, J. Moorkens, A. Guerberof, M. Nurminen, L. Marg, & M. L. Forcada (Eds.), Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (pp. 81-90).

Edman, Lukas ; Toral Ruiz, Antonio ; Noord, van, Gertjan. / Low-Resource Unsupervised NMT : Diagnosing the Problem and Providing a Linguistically Motivated Solution. Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. editor / André Martins ; Helena Moniz ; Sara Fumega ; Bruno Martins ; Fernando Batista ; Luisa Coheur ; Carla Parra ; Isabel Trancoso ; Marco Turchi ; Arianna Bisazza ; Joss Moorkens ; Ana Guerberof ; Mary Nurminen ; Lena Marg ; Mikel L. Forcada. Lisbon, 2020. pp. 81-90

@inproceedings{680bb04442d64f279a25236912b32c17,

title = "Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution",

abstract = "Unsupervised Machine Translation hasbeen advancing our ability to translatewithout parallel data, but state-of-the-artmethods assume an abundance of mono-lingual data. This paper investigates thescenario where monolingual data is lim-ited as well, finding that current unsuper-vised methods suffer in performance un-der this stricter setting. We find that theperformance loss originates from the poorquality of the pretrained monolingual em-beddings, and we propose using linguis-tic information in the embedding train-ing scheme. To support this, we look attwo linguistic features that may help im-prove alignment quality: dependency in-formation and sub-word information. Us-ing dependency-based embeddings resultsin a complementary word representationwhich offers a boost in performance ofaround 1.5 BLEU points compared to stan-dardWORD2VECwhen monolingual datais limited to 1 million sentences per lan-guage. We also find that the inclusion ofsub-word information is crucial to improv-ing the quality of the embeddings",

author = "Lukas Edman and {Toral Ruiz}, Antonio and {Noord, van}, Gertjan",

year = "2020",

language = "English",

pages = "81--90",

editor = "Andr{\'e} Martins and Helena Moniz and Sara Fumega and Bruno Martins and Fernando Batista and Luisa Coheur and Carla Parra and Isabel Trancoso and Marco Turchi and Arianna Bisazza and Joss Moorkens and Ana Guerberof and Mary Nurminen and Lena Marg and Forcada, {Mikel L.}",

booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",

note = "22nd Annual Conference of the European Association for Machine Translation, EAMT 2020 ; Conference date: 03-11-2020 Through 05-11-2020",

url = "https://eamt2020.inesc-id.pt/",

}

Edman, L, Toral Ruiz, A & Noord, van, G 2020, Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution. in A Martins, H Moniz, S Fumega, B Martins, F Batista, L Coheur, C Parra, I Trancoso, M Turchi, A Bisazza, J Moorkens, A Guerberof, M Nurminen, L Marg & ML Forcada (eds), Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. Lisbon, pp. 81-90, 22nd Annual Conference of the European Association for Machine Translation, 03/11/2020.

Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution. / Edman, Lukas; Toral Ruiz, Antonio; Noord, van, Gertjan.
Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. ed. / André Martins; Helena Moniz; Sara Fumega; Bruno Martins; Fernando Batista; Luisa Coheur; Carla Parra; Isabel Trancoso; Marco Turchi; Arianna Bisazza; Joss Moorkens; Ana Guerberof; Mary Nurminen; Lena Marg; Mikel L. Forcada. Lisbon, 2020. p. 81-90.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Low-Resource Unsupervised NMT

T2 - 22nd Annual Conference of the European Association for Machine Translation

AU - Edman, Lukas

AU - Toral Ruiz, Antonio

AU - Noord, van, Gertjan

PY - 2020

Y1 - 2020

N2 - Unsupervised Machine Translation hasbeen advancing our ability to translatewithout parallel data, but state-of-the-artmethods assume an abundance of mono-lingual data. This paper investigates thescenario where monolingual data is lim-ited as well, finding that current unsuper-vised methods suffer in performance un-der this stricter setting. We find that theperformance loss originates from the poorquality of the pretrained monolingual em-beddings, and we propose using linguis-tic information in the embedding train-ing scheme. To support this, we look attwo linguistic features that may help im-prove alignment quality: dependency in-formation and sub-word information. Us-ing dependency-based embeddings resultsin a complementary word representationwhich offers a boost in performance ofaround 1.5 BLEU points compared to stan-dardWORD2VECwhen monolingual datais limited to 1 million sentences per lan-guage. We also find that the inclusion ofsub-word information is crucial to improv-ing the quality of the embeddings

AB - Unsupervised Machine Translation hasbeen advancing our ability to translatewithout parallel data, but state-of-the-artmethods assume an abundance of mono-lingual data. This paper investigates thescenario where monolingual data is lim-ited as well, finding that current unsuper-vised methods suffer in performance un-der this stricter setting. We find that theperformance loss originates from the poorquality of the pretrained monolingual em-beddings, and we propose using linguis-tic information in the embedding train-ing scheme. To support this, we look attwo linguistic features that may help im-prove alignment quality: dependency in-formation and sub-word information. Us-ing dependency-based embeddings resultsin a complementary word representationwhich offers a boost in performance ofaround 1.5 BLEU points compared to stan-dardWORD2VECwhen monolingual datais limited to 1 million sentences per lan-guage. We also find that the inclusion ofsub-word information is crucial to improv-ing the quality of the embeddings

M3 - Conference contribution

SP - 81

EP - 90

BT - Proceedings of the 22nd Annual Conference of the European Association for Machine Translation

A2 - Martins, André

A2 - Moniz, Helena

A2 - Fumega, Sara

A2 - Martins, Bruno

A2 - Batista, Fernando

A2 - Coheur, Luisa

A2 - Parra, Carla

A2 - Trancoso, Isabel

A2 - Turchi, Marco

A2 - Bisazza, Arianna

A2 - Moorkens, Joss

A2 - Guerberof, Ana

A2 - Nurminen, Mary

A2 - Marg, Lena

A2 - Forcada, Mikel L.

CY - Lisbon

Y2 - 3 November 2020 through 5 November 2020

ER -

Edman L, Toral Ruiz A, Noord, van G. Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution. In Martins A, Moniz H, Fumega S, Martins B, Batista F, Coheur L, Parra C, Trancoso I, Turchi M, Bisazza A, Moorkens J, Guerberof A, Nurminen M, Marg L, Forcada ML, editors, Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. Lisbon. 2020. p. 81-90

Low-Resource Unsupervised NMT: Diagnosing the Problem and Providing a Linguistically Motivated Solution

Abstract

Conference

Access to Document

Handle.net

Fingerprint

Cite this