Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages

Prajit Dhar; Arianna Bisazza; Gertjan van Noord

Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages

Prajit Dhar, Arianna Bisazza, Gertjan van Noord

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

2 Citations (Scopus)

88 Downloads (Pure)

Abstract

Dravidian languages, such as Kannada and Tamil, are notoriously difficult to translate by state-of-the-art neural models. This stems from the fact that these languages are morphologically very rich as well as being low-resourced. In this paper, we focus on subword segmentation and evaluate Linguistically Motivated Vocabulary Reduction (LMVR) against the more commonly used SentencePiece (SP) for the task of translating from English into four different Dravidian languages. Additionally we investigate the optimal subword vocabulary size for each language. We find that SP is the overall best choice for segmentation, and that larger dictionary sizes lead to higher translation quality.

Original language	English
Title of host publication	Proceedings of the 8th Workshop on Asian Translation (WAT2021)
Editors	Toshiaki Nakazawa, Hideki Nakayama, Isao Goto, Hideya Mino, Chenchen Ding, Raj Dabre, Anoop Kunchukuttan, Shohei Higashiyama, Hiroshi Manabe, Win Pa Pa, Shantipriya Parida, Ondřej Bojar, Chenhui Chu, Akiko Eriguchi, Kaori Abe, Yusuke Oda, Katsuhito Sudoh, Sadao Kurohashi, Pushpak Bhattacharyya
Publisher	Association for Computational Linguistics (ACL)
Pages	181-190
Number of pages	10
Publication status	Published - 5-Aug-2021

Access to Document

Optimal Word Segmentation for Neural Machine Translation into Dravidian LanguagesFinal publisher's version, 933 KBLicence: CC BY

Handle.net

Persistent link

Cite this

Dhar, P., Bisazza, A., & van Noord, G. (2021). Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages. In T. Nakazawa, H. Nakayama, I. Goto, H. Mino, C. Ding, R. Dabre, A. Kunchukuttan, S. Higashiyama, H. Manabe, W. Pa Pa, S. Parida, O. Bojar, C. Chu, A. Eriguchi, K. Abe, Y. Oda, K. Sudoh, S. Kurohashi, & P. Bhattacharyya (Eds.), Proceedings of the 8th Workshop on Asian Translation (WAT2021) (pp. 181-190). Association for Computational Linguistics (ACL).

Dhar, Prajit ; Bisazza, Arianna ; van Noord, Gertjan. / Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages. Proceedings of the 8th Workshop on Asian Translation (WAT2021). editor / Toshiaki Nakazawa ; Hideki Nakayama ; Isao Goto ; Hideya Mino ; Chenchen Ding ; Raj Dabre ; Anoop Kunchukuttan ; Shohei Higashiyama ; Hiroshi Manabe ; Win Pa Pa ; Shantipriya Parida ; Ondřej Bojar ; Chenhui Chu ; Akiko Eriguchi ; Kaori Abe ; Yusuke Oda ; Katsuhito Sudoh ; Sadao Kurohashi ; Pushpak Bhattacharyya. Association for Computational Linguistics (ACL), 2021. pp. 181-190

@inproceedings{0d4b6254b5654673a48e00a98e4adb54,

title = "Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages",

abstract = "Dravidian languages, such as Kannada and Tamil, are notoriously difficult to translate by state-of-the-art neural models. This stems from the fact that these languages are morphologically very rich as well as being low-resourced. In this paper, we focus on subword segmentation and evaluate Linguistically Motivated Vocabulary Reduction (LMVR) against the more commonly used SentencePiece (SP) for the task of translating from English into four different Dravidian languages. Additionally we investigate the optimal subword vocabulary size for each language. We find that SP is the overall best choice for segmentation, and that larger dictionary sizes lead to higher translation quality.",

author = "Prajit Dhar and Arianna Bisazza and {van Noord}, Gertjan",

year = "2021",

month = aug,

day = "5",

language = "English",

pages = "181--190",

editor = "Toshiaki Nakazawa and Hideki Nakayama and Isao Goto and Hideya Mino and Chenchen Ding and Raj Dabre and Anoop Kunchukuttan and Shohei Higashiyama and Hiroshi Manabe and {Pa Pa}, Win and Shantipriya Parida and Ond{\v r}ej Bojar and Chenhui Chu and Akiko Eriguchi and Kaori Abe and Yusuke Oda and Katsuhito Sudoh and Sadao Kurohashi and Pushpak Bhattacharyya",

booktitle = "Proceedings of the 8th Workshop on Asian Translation (WAT2021)",

publisher = "Association for Computational Linguistics (ACL)",

}

Dhar, P, Bisazza, A & van Noord, G 2021, Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages. in T Nakazawa, H Nakayama, I Goto, H Mino, C Ding, R Dabre, A Kunchukuttan, S Higashiyama, H Manabe, W Pa Pa, S Parida, O Bojar, C Chu, A Eriguchi, K Abe, Y Oda, K Sudoh, S Kurohashi & P Bhattacharyya (eds), Proceedings of the 8th Workshop on Asian Translation (WAT2021). Association for Computational Linguistics (ACL), pp. 181-190.

Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages. / Dhar, Prajit; Bisazza, Arianna ; van Noord, Gertjan.
Proceedings of the 8th Workshop on Asian Translation (WAT2021). ed. / Toshiaki Nakazawa; Hideki Nakayama; Isao Goto; Hideya Mino; Chenchen Ding; Raj Dabre; Anoop Kunchukuttan; Shohei Higashiyama; Hiroshi Manabe; Win Pa Pa; Shantipriya Parida; Ondřej Bojar; Chenhui Chu; Akiko Eriguchi; Kaori Abe; Yusuke Oda; Katsuhito Sudoh; Sadao Kurohashi; Pushpak Bhattacharyya. Association for Computational Linguistics (ACL), 2021. p. 181-190.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages

AU - Dhar, Prajit

AU - Bisazza, Arianna

AU - van Noord, Gertjan

PY - 2021/8/5

Y1 - 2021/8/5

N2 - Dravidian languages, such as Kannada and Tamil, are notoriously difficult to translate by state-of-the-art neural models. This stems from the fact that these languages are morphologically very rich as well as being low-resourced. In this paper, we focus on subword segmentation and evaluate Linguistically Motivated Vocabulary Reduction (LMVR) against the more commonly used SentencePiece (SP) for the task of translating from English into four different Dravidian languages. Additionally we investigate the optimal subword vocabulary size for each language. We find that SP is the overall best choice for segmentation, and that larger dictionary sizes lead to higher translation quality.

AB - Dravidian languages, such as Kannada and Tamil, are notoriously difficult to translate by state-of-the-art neural models. This stems from the fact that these languages are morphologically very rich as well as being low-resourced. In this paper, we focus on subword segmentation and evaluate Linguistically Motivated Vocabulary Reduction (LMVR) against the more commonly used SentencePiece (SP) for the task of translating from English into four different Dravidian languages. Additionally we investigate the optimal subword vocabulary size for each language. We find that SP is the overall best choice for segmentation, and that larger dictionary sizes lead to higher translation quality.

M3 - Conference contribution

SP - 181

EP - 190

BT - Proceedings of the 8th Workshop on Asian Translation (WAT2021)

A2 - Nakazawa, Toshiaki

A2 - Nakayama, Hideki

A2 - Goto, Isao

A2 - Mino, Hideya

A2 - Ding, Chenchen

A2 - Dabre, Raj

A2 - Kunchukuttan, Anoop

A2 - Higashiyama, Shohei

A2 - Manabe, Hiroshi

A2 - Pa Pa, Win

A2 - Parida, Shantipriya

A2 - Bojar, Ondřej

A2 - Chu, Chenhui

A2 - Eriguchi, Akiko

A2 - Abe, Kaori

A2 - Oda, Yusuke

A2 - Sudoh, Katsuhito

A2 - Kurohashi, Sadao

A2 - Bhattacharyya, Pushpak

PB - Association for Computational Linguistics (ACL)

ER -

Dhar P, Bisazza A , van Noord G. Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages. In Nakazawa T, Nakayama H, Goto I, Mino H, Ding C, Dabre R, Kunchukuttan A, Higashiyama S, Manabe H, Pa Pa W, Parida S, Bojar O, Chu C, Eriguchi A, Abe K, Oda Y, Sudoh K, Kurohashi S, Bhattacharyya P, editors, Proceedings of the 8th Workshop on Asian Translation (WAT2021). Association for Computational Linguistics (ACL). 2021. p. 181-190