What Level of Quality Can Neural Machine Translation Attain on Literary Text?

Antonio Toral, Andy Way

    Research output: Chapter in Book/Report/Conference proceedingChapterAcademicpeer-review

    240 Downloads (Pure)

    Abstract

    Given the rise of the new neural approach to machine translation (NMT) and its promising performance on different text types, we assess the translation quality it can attain on what is perceived to be the greatest challenge for MT: literary text. Specifically, we target novels, arguably the most popular type of literary text. We build a literary-adapted NMT system for the English-to-Catalan translation direction and evaluate it against a system pertaining to the previous dominant paradigm in MT: statistical phrase-based MT (PBSMT). To this end, for the first time we train MT systems, both NMT and PBSMT, on large amounts of literary text (over 100 million words) and evaluate them on a set of 12 widely known novels spanning from the 1920s to the present day. According to the BLEU automatic evaluation metric, NMT is significantly better than PBSMT (p < 0.01) on all the novels considered. Overall, NMT results in a 11% relative improvement (3 points absolute) over PBSMT. A complementary human evaluation on three of the books shows that between 17% and 34% of the translations, depending on the book, produced by NMT (versus 8% and 20% with PBSMT) are perceived by native speakers of the target language to be of equivalent quality to translations produced by a professional human translator.
    Original languageEnglish
    Title of host publicationTranslation Quality Assessment
    Subtitle of host publicationFrom Principles to Practice
    EditorsJoss Moorkens, Sheila Castilho, Federico Gaspari, Stephen Doherty
    Place of PublicationCham
    PublisherSpringer International Publishing AG
    Pages263-287
    Number of pages25
    ISBN (Electronic) 978-3-319-91241-7
    ISBN (Print)978-3-319-91241-7
    DOIs
    Publication statusPublished - 2018

    Publication series

    NameMachine Translation: Technologies and Applications
    Volume1

    Keywords

    • Translation quality assessment
    • Principles to practice
    • Literature translation
    • Neural machine translation
    • Pairwise ranking
    • Phrase-based statistical machine translation

    Fingerprint

    Dive into the research topics of 'What Level of Quality Can Neural Machine Translation Attain on Literary Text?'. Together they form a unique fingerprint.

    Cite this