Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation

Antonio Toral, Sheila Castilho, Ke Hu, Andy Way

    Research output: Contribution to conferencePaperAcademic

    105 Citations (Scopus)
    270 Downloads (Pure)

    Abstract

    We reassess a recent study (Hassan et al., 2018) that claimed that machine translation (MT) has reached human parity for the translation of news from Chinese into English, using pairwise ranking and considering three variables that were not taken into account in that previous study: the language in which the
    source side of the test set was originally written, the translation proficiency of the evaluators, and the provision of inter-sentential context. If we consider only original source text (i.e. not translated from another language, or translationese), then we find evidence showing that human parity has not been achieved. We compare the judgments of professional translators against those of non-experts and discover that those of the experts result in higher
    inter-annotator agreement and better discrimination between human and machine translations. In addition, we analyse the human translations of the test set and identify important translation issues. Finally, based on these findings, we provide a set of recommendations for future human evaluations of MT.
    Original languageEnglish
    Pages113-123
    Number of pages11
    Publication statusPublished - 31-Oct-2018
    EventTHIRD CONFERENCE ON MACHINE TRANSLATION - Brussels, Belgium
    Duration: 31-Oct-20181-Nov-2018
    http://www.statmt.org/wmt18/

    Conference

    ConferenceTHIRD CONFERENCE ON MACHINE TRANSLATION
    Abbreviated titleWMT18
    Country/TerritoryBelgium
    CityBrussels
    Period31/10/201801/11/2018
    Internet address

    Fingerprint

    Dive into the research topics of 'Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation'. Together they form a unique fingerprint.

    Cite this