Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation

Antonio Toral, Sheila Castilho, Ke Hu, Andy Way

    OnderzoeksoutputAcademic

    143 Downloads (Pure)

    Samenvatting

    We reassess a recent study (Hassan et al., 2018) that claimed that machine translation (MT) has reached human parity for the translation of news from Chinese into English, using pairwise ranking and considering three variables that were not taken into account in that previous study: the language in which the
    source side of the test set was originally written, the translation proficiency of the evaluators, and the provision of inter-sentential context. If we consider only original source text (i.e. not translated from another language, or translationese), then we find evidence showing that human parity has not been achieved. We compare the judgments of professional translators against those of non-experts and discover that those of the experts result in higher
    inter-annotator agreement and better discrimination between human and machine translations. In addition, we analyse the human translations of the test set and identify important translation issues. Finally, based on these findings, we provide a set of recommendations for future human evaluations of MT.
    Originele taal-2English
    Pagina's113-123
    Aantal pagina's11
    StatusPublished - 31-okt-2018
    EvenementTHIRD CONFERENCE ON MACHINE TRANSLATION - Brussels, Belgium
    Duur: 31-okt-20181-nov-2018
    http://www.statmt.org/wmt18/

    Conference

    ConferenceTHIRD CONFERENCE ON MACHINE TRANSLATION
    Verkorte titelWMT18
    LandBelgium
    StadBrussels
    Periode31/10/201801/11/2018
    Internet adres

    Citeer dit