Evaluating Algorithms for the Generation of Referring Expressions: Going beyond Toy Domains.

Ielka van der Sluis, A. Gatt, K. van Deemter

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    13 Citations (Scopus)
    88 Downloads (Pure)


    We describe a corpus-based evaluation method-
    ology, applied to a number of classic algorithms
    in the generation of referring expressions. Fol-
    lowing up on earlier work involving very simple
    domains, this paper deals with the issues asso-
    ciated with domains that contain ‘real-life’ ob-
    jects of some complexity. Results indicate that
    state of the art algorithms perform very differ-
    ently when applied to a complex domain. More-
    over, if a version of the Incremental Algorithm is
    used then it becomes of huge importance to se-
    lect a good preference order, because some pref-
    erence orders are prone to generating very un-
    natural output. Finding good preference orders,
    however, can be difficult, as we show. These re-
    sults should contribute to a growing debate on
    the evaluation of nlg systems, arguing in favour
    of carefully constructed balanced and semanti-
    cally transparent corpora.
    Original languageEnglish
    Title of host publicationProceedings of the RANLP-2007
    Place of PublicationBorovets, Bulgaria
    Publication statusPublished - 2007

    Cite this