Evaluating Text Generation from Discourse Representation Structures

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

36 Downloads (Pure)

Abstract

We present an end-to-end neural approach to generate English sentences from formal meaning representations, Discourse Representation Structures (DRSs). We use a rather standard bi-LSTM sequence-to-sequence model, work with a linearized DRS input representation, and evaluate character-level and word-level decoders. We obtain very encouraging results in terms of reference-based automatic metrics such as BLEU. But because such metrics only evaluate the surface level of generated output, we develop a new metric, ROSE, that targets specific semantic phenomena. We do this with five DRS generation challenge sets focusing on tense, grammatical number, polarity, named entities and quantities. The aim of these challenge sets is to assess the neural generator’s systematicity and generalization to unseen inputs.
Original languageEnglish
Title of host publicationProceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
EditorsAntoine Bosselut, Esin Durmus, Varun Prashant Gangal, Sebastian Gehrmann, Yacine Jernite, Laura Perez-Beltrachini, Samira Shaikh, Wei Xu
PublisherAssociation for Computational Linguistics (ACL)
Pages73-83
Number of pages11
DOIs
Publication statusPublished - 2021

Cite this