Invisible to People but not to Machines: Evaluation of Style-aware Headline Generation in Absence of Reliable Human Judgment

Lorenzo De Mattei, Michele Cafagna, Felice Dell’Orletta, Malvina Nissim

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    2 Citations (Scopus)
    3 Downloads (Pure)


    We automatically generate headlines that are expected to comply with the specific styles of two different Italian newspapers. Through a data alignment strategy and different training/testing settings, we aim at decoupling content from style and preserve the latter in generation. In order to evaluate the generated headlines’ quality in terms of their specific newspaper-compliance, we devise a fine-grained evaluation strategy based on automatic classification. We observe that our models do indeed learn newspaper-specific style.Importantly, we also observe that humans aren’t reliable judges for this task, since although familiar with the newspapers, they are notable to discern their specific styles even in the original human-written headlines. The utility of automatic evaluation goes therefore beyond saving the costs and hurdles of manual annotation, and deserves particular care in its design.
    Original languageEnglish
    Title of host publicationProceedings of The 12th Language Resources and Evaluation Conference
    Subtitle of host publicationLREC 2020
    PublisherEuropean Language Resources Association (ELRA)
    Number of pages9
    Publication statusPublished - 2020
    Event12th Language Resources and Evaluation Conference
    : LREC 2020
    - Marseille, France
    Duration: 11-May-202016-May-2020


    Conference12th Language Resources and Evaluation Conference
    Internet address

    Cite this