Text-Picture Relations in Multimodal Instructions

Ielka van der Sluis, Anne Nienke Eppinga, Gisela Redeker

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    339 Downloads (Pure)


    This paper presents a multi-method approach to the description and evaluation of multimodal content in situated contexts. First, a corpus of 15 simple first-aid instructions that include texts and pictures is shown to exhibit variation in content and presentation. We then report a user study in which four versions of a tick-removal instruction were used to test the effects of the relative placement of text and pictures in a particular instruction. The participants’ processing of the instruction and their task performance were video-recorded and registered with an eye tracker. Questionnaires and interviews were used to measure comprehension,
    recall and the instruction’s attractiveness. Results show that users
    first read at least some of the text before looking at the pictures, and prefer to have the pictures placed to the right or below the text.
    Original languageEnglish
    Title of host publicationProceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication
    Place of PublicationMontpellier, France.
    Number of pages6
    Publication statusPublished - 19-Sept-2017
    EventFMSC workshop on Foundations of Situated or Multimodal Communication - Montpellier , Montpellier , France
    Duration: 19-Sept-201719-Sept-2018


    WorkshopFMSC workshop on Foundations of Situated or Multimodal Communication


    • multimodal
    • instructions
    • text-picture relations

    Cite this