Text-Picture Relations in Multimodal Instructions

Ielka van der Sluis, Anne Nienke Eppinga, Gisela Redeker

    OnderzoeksoutputAcademicpeer review

    197 Downloads (Pure)

    Samenvatting

    This paper presents a multi-method approach to the description and evaluation of multimodal content in situated contexts. First, a corpus of 15 simple first-aid instructions that include texts and pictures is shown to exhibit variation in content and presentation. We then report a user study in which four versions of a tick-removal instruction were used to test the effects of the relative placement of text and pictures in a particular instruction. The participants’ processing of the instruction and their task performance were video-recorded and registered with an eye tracker. Questionnaires and interviews were used to measure comprehension,
    recall and the instruction’s attractiveness. Results show that users
    first read at least some of the text before looking at the pictures, and prefer to have the pictures placed to the right or below the text.
    Originele taal-2English
    TitelProceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication
    Plaats van productieMontpellier, France.
    Aantal pagina's6
    StatusPublished - 19-sep-2017
    EvenementFMSC workshop on Foundations of Situated or Multimodal Communication - Montpellier , Montpellier , France
    Duur: 19-sep-201719-sep-2018

    Workshop

    WorkshopFMSC workshop on Foundations of Situated or Multimodal Communication
    Land/RegioFrance
    StadMontpellier
    Periode19/09/201719/09/2018

    Citeer dit