Text-Picture Relations in Multimodal Instructions

Ielka van der Sluis; Anne Nienke Eppinga; Gisela Redeker

Text-Picture Relations in Multimodal Instructions

Ielka van der Sluis, Anne Nienke Eppinga, Gisela Redeker

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

339 Downloads (Pure)

Abstract

This paper presents a multi-method approach to the description and evaluation of multimodal content in situated contexts. First, a corpus of 15 simple first-aid instructions that include texts and pictures is shown to exhibit variation in content and presentation. We then report a user study in which four versions of a tick-removal instruction were used to test the effects of the relative placement of text and pictures in a particular instruction. The participants’ processing of the instruction and their task performance were video-recorded and registered with an eye tracker. Questionnaires and interviews were used to measure comprehension,
recall and the instruction’s attractiveness. Results show that users
first read at least some of the text before looking at the pictures, and prefer to have the pictures placed to the right or below the text.

Original language	English
Title of host publication	Proceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication
Place of Publication	Montpellier, France.
Number of pages	6
Publication status	Published - 19-Sept-2017
Event	FMSC workshop on Foundations of Situated or Multimodal Communication - Montpellier , Montpellier , France Duration: 19-Sept-2017 → 19-Sept-2018

Workshop

Workshop	FMSC workshop on Foundations of Situated or Multimodal Communication
Country/Territory	France
City	Montpellier
Period	19/09/2017 → 19/09/2018

Keywords

multimodal
instructions
text-picture relations

Access to Document

Text-Picture Relations in Multimodal InstructionsFinal publisher's version, 4.22 MBLicence: Unspecified

Handle.net

Persistent link

Cite this

@inproceedings{271fc38f41404f3c8af83466dea8f774,

title = "Text-Picture Relations in Multimodal Instructions",

abstract = "This paper presents a multi-method approach to the description and evaluation of multimodal content in situated contexts. First, a corpus of 15 simple first-aid instructions that include texts and pictures is shown to exhibit variation in content and presentation. We then report a user study in which four versions of a tick-removal instruction were used to test the effects of the relative placement of text and pictures in a particular instruction. The participants{\textquoteright} processing of the instruction and their task performance were video-recorded and registered with an eye tracker. Questionnaires and interviews were used to measure comprehension,recall and the instruction{\textquoteright}s attractiveness. Results show that usersfirst read at least some of the text before looking at the pictures, and prefer to have the pictures placed to the right or below the text.",

keywords = "multimodal, instructions, text-picture relations",

author = "{van der Sluis}, Ielka and Eppinga, {Anne Nienke} and Gisela Redeker",

year = "2017",

month = sep,

day = "19",

language = "English",

booktitle = "Proceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication",

note = "FMSC workshop on Foundations of Situated or Multimodal Communication ; Conference date: 19-09-2017 Through 19-09-2018",

}

van der Sluis, I, Eppinga, AN & Redeker, G 2017, Text-Picture Relations in Multimodal Instructions. in Proceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication. Montpellier, France., FMSC workshop on Foundations of Situated or Multimodal Communication, Montpellier , France, 19/09/2017.

Text-Picture Relations in Multimodal Instructions. / van der Sluis, Ielka; Eppinga, Anne Nienke; Redeker, Gisela.
Proceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication. Montpellier, France., 2017.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Text-Picture Relations in Multimodal Instructions

AU - van der Sluis, Ielka

AU - Eppinga, Anne Nienke

AU - Redeker, Gisela

PY - 2017/9/19

Y1 - 2017/9/19

N2 - This paper presents a multi-method approach to the description and evaluation of multimodal content in situated contexts. First, a corpus of 15 simple first-aid instructions that include texts and pictures is shown to exhibit variation in content and presentation. We then report a user study in which four versions of a tick-removal instruction were used to test the effects of the relative placement of text and pictures in a particular instruction. The participants’ processing of the instruction and their task performance were video-recorded and registered with an eye tracker. Questionnaires and interviews were used to measure comprehension,recall and the instruction’s attractiveness. Results show that usersfirst read at least some of the text before looking at the pictures, and prefer to have the pictures placed to the right or below the text.

AB - This paper presents a multi-method approach to the description and evaluation of multimodal content in situated contexts. First, a corpus of 15 simple first-aid instructions that include texts and pictures is shown to exhibit variation in content and presentation. We then report a user study in which four versions of a tick-removal instruction were used to test the effects of the relative placement of text and pictures in a particular instruction. The participants’ processing of the instruction and their task performance were video-recorded and registered with an eye tracker. Questionnaires and interviews were used to measure comprehension,recall and the instruction’s attractiveness. Results show that usersfirst read at least some of the text before looking at the pictures, and prefer to have the pictures placed to the right or below the text.

KW - multimodal

KW - instructions

KW - text-picture relations

M3 - Conference contribution

BT - Proceedings of the 12th International Workshop on Computational Semantics (IWCS 2017): Proceedings of the FMSC workshop on Foundations of Situated or Multimodal Communication

CY - Montpellier, France.

T2 - FMSC workshop on Foundations of Situated or Multimodal Communication

Y2 - 19 September 2017 through 19 September 2018

ER -