Degraded visual and auditory input individually impair audiovisual emotion recognition from speech-like stimuli, but no evidence for an exacerbated effect from combined degradation

Minke J de Boer*, Tim Jürgens, Frans W Cornelissen, Deniz Başkent

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

20 Downloads (Pure)

Abstract

Emotion recognition requires optimal integration of the multisensory signals from vision and hearing. A sensory loss in either or both modalities can lead to changes in integration and related perceptual strategies. To investigate potential acute effects of combined impairments due to sensory information loss only, we degraded the visual and auditory information in audiovisual video-recordings, and presented these to a group of healthy young volunteers. These degradations intended to approximate some aspects of vision and hearing impairment in simulation. Other aspects, related to advanced age, potential health issues, but also long-term adaptation and cognitive compensation strategies, were not included in the simulations. Besides accuracy of emotion recognition, eye movements were recorded to capture perceptual strategies. Our data show that emotion recognition performance decreases when degraded visual and auditory information are presented in isolation, but simultaneously degrading both modalities does not exacerbate these isolated effects. Moreover, degrading the visual information strongly impacts recognition performance and on viewing behavior. In contrast, degrading auditory information alongside normal or degraded video had little (additional) effect on performance or gaze. Nevertheless, our results hold promise for visually impaired individuals, because the addition of any audio to any video greatly facilitates performance, even though adding audio does not completely compensate for the negative effects of video degradation. Additionally, observers modified their viewing behavior to degraded video in order to maximize their performance. Therefore, optimizing the hearing of visually impaired individuals and teaching them such optimized viewing behavior could be worthwhile endeavors for improving emotion recognition.

Original languageEnglish
Pages (from-to)51-62
Number of pages12
JournalVision Research
Volume180
Early online date24-Dec-2020
DOIs
Publication statusPublished - Mar-2021

Keywords

  • Emotion perception
  • Eye-tracking
  • Central scotoma
  • Age-related hearing loss
  • Audiovisual
  • Dynamic
  • HEARING-LOSS
  • MACULAR DEGENERATION
  • CENTRAL SCOTOMAS
  • EYE-MOVEMENTS
  • LOW-VISION
  • AGE
  • PERCEPTION
  • SEARCH
  • PERFORMANCE
  • INTEGRATION

Cite this