Keeping track of emotions: audiovisual integration for emotion recognition and compensation for sensory degradations captured by perceptual strategies

The majority of emotional expressions are multimodal and dynamic in nature. Emotion recognition, therefore, requires integration of these multimodal signals. Sensory impairments likely affect emotion recognition, but although sensory impairments are common in older adults, it is unknown how they affect emotion recognition. As more people reach old age, accompanied by an increase in the prevalence of sensory impairments, it is urgent to comprehensively understand audiovisual integration, especially in older individuals. My thesis sought to create a basic understanding of audiovisual integration for emotion recognition and study how audiovisual interactions change with simulated sensory impairments. A secondary aim was to understand how age affects these outcomes.
To systematically address these aims, I examined how well observers recognize emotions, presented via videos, and how emotion recognition accuracy and perceptual strategies, assessed via eye-tracking, vary under changing availability and reliability of the visual and auditory information.
The research presented in my thesis shows that audiovisual integration and compensation abilities remain intact with age, despite a general decline in recognition accuracy. Compensation for degraded audio is possible by relying more on visual signals, but not vice versa. Older observers adapt their perceptual strategies in a different, perhaps less efficient, manner than younger observers. Importantly, I demonstrate that it is crucial to use additional measurements besides recognition accuracy in order to understand audiovisual integration mechanisms. Measurements such as eye-tracking allow examining whether the reliance on visual and auditory signals alters with age and (simulated) sensory impairments, even when lacking a change in accuracy.