Applying Deconvolution and Nonlinear Modeling to Coregistered Electroencephalogram/Eye-Tracking Experiments in Vision Research

When combining eye tracking and EEG, two fundamental problems remain to be solved: (a) varying temporal overlap between brain responses evoked, for example, by consecutive fixations in natural viewing and (b) the numerous and often nonlinear influences of low-level stimulus properties and eye movement properties (like saccade amplitude) on the neural responses. To address these two problems, we have recently published the freely available open source “unfold” toolbox (www.unfoldtoolbox.org) which unifies the linear deconvolution framework (to disentangle overlapping potentials) with nonlinear regression (generalized additive modeling, to control for nonlinear confounds). In this talk, we will illustrate how this approach can be used to address theoretically interesting questions in vision research using data from face perception, scene viewing, and natural sentence reading. First, we will demonstrate how deconvolution can be used to account for, and analyze, overlapping brain potentials produced by involuntary (micro)saccades in a typical face recognition experiment. Then, we will disentangle multiple nonlinear influences of saccade parameters on fixation-related potentials during natural scene viewing. Finally, we will isolate the neural correlates of preview benefit in natural reading and separate them from the artifactual effects of varying fixation durations. Our approach shows a principal way to measure reliable fixation-related potentials during natural vision.