A comparison between factor analysis and item response theory modeling in scale analysis

Factor analysis (FA) and item response theory (IRT) modeling are compared (a) theoretically by examining the model definitions and considering the most commonly applied estimation methods for the models, (b) empirically by reviewing journal articles on scaling research to show what is actually done in practice, (c) in model estimation performance by means of a Monte Carlo simulation study to investigate the robustness of either approach against violations of distributional assumptions, and (d) in practice by applying the models to empirical data sets.

In a simulation study, the effects of four explanatory variables (latent variable (LV) distribution, item response distribution, scale strength, and sample size) on model estimation are investigated by applying four selected scaling models on samples of generated data: FA of the sample covariance matrix (FA-lin), FA of the estimated polychoric correlation matrix (FA-poly), the graded response IRT model (IRT-grm), and the nonparametric Mokken IRT model extended to polytomous items (IRT-mok).

For normal data it is shown that both FA-poly and IRT-grm perform well. Under conditions of a skew-normal LV, the performance of all parametric models deteriorates compared to normal LV conditions, most notably when combined with skewed item variables. IRT-grm performs best in such circumstances. Nonnormal LV and item distributions do not pose any estimation problems for IRT-mok.

Guidelines are provided for applied researchers employing scale analysis: Take a large enough sample size, use available substantive knowledge, inspect the sample data, choose a model based on specific characteristics of the data, assess model fit, and use model-estimated LV scores.