Assessing the internal consistency reliability of ecological momentary assessment measures: Insights from the WARN-D study

Intensive longitudinal research has become increasingly popular in the social and clinical sciences in recent years. However, this popularity has brought about many challenges for both methodological and empirical researchers, including challenges regarding measurement. In this preregistered study, we are particularly interested in the assessment of the reliability when multiple items are used to measure the same construct in intensive longitudinal data. This is important because reliability estimates are necessary (albeit not sufficient) to evaluate the quality of measures. Here, we evaluate the internal consistency reliability of scales used during Stage 2 of the WARN-D study, a 3-month period of daily and weekly measurements. The WARN-D study is a prospective 2-year study of approximately 1,750 students conducted in the Netherlands, aiming at building an early warning system for depression. Stage 2 includes 3 months of data on positive and negative affect measured four times a day and depression and anxiety measured once a week. To assess the reliability of each scale, we use six different statistical approaches including three simpler approaches that estimate the reliability at the between-person and within-person levels and three idiographic approaches that estimate person-specific reliability coefficients. This article also serves as a tutorial guide for substantive researchers, providing annotated code to facilitate estimating and reporting the reliability of ecological momentary assessment measures. We encourage all researchers to report the reliability of their data when applying the introduced statistical approaches, contributing to a collaborative effort toward developing more reliable measures in psychological and behavioral science. (PsycInfo Database Record (c) 2025 APA, all rights reserved).