Once is not enough: Establishing reliability criteria for teacher evaluation based on classroom observations

Classroom observation is the most implemented method to evaluate teaching. To ensure reliability, researchers often train observers extensively. However, schools have limited resources to train observers and often lesson observation is performed by limitedly trained or untrained colleagues. In this study an evaluation procedure is implemented which is dependent on classroom observation by limitedly trained (3hours) peers. To study whether observations
have sufficient reliability, two different criteria are specified: one more lenient for formative evaluation and the other one strict for summative evaluation. The study aims to explore whether these criteria are realistic for schools. The sample contains 198 lesson observations of 69 teachers, by 62 peer-colleagues. Two different aspects of reliability are studied: (1) generalizability: the extend to which other lessons observed by other peers would give the same evaluation result, (2) person fit: the extend to which observations fit model assumptions. The results show that three peer-observers are required to achieve sufficient reliability for formative purposes, while more than 10 are required to achieve sufficient reliability.