The reliability of logbook data of medical students: An estimation of interobserver agreement, sensitivity and specificity

Objective Logbooks are widely used in medical schools as an evaluation tool to assess students' progress towards objectives. To estimate whether students fill in their logbooks reliably, we measured interobserver agreement by comparing doctors' data and students' data.

Method Completed logbooks were collected at two subdivisions of the department of Internal Medicine at the University Hospital of Groningen. The logbook contains 231 preprinted diseases. Doctors and students recorded the diseases they had encountered. Interobserver agreement, expressed by the Jaccard coefficient (J), was calculated for the complete set of diseases and for a subset of core diseases. To assess the kinds of errors which students made, sensitivity and specificity were determined.

Results Logbook data of doctors and students are not fully consistent (mean J for the complete set of diseases was .23 and for the core diseases .36). The quality of the logbook data is high in the sense that students do not record many false identifications (mean specificity for the complete set of diseases and for the core diseases were .96 and .93, respectively); the quality is poor in the sense that students do not record all the diseases which could be seen at the department (mean sensitivity for the complete set of diseases is .36 and for the core diseases it is .51).

Conclusion This study shows inconsistencies in recording diseases in a logbook by students compared with doctors. In particular the diseases which are present at a department are under-reported by students. Supervision and feedback are important mechanisms to optimize the students' use of (1) all diseases which could be encountered and (2) the logbook.