The relation between prediction model performance measures and patient selection outcomes for proton therapy in head and neck cancer

Artuur Marijn Leeuwenberg*, Johannes Bernardus Reitsma, Lisa Griet Lydia Jozef Van den Bosch, Jeroen Hoogland, Arjen van der Schaaf, Frank Jozef Pieter Hoebers, Oda Bemadette Wijers, Johannes Albertus Langendijk, Karel Gerardus Maria Moons, Ewoud Schuit

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

2 Downloads (Pure)


Background: Normal-tissue complication probability (NTCP) models predict complication risk in patients receiving radiotherapy, considering radiation dose to healthy tissues, and are used to select patients for proton therapy, based on their expected reduction in risk after proton therapy versus photon radiotherapy (ΔNTCP). Recommended model evaluation measures include area under the receiver operating characteristic curve (AUC), overall calibration (CITL), and calibration slope (CS), whose precise relation to patient selection is still unclear. We investigated how each measure relates to patient selection outcomes. Methods: The model validation and consequent patient selection process was simulated within empirical head and neck cancer patient data. By manipulating performance measures independently via model perturbations, the relation between model performance and patient selection was studied. Results: Small reductions in AUC (-0.02) yielded mean changes in ΔNTCP between 0.9–3.2 %, and single-model patient selection differences between 2–19 %. Deviations (-0.2 or +0.2) in CITL or CS yielded mean changes in ΔNTCP between 0.3–1.4 %, and single-model patient selection differences between 1–10 %. Conclusions: Each measure independently impacts ΔNTCP and patient selection and should thus be assessed in a representative sufficiently large external sample. Our suggested practical model selection approach is considering the model with the highest AUC, and recalibrating it if needed.

Original languageEnglish
Article number109449
Number of pages10
JournalRadiotherapy and Oncology
Publication statusPublished - Feb-2023


  • Head and neck cancer
  • Individualized treatment decisions
  • Normal tissue complication probability models
  • Prediction performance measures

Cite this