Reproducibility of radiomics quality score: an intra- and inter-rater reliability study

EuSoMII Radiomics Auditing Group; Tugba Akinci D'Antonoli; Armando Ugo Cavallo; Federica Vernuccio; Arnaldo Stanzione; Michail E Klontzas; Roberto Cannella; Lorenzo Ugga; Agah Baran; Salvatore Claudio Fanni; Ekaterina Petrash; Ilaria Ambrosini; Luca Alessandro Cappellini; Peter van Ooijen; Elmar Kotter; Daniel Pinto dos Santos; Renato Cuocolo

doi:10.1007/s00330-023-10217-x

Reproducibility of radiomics quality score: an intra- and inter-rater reliability study

EuSoMII Radiomics Auditing Group, Tugba Akinci D'Antonoli, Armando Ugo Cavallo, Federica Vernuccio, Arnaldo Stanzione, Michail E Klontzas, Roberto Cannella, Lorenzo Ugga, Agah Baran, Salvatore Claudio Fanni, Ekaterina Petrash, Ilaria Ambrosini, Luca Alessandro Cappellini, Peter van Ooijen, Elmar Kotter, Daniel Pinto dos Santos, Renato Cuocolo

Research output: Contribution to journal › Article › Academic › peer-review

30 Citations (Scopus)

71 Downloads (Pure)

Abstract

OBJECTIVES: To investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.

METHODS: Nine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.

RESULTS: The inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).

CONCLUSIONS: Reproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.

CLINICAL RELEVANCE STATEMENT: There is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.

KEY POINTS: • Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

Original language	English
Pages (from-to)	2791–2804
Number of pages	14
Journal	European Radiology
Volume	34
Early online date	21-Sept-2023
DOIs	https://doi.org/10.1007/s00330-023-10217-x
Publication status	Published - Apr-2024

Access to Document

10.1007/s00330-023-10217-xLicence: CC BY

Reproducibility of radiomics quality scoreFinal publisher's version, 1.86 MBLicence: CC BY

Handle.net

Persistent link

Cite this

EuSoMII Radiomics Auditing Group, Akinci D'Antonoli, T., Cavallo, A. U., Vernuccio, F., Stanzione, A., Klontzas, M. E., Cannella, R., Ugga, L., Baran, A., Fanni, S. C., Petrash, E., Ambrosini, I., Cappellini, L. A., van Ooijen, P., Kotter, E., Pinto dos Santos, D., & Cuocolo, R. (2024). Reproducibility of radiomics quality score: an intra- and inter-rater reliability study. European Radiology, 34, 2791–2804. https://doi.org/10.1007/s00330-023-10217-x

@article{31cd642a5aa34bcbbee09f92cde14a2f,

title = "Reproducibility of radiomics quality score: an intra- and inter-rater reliability study",

abstract = "OBJECTIVES: To investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.METHODS: Nine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.RESULTS: The inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).CONCLUSIONS: Reproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.CLINICAL RELEVANCE STATEMENT: There is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.KEY POINTS: • Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.",

author = "\{EuSoMII Radiomics Auditing Group\} and \{Akinci D'Antonoli\}, Tugba and Cavallo, \{Armando Ugo\} and Federica Vernuccio and Arnaldo Stanzione and Klontzas, \{Michail E\} and Roberto Cannella and Lorenzo Ugga and Agah Baran and Fanni, \{Salvatore Claudio\} and Ekaterina Petrash and Ilaria Ambrosini and Cappellini, \{Luca Alessandro\} and \{van Ooijen\}, Peter and Elmar Kotter and \{Pinto dos Santos\}, Daniel and Renato Cuocolo",

note = "{\textcopyright} 2023. The Author(s).",

year = "2024",

month = apr,

doi = "10.1007/s00330-023-10217-x",

language = "English",

volume = "34",

pages = "2791–2804",

journal = "European Radiology",

issn = "0938-7994",

publisher = "Springer",

}

EuSoMII Radiomics Auditing Group, Akinci D'Antonoli, T, Cavallo, AU, Vernuccio, F, Stanzione, A, Klontzas, ME, Cannella, R, Ugga, L, Baran, A, Fanni, SC, Petrash, E, Ambrosini, I, Cappellini, LA, van Ooijen, P, Kotter, E, Pinto dos Santos, D & Cuocolo, R 2024, 'Reproducibility of radiomics quality score: an intra- and inter-rater reliability study', European Radiology, vol. 34, pp. 2791–2804. https://doi.org/10.1007/s00330-023-10217-x

TY - JOUR

T1 - Reproducibility of radiomics quality score

T2 - an intra- and inter-rater reliability study

AU - EuSoMII Radiomics Auditing Group

AU - Akinci D'Antonoli, Tugba

AU - Cavallo, Armando Ugo

AU - Vernuccio, Federica

AU - Stanzione, Arnaldo

AU - Klontzas, Michail E

AU - Cannella, Roberto

AU - Ugga, Lorenzo

AU - Baran, Agah

AU - Fanni, Salvatore Claudio

AU - Petrash, Ekaterina

AU - Ambrosini, Ilaria

AU - Cappellini, Luca Alessandro

AU - van Ooijen, Peter

AU - Kotter, Elmar

AU - Pinto dos Santos, Daniel

AU - Cuocolo, Renato

PY - 2024/4

Y1 - 2024/4

N2 - OBJECTIVES: To investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.METHODS: Nine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.RESULTS: The inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).CONCLUSIONS: Reproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.CLINICAL RELEVANCE STATEMENT: There is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.KEY POINTS: • Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

AB - OBJECTIVES: To investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.METHODS: Nine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.RESULTS: The inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).CONCLUSIONS: Reproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.CLINICAL RELEVANCE STATEMENT: There is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.KEY POINTS: • Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

U2 - 10.1007/s00330-023-10217-x

DO - 10.1007/s00330-023-10217-x

M3 - Article

C2 - 37733025

SN - 0938-7994

VL - 34

SP - 2791

EP - 2804

JO - European Radiology

JF - European Radiology

ER -

Reproducibility of radiomics quality score: an intra- and inter-rater reliability study

Abstract

Access to Document

Handle.net

Fingerprint

Cite this