Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CT

Nikos Sourlos; GertJan Pelgrim; Hendrik Joost Wisselink; Xiaofei Yang; Gonda de Jonge; Mieneke Rook; Mathias Prokop; Grigory Sidorenkov; Marcel van Tuinen; Rozemarijn Vliegenthart; Peter M. A. van Ooijen

doi:10.1186/s41747-024-00459-9

Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CT

Nikos Sourlos, GertJan Pelgrim, Hendrik Joost Wisselink, Xiaofei Yang, Gonda de Jonge, Mieneke Rook, Mathias Prokop, Grigory Sidorenkov, Marcel van Tuinen, Rozemarijn Vliegenthart, Peter M. A. van Ooijen^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

30 Downloads (Pure)

Abstract

BACKGROUND: Emphysema influences the appearance of lung tissue in computed tomography (CT). We evaluated whether this affects lung nodule detection by artificial intelligence (AI) and human readers (HR).

METHODS: Individuals were selected from the "Lifelines" cohort who had undergone low-dose chest CT. Nodules in individuals without emphysema were matched to similar-sized nodules in individuals with at least moderate emphysema. AI results for nodular findings of 30-100 mm 3 and 101-300 mm 3 were compared to those of HR; two expert radiologists blindly reviewed discrepancies. Sensitivity and false positives (FPs)/scan were compared for emphysema and non-emphysema groups.

RESULTS: Thirty-nine participants with and 82 without emphysema were included (n = 121, aged 61 ± 8 years (mean ± standard deviation), 58/121 males (47.9%)). AI and HR detected 196 and 206 nodular findings, respectively, yielding 109 concordant nodules and 184 discrepancies, including 118 true nodules. For AI, sensitivity was 0.68 (95% confidence interval 0.57-0.77) in emphysema versus 0.71 (0.62-0.78) in non-emphysema, with FPs/scan 0.51 and 0.22, respectively (p = 0.028). For HR, sensitivity was 0.76 (0.65-0.84) and 0.80 (0.72-0.86), with FPs/scan of 0.15 and 0.27 (p = 0.230). Overall sensitivity was slightly higher for HR than for AI, but this difference disappeared after the exclusion of benign lymph nodes. FPs/scan were higher for AI in emphysema than in non-emphysema (p = 0.028), while FPs/scan for HR were higher than AI for 30-100 mm 3 nodules in non-emphysema (p = 0.009).

CONCLUSIONS: AI resulted in more FPs/scan in emphysema compared to non-emphysema, a difference not observed for HR.

RELEVANCE STATEMENT: In the creation of a benchmark dataset to validate AI software for lung nodule detection, the inclusion of emphysema cases is important due to the additional number of FPs.

KEY POINTS: • The sensitivity of nodule detection by AI was similar in emphysema and non-emphysema. • AI had more FPs/scan in emphysema compared to non-emphysema. • Sensitivity and FPs/scan by the human reader were comparable for emphysema and non-emphysema. • Emphysema and non-emphysema representation in benchmark dataset is important for validating AI.

Original language	English
Article number	63
Number of pages	11
Journal	European Radiology Experimental
Volume	8
DOIs	https://doi.org/10.1186/s41747-024-00459-9
Publication status	Published - 20-May-2024

Keywords

Humans
Male
Middle Aged
Female
Tomography, X-Ray Computed/methods
Artificial Intelligence
Pulmonary Emphysema/diagnostic imaging
Software
Sensitivity and Specificity
Lung Neoplasms/diagnostic imaging
Aged
Radiation Dosage
Solitary Pulmonary Nodule/diagnostic imaging
Radiographic Image Interpretation, Computer-Assisted/methods

Access to Document

10.1186/s41747-024-00459-9Licence: CC BY

Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CTFinal publisher's version, 2.36 MBLicence: CC BY

Handle.net

Persistent link

Cite this

Sourlos, N., Pelgrim, G., Wisselink, H. J., Yang, X., de Jonge, G., Rook, M., Prokop, M., Sidorenkov, G., van Tuinen, M., Vliegenthart, R., & van Ooijen, P. M. A. (2024). Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CT. European Radiology Experimental, 8, Article 63. https://doi.org/10.1186/s41747-024-00459-9

@article{6e60340fa7c14fa18d8e68a405145717,

title = "Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CT",

abstract = "BACKGROUND: Emphysema influences the appearance of lung tissue in computed tomography (CT). We evaluated whether this affects lung nodule detection by artificial intelligence (AI) and human readers (HR).METHODS: Individuals were selected from the {"}Lifelines{"} cohort who had undergone low-dose chest CT. Nodules in individuals without emphysema were matched to similar-sized nodules in individuals with at least moderate emphysema. AI results for nodular findings of 30-100 mm 3 and 101-300 mm 3 were compared to those of HR; two expert radiologists blindly reviewed discrepancies. Sensitivity and false positives (FPs)/scan were compared for emphysema and non-emphysema groups. RESULTS: Thirty-nine participants with and 82 without emphysema were included (n = 121, aged 61 ± 8 years (mean ± standard deviation), 58/121 males (47.9%)). AI and HR detected 196 and 206 nodular findings, respectively, yielding 109 concordant nodules and 184 discrepancies, including 118 true nodules. For AI, sensitivity was 0.68 (95% confidence interval 0.57-0.77) in emphysema versus 0.71 (0.62-0.78) in non-emphysema, with FPs/scan 0.51 and 0.22, respectively (p = 0.028). For HR, sensitivity was 0.76 (0.65-0.84) and 0.80 (0.72-0.86), with FPs/scan of 0.15 and 0.27 (p = 0.230). Overall sensitivity was slightly higher for HR than for AI, but this difference disappeared after the exclusion of benign lymph nodes. FPs/scan were higher for AI in emphysema than in non-emphysema (p = 0.028), while FPs/scan for HR were higher than AI for 30-100 mm 3 nodules in non-emphysema (p = 0.009). CONCLUSIONS: AI resulted in more FPs/scan in emphysema compared to non-emphysema, a difference not observed for HR.RELEVANCE STATEMENT: In the creation of a benchmark dataset to validate AI software for lung nodule detection, the inclusion of emphysema cases is important due to the additional number of FPs.KEY POINTS: • The sensitivity of nodule detection by AI was similar in emphysema and non-emphysema. • AI had more FPs/scan in emphysema compared to non-emphysema. • Sensitivity and FPs/scan by the human reader were comparable for emphysema and non-emphysema. • Emphysema and non-emphysema representation in benchmark dataset is important for validating AI.",

keywords = "Humans, Male, Middle Aged, Female, Tomography, X-Ray Computed/methods, Artificial Intelligence, Pulmonary Emphysema/diagnostic imaging, Software, Sensitivity and Specificity, Lung Neoplasms/diagnostic imaging, Aged, Radiation Dosage, Solitary Pulmonary Nodule/diagnostic imaging, Radiographic Image Interpretation, Computer-Assisted/methods",

author = "Nikos Sourlos and GertJan Pelgrim and Wisselink, {Hendrik Joost} and Xiaofei Yang and {de Jonge}, Gonda and Mieneke Rook and Mathias Prokop and Grigory Sidorenkov and {van Tuinen}, Marcel and Rozemarijn Vliegenthart and {van Ooijen}, {Peter M. A.}",

note = "{\textcopyright} 2024. The Author(s).",

year = "2024",

month = may,

day = "20",

doi = "10.1186/s41747-024-00459-9",

language = "English",

volume = "8",

journal = "European Radiology Experimental",

issn = "2509-9280",

publisher = "Springer",

}

TY - JOUR

T1 - Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CT

AU - Sourlos, Nikos

AU - Pelgrim, GertJan

AU - Wisselink, Hendrik Joost

AU - Yang, Xiaofei

AU - de Jonge, Gonda

AU - Rook, Mieneke

AU - Prokop, Mathias

AU - Sidorenkov, Grigory

AU - van Tuinen, Marcel

AU - Vliegenthart, Rozemarijn

AU - van Ooijen, Peter M. A.

PY - 2024/5/20

Y1 - 2024/5/20

N2 - BACKGROUND: Emphysema influences the appearance of lung tissue in computed tomography (CT). We evaluated whether this affects lung nodule detection by artificial intelligence (AI) and human readers (HR).METHODS: Individuals were selected from the "Lifelines" cohort who had undergone low-dose chest CT. Nodules in individuals without emphysema were matched to similar-sized nodules in individuals with at least moderate emphysema. AI results for nodular findings of 30-100 mm 3 and 101-300 mm 3 were compared to those of HR; two expert radiologists blindly reviewed discrepancies. Sensitivity and false positives (FPs)/scan were compared for emphysema and non-emphysema groups. RESULTS: Thirty-nine participants with and 82 without emphysema were included (n = 121, aged 61 ± 8 years (mean ± standard deviation), 58/121 males (47.9%)). AI and HR detected 196 and 206 nodular findings, respectively, yielding 109 concordant nodules and 184 discrepancies, including 118 true nodules. For AI, sensitivity was 0.68 (95% confidence interval 0.57-0.77) in emphysema versus 0.71 (0.62-0.78) in non-emphysema, with FPs/scan 0.51 and 0.22, respectively (p = 0.028). For HR, sensitivity was 0.76 (0.65-0.84) and 0.80 (0.72-0.86), with FPs/scan of 0.15 and 0.27 (p = 0.230). Overall sensitivity was slightly higher for HR than for AI, but this difference disappeared after the exclusion of benign lymph nodes. FPs/scan were higher for AI in emphysema than in non-emphysema (p = 0.028), while FPs/scan for HR were higher than AI for 30-100 mm 3 nodules in non-emphysema (p = 0.009). CONCLUSIONS: AI resulted in more FPs/scan in emphysema compared to non-emphysema, a difference not observed for HR.RELEVANCE STATEMENT: In the creation of a benchmark dataset to validate AI software for lung nodule detection, the inclusion of emphysema cases is important due to the additional number of FPs.KEY POINTS: • The sensitivity of nodule detection by AI was similar in emphysema and non-emphysema. • AI had more FPs/scan in emphysema compared to non-emphysema. • Sensitivity and FPs/scan by the human reader were comparable for emphysema and non-emphysema. • Emphysema and non-emphysema representation in benchmark dataset is important for validating AI.

AB - BACKGROUND: Emphysema influences the appearance of lung tissue in computed tomography (CT). We evaluated whether this affects lung nodule detection by artificial intelligence (AI) and human readers (HR).METHODS: Individuals were selected from the "Lifelines" cohort who had undergone low-dose chest CT. Nodules in individuals without emphysema were matched to similar-sized nodules in individuals with at least moderate emphysema. AI results for nodular findings of 30-100 mm 3 and 101-300 mm 3 were compared to those of HR; two expert radiologists blindly reviewed discrepancies. Sensitivity and false positives (FPs)/scan were compared for emphysema and non-emphysema groups. RESULTS: Thirty-nine participants with and 82 without emphysema were included (n = 121, aged 61 ± 8 years (mean ± standard deviation), 58/121 males (47.9%)). AI and HR detected 196 and 206 nodular findings, respectively, yielding 109 concordant nodules and 184 discrepancies, including 118 true nodules. For AI, sensitivity was 0.68 (95% confidence interval 0.57-0.77) in emphysema versus 0.71 (0.62-0.78) in non-emphysema, with FPs/scan 0.51 and 0.22, respectively (p = 0.028). For HR, sensitivity was 0.76 (0.65-0.84) and 0.80 (0.72-0.86), with FPs/scan of 0.15 and 0.27 (p = 0.230). Overall sensitivity was slightly higher for HR than for AI, but this difference disappeared after the exclusion of benign lymph nodes. FPs/scan were higher for AI in emphysema than in non-emphysema (p = 0.028), while FPs/scan for HR were higher than AI for 30-100 mm 3 nodules in non-emphysema (p = 0.009). CONCLUSIONS: AI resulted in more FPs/scan in emphysema compared to non-emphysema, a difference not observed for HR.RELEVANCE STATEMENT: In the creation of a benchmark dataset to validate AI software for lung nodule detection, the inclusion of emphysema cases is important due to the additional number of FPs.KEY POINTS: • The sensitivity of nodule detection by AI was similar in emphysema and non-emphysema. • AI had more FPs/scan in emphysema compared to non-emphysema. • Sensitivity and FPs/scan by the human reader were comparable for emphysema and non-emphysema. • Emphysema and non-emphysema representation in benchmark dataset is important for validating AI.

KW - Humans

KW - Male

KW - Middle Aged

KW - Female

KW - Tomography, X-Ray Computed/methods

KW - Artificial Intelligence

KW - Pulmonary Emphysema/diagnostic imaging

KW - Software

KW - Sensitivity and Specificity

KW - Lung Neoplasms/diagnostic imaging

KW - Aged

KW - Radiation Dosage

KW - Solitary Pulmonary Nodule/diagnostic imaging

KW - Radiographic Image Interpretation, Computer-Assisted/methods

U2 - 10.1186/s41747-024-00459-9

DO - 10.1186/s41747-024-00459-9

M3 - Article

C2 - 38764066

SN - 2509-9280

VL - 8

JO - European Radiology Experimental

JF - European Radiology Experimental

M1 - 63

ER -

Effect of emphysema on AI software and human reader performance in lung nodule detection from low-dose chest CT

Abstract

Keywords

Access to Document

Handle.net

Fingerprint

Cite this