LUNGx Challenge for computerized lung nodule classification
Abstract
The purpose of this work is to describe the LUNGx Challenge for the computerized classification of lung nodules on diagnostic computed tomography (CT) scans as benign or malignant and report the performance of participants’ computerized methods along with that of six radiologists who participated in an observer study performing the same Challenge task on the same dataset. The Challenge provided sets of calibration and testing scans, established a performance assessment process, and created an infrastructure for case dissemination and result submission. We present ten groups that applied their own methods to 73 lung nodules (37 benign and 36 malignant) that were selected to achieve approximate size matching between the two cohorts. Area under the receiver operating characteristic curve (AUC) values for these methods ranged from 0.50 to 0.68; only three methods performed statistically better than random guessing. The radiologists’ AUC values ranged from 0.70 to 0.85; three radiologists performed statistically better than the best-performing computer method. The LUNGx Challenge compared the performance of computerized methods in the task of differentiating benign from malignant lung nodules on CT scans, placed in the context of the performance of radiologists on the same task. Lastly, the continued public availability of the Challenge casesmore »
- Authors:
-
- Univ. of Chicago, IL (United States). Department of Radiology
- Univ. of Michigan, Ann Arbor, MI (United States). Department of Radiology
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Health Data Sciences Institute, Biomedical Science and Engineering Center
- National Cancer Institute, Cancer Imaging Program, Division of Cancer Treatment and Diagnosis, Bethesda, MD (United States)
- Leidos Biomedical Research, Inc., Frederick National Laboratory for Cancer Research, Cancer Imaging Program, Frederick, MD (United States)
- Publication Date:
- Research Org.:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1338539
- Grant/Contract Number:
- AC05-00OR22725
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Journal of Medical Imaging
- Additional Journal Information:
- Journal Volume: 3; Journal Issue: 4; Conference: LUNGx Challenge for computerized lung nodule classification; Journal ID: ISSN 2329-4302
- Publisher:
- SPIE
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 62 RADIOLOGY AND NUCLEAR MEDICINE; 97 MATHEMATICS AND COMPUTING
Citation Formats
Armato, Samuel G., Drukker, Karen, Li, Feng, Hadjiiski, Lubomir, Tourassi, Georgia D., Engelmann, Roger M., Giger, Maryellen L., Redmond, George, Farahani, Keyvan, Kirby, Justin S., and Clarke, Laurence P. LUNGx Challenge for computerized lung nodule classification. United States: N. p., 2016.
Web. doi:10.1117/1.JMI.3.4.044506.
Armato, Samuel G., Drukker, Karen, Li, Feng, Hadjiiski, Lubomir, Tourassi, Georgia D., Engelmann, Roger M., Giger, Maryellen L., Redmond, George, Farahani, Keyvan, Kirby, Justin S., & Clarke, Laurence P. LUNGx Challenge for computerized lung nodule classification. United States. https://doi.org/10.1117/1.JMI.3.4.044506
Armato, Samuel G., Drukker, Karen, Li, Feng, Hadjiiski, Lubomir, Tourassi, Georgia D., Engelmann, Roger M., Giger, Maryellen L., Redmond, George, Farahani, Keyvan, Kirby, Justin S., and Clarke, Laurence P. Mon .
"LUNGx Challenge for computerized lung nodule classification". United States. https://doi.org/10.1117/1.JMI.3.4.044506. https://www.osti.gov/servlets/purl/1338539.
@article{osti_1338539,
title = {LUNGx Challenge for computerized lung nodule classification},
author = {Armato, Samuel G. and Drukker, Karen and Li, Feng and Hadjiiski, Lubomir and Tourassi, Georgia D. and Engelmann, Roger M. and Giger, Maryellen L. and Redmond, George and Farahani, Keyvan and Kirby, Justin S. and Clarke, Laurence P.},
abstractNote = {The purpose of this work is to describe the LUNGx Challenge for the computerized classification of lung nodules on diagnostic computed tomography (CT) scans as benign or malignant and report the performance of participants’ computerized methods along with that of six radiologists who participated in an observer study performing the same Challenge task on the same dataset. The Challenge provided sets of calibration and testing scans, established a performance assessment process, and created an infrastructure for case dissemination and result submission. We present ten groups that applied their own methods to 73 lung nodules (37 benign and 36 malignant) that were selected to achieve approximate size matching between the two cohorts. Area under the receiver operating characteristic curve (AUC) values for these methods ranged from 0.50 to 0.68; only three methods performed statistically better than random guessing. The radiologists’ AUC values ranged from 0.70 to 0.85; three radiologists performed statistically better than the best-performing computer method. The LUNGx Challenge compared the performance of computerized methods in the task of differentiating benign from malignant lung nodules on CT scans, placed in the context of the performance of radiologists on the same task. Lastly, the continued public availability of the Challenge cases will provide a valuable resource for the medical imaging research community.},
doi = {10.1117/1.JMI.3.4.044506},
journal = {Journal of Medical Imaging},
number = 4,
volume = 3,
place = {United States},
year = {Mon Dec 19 00:00:00 EST 2016},
month = {Mon Dec 19 00:00:00 EST 2016}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journalFigures / Tables:
Figure 1: The interface developed for the observer study allowed a user to raster through all section images of a scan, manipulate the visualization settings, and view relevant patient and image-acquisition information from the image DICOM headers. Nodules for evaluation were demarcated with blue crosshairs. Radiologists used the slider barmore »
All figures and tables
(4 total)
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: The ANODE09 study
journal, December 2010
- van Ginneken, Bram; Armato, Samuel G.; de Hoop, Bartjan
- Medical Image Analysis, Vol. 14, Issue 6
Extraction of Airways From CT (EXACT'09)
journal, November 2012
- Lo, Pechin; van Ginneken, Bram; Reinhardt, Joseph M.
- IEEE Transactions on Medical Imaging, Vol. 31, Issue 11
Computer Aided Characterization of the Solitary Pulmonary Nodule Using Volumetric and Contrast Enhancement Features1
journal, October 2005
- Shah, Sumit K.; McNitt-Gray, Michael F.; Rogers, Sarah R.
- Academic Radiology, Vol. 12, Issue 10
Computer-aided diagnosis of pulmonary nodules on CT scans: Segmentation and classification using 3D active contours: Computer-aided diagnosis of lung nodules on CT
journal, June 2006
- Way, Ted W.; Hadjiiski, Lubomir M.; Sahiner, Berkman
- Medical Physics, Vol. 33, Issue 7Part1
Genetic programming-based feature transform and classification for the automatic detection of pulmonary nodules on computed tomography images
journal, December 2012
- Choi, Wook-Jin; Choi, Tae-Sun
- Information Sciences, Vol. 212
Feature Selection and Performance Evaluation of Support Vector Machine (SVM)-Based Classifier for Differentiating Benign and Malignant Pulmonary Nodules by Computed Tomography
journal, February 2009
- Zhu, Yanjie; Tan, Yongqiang; Hua, Yanqing
- Journal of Digital Imaging, Vol. 23, Issue 1
Reliable and Computationally Efficient Maximum-Likelihood Estimation of “Proper” Binormal ROC Curves
journal, July 2007
- Pesce, Lorenzo L.; Metz, Charles E.
- Academic Radiology, Vol. 14, Issue 7
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
journal, July 2013
- Clark, Kenneth; Vendt, Bruce; Smith, Kirk
- Journal of Digital Imaging, Vol. 26, Issue 6
ROC Methodology in Radiologic Imaging
journal, January 1986
- Metz, Charles E.
- Investigative Radiology, Vol. 21, Issue 9
Effect of case selection on the performance of computer-aided detection schemes
journal, February 1994
- Nishikawa, Robert M.; Giger, Maryellen L.; Doi, Kunio
- Medical Physics, Vol. 21, Issue 2
A Query Tool for Investigator Access to the Data and Images of the National Lung Screening Trial
journal, March 2015
- Commean, Paul K.; Rathmell, Joshua M.; Clark, Ken W.
- Journal of Digital Imaging, Vol. 28, Issue 4
Solid, Part-Solid, or Non-Solid?: Classification of Pulmonary Nodules in Low-Dose Chest Computed Tomography by a Computer-Aided Diagnosis System
journal, January 2015
- Jacobs, Colin; van Rikxoort, Eva M.; Scholten, Ernst Th.
- Investigative Radiology, Vol. 50, Issue 3
Characterization of pulmonary nodules: effects of size and feature type on reported performance
conference, April 2008
- Jirapatnakul, Artit C.; Reeves, Anthony P.; Apanasovich, Tatiyana V.
- Medical Imaging, SPIE Proceedings
Research Imaging in an Academic Medical Center
journal, June 2012
- Armato, Samuel G.; Gruszauskas, Nicholas P.; MacMahon, Heber
- Academic Radiology, Vol. 19, Issue 6
The Effect of Verification on the Assessment of Imaging Techniques
journal, January 1983
- Revesz, George; Kundel, Harold L.; Bonitatibus, Michael
- Investigative Radiology, Vol. 18, Issue 2
A Mathematical Model for Predicting Malignancy of Solitary Pulmonary Nodules
journal, February 2012
- Li, Yun; Wang, Jun
- World Journal of Surgery, Vol. 36, Issue 4
Quantitative imaging features to predict cancer status in lung nodules
conference, March 2016
- Liu, Ying; Balagurunathan, Yoganand; Atwater, Thomas
- SPIE Medical Imaging, SPIE Proceedings
Probability of Cancer in Pulmonary Nodules Detected on First Screening CT
journal, September 2013
- McWilliams, Annette; Tammemagi, Martin C.; Mayo, John R.
- New England Journal of Medicine, Vol. 369, Issue 10
Novel and Convenient Method to Evaluate the Character of Solitary Pulmonary Nodule-Comparison of Three Mathematical Prediction Models and Further Stratification of Risk Factors
journal, October 2013
- Xiao, Fei; Liu, Deruo; Guo, Yongqing
- PLoS ONE, Vol. 8, Issue 10
A pattern classification approach to characterizing solitary pulmonary nodules imaged on high resolution CT: Preliminary results
journal, June 1999
- McNitt-Gray, Michael F.; Hart, Eric M.; Wyckoff, Nathaniel
- Medical Physics, Vol. 26, Issue 6
Comparing algorithms for automated vessel segmentation in computed tomography scans of the lung: the VESSEL12 study
journal, October 2014
- Rudyanto, Rina D.; Kerkstra, Sjoerd; van Rikxoort, Eva M.
- Medical Image Analysis, Vol. 18, Issue 7
New Guidelines to Evaluate the Response to Treatment in Solid Tumors
journal, February 2000
- Therasse, Patrick; Arbuck, Susan G.; Eisenhauer, Elizabeth A.
- JNCI: Journal of the National Cancer Institute, Vol. 92, Issue 3
Neural Network Ensemble-Based Computer-Aided Diagnosis for Differentiation of Lung Nodules on CT Images
journal, May 2010
- Chen, Hui; Xu, Yan; Ma, Yujing
- Academic Radiology, Vol. 17, Issue 5
SPIE-AAPM-NCI Lung Nodule Classification Challenge Dataset
dataset, January 2015
- P., Armato III, Samuel G. ; Hadjiiski, Lubomir; Tourassi, Georgia D. ; Drukker, Karen; Giger, Maryellen L. ; Li, Feng; Redmond, George; Farahani, Keyvan; Kirby, Justin S. ; Clarke, Laurence
- The Cancer Imaging Archive
The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans: The LIDC/IDRI thoracic CT database of lung nodules
journal, January 2011
- Armato, Samuel G.; McLennan, Geoffrey; Bidaut, Luc
- Medical Physics, Vol. 38, Issue 2
Assessing probability of malignancy in solid solitary pulmonary nodules with a new Bayesian calculator: improving diagnostic accuracy by means of expanded and updated features
journal, September 2014
- Soardi, G. A.; Perandini, Simone; Motton, M.
- European Radiology, Vol. 25, Issue 1
Automated lung nodule classification following automated nodule detection on CT: A serial approach
journal, May 2003
- Armato, Samuel G.; Altman, Michael B.; Wilkie, Joel
- Medical Physics, Vol. 30, Issue 6
Guest Editorial: LUNGx Challenge for computerized lung nodule classification: reflections and lessons learned
journal, April 2015
- Armato, Samuel G.; Hadjiiski, Lubomir; Tourassi, Georgia D.
- Journal of Medical Imaging, Vol. 2, Issue 2
Lung Lesion Extraction Using a Toboggan Based Growing Automatic Segmentation Approach
journal, January 2016
- Jiangdian Song,
- IEEE Transactions on Medical Imaging, Vol. 35, Issue 1
Improved pulmonary nodule classification utilizing quantitative lung parenchyma features
journal, September 2015
- Dilger, Samantha K. N.; Uthoff, Johanna; Judisch, Alexandra
- Journal of Medical Imaging, Vol. 2, Issue 4
Variations in measured performance of CAD schemes due to database composition and scoring protocol
conference, June 1998
- Nishikawa, Robert M.; Yarusso, Laura M.
- Medical Imaging '98, SPIE Proceedings
Solid pulmonary nodule risk assessment and decision analysis: comparison of four prediction models in 285 cases
journal, December 2015
- Perandini, Simone; Soardi, Gian Alberto; Motton, Massimiliano
- European Radiology, Vol. 26, Issue 9
The Lung Reporting and Data System (LU-RADS): A Proposal for Computed Tomography Screening
journal, May 2014
- Manos, Daria; Seely, Jean M.; Taylor, Jana
- Canadian Association of Radiologists Journal, Vol. 65, Issue 2
Works referencing / citing this record:
Radiomics analysis of pulmonary nodules in low-dose CT for early detection of lung cancer
journal, March 2018
- Choi, Wookjin; Oh, Jung Hun; Riyahi, Sadegh
- Medical Physics, Vol. 45, Issue 4
Machine learning approach for distinguishing malignant and benign lung nodules utilizing standardized perinodular parenchymal features from CT
journal, June 2019
- Uthoff, Johanna; Stephens, Matthew J.; Newell, John D.
- Medical Physics
Evolving the pulmonary nodules diagnosis from classical approaches to deep learning-aided decision support: three decades’ development course and future prospect
journal, November 2019
- Liu, Bo; Chi, Wenhao; Li, Xinran
- Journal of Cancer Research and Clinical Oncology, Vol. 146, Issue 1
Novel convolutional neural network architecture for improved pulmonary nodule classification on computed tomography
journal, February 2020
- Wang, Yi; Zhang, Hao; Chae, Kum Ju
- Multidimensional Systems and Signal Processing, Vol. 31, Issue 3
Lung Nodule: Imaging Features and Evaluation in the Age of Machine Learning
journal, July 2019
- Sayyouh, M.; Hadjiiyski, L. M.; Chan, H-P.
- Current Pulmonology Reports, Vol. 8, Issue 3
Pulmonary nodule classification in lung cancer screening with three-dimensional convolutional neural networks
journal, October 2017
- Liu, Shuang; Xie, Yiting; Jirapatnakul, Artit
- Journal of Medical Imaging, Vol. 4, Issue 04
Lessons on managing pulmonary nodules from NELSON: we have come a long way
journal, March 2019
- Horst, Carolyn; Nair, Arjun; Janes, Sam M.
- Thorax, Vol. 74, Issue 5
Artificial intelligence in oncology, its scope and future prospects with specific reference to radiation oncology
journal, July 2019
- Rattan, Rajit; Kataria, Tejinder; Banerjee, Susovan
- BJR|Open, Vol. 1, Issue 1
An Appraisal of Lung Nodules Automatic Classification Algorithms for CT Images
journal, January 2019
- Wang, Xinqi; Mao, Keming; Wang, Lizhe
- Sensors, Vol. 19, Issue 1
Figures / Tables found in this record:
Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.