LUNGx Challenge for computerized lung nodule classification

Armato, Samuel G.; Drukker, Karen; Li, Feng; Hadjiiski, Lubomir; Tourassi, Georgia D.; Engelmann, Roger M.; Giger, Maryellen L.; Redmond, George; Farahani, Keyvan; Kirby, Justin S.; Clarke, Laurence P.

doi:10.1117/1.JMI.3.4.044506

LUNGx Challenge for computerized lung nodule classification

Journal Article · Mon Dec 19 00:00:00 EST 2016 · Journal of Medical Imaging

DOI:https://doi.org/10.1117/1.JMI.3.4.044506· OSTI ID:1338539

Armato, Samuel G. ^[1]; Drukker, Karen ^[1]; Li, Feng ^[1]; Hadjiiski, Lubomir ^[2]; Tourassi, Georgia D. ^[3]; Engelmann, Roger M. ^[1]; Giger, Maryellen L. ^[1]; Redmond, George ^[4]; Farahani, Keyvan ^[4]; Kirby, Justin S. ^[5]; Clarke, Laurence P. ^[4]

Univ. of Chicago, IL (United States). Department of Radiology
Univ. of Michigan, Ann Arbor, MI (United States). Department of Radiology
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Health Data Sciences Institute, Biomedical Science and Engineering Center
National Cancer Institute, Cancer Imaging Program, Division of Cancer Treatment and Diagnosis, Bethesda, MD (United States)
Leidos Biomedical Research, Inc., Frederick National Laboratory for Cancer Research, Cancer Imaging Program, Frederick, MD (United States)

The purpose of this work is to describe the LUNGx Challenge for the computerized classification of lung nodules on diagnostic computed tomography (CT) scans as benign or malignant and report the performance of participants’ computerized methods along with that of six radiologists who participated in an observer study performing the same Challenge task on the same dataset. The Challenge provided sets of calibration and testing scans, established a performance assessment process, and created an infrastructure for case dissemination and result submission. We present ten groups that applied their own methods to 73 lung nodules (37 benign and 36 malignant) that were selected to achieve approximate size matching between the two cohorts. Area under the receiver operating characteristic curve (AUC) values for these methods ranged from 0.50 to 0.68; only three methods performed statistically better than random guessing. The radiologists’ AUC values ranged from 0.70 to 0.85; three radiologists performed statistically better than the best-performing computer method. The LUNGx Challenge compared the performance of computerized methods in the task of differentiating benign from malignant lung nodules on CT scans, placed in the context of the performance of radiologists on the same task. Lastly, the continued public availability of the Challenge cases will provide a valuable resource for the medical imaging research community.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1338539

Journal Information:: Journal of Medical Imaging, Journal Name: Journal of Medical Imaging Journal Issue: 4 Vol. 3; ISSN 2329-4302

Publisher:: SPIECopyright Statement

Country of Publication:: United States

Language:: English

References (33)

A Mathematical Model for Predicting Malignancy of Solitary Pulmonary Nodules Li, Yun; Wang, Jun World Journal of Surgery, Vol. 36, Issue 4 https://doi.org/10.1007/s00268-012-1449-8	journal	February 2012
Assessing probability of malignancy in solid solitary pulmonary nodules with a new Bayesian calculator: improving diagnostic accuracy by means of expanded and updated features Soardi, G. A.; Perandini, Simone; Motton, M. European Radiology, Vol. 25, Issue 1 https://doi.org/10.1007/s00330-014-3396-2	journal	September 2014
Solid pulmonary nodule risk assessment and decision analysis: comparison of four prediction models in 285 cases Perandini, Simone; Soardi, Gian Alberto; Motton, Massimiliano European Radiology, Vol. 26, Issue 9 https://doi.org/10.1007/s00330-015-4138-9	journal	December 2015
Feature Selection and Performance Evaluation of Support Vector Machine (SVM)-Based Classifier for Differentiating Benign and Malignant Pulmonary Nodules by Computed Tomography Zhu, Yanjie; Tan, Yongqiang; Hua, Yanqing Journal of Digital Imaging, Vol. 23, Issue 1 https://doi.org/10.1007/s10278-009-9185-9	journal	February 2009
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository Clark, Kenneth; Vendt, Bruce; Smith, Kirk Journal of Digital Imaging, Vol. 26, Issue 6 https://doi.org/10.1007/s10278-013-9622-7	journal	July 2013
A Query Tool for Investigator Access to the Data and Images of the National Lung Screening Trial Commean, Paul K.; Rathmell, Joshua M.; Clark, Ken W. Journal of Digital Imaging, Vol. 28, Issue 4 https://doi.org/10.1007/s10278-015-9785-5	journal	March 2015
Computer Aided Characterization of the Solitary Pulmonary Nodule Using Volumetric and Contrast Enhancement Features1 Shah, Sumit K.; McNitt-Gray, Michael F.; Rogers, Sarah R. Academic Radiology, Vol. 12, Issue 10 https://doi.org/10.1016/j.acra.2005.06.005	journal	October 2005
Reliable and Computationally Efficient Maximum-Likelihood Estimation of “Proper” Binormal ROC Curves Pesce, Lorenzo L.; Metz, Charles E. Academic Radiology, Vol. 14, Issue 7 https://doi.org/10.1016/j.acra.2007.03.012	journal	July 2007
Neural Network Ensemble-Based Computer-Aided Diagnosis for Differentiation of Lung Nodules on CT Images Chen, Hui; Xu, Yan; Ma, Yujing Academic Radiology, Vol. 17, Issue 5 https://doi.org/10.1016/j.acra.2009.12.009	journal	May 2010
Research Imaging in an Academic Medical Center Armato, Samuel G.; Gruszauskas, Nicholas P.; MacMahon, Heber Academic Radiology, Vol. 19, Issue 6 https://doi.org/10.1016/j.acra.2012.02.002	journal	June 2012
The Lung Reporting and Data System (LU-RADS): A Proposal for Computed Tomography Screening Manos, Daria; Seely, Jean M.; Taylor, Jana Canadian Association of Radiologists Journal, Vol. 65, Issue 2 https://doi.org/10.1016/j.carj.2014.03.004	journal	May 2014
Genetic programming-based feature transform and classification for the automatic detection of pulmonary nodules on computed tomography images Choi, Wook-Jin; Choi, Tae-Sun Information Sciences, Vol. 212 https://doi.org/10.1016/j.ins.2012.05.008	journal	December 2012
Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: The ANODE09 study van Ginneken, Bram; Armato, Samuel G.; de Hoop, Bartjan Medical Image Analysis, Vol. 14, Issue 6 https://doi.org/10.1016/j.media.2010.05.005	journal	December 2010
Comparing algorithms for automated vessel segmentation in computed tomography scans of the lung: the VESSEL12 study Rudyanto, Rina D.; Kerkstra, Sjoerd; van Rikxoort, Eva M. Medical Image Analysis, Vol. 18, Issue 7 https://doi.org/10.1016/j.media.2014.07.003	journal	October 2014
Probability of Cancer in Pulmonary Nodules Detected on First Screening CT McWilliams, Annette; Tammemagi, Martin C.; Mayo, John R. New England Journal of Medicine, Vol. 369, Issue 10 https://doi.org/10.1056/NEJMoa1214726	journal	September 2013
New Guidelines to Evaluate the Response to Treatment in Solid Tumors Therasse, Patrick; Arbuck, Susan G.; Eisenhauer, Elizabeth A. JNCI: Journal of the National Cancer Institute, Vol. 92, Issue 3 https://doi.org/10.1093/jnci/92.3.205	journal	February 2000
The Effect of Verification on the Assessment of Imaging Techniques Revesz, George; Kundel, Harold L.; Bonitatibus, Michael Investigative Radiology, Vol. 18, Issue 2 https://doi.org/10.1097/00004424-198303000-00018	journal	January 1983
ROC Methodology in Radiologic Imaging Metz, Charles E. Investigative Radiology, Vol. 21, Issue 9 https://doi.org/10.1097/00004424-198609000-00009	journal	January 1986
Solid, Part-Solid, or Non-Solid?: Classification of Pulmonary Nodules in Low-Dose Chest Computed Tomography by a Computer-Aided Diagnosis System Jacobs, Colin; van Rikxoort, Eva M.; Scholten, Ernst Th. Investigative Radiology, Vol. 50, Issue 3 https://doi.org/10.1097/RLI.0000000000000121	journal	January 2015
Extraction of Airways From CT (EXACT'09) Lo, Pechin; van Ginneken, Bram; Reinhardt, Joseph M. IEEE Transactions on Medical Imaging, Vol. 31, Issue 11 https://doi.org/10.1109/TMI.2012.2209674	journal	November 2012
Lung Lesion Extraction Using a Toboggan Based Growing Automatic Segmentation Approach No authors listed IEEE Transactions on Medical Imaging, Vol. 35, Issue 1 https://doi.org/10.1109/TMI.2015.2474119	journal	January 2016
Guest Editorial: LUNGx Challenge for computerized lung nodule classification: reflections and lessons learned Armato, Samuel G.; Hadjiiski, Lubomir; Tourassi, Georgia D. Journal of Medical Imaging, Vol. 2, Issue 2 https://doi.org/10.1117/1.JMI.2.2.020103	journal	April 2015
Improved pulmonary nodule classification utilizing quantitative lung parenchyma features Dilger, Samantha K. N.; Uthoff, Johanna; Judisch, Alexandra Journal of Medical Imaging, Vol. 2, Issue 4 https://doi.org/10.1117/1.JMI.2.4.041004	journal	September 2015
Quantitative imaging features to predict cancer status in lung nodules Liu, Ying; Balagurunathan, Yoganand; Atwater, Thomas SPIE Medical Imaging, SPIE Proceedings https://doi.org/10.1117/12.2216809	conference	March 2016
Variations in measured performance of CAD schemes due to database composition and scoring protocol Nishikawa, Robert M.; Yarusso, Laura M. Medical Imaging '98, SPIE Proceedings https://doi.org/10.1117/12.310894	conference	June 1998
Characterization of pulmonary nodules: effects of size and feature type on reported performance Jirapatnakul, Artit C.; Reeves, Anthony P.; Apanasovich, Tatiyana V. Medical Imaging, SPIE Proceedings https://doi.org/10.1117/12.770514	conference	April 2008
Automated lung nodule classification following automated nodule detection on CT: A serial approach Armato, Samuel G.; Altman, Michael B.; Wilkie, Joel Medical Physics, Vol. 30, Issue 6 https://doi.org/10.1118/1.1573210	journal	May 2003
Computer-aided diagnosis of pulmonary nodules on CT scans: Segmentation and classification using 3D active contours: Computer-aided diagnosis of lung nodules on CT Way, Ted W.; Hadjiiski, Lubomir M.; Sahiner, Berkman Medical Physics, Vol. 33, Issue 7Part1 https://doi.org/10.1118/1.2207129	journal	June 2006
The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans: The LIDC/IDRI thoracic CT database of lung nodules Armato, Samuel G.; McLennan, Geoffrey; Bidaut, Luc Medical Physics, Vol. 38, Issue 2 https://doi.org/10.1118/1.3528204	journal	January 2011
Effect of case selection on the performance of computer-aided detection schemes Nishikawa, Robert M.; Giger, Maryellen L.; Doi, Kunio Medical Physics, Vol. 21, Issue 2 https://doi.org/10.1118/1.597287	journal	February 1994
A pattern classification approach to characterizing solitary pulmonary nodules imaged on high resolution CT: Preliminary results McNitt-Gray, Michael F.; Hart, Eric M.; Wyckoff, Nathaniel Medical Physics, Vol. 26, Issue 6 https://doi.org/10.1118/1.598603	journal	June 1999
Novel and Convenient Method to Evaluate the Character of Solitary Pulmonary Nodule-Comparison of Three Mathematical Prediction Models and Further Stratification of Risk Factors Xiao, Fei; Liu, Deruo; Guo, Yongqing PLoS ONE, Vol. 8, Issue 10 https://doi.org/10.1371/journal.pone.0078271	journal	October 2013
SPIE-AAPM-NCI Lung Nodule Classification Challenge Dataset P., Armato III, Samuel G. ; Hadjiiski, Lubomir; Tourassi, Georgia D. ; Drukker, Karen; Giger, Maryellen L. ; Li, Feng; Redmond, George; Farahani, Keyvan; Kirby, Justin S. ; Clarke, Laurence The Cancer Imaging Archive https://doi.org/10.7937/K9/TCIA.2015.UZLSU3FL	dataset	January 2015

Cited By (9)

Radiomics analysis of pulmonary nodules in low-dose CT for early detection of lung cancer Choi, Wookjin; Oh, Jung Hun; Riyahi, Sadegh Medical Physics, Vol. 45, Issue 4 https://doi.org/10.1002/mp.12820	journal	March 2018
Machine learning approach for distinguishing malignant and benign lung nodules utilizing standardized perinodular parenchymal features from CT Uthoff, Johanna; Stephens, Matthew J.; Newell, John D. Medical Physics https://doi.org/10.1002/mp.13592	journal	June 2019
Evolving the pulmonary nodules diagnosis from classical approaches to deep learning-aided decision support: three decades’ development course and future prospect Liu, Bo; Chi, Wenhao; Li, Xinran Journal of Cancer Research and Clinical Oncology, Vol. 146, Issue 1 https://doi.org/10.1007/s00432-019-03098-5	journal	November 2019
Novel convolutional neural network architecture for improved pulmonary nodule classification on computed tomography Wang, Yi; Zhang, Hao; Chae, Kum Ju Multidimensional Systems and Signal Processing, Vol. 31, Issue 3 https://doi.org/10.1007/s11045-020-00703-6	journal	February 2020
Lung Nodule: Imaging Features and Evaluation in the Age of Machine Learning Sayyouh, M.; Hadjiiyski, L. M.; Chan, H-P. Current Pulmonology Reports, Vol. 8, Issue 3 https://doi.org/10.1007/s13665-019-00229-8	journal	July 2019
Pulmonary nodule classification in lung cancer screening with three-dimensional convolutional neural networks Liu, Shuang; Xie, Yiting; Jirapatnakul, Artit Journal of Medical Imaging, Vol. 4, Issue 04 https://doi.org/10.1117/1.jmi.4.4.041308	journal	October 2017
Lessons on managing pulmonary nodules from NELSON: we have come a long way Horst, Carolyn; Nair, Arjun; Janes, Sam M. Thorax, Vol. 74, Issue 5 https://doi.org/10.1136/thoraxjnl-2018-212783	journal	March 2019
Artificial intelligence in oncology, its scope and future prospects with specific reference to radiation oncology Rattan, Rajit; Kataria, Tejinder; Banerjee, Susovan BJR\|Open, Vol. 1, Issue 1 https://doi.org/10.1259/bjro.20180031	journal	July 2019
An Appraisal of Lung Nodules Automatic Classification Algorithms for CT Images Wang, Xinqi; Mao, Keming; Wang, Lizhe Sensors, Vol. 19, Issue 1 https://doi.org/10.3390/s19010194	journal	January 2019