Selective Information Extraction Strategies for Cancer Pathology Reports with Convolutional Neural Networks

Yoon, Hong-Jun; Qiu, John X.; Christian, Blair; Hinkle, Jacob; Alamudun, Folami; Tourassi, Georgia

doi:10.1007/978-3-030-16841-4_9

Title: Selective Information Extraction Strategies for Cancer Pathology Reports with Convolutional Neural Networks

Abstract

To trust model predictions, it is important to ensure new data scored by the model comes from the same population used for model training. If the model is used to score new data different than the model’s training data, then predictions and model performance metrics cannot be trusted. Identifying and excluding these anomalous data points is an important task when using models in the real world. Traditional machine learning algorithms and classifiers don’t have the capability to abstain in this case. Here we propose a data-novelty detection algorithm for the Convolutional Neural Network classifier, yielding a rejection score for each new data point scored. It is a post-modeling procedure which examines the distribution of convolution filters to determine if the prediction should be trusted. We apply this algorithm to an information extraction model for a natural language text corpus. We evaluated the algorithm performance using a primary cancer site classification model applied to cancer pathology reports. Results demonstrate that the algorithm is an effective way to exclude cancer pathology reports from model scoring when they do not contain the expected information necessary to accurately classify the primary cancer type.

Authors:

^[1]; Qiu, John X. ^[1];

^[1];

^[1]

ORNL

Publication Date:: 2019-04-01

Research Org.:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1509553

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: INNS Big Data and Deep Learning 2019 - Genoa, , Italy - 4/16/2019 4:00:00 AM-4/18/2019 4:00:00 AM

Country of Publication:: United States

Language:: English

Citation Formats


                    Yoon, Hong-Jun, Qiu, John X., Christian, Blair, Hinkle, Jacob, Alamudun, Folami, and Tourassi, Georgia. Selective Information Extraction Strategies for Cancer Pathology Reports with Convolutional Neural Networks.  United States: N. p., 2019. 
        Web.  doi:10.1007/978-3-030-16841-4_9.

Copy to clipboard


                    Yoon, Hong-Jun, Qiu, John X., Christian, Blair, Hinkle, Jacob, Alamudun, Folami, & Tourassi, Georgia. Selective Information Extraction Strategies for Cancer Pathology Reports with Convolutional Neural Networks.  United States.  https://doi.org/10.1007/978-3-030-16841-4_9

Copy to clipboard


                    Yoon, Hong-Jun, Qiu, John X., Christian, Blair, Hinkle, Jacob, Alamudun, Folami, and Tourassi, Georgia. 2019.  
        "Selective Information Extraction Strategies for Cancer Pathology Reports with Convolutional Neural Networks".  United States.  https://doi.org/10.1007/978-3-030-16841-4_9.  https://www.osti.gov/servlets/purl/1509553.

Copy to clipboard


                    
@article{osti_1509553,

  title        = {Selective Information Extraction Strategies for Cancer Pathology Reports with Convolutional Neural Networks},

  author       = {Yoon, Hong-Jun and Qiu, John X. and Christian, Blair and Hinkle, Jacob and Alamudun, Folami and Tourassi, Georgia},

  abstractNote = {To trust model predictions, it is important to ensure new data scored by the model comes from the same population used for model training. If the model is used to score new data different than the model’s training data, then predictions and model performance metrics cannot be trusted. Identifying and excluding these anomalous data points is an important task when using models in the real world. Traditional machine learning algorithms and classifiers don’t have the capability to abstain in this case. Here we propose a data-novelty detection algorithm for the Convolutional Neural Network classifier, yielding a rejection score for each new data point scored. It is a post-modeling procedure which examines the distribution of convolution filters to determine if the prediction should be trusted. We apply this algorithm to an information extraction model for a natural language text corpus. We evaluated the algorithm performance using a primary cancer site classification model applied to cancer pathology reports. Results demonstrate that the algorithm is an effective way to exclude cancer pathology reports from model scoring when they do not contain the expected information necessary to accurately classify the primary cancer type.},

  doi          = {10.1007/978-3-030-16841-4_9},

  url          = {https://www.osti.gov/biblio/1509553},
  journal      = {},

  issn         = {1064-3745},

  number       = ,

  volume       = ,

  place        = {United States},

  year         = {2019},

  month        = {4}

}

Copy to clipboard

Conference:

View Conference (0.33 MB)

https://doi.org/10.1007/978-3-030-16841-4_9

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

On the Exact Variance of Products
journal, December 1960

Goodman, Leo A.
Journal of the American Statistical Association, Vol. 55, Issue 292
https://doi.org/10.1080/01621459.1960.10483369

The 2007 WHO Classification of Tumours of the Central Nervous System
journal, July 2007

Louis, David N.; Ohgaki, Hiroko; Wiestler, Otmar D.
Acta Neuropathologica, Vol. 114, Issue 2
https://doi.org/10.1007/s00401-007-0243-4

Extracting Information from Textual Documents in the Electronic Health Record: A Review of Recent Research
journal, January 2008

Meystre, S. M.; Savova, G. K.; Kipper-Schuler, K. C.
Yearbook of Medical Informatics, Vol. 17, Issue 01
https://doi.org/10.1055/s-0038-1638592

Deep learning
journal, May 2015

LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
Nature, Vol. 521, Issue 7553
https://doi.org/10.1038/nature14539

Bayesian Data Analysis
book, November 2013

Gelman, Andrew; Carlin, John B.; Stern, Hal S.
https://doi.org/10.1201/b16018

Filter pruning of Convolutional Neural Networks for text classification: A case study of cancer pathology report comprehension
conference, March 2018

Yoon, Hong-Jun; Robinson, Sarah; Christian, J. Blair
2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI)
https://doi.org/10.1109/BHI.2018.8333439

Convolutional Neural Networks for Sentence Classification
conference, January 2014

Kim, Yoon
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)
https://doi.org/10.3115/v1/D14-1181

Similar records in OSTI.GOV collections:

Filter pruning of Convolutional Neural Networks for text classification: A case study of cancer pathology report comprehension

Conference Yoon, Hong-Jun; Robinson, Sarah; Christian, Blair; ...

Convolutional Neural Networks (CNN) have recently demonstrated effective performance in many Natural Language Processing tasks. In this study, we explore a novel approach for pruning a CNN's convolution filters using our new data-driven utility score. We have applied this technique to an information extraction task of classifying a dataset of cancer pathology reports by cancer type, a highly imbalanced dataset. Compared to standard CNN training, our new algorithm resulted in a nearly .07 increase in the micro-averaged F1-score and a strong .22 increase in the macro-averaged F1-score using a model with nearly a third fewer network weights. We show howmore »« less
https://doi.org/10.1109/BHI.2018.8333439

Full Text Available
Automatic extraction of cancer registry reportable information from free-text pathology reports using multitask convolutional neural networks

Journal Article Alawad, Mohammed; Gao, Shang; Qiu, John; ... - Journal of the American Medical Informatics Association

Objective: In this work we implement 2 different multitask learning (MTL) techniques, hard parameter sharing and cross-stitch, to train a word-level convolutional neural network (CNN) specifically designed for automatic extraction of cancer data from unstructured text in pathology reports. We show the importance of learning related information extraction (IE) tasks leveraging shared representations across the tasks to achieve state-of-the-art performance in classification accuracy and computational efficiency. Materials and Methods: Multitask CNN (MTCNN) attempts to tackle document information extraction by learning to extract multiple key cancer characteristics simultaneously. We trained our MTCNN to perform 5 information extraction tasks: (1) primary cancermore »« less
Cited by 22
https://doi.org/10.1093/jamia/ocz153
Coarse-to-Fine Multi-Task Training of Convolutional Neural Networks for Automated Information Extraction from Cancer Pathology Reports

Conference Alawad, Mohammed; Yoon, Hong-Jun; Tourassi, Georgia

Information extraction and coding of free-text pathology reports is an important activity for cancer registries to support national cancer surveillance. Cancer registrars must process high volumes of pathology reports on an annual basis. In this study, we investigated an automated approach using a coarse-to-fine training of convolutional neural networks (CNNs) for extracting the primary site, histological grade and laterality from unstructured cancer pathology text reports. Our proposed training scheme consists of two stages. In the first stage, the multi-task learning (MTL) with hard parameter sharing approach is used to train a multi-task MT-CNN model for all the tasks. Then, themore »« less
https://doi.org/10.1109/BHI.2018.8333408

Full Text Available
Model-based Hyperparameter Optimization of Convolutional Neural Networks for Information Extraction from Cancer Pathology Reports on HPC

Conference Yoon, Hong-Jun; Gounley, John; Gao, Shang; ...

Finding optimal hyperparameters is necessary to identify the best performing deep learning models but the process is costly. In this paper, we applied model-based optimization, also known as Bayesian optimization, using the CANDLE framework implemented on a High-Performance Computing environment. As a use case we selected information extraction from cancer pathology reports using a multi-task convolutional neural network, and hierarchical convolutional attention network to be optimized. We utilized a synthesized text corpus of 8,000 training cases and 2,000 validation cases with four types of clinical task labels including primary cancer site, laterality, behavior, and histological grade. We demonstrated that hyperparametermore »« less
https://doi.org/10.1109/BHI.2019.8834674

Full Text Available
Automatic information extraction from childhood cancer pathology reports

Journal Article Yoon, Hong-Jun; Peluso, Alina; Durbin, Eric; ... - JAMIA Open

The International Classification of Childhood Cancer (ICCC) facilitates the effective classification of a heterogeneous group of cancers in the important pediatric population. However, there has been no development of machine learning models for the ICCC classification. We developed deep learning-based information extraction models from cancer pathology reports based on the ICD-O-3 coding standard. In this article, we describe extending the models to perform ICCC classification. We developed 2 models, ICD-O-3 classification and ICCC recoding (Model 1) and direct ICCC classification (Model 2), and 4 scenarios subject to the training sample size. We evaluated these models with a corpus consisting ofmore »« less
https://doi.org/10.1093/jamiaopen/ooac049

Full Text Available

Similar Records

Title: Selective Information Extraction Strategies for Cancer Pathology Reports with Convolutional Neural Networks

Abstract

Citation Formats

On the Exact Variance of Products journal, December 1960

The 2007 WHO Classification of Tumours of the Central Nervous System journal, July 2007

Extracting Information from Textual Documents in the Electronic Health Record: A Review of Recent Research journal, January 2008

Deep learning journal, May 2015

Bayesian Data Analysis book, November 2013

Filter pruning of Convolutional Neural Networks for text classification: A case study of cancer pathology report comprehension conference, March 2018

Convolutional Neural Networks for Sentence Classification conference, January 2014

On the Exact Variance of Products
journal, December 1960

The 2007 WHO Classification of Tumours of the Central Nervous System
journal, July 2007

Extracting Information from Textual Documents in the Electronic Health Record: A Review of Recent Research
journal, January 2008

Deep learning
journal, May 2015

Bayesian Data Analysis
book, November 2013

Filter pruning of Convolutional Neural Networks for text classification: A case study of cancer pathology report comprehension
conference, March 2018

Convolutional Neural Networks for Sentence Classification
conference, January 2014