Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Class imbalance in out-of-distribution datasets: Improving the robustness of the TextCNN for the classification of rare cancer types

Journal Article · · Journal of Biomedical Informatics
 [1];  [2];  [3];  [4];  [5];  [6];  [7];  [8];  [9];  [10];  [11];  [12];  [2];  [2]
  1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States); Univ. of Tennessee, Knoxville, TN (United States)
  2. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  3. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States); Vanderbilt Univ., Nashville, TN (United States)
  4. Univ. of Kentucky, Lexington, KY (United States)
  5. Louisiana State Univ., New Orleans, LA (United States)
  6. Rutgers Univ., New Brunswick, NJ (United States)
  7. Univ. of Utah, Salt Lake City, UT (United States)
  8. Fred Hutchison Cancer Research Center, Seattle, WA (United States)
  9. Univ. of New Mexico, Albuquerque, NM (United States)
  10. California Dept. of Public Health, Sacremento, CA (United States)
  11. Information Management Services, Inc., Calverton, MD (United States)
  12. National Cancer Institute, Bethesda, MD (United States)
+ Show Author Affiliations

In the last decade, the widespread adoption of electronic health record documentation has created huge opportunities for information mining. Natural language processing (NLP) techniques using machine and deep learning are becoming increasingly widespread for information extraction tasks from unstructured clinical notes. Disparities in performance when deploying machine learning models in the real world have recently received considerable attention. In the clinical NLP domain, the robustness of convolutional neural networks (CNNs) for classifying cancer pathology reports under natural distribution shifts remains understudied. In this research, we aim to quantify and improve the performance of the CNN for text classification on out-of-distribution (OOD) datasets resulting from the natural evolution of clinical text in pathology reports. We identified class imbalance due to different prevalence of cancer types as one of the sources of performance drop and analyzed the impact of previous methods for addressing class imbalance when deploying models in real-world domains. Our results show that our novel class-specialized ensemble technique outperforms other methods for the classification of rare cancer types in terms of macro F1 scores. We also found that traditional ensemble methods perform better in top classes, leading to higher micro F1 scores. Based on our findings, we formulate a series of recommendations for other ML practitioners on how to build robust models with extremely imbalanced datasets in biomedical NLP applications.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Argonne National Laboratory (ANL), Argonne, IL (United States); Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States); Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Organization:
USDOE Office of Science (SC); USDOE National Nuclear Security Administration (NNSA); Centers for Disease Control and Prevention (CDC); National Cancer Institute (NCI); University of Utah; Huntsman Cancer Foundation
Grant/Contract Number:
AC05-00OR22725; AC02-06CH11357; AC52-07NA27344; AC52-06NA25396
OSTI ID:
1884003
Journal Information:
Journal of Biomedical Informatics, Journal Name: Journal of Biomedical Informatics Journal Issue: 1 Vol. 125; ISSN 1532-0464
Publisher:
ElsevierCopyright Statement
Country of Publication:
United States
Language:
English

References (24)

Bagging predictors journal August 1996
Classifying medical relations in clinical text via convolutional neural networks journal January 2019
Classifying cancer pathology reports with hierarchical self-attention networks journal November 2019
Dealing with Data Imbalance in Text Classification journal January 2019
Shortcut learning in deep neural networks journal November 2020
Hierarchical attention networks for information extraction from cancer pathology reports journal November 2017
Automatic extraction of cancer registry reportable information from free-text pathology reports using multitask convolutional neural networks journal November 2019
On Robustness and Transferability of Convolutional Neural Networks conference June 2021
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization conference October 2021
Clinical Text Classification with Rule-based Features and Knowledge-guided Convolutional Neural Networks conference June 2018
On the Class Imbalance Problem conference October 2008
Deep Learning for Automated Extraction of Primary Sites From Cancer Pathology Reports journal January 2018
Measuring Domain Shift for Deep Learning in Histopathology journal February 2021
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning journal August 2019
Experimental perspectives on learning from imbalanced data conference January 2007
Convolutional neural networks for biomedical text classification conference September 2015
Use of Natural Language Processing to Extract Clinical Cancer Phenotypes from Electronic Medical Records journal November 2019
SMOTE for high-dimensional class-imbalanced data journal March 2013
Deep active learning for classifying cancer pathology reports journal March 2021
Survey on deep learning with class imbalance journal March 2019
SMOTE: Synthetic Minority Over-sampling Technique journal January 2002
Hierarchical Convolutional Attention Networks for Text Classification conference January 2018
Data Sampling Methods to Deal With the Big Data Multi-Class Imbalance Problem journal February 2020
Deep active learning for classifying cancer pathology reports collection January 2021

Similar Records

Characterizing Quantum Classifier Utility in Natural Language Processing Workflows
Conference · Fri Sep 01 00:00:00 EDT 2023 · OSTI ID:2397467

Related Subjects

60 APPLIED LIFE SCIENCES
CNN
NLP
class imbalance
deep learning
ensemble
text classification