skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Pitfalls in Prediction Modeling for Normal Tissue Toxicity in Radiation Therapy: An Illustration With the Individual Radiation Sensitivity and Mammary Carcinoma Risk Factor Investigation Cohorts

Abstract

Purpose: To identify the main causes underlying the failure of prediction models for radiation therapy toxicity to replicate. Methods and Materials: Data were used from two German cohorts, Individual Radiation Sensitivity (ISE) (n=418) and Mammary Carcinoma Risk Factor Investigation (MARIE) (n=409), of breast cancer patients with similar characteristics and radiation therapy treatments. The toxicity endpoint chosen was telangiectasia. The LASSO (least absolute shrinkage and selection operator) logistic regression method was used to build a predictive model for a dichotomized endpoint (Radiation Therapy Oncology Group/European Organization for the Research and Treatment of Cancer score 0, 1, or ≥2). Internal areas under the receiver operating characteristic curve (inAUCs) were calculated by a naïve approach whereby the training data (ISE) were also used for calculating the AUC. Cross-validation was also applied to calculate the AUC within the same cohort, a second type of inAUC. Internal AUCs from cross-validation were calculated within ISE and MARIE separately. Models trained on one dataset (ISE) were applied to a test dataset (MARIE) and AUCs calculated (exAUCs). Results: Internal AUCs from the naïve approach were generally larger than inAUCs from cross-validation owing to overfitting the training data. Internal AUCs from cross-validation were also generally larger than the exAUCs,more » reflecting heterogeneity in the predictors between cohorts. The best models with largest inAUCs from cross-validation within both cohorts had a number of common predictors: hypertension, normalized total boost, and presence of estrogen receptors. Surprisingly, the effect (coefficient in the prediction model) of hypertension on telangiectasia incidence was positive in ISE and negative in MARIE. Other predictors were also not common between the 2 cohorts, illustrating that overcoming overfitting does not solve the problem of replication failure of prediction models completely. Conclusions: Overfitting and cohort heterogeneity are the 2 main causes of replication failure of prediction models across cohorts. Cross-validation and similar techniques (eg, bootstrapping) cope with overfitting, but the development of validated predictive models for radiation therapy toxicity requires strategies that deal with cohort heterogeneity.« less

Authors:
 [1];  [2];  [1];  [3];  [4];  [5]; ; ;  [6];  [7];  [1]
  1. Department of Basic Medical Sciences, Faculty of Health Sciences, Ghent University, Ghent (Belgium)
  2. (Belgium)
  3. Department of Mathematical Modeling, Statistics, and Bioinformatics, Faculty of Bioscience Engineering, Ghent University, Ghent (Belgium)
  4. (Australia)
  5. Department of Data Analysis, Faculty of Psychology and Educational Sciences, Ghent University, Ghent (Belgium)
  6. Division of Cancer Epidemiology, German Cancer Research Center, Heidelberg (Germany)
  7. Translational Radiobiology Group, Institute of Cancer Sciences, Radiotherapy Related Research, Christie Hospital NHS Trust, University of Manchester, Manchester (United Kingdom)
Publication Date:
OSTI Identifier:
22648766
Resource Type:
Journal Article
Resource Relation:
Journal Name: International Journal of Radiation Oncology, Biology and Physics; Journal Volume: 95; Journal Issue: 5; Other Information: Copyright (c) 2016 Elsevier Science B.V., Amsterdam, The Netherlands, All rights reserved.; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
62 RADIOLOGY AND NUCLEAR MEDICINE; FORECASTING; MAMMARY GLANDS; NEOPLASMS; RADIATION HAZARDS; RADIOSENSITIVITY; RADIOTHERAPY; SIMULATION; TOXICITY; TRAINING

Citation Formats

Mbah, Chamberlain, E-mail: chamberlain.mbah@ugent.be, Department of Mathematical Modeling, Statistics, and Bioinformatics, Faculty of Bioscience Engineering, Ghent University, Ghent, Thierens, Hubert, Thas, Olivier, National Institute for Applied Statistics Research Australia, University of Wollongong, Wollongong, New South Wales, De Neve, Jan, Chang-Claude, Jenny, Seibold, Petra, Botma, Akke, West, Catharine, and De Ruyck, Kim. Pitfalls in Prediction Modeling for Normal Tissue Toxicity in Radiation Therapy: An Illustration With the Individual Radiation Sensitivity and Mammary Carcinoma Risk Factor Investigation Cohorts. United States: N. p., 2016. Web. doi:10.1016/J.IJROBP.2016.03.034.
Mbah, Chamberlain, E-mail: chamberlain.mbah@ugent.be, Department of Mathematical Modeling, Statistics, and Bioinformatics, Faculty of Bioscience Engineering, Ghent University, Ghent, Thierens, Hubert, Thas, Olivier, National Institute for Applied Statistics Research Australia, University of Wollongong, Wollongong, New South Wales, De Neve, Jan, Chang-Claude, Jenny, Seibold, Petra, Botma, Akke, West, Catharine, & De Ruyck, Kim. Pitfalls in Prediction Modeling for Normal Tissue Toxicity in Radiation Therapy: An Illustration With the Individual Radiation Sensitivity and Mammary Carcinoma Risk Factor Investigation Cohorts. United States. doi:10.1016/J.IJROBP.2016.03.034.
Mbah, Chamberlain, E-mail: chamberlain.mbah@ugent.be, Department of Mathematical Modeling, Statistics, and Bioinformatics, Faculty of Bioscience Engineering, Ghent University, Ghent, Thierens, Hubert, Thas, Olivier, National Institute for Applied Statistics Research Australia, University of Wollongong, Wollongong, New South Wales, De Neve, Jan, Chang-Claude, Jenny, Seibold, Petra, Botma, Akke, West, Catharine, and De Ruyck, Kim. 2016. "Pitfalls in Prediction Modeling for Normal Tissue Toxicity in Radiation Therapy: An Illustration With the Individual Radiation Sensitivity and Mammary Carcinoma Risk Factor Investigation Cohorts". United States. doi:10.1016/J.IJROBP.2016.03.034.
@article{osti_22648766,
title = {Pitfalls in Prediction Modeling for Normal Tissue Toxicity in Radiation Therapy: An Illustration With the Individual Radiation Sensitivity and Mammary Carcinoma Risk Factor Investigation Cohorts},
author = {Mbah, Chamberlain, E-mail: chamberlain.mbah@ugent.be and Department of Mathematical Modeling, Statistics, and Bioinformatics, Faculty of Bioscience Engineering, Ghent University, Ghent and Thierens, Hubert and Thas, Olivier and National Institute for Applied Statistics Research Australia, University of Wollongong, Wollongong, New South Wales and De Neve, Jan and Chang-Claude, Jenny and Seibold, Petra and Botma, Akke and West, Catharine and De Ruyck, Kim},
abstractNote = {Purpose: To identify the main causes underlying the failure of prediction models for radiation therapy toxicity to replicate. Methods and Materials: Data were used from two German cohorts, Individual Radiation Sensitivity (ISE) (n=418) and Mammary Carcinoma Risk Factor Investigation (MARIE) (n=409), of breast cancer patients with similar characteristics and radiation therapy treatments. The toxicity endpoint chosen was telangiectasia. The LASSO (least absolute shrinkage and selection operator) logistic regression method was used to build a predictive model for a dichotomized endpoint (Radiation Therapy Oncology Group/European Organization for the Research and Treatment of Cancer score 0, 1, or ≥2). Internal areas under the receiver operating characteristic curve (inAUCs) were calculated by a naïve approach whereby the training data (ISE) were also used for calculating the AUC. Cross-validation was also applied to calculate the AUC within the same cohort, a second type of inAUC. Internal AUCs from cross-validation were calculated within ISE and MARIE separately. Models trained on one dataset (ISE) were applied to a test dataset (MARIE) and AUCs calculated (exAUCs). Results: Internal AUCs from the naïve approach were generally larger than inAUCs from cross-validation owing to overfitting the training data. Internal AUCs from cross-validation were also generally larger than the exAUCs, reflecting heterogeneity in the predictors between cohorts. The best models with largest inAUCs from cross-validation within both cohorts had a number of common predictors: hypertension, normalized total boost, and presence of estrogen receptors. Surprisingly, the effect (coefficient in the prediction model) of hypertension on telangiectasia incidence was positive in ISE and negative in MARIE. Other predictors were also not common between the 2 cohorts, illustrating that overcoming overfitting does not solve the problem of replication failure of prediction models completely. Conclusions: Overfitting and cohort heterogeneity are the 2 main causes of replication failure of prediction models across cohorts. Cross-validation and similar techniques (eg, bootstrapping) cope with overfitting, but the development of validated predictive models for radiation therapy toxicity requires strategies that deal with cohort heterogeneity.},
doi = {10.1016/J.IJROBP.2016.03.034},
journal = {International Journal of Radiation Oncology, Biology and Physics},
number = 5,
volume = 95,
place = {United States},
year = 2016,
month = 8
}
  • Purpose: To identify dosimetric parameters that correlate with acute hematologic toxicity (HT) in patients with squamous cell carcinoma of the anal canal treated with definitive chemoradiotherapy (CRT). Methods and Materials: We analyzed 33 patients receiving CRT. Pelvic bone (PBM) was contoured for each patient and divided into subsites: ilium, lower pelvis (LP), and lumbosacral spine (LSS). The volume of each region receiving at least 5, 10, 15, 20, 30, and 40 Gy was calculated. Endpoints included grade {>=}3 HT (HT3+) and hematologic event (HE), defined as any grade {>=}2 HT with a modification in chemotherapy dose. Normal tissue complication probabilitymore » (NTCP) was evaluated with the Lyman-Kutcher-Burman (LKB) model. Logistic regression was used to test associations between HT and dosimetric/clinical parameters. Results: Nine patients experienced HT3+ and 15 patients experienced HE. Constrained optimization of the LKB model for HT3+ yielded the parameters m = 0.175, n = 1, and TD{sub 50} = 32 Gy. With this model, mean PBM doses of 25 Gy, 27.5 Gy, and 31 Gy result in a 10%, 20%, and 40% risk of HT3+, respectively. Compared with patients with mean PBM dose of <30 Gy, patients with mean PBM dose {>=}30 Gy had a 14-fold increase in the odds of developing HT3+ (p = 0.005). Several low-dose radiation parameters (i.e., PBM-V10) were associated with the development of HT3+ and HE. No association was found with the ilium, LP, or clinical factors. Conclusions: LKB modeling confirms the expectation that PBM acts like a parallel organ, implying that the mean dose to the organ is a useful predictor for toxicity. Low-dose radiation to the PBM was also associated with clinically significant HT. Keeping the mean PBM dose <22.5 Gy and <25 Gy is associated with a 5% and 10% risk of HT, respectively.« less
  • Purpose: To describe the probability of RILD by application of the Lyman-Kutcher-Burman normal-tissue complication (NTCP) model for primary liver carcinoma (PLC) treated with hypofractionated three-dimensional conformal radiotherapy (3D-CRT). Methods and Materials: A total of 109 PLC patients treated by 3D-CRT were followed for RILD. Of these patients, 93 were in liver cirrhosis of Child-Pugh Grade A, and 16 were in Child-Pugh Grade B. The Michigan NTCP model was used to predict the probability of RILD, and then the modified Lyman NTCP model was generated for Child-Pugh A and Child-Pugh B patients by maximum-likelihood analysis. Results: Of all patients, 17 developedmore » RILD in which 8 were of Child-Pugh Grade A, and 9 were of Child-Pugh Grade B. The prediction of RILD by the Michigan model was underestimated for PLC patients. The modified n, m, TD{sub 5} (1) were 1.1, 0.28, and 40.5 Gy and 0.7, 0.43, and 23 Gy for patients with Child-Pugh A and B, respectively, which yielded better estimations of RILD probability. The hepatic tolerable doses (TD{sub 5}) would be MDTNL of 21 Gy and 6 Gy, respectively, for Child-Pugh A and B patients. Conclusions: The Michigan model was probably not fit to predict RILD in PLC patients. A modified Lyman NTCP model for RILD was recommended.« less
  • The goal of this study was to implement and validate a noninvasive, quantitative ultrasonic technique for accurate and reproducible measurement of normal-tissue toxicity in radiation therapy. The authors adapted an existing ultrasonic tissue characterization (UTC) technique that used a calibrated 1D spectrum based on region-of-interest analysis. They modified the calibration procedure by using a reference phantom instead of a planar reflector. This UTC method utilized ultrasonic radio-frequency echo signals to generate spectral parameters related to the physical properties (e.g., size, shape, and relative acoustic impedance) of tissue microstructures. Three spectral parameters were investigated for quantification of normal-tissue injury: Spectral slope,more » intercept, and midband fit. They conducted a tissue-mimicking phantom study to verify the reproducibility of UTC measurements and initiated a clinical study of radiation-induced breast-tissue toxicity. Spectral parameter values from measurements on two phantoms were reproducible within 1% of each other. Eleven postradiation breast-cancer patients were studied and significant differences between the irradiated and untreated (contralateral) breasts were observed for spectral intercept (p=0.003) and midband fit (p<0.001) but not for slope (p=0.14). In comparison to the untreated breast, the average difference in the spectral intercept was 2.99{+-}0.75 dB and the average difference in the midband fit was 3.99{+-}0.65 dB. The preliminary clinical study demonstrated the feasibility of using the quantitative ultrasonic method to evaluate normal-tissue toxicity in radiation therapy.« less
  • By using a unique short-term culture system capable of growing both normal and malignant breast epithelial tissue, human recombinant tumor necrosis factor (TNF) showed preferential cytotoxicity to malignant cells as compared to the corresponding nonmalignant cells. Most of the malignant specimens were sensitive to TNF with 13 of 18 specimens showing 90% inhibition of clonal growth (ID{sub 90}). In contrast, all 13 nonmalignant specimens tested clustered at the resistant end of the TNF response spectrum. This differential sensitivity to TNF was seen in three cases in which malignant and nonmalignant breast epithelial tissues from the same patient were studied. Tomore » investigate the mechanism of resistance to TNF by normal cells, the presence of receptors for TNF was determined. Five of six cultures showed specific binding of {sup 125}I-labeled TNF and there was no relationship between the degree of resistance and the degree of specific binding. Simultaneous comparison of tumor responsiveness to doxorubicin and TNF revealed a positive correlation in ID{sub 90} values; these results may have important implications for the clinical use of TNF in cancer patients heavily pretreated with doxorubicin.« less