Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine
Abstract
In the past, precision medicine involves classifying patients to identify subpopulations that respond favorably to specific therapeutics. We discuss precision medicine as a dynamic feedback control problem, where treatment administered to a patient is guided by measurements taken during the course of treatment. We consider sepsis, a life-threatening condition in which dysregulation of the immune system causes tissue damage. We leverage an existing simulation of the innate immune response to infection and apply deep reinforcement learning (DRL) to discover an adaptive personalized treatment policy that specifies effective multicytokine therapy to simulated sepsis patients based on systemic measurements. The learned policy achieves a dramatic reduction in mortality rate over a set of 500 simulated patients relative to standalone antibiotic therapy. Advantages of our approach are threefold: (1) the use of simulation allows exploring therapeutic strategies beyond clinical practice and available data, (2) advances in DRL accommodate learning complex therapeutic strategies for complex biological systems, and (3) optimized treatments respond to a patient's individual disease progression over time, therefore, capturing both differences across patients and the inherent randomness of disease progression within a single patient. We believe that this work motivates both considering adaptive personalized multicytokine mediation therapy for sepsis and exploitingmore »
- Authors:
-
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States); Georgia Inst. of Technology, Atlanta, GA (United States)
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States); Univ. of Toronto, ON (Canada)
- Univ. of Vermont, Burlington, VT (United States); Univ. of Chicago, IL (United States)
- Publication Date:
- Research Org.:
- Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
- Sponsoring Org.:
- USDOE National Nuclear Security Administration (NNSA); National Institutes of Health (NIH)
- OSTI Identifier:
- 1477825
- Report Number(s):
- LLNL-JRNL-745693
Journal ID: ISSN 1557-8666; 900608
- Grant/Contract Number:
- AC52-07NA27344
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Journal of Computational Biology
- Additional Journal Information:
- Journal Volume: 26; Journal Issue: 6; Journal ID: ISSN 1557-8666
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 59 BASIC BIOLOGICAL SCIENCES; 97 MATHEMATICS AND COMPUTING; agent-based model; deep reinforcement learning; precision medicine; sepsis
Citation Formats
Petersen, Brenden K., Yang, Jiachen, Grathwohl, Will S., Cockrell, Chase, Santiago, Claudio, An, Gary, and Faissol, Daniel M. Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine. United States: N. p., 2019.
Web. doi:10.1089/cmb.2018.0168.
Petersen, Brenden K., Yang, Jiachen, Grathwohl, Will S., Cockrell, Chase, Santiago, Claudio, An, Gary, & Faissol, Daniel M. Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine. United States. https://doi.org/10.1089/cmb.2018.0168
Petersen, Brenden K., Yang, Jiachen, Grathwohl, Will S., Cockrell, Chase, Santiago, Claudio, An, Gary, and Faissol, Daniel M. Thu .
"Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine". United States. https://doi.org/10.1089/cmb.2018.0168. https://www.osti.gov/servlets/purl/1477825.
@article{osti_1477825,
title = {Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine},
author = {Petersen, Brenden K. and Yang, Jiachen and Grathwohl, Will S. and Cockrell, Chase and Santiago, Claudio and An, Gary and Faissol, Daniel M.},
abstractNote = {In the past, precision medicine involves classifying patients to identify subpopulations that respond favorably to specific therapeutics. We discuss precision medicine as a dynamic feedback control problem, where treatment administered to a patient is guided by measurements taken during the course of treatment. We consider sepsis, a life-threatening condition in which dysregulation of the immune system causes tissue damage. We leverage an existing simulation of the innate immune response to infection and apply deep reinforcement learning (DRL) to discover an adaptive personalized treatment policy that specifies effective multicytokine therapy to simulated sepsis patients based on systemic measurements. The learned policy achieves a dramatic reduction in mortality rate over a set of 500 simulated patients relative to standalone antibiotic therapy. Advantages of our approach are threefold: (1) the use of simulation allows exploring therapeutic strategies beyond clinical practice and available data, (2) advances in DRL accommodate learning complex therapeutic strategies for complex biological systems, and (3) optimized treatments respond to a patient's individual disease progression over time, therefore, capturing both differences across patients and the inherent randomness of disease progression within a single patient. We believe that this work motivates both considering adaptive personalized multicytokine mediation therapy for sepsis and exploiting simulation with DRL for precision medicine more broadly.},
doi = {10.1089/cmb.2018.0168},
journal = {Journal of Computational Biology},
number = 6,
volume = 26,
place = {United States},
year = {Thu Jun 06 00:00:00 EDT 2019},
month = {Thu Jun 06 00:00:00 EDT 2019}
}
Web of Science
Works referenced in this record:
In silico experiments of existing and hypothetical cytokine-directed clinical trials using agent-based modeling*
journal, January 2004
- An, Gary
- Critical Care Medicine, Vol. 32, Issue 10
Optimization and Control of Agent-Based Models in Biology: A Perspective
journal, November 2016
- An, G.; Fitzpatrick, B. G.; Christley, S.
- Bulletin of Mathematical Biology, Vol. 79, Issue 1
Agent‐based models in translational systems biology
journal, September 2009
- An, Gary; Mi, Qi; Dutta‐Moscato, Joyeeta
- Wiley Interdisciplinary Reviews: Systems Biology and Medicine, Vol. 1, Issue 2
A systematic review and meta-analysis of early goal-directed therapy for septic shock: the ARISE, ProCESS and ProMISe Investigators
journal, May 2015
- Angus, D. C.; Barnato, A. E.; Bell, D.
- Intensive Care Medicine, Vol. 41, Issue 9
Multi-cell Agent-based Simulation of the Microvasculature to Study the Dynamics of Circulating Inflammatory Cell Trafficking
journal, April 2007
- Bailey, Alexander M.; Thorne, Bryan C.; Peirce, Shayn M.
- Annals of Biomedical Engineering, Vol. 35, Issue 6
An enhanced agent based model of the immune system response
journal, December 2006
- Baldazzi, V.; Castiglione, F.; Bernaschi, M.
- Cellular Immunology, Vol. 244, Issue 2
Agent-based modeling of host–pathogen systems: The successes and challenges
journal, April 2009
- Bauer, Amy L.; Beauchemin, Catherine A. A.; Perelson, Alan S.
- Information Sciences, Vol. 179, Issue 10
Sepsis reconsidered: Identifying novel metrics for behavioral landscape characterization with a high-performance computing implementation of an agent-based model
journal, October 2017
- Cockrell, Chase; An, Gary
- Journal of Theoretical Biology, Vol. 430
Agent-based modeling: a systematic assessment of use cases and requirements for enhancing pharmaceutical research and development productivity: Agent-based modeling
journal, June 2013
- Hunt, C. Anthony; Kennedy, Ryan C.; Kim, Sean H. J.
- Wiley Interdisciplinary Reviews: Systems Biology and Medicine, Vol. 5, Issue 4
Clinical trials of mediator-directed therapy in sepsis: what have we learned?
journal, February 2000
- Marshall, J. C.
- Intensive Care Medicine, Vol. 26, Issue 0
The Next Generation of Sepsis Clinical Trial Designs: What Is Next After the Demise of Recombinant Human Activated Protein C?*
journal, January 2014
- Opal, Steven M.; Dellinger, R. Phillip; Vincent, Jean-Louis
- Critical Care Medicine, Vol. 42, Issue 7
The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)
journal, February 2016
- Singer, Mervyn; Deutschman, Clifford S.; Seymour, Christopher Warren
- JAMA, Vol. 315, Issue 8
Simulating cancer growth with multiscale agent-based modeling
journal, February 2015
- Wang, Zhihui; Butner, Joseph D.; Kerketta, Romica
- Seminars in Cancer Biology, Vol. 30
Pharmacoeconomic Implications of New Therapies in Sepsis
journal, January 2004
- Wood, Kelly A.; Angus, Derek C.
- PharmacoEconomics, Vol. 22, Issue 14
Development of a three-dimensional multiscale agent-based tumor model: Simulating gene-protein interaction profiles, cell phenotypes and multicellular patterns in brain cancer
journal, January 2007
- Zhang, Le; Athale, Chaitanya A.; Deisboeck, Thomas S.
- Journal of Theoretical Biology, Vol. 244, Issue 1
Works referencing / citing this record:
Agent‐based models of inflammation in translational systems biology: A decade later
journal, June 2019
- Vodovotz, Yoram; An, Gary
- Wiley Interdisciplinary Reviews: Systems Biology and Medicine, Vol. 11, Issue 6
Supporting Computational Apprenticeship Through Educational and Software Infrastructure: A Case Study in a Mathematical Oncology Research Lab
journal, February 2021
- Madamanchi, Aasakiran; Thomas, Madison; Magana, Alejandra
- PRIMUS