DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine

Abstract

In the past, precision medicine involves classifying patients to identify subpopulations that respond favorably to specific therapeutics. We discuss precision medicine as a dynamic feedback control problem, where treatment administered to a patient is guided by measurements taken during the course of treatment. We consider sepsis, a life-threatening condition in which dysregulation of the immune system causes tissue damage. We leverage an existing simulation of the innate immune response to infection and apply deep reinforcement learning (DRL) to discover an adaptive personalized treatment policy that specifies effective multicytokine therapy to simulated sepsis patients based on systemic measurements. The learned policy achieves a dramatic reduction in mortality rate over a set of 500 simulated patients relative to standalone antibiotic therapy. Advantages of our approach are threefold: (1) the use of simulation allows exploring therapeutic strategies beyond clinical practice and available data, (2) advances in DRL accommodate learning complex therapeutic strategies for complex biological systems, and (3) optimized treatments respond to a patient's individual disease progression over time, therefore, capturing both differences across patients and the inherent randomness of disease progression within a single patient. We believe that this work motivates both considering adaptive personalized multicytokine mediation therapy for sepsis and exploitingmore » simulation with DRL for precision medicine more broadly.« less

Authors:
 [1];  [2];  [3];  [4];  [1];  [4];  [1]
  1. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
  2. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States); Georgia Inst. of Technology, Atlanta, GA (United States)
  3. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States); Univ. of Toronto, ON (Canada)
  4. Univ. of Vermont, Burlington, VT (United States); Univ. of Chicago, IL (United States)
Publication Date:
Research Org.:
Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA); National Institutes of Health (NIH)
OSTI Identifier:
1477825
Report Number(s):
LLNL-JRNL-745693
Journal ID: ISSN 1557-8666; 900608
Grant/Contract Number:  
AC52-07NA27344
Resource Type:
Accepted Manuscript
Journal Name:
Journal of Computational Biology
Additional Journal Information:
Journal Volume: 26; Journal Issue: 6; Journal ID: ISSN 1557-8666
Country of Publication:
United States
Language:
English
Subject:
59 BASIC BIOLOGICAL SCIENCES; 97 MATHEMATICS AND COMPUTING; agent-based model; deep reinforcement learning; precision medicine; sepsis

Citation Formats

Petersen, Brenden K., Yang, Jiachen, Grathwohl, Will S., Cockrell, Chase, Santiago, Claudio, An, Gary, and Faissol, Daniel M. Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine. United States: N. p., 2019. Web. doi:10.1089/cmb.2018.0168.
Petersen, Brenden K., Yang, Jiachen, Grathwohl, Will S., Cockrell, Chase, Santiago, Claudio, An, Gary, & Faissol, Daniel M. Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine. United States. https://doi.org/10.1089/cmb.2018.0168
Petersen, Brenden K., Yang, Jiachen, Grathwohl, Will S., Cockrell, Chase, Santiago, Claudio, An, Gary, and Faissol, Daniel M. Thu . "Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine". United States. https://doi.org/10.1089/cmb.2018.0168. https://www.osti.gov/servlets/purl/1477825.
@article{osti_1477825,
title = {Deep Reinforcement Learning and Simulation as a Path Toward Precision Medicine},
author = {Petersen, Brenden K. and Yang, Jiachen and Grathwohl, Will S. and Cockrell, Chase and Santiago, Claudio and An, Gary and Faissol, Daniel M.},
abstractNote = {In the past, precision medicine involves classifying patients to identify subpopulations that respond favorably to specific therapeutics. We discuss precision medicine as a dynamic feedback control problem, where treatment administered to a patient is guided by measurements taken during the course of treatment. We consider sepsis, a life-threatening condition in which dysregulation of the immune system causes tissue damage. We leverage an existing simulation of the innate immune response to infection and apply deep reinforcement learning (DRL) to discover an adaptive personalized treatment policy that specifies effective multicytokine therapy to simulated sepsis patients based on systemic measurements. The learned policy achieves a dramatic reduction in mortality rate over a set of 500 simulated patients relative to standalone antibiotic therapy. Advantages of our approach are threefold: (1) the use of simulation allows exploring therapeutic strategies beyond clinical practice and available data, (2) advances in DRL accommodate learning complex therapeutic strategies for complex biological systems, and (3) optimized treatments respond to a patient's individual disease progression over time, therefore, capturing both differences across patients and the inherent randomness of disease progression within a single patient. We believe that this work motivates both considering adaptive personalized multicytokine mediation therapy for sepsis and exploiting simulation with DRL for precision medicine more broadly.},
doi = {10.1089/cmb.2018.0168},
journal = {Journal of Computational Biology},
number = 6,
volume = 26,
place = {United States},
year = {Thu Jun 06 00:00:00 EDT 2019},
month = {Thu Jun 06 00:00:00 EDT 2019}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 18 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Optimization and Control of Agent-Based Models in Biology: A Perspective
journal, November 2016

  • An, G.; Fitzpatrick, B. G.; Christley, S.
  • Bulletin of Mathematical Biology, Vol. 79, Issue 1
  • DOI: 10.1007/s11538-016-0225-6

Agent‐based models in translational systems biology
journal, September 2009

  • An, Gary; Mi, Qi; Dutta‐Moscato, Joyeeta
  • Wiley Interdisciplinary Reviews: Systems Biology and Medicine, Vol. 1, Issue 2
  • DOI: 10.1002/wsbm.45

A systematic review and meta-analysis of early goal-directed therapy for septic shock: the ARISE, ProCESS and ProMISe Investigators
journal, May 2015


Multi-cell Agent-based Simulation of the Microvasculature to Study the Dynamics of Circulating Inflammatory Cell Trafficking
journal, April 2007

  • Bailey, Alexander M.; Thorne, Bryan C.; Peirce, Shayn M.
  • Annals of Biomedical Engineering, Vol. 35, Issue 6
  • DOI: 10.1007/s10439-007-9266-1

An enhanced agent based model of the immune system response
journal, December 2006


Agent-based modeling of host–pathogen systems: The successes and challenges
journal, April 2009

  • Bauer, Amy L.; Beauchemin, Catherine A. A.; Perelson, Alan S.
  • Information Sciences, Vol. 179, Issue 10
  • DOI: 10.1016/j.ins.2008.11.012

Agent-based modeling: a systematic assessment of use cases and requirements for enhancing pharmaceutical research and development productivity: Agent-based modeling
journal, June 2013

  • Hunt, C. Anthony; Kennedy, Ryan C.; Kim, Sean H. J.
  • Wiley Interdisciplinary Reviews: Systems Biology and Medicine, Vol. 5, Issue 4
  • DOI: 10.1002/wsbm.1222

Clinical trials of mediator-directed therapy in sepsis: what have we learned?
journal, February 2000


The Next Generation of Sepsis Clinical Trial Designs: What Is Next After the Demise of Recombinant Human Activated Protein C?*
journal, January 2014


The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)
journal, February 2016

  • Singer, Mervyn; Deutschman, Clifford S.; Seymour, Christopher Warren
  • JAMA, Vol. 315, Issue 8
  • DOI: 10.1001/jama.2016.0287

Simulating cancer growth with multiscale agent-based modeling
journal, February 2015


Pharmacoeconomic Implications of New Therapies in Sepsis
journal, January 2004


Development of a three-dimensional multiscale agent-based tumor model: Simulating gene-protein interaction profiles, cell phenotypes and multicellular patterns in brain cancer
journal, January 2007

  • Zhang, Le; Athale, Chaitanya A.; Deisboeck, Thomas S.
  • Journal of Theoretical Biology, Vol. 244, Issue 1
  • DOI: 10.1016/j.jtbi.2006.06.034

Works referencing / citing this record:

Agent‐based models of inflammation in translational systems biology: A decade later
journal, June 2019

  • Vodovotz, Yoram; An, Gary
  • Wiley Interdisciplinary Reviews: Systems Biology and Medicine, Vol. 11, Issue 6
  • DOI: 10.1002/wsbm.1460