Hierarchical deep reinforcement learning reveals a modular mechanism of cell movement

Wang, Zi; Xu, Yichi; Wang, Dali; Yang, Jiawei; Bao, Zhirong

doi:10.1038/s42256-021-00431-x

Hierarchical deep reinforcement learning reveals a modular mechanism of cell movement

Journal Article · Mon Jan 10 00:00:00 EST 2022 · Nature Machine Intelligence

DOI:https://doi.org/10.1038/s42256-021-00431-x· OSTI ID:1869083

Wang, Zi ^[1]; Xu, Yichi ^[2]; ^[3]; Yang, Jiawei ^[1]; Bao, Zhirong ^[2]

Univ. of Tennessee, Knoxville, TN (United States)
Sloan Kettering Inst., New York, NY (United States)
Univ. of Tennessee, Knoxville, TN (United States); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Time-lapse images of cells and tissues contain rich information about dynamic cell behaviours, which reflect the underlying processes of proliferation, differentiation and morphogenesis. However, we lack computational tools for effective inference. Here we exploit deep reinforcement learning (DRL) to infer cell–cell interactions and collective cell behaviours in tissue morphogenesis from three-dimensional (3D) time-lapse images. We use hierarchical DRL (HDRL), known for multiscale learning and data efficiency, to examine cell migrations based on images with a ubiquitous nuclear label and simple rules formulated from empirical statistics of the images. When applied to Caenorhabditis elegans embryogenesis, HDRL reveals a multiphase, modular organization of cell movement. Imaging with additional cellular markers confirms the modular organization as a novel migration mechanism, which we term sequential rosettes. Furthermore, HDRL forms a transferable model that successfully differentiates sequential rosettes-based migration from others. Our study demonstrates a powerful approach to infer the underlying biology from time-lapse imaging without prior knowledge.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: National Institutes of Health (NIH); USDOE Office of Science (SC)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1869083

Journal Information:: Nature Machine Intelligence, Journal Name: Nature Machine Intelligence Journal Issue: 1 Vol. 4; ISSN 2522-5839

Publisher:: Springer NatureCopyright Statement

Country of Publication:: United States

Language:: English

References (32)

The embryonic cell lineage of the nematode Caenorhabditis elegans Sulston, J. E.; Schierenberg, E.; White, J. G. Developmental Biology, Vol. 100, Issue 1 https://doi.org/10.1016/0012-1606(83)90201-4	journal	November 1983
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning Sutton, Richard S.; Precup, Doina; Singh, Satinder Artificial Intelligence, Vol. 112, Issue 1-2 https://doi.org/10.1016/S0004-3702(99)00052-1	journal	August 1999
Spatial and Temporal Controls Target pal-1 Blastomere-Specification Activity to a Single Blastomere Lineage in C. elegans Embryos Hunter, Craig P.; Kenyon, Cynthia Cell, Vol. 87, Issue 2 https://doi.org/10.1016/S0092-8674(00)81340-9	journal	October 1996
The Regulatory Landscape of Lineage Differentiation in a Metazoan Embryo Du, Zhuo; Santella, Anthony; He, Fei Developmental Cell, Vol. 34, Issue 5 https://doi.org/10.1016/j.devcel.2015.07.014	journal	September 2015
PCP and SAX-3/Robo Pathways Cooperate to Regulate Convergent Extension-Based Nerve Cord Assembly in C. elegans Shah, Pavak K.; Tanner, Matthew R.; Kovacevic, Ismar Developmental Cell, Vol. 41, Issue 2 https://doi.org/10.1016/j.devcel.2017.03.024	journal	April 2017
A positional Toll receptor code directs convergent extension in Drosophila Paré, Adam C.; Vichas, Athea; Fincher, Christopher T. Nature, Vol. 515, Issue 7528 https://doi.org/10.1038/nature13953	journal	November 2014
Human-level control through deep reinforcement learning Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David Nature, Vol. 518, Issue 7540 https://doi.org/10.1038/nature14236	journal	February 2015
Mastering the game of Go with deep neural networks and tree search Silver, David; Huang, Aja; Maddison, Chris J. Nature, Vol. 529, Issue 7587 https://doi.org/10.1038/nature16961	journal	January 2016
Prospective identification of hematopoietic lineage choice by deep learning Buggenthin, Felix; Buettner, Florian; Hoppe, Philipp S. Nature Methods, Vol. 14, Issue 4 https://doi.org/10.1038/nmeth.4182	journal	February 2017
Mechanobiology of collective cell behaviours Ladoux, Benoit; Mège, René-Marc Nature Reviews Molecular Cell Biology, Vol. 18, Issue 12 https://doi.org/10.1038/nrm.2017.98	journal	November 2017
Vector-based navigation using grid-like representations in artificial agents Banino, Andrea; Barry, Caswell; Uria, Benigno Nature, Vol. 557, Issue 7705 https://doi.org/10.1038/s41586-018-0102-6	journal	May 2018
Array programming with NumPy Harris, Charles R.; Millman, K. Jarrod; van der Walt, Stéfan J. Nature, Vol. 585, Issue 7825 https://doi.org/10.1038/s41586-020-2649-2	journal	September 2020
Deep learning for cellular image analysis Moen, Erick; Bannon, Dylan; Kudo, Takamasa Nature Methods, Vol. 16, Issue 12 https://doi.org/10.1038/s41592-019-0403-1	journal	May 2019
Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction Belthangady, Chinmay; Royer, Loic A. Nature Methods, Vol. 16, Issue 12 https://doi.org/10.1038/s41592-019-0458-z	journal	July 2019
Reinforcement learning in artificial and biological systems Neftci, Emre O.; Averbeck, Bruno B. Nature Machine Intelligence, Vol. 1, Issue 3 https://doi.org/10.1038/s42256-019-0025-4	journal	March 2019
Automated cell lineage tracing in Caenorhabditis elegans Bao, Z.; Murray, J. I.; Boyle, T. Proceedings of the National Academy of Sciences, Vol. 103, Issue 8 https://doi.org/10.1073/pnas.0511111103	journal	February 2006
Inverted selective plane illumination microscopy (iSPIM) enables coupled cell identity lineaging and neurodevelopmental imaging in Caenorhabditis elegans Wu, Y.; Ghitani, A.; Christensen, R. Proceedings of the National Academy of Sciences, Vol. 108, Issue 43 https://doi.org/10.1073/pnas.1108494108	journal	October 2011
Deep reinforcement learning of cell movement in the early stage of C.elegans embryogenesis Wang, Zi; Wang, Dali; Li, Chengcheng Bioinformatics, Vol. 34, Issue 18 https://doi.org/10.1093/bioinformatics/bty323	journal	April 2018
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates Gu, Shixiang; Holly, Ethan; Lillicrap, Timothy 2017 IEEE International Conference on Robotics and Automation (ICRA) https://doi.org/10.1109/ICRA.2017.7989385	conference	May 2017
Review of Deep Reinforcement Learning for Robot Manipulation Nguyen, Hai; La, Hung 2019 Third IEEE International Conference on Robotic Computing (IRC) https://doi.org/10.1109/IRC.2019.00120	conference	February 2019
Deep Reinforcement Learning: A Brief Survey Arulkumaran, Kai; Deisenroth, Marc Peter; Brundage, Miles IEEE Signal Processing Magazine, Vol. 34, Issue 6 https://doi.org/10.1109/MSP.2017.2743240	journal	November 2017
Imaging Morphogenesis: Technological Advances and Biological Insights Keller, Philipp J. Science, Vol. 340, Issue 6137 https://doi.org/10.1126/science.1234168	journal	June 2013
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play Silver, David; Hubert, Thomas; Schrittwieser, Julian Science, Vol. 362, Issue 6419 https://doi.org/10.1126/science.aar6404	journal	December 2018
Cell Neighbor Determination in the Metazoan Embryo System Wang, Zi; Wang, Dali; Li, Husheng BCB '17: 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology,and Health Informatics https://doi.org/10.1145/3107411.3107465	conference	August 2017
A hybrid blob-slice model for accurate and efficient detection of fluorescence labeled nuclei in 3D Santella, Anthony; Du, Zhuo; Nowotschin, Sonja BMC Bioinformatics, Vol. 11, Issue 1 https://doi.org/10.1186/1471-2105-11-580	journal	November 2010
A semi-local neighborhood-based framework for probabilistic cell lineage tracing Santella, Anthony; Du, Zhuo; Bao, Zhirong BMC Bioinformatics, Vol. 15, Issue 1 https://doi.org/10.1186/1471-2105-15-217	journal	January 2014
WormGUIDES: an interactive single cell developmental atlas and tool for collaborative multidimensional data exploration Santella, Anthony; Catena, Raúl; Kovacevic, Ismar BMC Bioinformatics, Vol. 16, Issue 1 https://doi.org/10.1186/s12859-015-0627-8	journal	June 2015
AceTree: a major update and case study in the long term maintenance of open-source scientific software Katzman, Braden; Tang, Doris; Santella, Anthony BMC Bioinformatics, Vol. 19, Issue 1 https://doi.org/10.1186/s12859-018-2127-0	journal	April 2018
Systematic quantification of developmental phenotypes at single-cell resolution during embryogenesis Moore, J. L.; Du, Z.; Bao, Z. Development, Vol. 140, Issue 15 https://doi.org/10.1242/dev.096040	journal	July 2013
An Observation-Driven Agent-Based Modeling and Analysis Framework for C. elegans Embryogenesis Wang, Zi; Ramsey, Benjamin J.; Wang, Dali PLOS ONE, Vol. 11, Issue 11 https://doi.org/10.1371/journal.pone.0166551	journal	November 2016
A Deep Hierarchical Approach to Lifelong Learning in Minecraft Tessler, Chen; Givony, Shahar; Zahavy, Tom Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31, Issue 1 https://doi.org/10.1609/aaai.v31i1.10744	journal	February 2017
Cadherin preserves cohesion across involuting tissues during C. elegans neurulation Barnes, Kristopher M.; Fan, Li; Moyle, Mark W. eLife, Vol. 9 https://doi.org/10.7554/eLife.58626	journal	October 2020

Similar Records

Augmenting C. elegans Microscopic Dataset for Accelerated Pattern Recognition

Conference · Wed May 01 00:00:00 EDT 2019 · OSTI ID:1564184

An Observation Data Driven Simulation and Analysis Framework for Early Stage C. elegans Embryogenesis

Journal Article · Mon Aug 27 20:00:00 EDT 2018 · Journal of Biomedical Science and Engineering · OSTI ID:1468155

Related Subjects

59 BASIC BIOLOGICAL SCIENCES
97 MATHEMATICS AND COMPUTING
body patterning
machine learning

Hierarchical deep reinforcement learning reveals a modular mechanism of cell movement

Citation Formats

References (32)

Similar Records

Related Subjects