Hierarchical deep reinforcement learning reveals a modular mechanism of cell movement
Abstract
Time-lapse images of cells and tissues contain rich information about dynamic cell behaviours, which reflect the underlying processes of proliferation, differentiation and morphogenesis. However, we lack computational tools for effective inference. Here we exploit deep reinforcement learning (DRL) to infer cell–cell interactions and collective cell behaviours in tissue morphogenesis from three-dimensional (3D) time-lapse images. We use hierarchical DRL (HDRL), known for multiscale learning and data efficiency, to examine cell migrations based on images with a ubiquitous nuclear label and simple rules formulated from empirical statistics of the images. When applied to Caenorhabditis elegans embryogenesis, HDRL reveals a multiphase, modular organization of cell movement. Imaging with additional cellular markers confirms the modular organization as a novel migration mechanism, which we term sequential rosettes. Furthermore, HDRL forms a transferable model that successfully differentiates sequential rosettes-based migration from others. Our study demonstrates a powerful approach to infer the underlying biology from time-lapse imaging without prior knowledge.
- Authors:
-
[3];
- Univ. of Tennessee, Knoxville, TN (United States)
- Sloan Kettering Inst., New York, NY (United States)
- Univ. of Tennessee, Knoxville, TN (United States); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Publication Date:
- Research Org.:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC); National Institutes of Health (NIH)
- OSTI Identifier:
- 1869083
- Grant/Contract Number:
- AC05-00OR22725; R01GM097576; P30CA008748
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Nature Machine Intelligence
- Additional Journal Information:
- Journal Volume: 4; Journal Issue: 1; Journal ID: ISSN 2522-5839
- Publisher:
- Springer Nature
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 59 BASIC BIOLOGICAL SCIENCES; 97 MATHEMATICS AND COMPUTING; body patterning; machine learning
Citation Formats
Wang, Zi, Xu, Yichi, Wang, Dali, Yang, Jiawei, and Bao, Zhirong. Hierarchical deep reinforcement learning reveals a modular mechanism of cell movement. United States: N. p., 2022.
Web. doi:10.1038/s42256-021-00431-x.
Wang, Zi, Xu, Yichi, Wang, Dali, Yang, Jiawei, & Bao, Zhirong. Hierarchical deep reinforcement learning reveals a modular mechanism of cell movement. United States. https://doi.org/10.1038/s42256-021-00431-x
Wang, Zi, Xu, Yichi, Wang, Dali, Yang, Jiawei, and Bao, Zhirong. Mon .
"Hierarchical deep reinforcement learning reveals a modular mechanism of cell movement". United States. https://doi.org/10.1038/s42256-021-00431-x. https://www.osti.gov/servlets/purl/1869083.
@article{osti_1869083,
title = {Hierarchical deep reinforcement learning reveals a modular mechanism of cell movement},
author = {Wang, Zi and Xu, Yichi and Wang, Dali and Yang, Jiawei and Bao, Zhirong},
abstractNote = {Time-lapse images of cells and tissues contain rich information about dynamic cell behaviours, which reflect the underlying processes of proliferation, differentiation and morphogenesis. However, we lack computational tools for effective inference. Here we exploit deep reinforcement learning (DRL) to infer cell–cell interactions and collective cell behaviours in tissue morphogenesis from three-dimensional (3D) time-lapse images. We use hierarchical DRL (HDRL), known for multiscale learning and data efficiency, to examine cell migrations based on images with a ubiquitous nuclear label and simple rules formulated from empirical statistics of the images. When applied to Caenorhabditis elegans embryogenesis, HDRL reveals a multiphase, modular organization of cell movement. Imaging with additional cellular markers confirms the modular organization as a novel migration mechanism, which we term sequential rosettes. Furthermore, HDRL forms a transferable model that successfully differentiates sequential rosettes-based migration from others. Our study demonstrates a powerful approach to infer the underlying biology from time-lapse imaging without prior knowledge.},
doi = {10.1038/s42256-021-00431-x},
journal = {Nature Machine Intelligence},
number = 1,
volume = 4,
place = {United States},
year = {Mon Jan 10 00:00:00 EST 2022},
month = {Mon Jan 10 00:00:00 EST 2022}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Mechanobiology of collective cell behaviours
journal, November 2017
- Ladoux, Benoit; Mège, René-Marc
- Nature Reviews Molecular Cell Biology, Vol. 18, Issue 12
Inverted selective plane illumination microscopy (iSPIM) enables coupled cell identity lineaging and neurodevelopmental imaging in Caenorhabditis elegans
journal, October 2011
- Wu, Y.; Ghitani, A.; Christensen, R.
- Proceedings of the National Academy of Sciences, Vol. 108, Issue 43
Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction
journal, July 2019
- Belthangady, Chinmay; Royer, Loic A.
- Nature Methods, Vol. 16, Issue 12
Reinforcement learning in artificial and biological systems
journal, March 2019
- Neftci, Emre O.; Averbeck, Bruno B.
- Nature Machine Intelligence, Vol. 1, Issue 3
Imaging Morphogenesis: Technological Advances and Biological Insights
journal, June 2013
- Keller, Philipp J.
- Science, Vol. 340, Issue 6137
Human-level control through deep reinforcement learning
journal, February 2015
- Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David
- Nature, Vol. 518, Issue 7540
Prospective identification of hematopoietic lineage choice by deep learning
journal, February 2017
- Buggenthin, Felix; Buettner, Florian; Hoppe, Philipp S.
- Nature Methods, Vol. 14, Issue 4
Mastering the game of Go with deep neural networks and tree search
journal, January 2016
- Silver, David; Huang, Aja; Maddison, Chris J.
- Nature, Vol. 529, Issue 7587
AceTree: a major update and case study in the long term maintenance of open-source scientific software
journal, April 2018
- Katzman, Braden; Tang, Doris; Santella, Anthony
- BMC Bioinformatics, Vol. 19, Issue 1
Cadherin preserves cohesion across involuting tissues during C. elegans neurulation
journal, October 2020
- Barnes, Kristopher M.; Fan, Li; Moyle, Mark W.
- eLife, Vol. 9
A hybrid blob-slice model for accurate and efficient detection of fluorescence labeled nuclei in 3D
journal, November 2010
- Santella, Anthony; Du, Zhuo; Nowotschin, Sonja
- BMC Bioinformatics, Vol. 11, Issue 1
PCP and SAX-3/Robo Pathways Cooperate to Regulate Convergent Extension-Based Nerve Cord Assembly in C. elegans
journal, April 2017
- Shah, Pavak K.; Tanner, Matthew R.; Kovacevic, Ismar
- Developmental Cell, Vol. 41, Issue 2
Systematic quantification of developmental phenotypes at single-cell resolution during embryogenesis
journal, July 2013
- Moore, J. L.; Du, Z.; Bao, Z.
- Development, Vol. 140, Issue 15
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
journal, August 1999
- Sutton, Richard S.; Precup, Doina; Singh, Satinder
- Artificial Intelligence, Vol. 112, Issue 1-2
Deep reinforcement learning of cell movement in the early stage of C.elegans embryogenesis
journal, April 2018
- Wang, Zi; Wang, Dali; Li, Chengcheng
- Bioinformatics, Vol. 34, Issue 18
Spatial and Temporal Controls Target pal-1 Blastomere-Specification Activity to a Single Blastomere Lineage in C. elegans Embryos
journal, October 1996
- Hunter, Craig P.; Kenyon, Cynthia
- Cell, Vol. 87, Issue 2
An Observation-Driven Agent-Based Modeling and Analysis Framework for C. elegans Embryogenesis
journal, November 2016
- Wang, Zi; Ramsey, Benjamin J.; Wang, Dali
- PLOS ONE, Vol. 11, Issue 11
Deep learning for cellular image analysis
journal, May 2019
- Moen, Erick; Bannon, Dylan; Kudo, Takamasa
- Nature Methods, Vol. 16, Issue 12
Cell Neighbor Determination in the Metazoan Embryo System
conference, August 2017
- Wang, Zi; Wang, Dali; Li, Husheng
- BCB '17: 8th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology,and Health Informatics
Array programming with NumPy
journal, September 2020
- Harris, Charles R.; Millman, K. Jarrod; van der Walt, Stéfan J.
- Nature, Vol. 585, Issue 7825
A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
journal, December 2018
- Silver, David; Hubert, Thomas; Schrittwieser, Julian
- Science, Vol. 362, Issue 6419
Review of Deep Reinforcement Learning for Robot Manipulation
conference, February 2019
- Nguyen, Hai; La, Hung
- 2019 Third IEEE International Conference on Robotic Computing (IRC)
A positional Toll receptor code directs convergent extension in Drosophila
journal, November 2014
- Paré, Adam C.; Vichas, Athea; Fincher, Christopher T.
- Nature, Vol. 515, Issue 7528
Automated cell lineage tracing in Caenorhabditis elegans
journal, February 2006
- Bao, Z.; Murray, J. I.; Boyle, T.
- Proceedings of the National Academy of Sciences, Vol. 103, Issue 8
A semi-local neighborhood-based framework for probabilistic cell lineage tracing
journal, January 2014
- Santella, Anthony; Du, Zhuo; Bao, Zhirong
- BMC Bioinformatics, Vol. 15, Issue 1
The Regulatory Landscape of Lineage Differentiation in a Metazoan Embryo
journal, September 2015
- Du, Zhuo; Santella, Anthony; He, Fei
- Developmental Cell, Vol. 34, Issue 5
The embryonic cell lineage of the nematode Caenorhabditis elegans
journal, November 1983
- Sulston, J. E.; Schierenberg, E.; White, J. G.
- Developmental Biology, Vol. 100, Issue 1
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
conference, May 2017
- Gu, Shixiang; Holly, Ethan; Lillicrap, Timothy
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
WormGUIDES: an interactive single cell developmental atlas and tool for collaborative multidimensional data exploration
journal, June 2015
- Santella, Anthony; Catena, Raúl; Kovacevic, Ismar
- BMC Bioinformatics, Vol. 16, Issue 1
Vector-based navigation using grid-like representations in artificial agents
journal, May 2018
- Banino, Andrea; Barry, Caswell; Uria, Benigno
- Nature, Vol. 557, Issue 7705
A Deep Hierarchical Approach to Lifelong Learning in Minecraft
journal, February 2017
- Tessler, Chen; Givony, Shahar; Zahavy, Tom
- Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31, Issue 1