Learning surface molecular structures via machine vision
Abstract
Recent advances in high resolution scanning transmission electron and scanning probe microscopies have allowed researchers to perform measurements of materials structural parameters and functional properties in real space with a picometre precision. In many technologically relevant atomic and/or molecular systems, however, the information of interest is distributed spatially in a non-uniform manner and may have a complex multi-dimensional nature. One of the critical issues, therefore, lies in being able to accurately identify (‘read out’) all the individual building blocks in different atomic/molecular architectures, as well as more complex patterns that these blocks may form, on a scale of hundreds and thousands of individual atomic/molecular units. Here we employ machine vision to read and recognize complex molecular assemblies on surfaces. Specifically, we combine Markov random field model and convolutional neural networks to classify structural and rotational states of all individual building blocks in molecular assembly on the metallic surface visualized in high-resolution scanning tunneling microscopy measurements. We show how the obtained full decoding of the system allows us to directly construct a pair density function—a centerpiece in analysis of disorder-property relationship paradigm—as well as to analyze spatial correlations between multiple order parameters at the nanoscale, and elucidate reaction pathway involving molecularmore »
- Authors:
-
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States); Univ. of Tennessee, Knoxville, TN (United States)
- Publication Date:
- Research Org.:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1376377
- Grant/Contract Number:
- AC05-00OR22725
- Resource Type:
- Accepted Manuscript
- Journal Name:
- npj Computational Materials
- Additional Journal Information:
- Journal Volume: 3; Journal Issue: 1; Journal ID: ISSN 2057-3960
- Publisher:
- Nature Publishing Group
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 36 MATERIALS SCIENCE; 37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CHEMISTRY; Carbon nanotubes and fullerenes; Molecular self-assembly; Scanning probe microscopy
Citation Formats
Ziatdinov, Maxim, Maksov, Artem, and Kalinin, Sergei V. Learning surface molecular structures via machine vision. United States: N. p., 2017.
Web. doi:10.1038/s41524-017-0038-7.
Ziatdinov, Maxim, Maksov, Artem, & Kalinin, Sergei V. Learning surface molecular structures via machine vision. United States. https://doi.org/10.1038/s41524-017-0038-7
Ziatdinov, Maxim, Maksov, Artem, and Kalinin, Sergei V. Thu .
"Learning surface molecular structures via machine vision". United States. https://doi.org/10.1038/s41524-017-0038-7. https://www.osti.gov/servlets/purl/1376377.
@article{osti_1376377,
title = {Learning surface molecular structures via machine vision},
author = {Ziatdinov, Maxim and Maksov, Artem and Kalinin, Sergei V.},
abstractNote = {Recent advances in high resolution scanning transmission electron and scanning probe microscopies have allowed researchers to perform measurements of materials structural parameters and functional properties in real space with a picometre precision. In many technologically relevant atomic and/or molecular systems, however, the information of interest is distributed spatially in a non-uniform manner and may have a complex multi-dimensional nature. One of the critical issues, therefore, lies in being able to accurately identify (‘read out’) all the individual building blocks in different atomic/molecular architectures, as well as more complex patterns that these blocks may form, on a scale of hundreds and thousands of individual atomic/molecular units. Here we employ machine vision to read and recognize complex molecular assemblies on surfaces. Specifically, we combine Markov random field model and convolutional neural networks to classify structural and rotational states of all individual building blocks in molecular assembly on the metallic surface visualized in high-resolution scanning tunneling microscopy measurements. We show how the obtained full decoding of the system allows us to directly construct a pair density function—a centerpiece in analysis of disorder-property relationship paradigm—as well as to analyze spatial correlations between multiple order parameters at the nanoscale, and elucidate reaction pathway involving molecular conformation changes. Here, the method represents a significant shift in our way of analyzing atomic and/or molecular resolved microscopic images and can be applied to variety of other microscopic measurements of structural, electronic, and magnetic orders in different condensed matter systems.},
doi = {10.1038/s41524-017-0038-7},
journal = {npj Computational Materials},
number = 1,
volume = 3,
place = {United States},
year = {Thu Aug 10 00:00:00 EDT 2017},
month = {Thu Aug 10 00:00:00 EDT 2017}
}
Web of Science
Figures / Tables:
Works referenced in this record:
Mapping Octahedral Tilts and Polarization Across a Domain Wall in BiFeO 3 from Z-Contrast Scanning Transmission Electron Microscopy Image Atomic Column Shape Analysis
journal, October 2010
- Borisevich, AlbinaY.; Ovchinnikov, Oleg S.; Chang, Hye Jung
- ACS Nano, Vol. 4, Issue 10
Bowl Inversion and Electronic Switching of Buckybowls on Gold
journal, September 2016
- Fujii, Shintaro; Ziatdinov, Maxim; Higashibayashi, Shuhei
- Journal of the American Chemical Society, Vol. 138, Issue 37
Microscopy: Hasten high resolution
journal, November 2014
- Pennycook, Stephen J.; Kalinin, Sergei V.
- Nature, Vol. 515, Issue 7528
Spontaneous Vortex Nanodomain Arrays at Ferroelectric Heterointerfaces
journal, February 2011
- Nelson, Christopher T.; Winchester, Benjamin; Zhang, Yi
- Nano Letters, Vol. 11, Issue 2
Direct Imaging of Covalent Bond Structure in Single-Molecule Chemical Reactions
journal, May 2013
- de Oteyza, D. G.; Gorman, P.; Chen, Y. -C.
- Science, Vol. 340, Issue 6139
Charge transfer and screening in individual molecules on metal substrates: A scanning tunneling spectroscopy and theoretical study
journal, September 2004
- Lu, Xinghua; Grobis, M.; Khoo, K. H.
- Physical Review B, Vol. 70, Issue 11
Local Indicators of Spatial Association-LISA
journal, April 1995
- Anselin, Luc
- Geographical Analysis, Vol. 27, Issue 2
Principal component and spatial correlation analysis of spectroscopic-imaging data in scanning probe microscopy
journal, February 2009
- Jesse, Stephen; Kalinin, Sergei V.
- Nanotechnology, Vol. 20, Issue 8
A Synthesis of Sumanene, a Fullerene Fragment
journal, September 2003
- Sakurai, H.
- Science, Vol. 301, Issue 5641
The Analysis of Spatial Association by Use of Distance Statistics
journal, July 1992
- Getis, Arthur; Ord, J. K.
- Geographical Analysis, Vol. 24, Issue 3
Interplay between defects, disorder and flexibility in metal-organic frameworks
journal, December 2016
- Bennett, Thomas D.; Cheetham, Anthony K.; Fuchs, Alain H.
- Nature Chemistry, Vol. 9, Issue 1
Ab initio quantum chemistry: Methodology and applications
journal, May 2005
- Friesner, R. A.
- Proceedings of the National Academy of Sciences, Vol. 102, Issue 19
Observing Atomic Collapse Resonances in Artificial Nuclei on Graphene
journal, March 2013
- Wang, Y.; Wong, D.; Shytov, A. V.
- Science, Vol. 340, Issue 6133
First-Principles Calculations of Complex Metal-Oxide Materials
journal, August 2010
- Rabe, Karin M.
- Annual Review of Condensed Matter Physics, Vol. 1, Issue 1
Markov Random Field Texture Models
journal, January 1983
- Cross, George R.; Jain, Anil K.
- IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-5, Issue 1
Bowl Inversion of Surface-Adsorbed Sumanene
journal, September 2014
- Jaafar, Rached; Pignedoli, Carlo A.; Bussi, Giovanni
- Journal of the American Chemical Society, Vol. 136, Issue 39
Big–deep–smart data in imaging for guiding materials design
journal, September 2015
- Kalinin, Sergei V.; Sumpter, Bobby G.; Archibald, Richard K.
- Nature Materials, Vol. 14, Issue 10
First-principles calculations for point defects in solids
journal, March 2014
- Freysoldt, Christoph; Grabowski, Blazej; Hickel, Tilmann
- Reviews of Modern Physics, Vol. 86, Issue 1
Visualization of the Molecular Jahn-Teller Effect in an Insulating K4C60 Monolayer
journal, October 2005
- Wachowiak, A.
- Science, Vol. 310, Issue 5747
Strain Doping: Reversible Single-Axis Control of a Complex Oxide Lattice via Helium Implantation
journal, June 2015
- Guo, Hangwen; Dong, Shuai; Rack, Philip D.
- Physical Review Letters, Vol. 114, Issue 25
The manipulation of C 60 in molecular arrays with an STM tip in regimes below the decomposition threshold
journal, January 2013
- Olyanich, Dmitry A.; Kotlyar, Vasily G.; Utas, Tatiana V.
- Nanotechnology, Vol. 24, Issue 5
A disorder-enhanced quasi-one-dimensional superconductor
journal, July 2016
- Petrović, A. P.; Ansermet, D.; Chernyshov, D.
- Nature Communications, Vol. 7, Issue 1
Combining satellite imagery and machine learning to predict poverty
journal, August 2016
- Jean, N.; Burke, M.; Xie, M.
- Science, Vol. 353, Issue 6301
The crystallography of correlated disorder
journal, May 2015
- Keen, David A.; Goodwin, Andrew L.
- Nature, Vol. 521, Issue 7552
On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs
journal, January 2001
- Weiss, Y.; Freeman, W. T.
- IEEE Transactions on Information Theory, Vol. 47, Issue 2
Design of crystal-like aperiodic solids with selective disorder–phonon coupling
journal, February 2016
- Overy, Alistair R.; Cairns, Andrew B.; Cliffe, Matthew J.
- Nature Communications, Vol. 7, Issue 1
Atomic-scale study of electric dipoles near charged and uncharged domain walls in ferroelectric films
journal, December 2007
- Jia, Chun-Lin; Mi, Shao-Bo; Urban, Knut
- Nature Materials, Vol. 7, Issue 1
Unit-cell scale mapping of ferroelectricity and tetragonality in epitaxial ultrathin ferroelectric films
journal, December 2006
- Jia, Chun-Lin; Nagarajan, Valanoor; He, Jia-Qing
- Nature Materials, Vol. 6, Issue 1
Defocused Emission Patterns from Chiral Fluorophores: Application to Chiral Axis Orientation Determination
journal, February 2011
- Cyphersmith, A.; Maksov, A.; Hassey-Paradise, R.
- The Journal of Physical Chemistry Letters, Vol. 2, Issue 6
Rigorous force field optimization principles based on statistical distance minimization
journal, October 2015
- Vlcek, Lukas; Chialvo, Ariel A.
- The Journal of Chemical Physics, Vol. 143, Issue 14
Scanning tunneling microscopy fingerprints of point defects in graphene: A theoretical prediction
journal, September 2007
- Amara, H.; Latil, S.; Meunier, V.
- Physical Review B, Vol. 76, Issue 11
Structure and energetics of the vacancy in graphite
journal, October 2003
- El-Barbary, A. A.; Telling, R. H.; Ewels, C. P.
- Physical Review B, Vol. 68, Issue 14
Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis
journal, May 2016
- Litjens, Geert; Sánchez, Clara I.; Timofeeva, Nadya
- Scientific Reports, Vol. 6, Issue 1
Direct observation of Σ7 domain boundary core structure in magnetic skyrmion lattice
journal, February 2016
- Matsumoto, Takao; So, Yeong-Gi; Kohno, Yuji
- Science Advances, Vol. 2, Issue 2
Strain Doping: Reversible Single-Axis Control of a Complex Oxide Lattice via Helium Implantation
text, January 2015
- Guo, Hangwen; Dong, Shuai; Rack, Philip D.
- arXiv
A disorder-enhanced quasi-one-dimensional superconductor
text, January 2016
- Petrović, A. P.; Ansermet, D.; Chernyshov, D.
- arXiv
Works referencing / citing this record:
Interface Characterization and Control of 2D Materials and Heterostructures
journal, July 2018
- Liu, Xiaolong; Hersam, Mark C.
- Advanced Materials, Vol. 30, Issue 39
Simulation and design of energy materials accelerated by machine learning
journal, June 2019
- Wang, Hongshuai; Ji, Yujin; Li, Youyong
- Wiley Interdisciplinary Reviews: Computational Molecular Science
Revealing ferroelectric switching character using deep recurrent neural networks
journal, October 2019
- Agar, Joshua C.; Naul, Brett; Pandya, Shishir
- Nature Communications, Vol. 10, Issue 1
Analyzing machine learning models to accelerate generation of fundamental materials insights
journal, March 2019
- Umehara, Mitsutaro; Stein, Helge S.; Guevarra, Dan
- npj Computational Materials, Vol. 5, Issue 1
Deep data analytics for genetic engineering of diatoms linking genotype to phenotype via machine learning
journal, June 2019
- Trofimov, Artem A.; Pawlicki, Alison A.; Borodinov, Nikolay
- npj Computational Materials, Vol. 5, Issue 1
Atom-by-atom fabrication with electron beams
journal, June 2019
- Dyck, Ondrej; Ziatdinov, Maxim; Lingerfelt, David B.
- Nature Reviews Materials, Vol. 4, Issue 7
Machine learning for molecular and materials science
journal, July 2018
- Butler, Keith T.; Davies, Daniel W.; Cartwright, Hugh
- Nature, Vol. 559, Issue 7715
Combining large-scale screening and machine learning to predict the metal-organic frameworks for organosulfurs removal from high-sour natural gas
journal, September 2019
- Liang, Hong; Yang, Wenyuan; Peng, Feng
- APL Materials, Vol. 7, Issue 9
Building and exploring libraries of atomic defects in graphene: Scanning transmission electron and scanning tunneling microscopy study
journal, September 2019
- Ziatdinov, Maxim; Dyck, Ondrej; Li, Xin
- Science Advances, Vol. 5, Issue 9
Automated structure discovery in atomic force microscopy
journal, February 2020
- Alldritt, Benjamin; Hapala, Prokop; Oinonen, Niko
- Science Advances, Vol. 6, Issue 9
Materials science in the artificial intelligence age: high-throughput library generation, machine learning, and a pathway from correlations to the underpinning physics
journal, July 2019
- Vasudevan, Rama K.; Choudhary, Kamal; Mehta, Apurva
- MRS Communications, Vol. 9, Issue 3
Building and exploring libraries of atomic defects in graphene: Scanning transmission electron and scanning tunneling microscopy study
journal, September 2019
- Ziatdinov, Maxim; Dyck, Ondrej; Li, Xin
- Science Advances, Vol. 5, Issue 9
AFLOW-ML: A RESTful API for machine-learning predictions of materials properties
preprint, January 2017
- Gossett, Eric; Toher, Cormac; Oses, Corey
- arXiv
Building and exploring libraries of atomic defects in graphene: scanning transmission electron and scanning tunneling microscopy study
preprint, January 2018
- Ziatdinov, Maxim; Dyck, Ondrej; Sumpter, Bobby G.
- arXiv
Automated Structure Discovery in Atomic Force Microscopy
text, January 2019
- Alldritt, Benjamin; Hapala, Prokop; Oinonena, Niko
- arXiv
Figures / Tables found in this record: