Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Learning surface molecular structures via machine vision

Abstract

Recent advances in high resolution scanning transmission electron and scanning probe microscopies have allowed researchers to perform measurements of materials structural parameters and functional properties in real space with a picometre precision. In many technologically relevant atomic and/or molecular systems, however, the information of interest is distributed spatially in a non-uniform manner and may have a complex multi-dimensional nature. One of the critical issues, therefore, lies in being able to accurately identify (‘read out’) all the individual building blocks in different atomic/molecular architectures, as well as more complex patterns that these blocks may form, on a scale of hundreds and thousands of individual atomic/molecular units. Here we employ machine vision to read and recognize complex molecular assemblies on surfaces. Specifically, we combine Markov random field model and convolutional neural networks to classify structural and rotational states of all individual building blocks in molecular assembly on the metallic surface visualized in high-resolution scanning tunneling microscopy measurements. We show how the obtained full decoding of the system allows us to directly construct a pair density function—a centerpiece in analysis of disorder-property relationship paradigm—as well as to analyze spatial correlations between multiple order parameters at the nanoscale, and elucidate reaction pathway involving molecularmore » conformation changes. Here, the method represents a significant shift in our way of analyzing atomic and/or molecular resolved microscopic images and can be applied to variety of other microscopic measurements of structural, electronic, and magnetic orders in different condensed matter systems.« less

Authors:
 [1]; ORCiD logo [2];  [1]
+ Show Author Affiliations
  1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  2. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States); Univ. of Tennessee, Knoxville, TN (United States)
Publication Date:
Research Org.:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1376377
Grant/Contract Number:  
AC05-00OR22725
Resource Type:
Accepted Manuscript
Journal Name:
npj Computational Materials
Additional Journal Information:
Journal Volume: 3; Journal Issue: 1; Journal ID: ISSN 2057-3960
Publisher:
Nature Publishing Group
Country of Publication:
United States
Language:
English
Subject:
36 MATERIALS SCIENCE; 37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CHEMISTRY; Carbon nanotubes and fullerenes; Molecular self-assembly; Scanning probe microscopy

Citation Formats

Ziatdinov, Maxim, Maksov, Artem, and Kalinin, Sergei V. Learning surface molecular structures via machine vision. United States: N. p., 2017. Web. doi:10.1038/s41524-017-0038-7.
Copy to clipboard
Ziatdinov, Maxim, Maksov, Artem, & Kalinin, Sergei V. Learning surface molecular structures via machine vision. United States. https://doi.org/10.1038/s41524-017-0038-7
Copy to clipboard
Ziatdinov, Maxim, Maksov, Artem, and Kalinin, Sergei V. Thu . "Learning surface molecular structures via machine vision". United States. https://doi.org/10.1038/s41524-017-0038-7. https://www.osti.gov/servlets/purl/1376377.
Copy to clipboard
@article{osti_1376377,
title = {Learning surface molecular structures via machine vision},
author = {Ziatdinov, Maxim and Maksov, Artem and Kalinin, Sergei V.},
abstractNote = {Recent advances in high resolution scanning transmission electron and scanning probe microscopies have allowed researchers to perform measurements of materials structural parameters and functional properties in real space with a picometre precision. In many technologically relevant atomic and/or molecular systems, however, the information of interest is distributed spatially in a non-uniform manner and may have a complex multi-dimensional nature. One of the critical issues, therefore, lies in being able to accurately identify (‘read out’) all the individual building blocks in different atomic/molecular architectures, as well as more complex patterns that these blocks may form, on a scale of hundreds and thousands of individual atomic/molecular units. Here we employ machine vision to read and recognize complex molecular assemblies on surfaces. Specifically, we combine Markov random field model and convolutional neural networks to classify structural and rotational states of all individual building blocks in molecular assembly on the metallic surface visualized in high-resolution scanning tunneling microscopy measurements. We show how the obtained full decoding of the system allows us to directly construct a pair density function—a centerpiece in analysis of disorder-property relationship paradigm—as well as to analyze spatial correlations between multiple order parameters at the nanoscale, and elucidate reaction pathway involving molecular conformation changes. Here, the method represents a significant shift in our way of analyzing atomic and/or molecular resolved microscopic images and can be applied to variety of other microscopic measurements of structural, electronic, and magnetic orders in different condensed matter systems.},
doi = {10.1038/s41524-017-0038-7},
journal = {npj Computational Materials},
number = 1,
volume = 3,
place = {United States},
year = {Thu Aug 10 00:00:00 EDT 2017},
month = {Thu Aug 10 00:00:00 EDT 2017}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1038/s41524-017-0038-7
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 65 works
Citation information provided by
Web of Science

Figures / Tables:

Figure 1 Figure 1: Description of the system and physical priors. Schematics of a sumanene molecule on a gold substrate and b bowl-up (U) and bowldown (D) conformational states. c Experimental STM image (raw data) of sumanene ad-layer structure on gold (111) surface. The image resolution is 910 px × 910 px.more » Inset shows a global Fast Fourier Transform of the STM image. The inner hexagon marked with yellow circles is due to formation of 2U1D structure in certain parts of the image. d Visual inspection of different conformational states (U and D) and rotational states from selected area of the STM image. The D states are marked by blue circles. The U states are marked by triangular whose orientation reflects a presence of different rotational states. e Principal component analysis on the dataset containing all the 938 molecules extracted from c. The first eigenvector is the mean value whereas the rest five eigenvectors correspond to the largest variances in the dataset (see also Supplementary Note 4) and can be linked to presence of different rotational classes (schematically indicated in the bottom right corner) observed e.g. in d« less
All figures and tables (8 total)
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

Bowl Inversion and Electronic Switching of Buckybowls on Gold
journal, September 2016

  • Fujii, Shintaro; Ziatdinov, Maxim; Higashibayashi, Shuhei
  • Journal of the American Chemical Society, Vol. 138, Issue 37
  • DOI: 10.1021/jacs.6b04741

Microscopy: Hasten high resolution
journal, November 2014

  • Pennycook, Stephen J.; Kalinin, Sergei V.
  • Nature, Vol. 515, Issue 7528
  • DOI: 10.1038/515487a

Spontaneous Vortex Nanodomain Arrays at Ferroelectric Heterointerfaces
journal, February 2011

  • Nelson, Christopher T.; Winchester, Benjamin; Zhang, Yi
  • Nano Letters, Vol. 11, Issue 2
  • DOI: 10.1021/nl1041808

Direct Imaging of Covalent Bond Structure in Single-Molecule Chemical Reactions
journal, May 2013

Local Indicators of Spatial Association-LISA
journal, April 1995

Principal component and spatial correlation analysis of spectroscopic-imaging data in scanning probe microscopy
journal, February 2009

A Synthesis of Sumanene, a Fullerene Fragment
journal, September 2003

The Analysis of Spatial Association by Use of Distance Statistics
journal, July 1992

Interplay between defects, disorder and flexibility in metal-organic frameworks
journal, December 2016

  • Bennett, Thomas D.; Cheetham, Anthony K.; Fuchs, Alain H.
  • Nature Chemistry, Vol. 9, Issue 1
  • DOI: 10.1038/nchem.2691

Ab initio quantum chemistry: Methodology and applications
journal, May 2005

Observing Atomic Collapse Resonances in Artificial Nuclei on Graphene
journal, March 2013

First-Principles Calculations of Complex Metal-Oxide Materials
journal, August 2010

Markov Random Field Texture Models
journal, January 1983

  • Cross, George R.; Jain, Anil K.
  • IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-5, Issue 1
  • DOI: 10.1109/TPAMI.1983.4767341

Bowl Inversion of Surface-Adsorbed Sumanene
journal, September 2014

  • Jaafar, Rached; Pignedoli, Carlo A.; Bussi, Giovanni
  • Journal of the American Chemical Society, Vol. 136, Issue 39
  • DOI: 10.1021/ja504126z

Big–deep–smart data in imaging for guiding materials design
journal, September 2015

  • Kalinin, Sergei V.; Sumpter, Bobby G.; Archibald, Richard K.
  • Nature Materials, Vol. 14, Issue 10
  • DOI: 10.1038/nmat4395

First-principles calculations for point defects in solids
journal, March 2014

  • Freysoldt, Christoph; Grabowski, Blazej; Hickel, Tilmann
  • Reviews of Modern Physics, Vol. 86, Issue 1
  • DOI: 10.1103/RevModPhys.86.253

Visualization of the Molecular Jahn-Teller Effect in an Insulating K4C60 Monolayer
journal, October 2005

Strain Doping: Reversible Single-Axis Control of a Complex Oxide Lattice via Helium Implantation
journal, June 2015

The manipulation of C 60 in molecular arrays with an STM tip in regimes below the decomposition threshold
journal, January 2013

A disorder-enhanced quasi-one-dimensional superconductor
journal, July 2016

  • Petrović, A. P.; Ansermet, D.; Chernyshov, D.
  • Nature Communications, Vol. 7, Issue 1
  • DOI: 10.1038/ncomms12262

Combining satellite imagery and machine learning to predict poverty
journal, August 2016

The crystallography of correlated disorder
journal, May 2015

On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs
journal, January 2001

  • Weiss, Y.; Freeman, W. T.
  • IEEE Transactions on Information Theory, Vol. 47, Issue 2
  • DOI: 10.1109/18.910585

Design of crystal-like aperiodic solids with selective disorder–phonon coupling
journal, February 2016

  • Overy, Alistair R.; Cairns, Andrew B.; Cliffe, Matthew J.
  • Nature Communications, Vol. 7, Issue 1
  • DOI: 10.1038/ncomms10445

Atomic-scale study of electric dipoles near charged and uncharged domain walls in ferroelectric films
journal, December 2007

  • Jia, Chun-Lin; Mi, Shao-Bo; Urban, Knut
  • Nature Materials, Vol. 7, Issue 1
  • DOI: 10.1038/nmat2080

Unit-cell scale mapping of ferroelectricity and tetragonality in epitaxial ultrathin ferroelectric films
journal, December 2006

  • Jia, Chun-Lin; Nagarajan, Valanoor; He, Jia-Qing
  • Nature Materials, Vol. 6, Issue 1
  • DOI: 10.1038/nmat1808

Defocused Emission Patterns from Chiral Fluorophores: Application to Chiral Axis Orientation Determination
journal, February 2011

  • Cyphersmith, A.; Maksov, A.; Hassey-Paradise, R.
  • The Journal of Physical Chemistry Letters, Vol. 2, Issue 6
  • DOI: 10.1021/jz2001024

Rigorous force field optimization principles based on statistical distance minimization
journal, October 2015

  • Vlcek, Lukas; Chialvo, Ariel A.
  • The Journal of Chemical Physics, Vol. 143, Issue 14
  • DOI: 10.1063/1.4932360

Scanning tunneling microscopy fingerprints of point defects in graphene: A theoretical prediction
journal, September 2007

Structure and energetics of the vacancy in graphite
journal, October 2003

Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis
journal, May 2016

  • Litjens, Geert; Sánchez, Clara I.; Timofeeva, Nadya
  • Scientific Reports, Vol. 6, Issue 1
  • DOI: 10.1038/srep26286

Direct observation of Σ7 domain boundary core structure in magnetic skyrmion lattice
journal, February 2016

A disorder-enhanced quasi-one-dimensional superconductor
text, January 2016

    Sort by:
    [ × clear filter / sort ]

    Works referencing / citing this record:

    Interface Characterization and Control of 2D Materials and Heterostructures
    journal, July 2018

    Simulation and design of energy materials accelerated by machine learning
    journal, June 2019

    • Wang, Hongshuai; Ji, Yujin; Li, Youyong
    • Wiley Interdisciplinary Reviews: Computational Molecular Science
    • DOI: 10.1002/wcms.1421

    Revealing ferroelectric switching character using deep recurrent neural networks
    journal, October 2019

    Analyzing machine learning models to accelerate generation of fundamental materials insights
    journal, March 2019

    • Umehara, Mitsutaro; Stein, Helge S.; Guevarra, Dan
    • npj Computational Materials, Vol. 5, Issue 1
    • DOI: 10.1038/s41524-019-0172-5

    Deep data analytics for genetic engineering of diatoms linking genotype to phenotype via machine learning
    journal, June 2019

    • Trofimov, Artem A.; Pawlicki, Alison A.; Borodinov, Nikolay
    • npj Computational Materials, Vol. 5, Issue 1
    • DOI: 10.1038/s41524-019-0202-3

    Atom-by-atom fabrication with electron beams
    journal, June 2019

    • Dyck, Ondrej; Ziatdinov, Maxim; Lingerfelt, David B.
    • Nature Reviews Materials, Vol. 4, Issue 7
    • DOI: 10.1038/s41578-019-0118-z

    Machine learning for molecular and materials science
    journal, July 2018

    Building and exploring libraries of atomic defects in graphene: Scanning transmission electron and scanning tunneling microscopy study
    journal, September 2019

    Automated structure discovery in atomic force microscopy
    journal, February 2020

    • Alldritt, Benjamin; Hapala, Prokop; Oinonen, Niko
    • Science Advances, Vol. 6, Issue 9
    • DOI: 10.1126/sciadv.aay6913

    Building and exploring libraries of atomic defects in graphene: Scanning transmission electron and scanning tunneling microscopy study
    journal, September 2019

    AFLOW-ML: A RESTful API for machine-learning predictions of materials properties
    preprint, January 2017

    Automated Structure Discovery in Atomic Force Microscopy
    text, January 2019

      Sort by:
      [ × clear filter / sort ]

      Figures / Tables found in this record:

      Figure 1 (p. 2)figure
      Figure 2 (p. 3)figure
      Table 1 (p. 4)table
      Figure 3 (p. 5)figure
      Figure 4 (p. 6)figure
      Figure 5 (p. 6)figure
      Figure 6 (p. 7)figure
      Figure 7 (p. 8)figure
        [ × clear filter / sort ]
        Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.

        Similar Records in DOE PAGES and OSTI.GOV collections: