Autonomous Molecular Structure Imaging with High-Resolution Atomic Force Microscopy for Molecular Mixture Discovery

Arias, Steven; Zhang, Yunlong; Zahl, Percy; Hollen, Shawna

doi:10.1021/acs.jpca.3c01685

Autonomous Molecular Structure Imaging with High-Resolution Atomic Force Microscopy for Molecular Mixture Discovery

Journal Article · Tue Jul 18 00:00:00 EDT 2023 · Journal of Physical Chemistry. A, Molecules, Spectroscopy, Kinetics, Environment, and General Theory

DOI:https://doi.org/10.1021/acs.jpca.3c01685· OSTI ID:2282115

^[1]; ^[2]; ^[3]; ^[1]

Univ. of New Hampshire, Durham, NH (United States)
ExxonMobil Research and Engineering Co., Annandale, NJ (United States)
Brookhaven National Laboratory (BNL), Upton, NY (United States). Center for Functional Nanomaterials (CFN)

Due to its single-molecule sensitivity, high-resolution atomic force microscopy (HR-AFM) has proved to be a valuable and uniquely advantageous tool to study complex molecular mixtures, which hold promise for developing clean energy and achieving environmental sustainability. However, significant challenges remain to achieve the full potential of the sophisticated and time-consuming experiments. Automation combined with machine learning (ML) and artificial intelligence (AI) is key to overcoming these challenges. Here we present Auto-HR-AFM, an AI tool to automatically collect HR-AFM images of petroleum-based mixtures. In this study, we trained an instance segmentation model to teach Auto-HR-AFM how to recognize features in HR-AFM images. Auto-HR-AFM then uses that information to optimize the imaging by adjusting the probe-molecule distance for each molecule in the run. Auto-HR-AFM is the initial tool that will lead to fully automated scanning probe microscopy (SPM) experiments, from start to finish. This automation will allow SPM to become a mainstream characterization technique for complex mixtures, an otherwise unattainable target.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States). Center for Functional Nanomaterials (CFN); Oak Ridge Institute for Science and Education (ORISE), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities (SUF); USDOE Office of Science (SC), Office of Workforce Development for Teachers & Scientists (WDTS)

Grant/Contract Number:: SC0012704; SC0014664

OSTI ID:: 2282115

Report Number(s):: BNL--225224-2024-JAAM

Journal Information:: Journal of Physical Chemistry. A, Molecules, Spectroscopy, Kinetics, Environment, and General Theory, Journal Name: Journal of Physical Chemistry. A, Molecules, Spectroscopy, Kinetics, Environment, and General Theory Journal Issue: 29 Vol. 127; ISSN 1089-5639

Publisher:: American Chemical SocietyCopyright Statement

Country of Publication:: United States

Language:: English

References (38)

Microsoft COCO: Common Objects in Context Lin, Tsung-Yi; Maire, Michael; Belongie, Serge ECCV 2014: Computer Vision – ECCV 2014 https://doi.org/10.1007/978-3-319-10602-1_48	book	January 2014
Petroleum pitch: Exploring a 50-year structure puzzle with real-space molecular imaging Chen, Pengcheng; Metz, Jordan N.; Mennito, Anthony S. Carbon, Vol. 161 https://doi.org/10.1016/j.carbon.2020.01.062	journal	May 2020
On the early stages of soot formation: Molecular structure elucidation by high-resolution atomic force microscopy Commodo, Mario; Kaiser, Katharina; De Falco, Gianluigi Combustion and Flame, Vol. 205 https://doi.org/10.1016/j.combustflame.2019.03.042	journal	July 2019
Automated tip functionalization via machine learning in scanning probe microscopy Alldritt, Benjamin; Urtev, Fedor; Oinonen, Niko Computer Physics Communications, Vol. 273 https://doi.org/10.1016/j.cpc.2021.108258	journal	April 2022
Automatic detection of Acacia longifolia invasive species based on UAV-acquired aerial imagery Gonçalves, Carolina; Santana, Pedro; Brandão, Tomás Information Processing in Agriculture, Vol. 9, Issue 2 https://doi.org/10.1016/j.inpa.2021.04.007	journal	June 2022
Insights into incipient soot formation by atomic force microscopy Schulz, Fabian; Commodo, Mario; Kaiser, Katharina Proceedings of the Combustion Institute, Vol. 37, Issue 1 https://doi.org/10.1016/j.proci.2018.06.100	journal	January 2019
Counting molecules: Python based scheme for automated enumeration and categorization of molecules in scanning tunneling microscopy images Hellerstedt, Jack; Cahlík, Aleš; Švec, Martin Software Impacts, Vol. 12 https://doi.org/10.1016/j.simpa.2022.100301	journal	May 2022
Overview of Asphaltene Nanostructures and Thermodynamic Applications Schuler, Bruno; Zhang, Yunlong; Liu, Fang Energy & Fuels, Vol. 34, Issue 12 https://doi.org/10.1021/acs.energyfuels.0c00874	journal	May 2020
Applications of Noncontact Atomic Force Microscopy in Petroleum Characterization: Opportunities and Challenges Zhang, Yunlong Energy & Fuels, Vol. 35, Issue 18 https://doi.org/10.1021/acs.energyfuels.1c02193	journal	September 2021
Perspectives on a New Age of Materials for Petroleum Zhang, Yunlong Energy & Fuels, Vol. 36, Issue 11 https://doi.org/10.1021/acs.energyfuels.2c01434	journal	May 2022
Heavy Oil Based Mixtures of Different Origins and Treatments Studied by Atomic Force Microscopy Schuler, Bruno; Fatayer, Shadi; Meyer, Gerhard Energy & Fuels, Vol. 31, Issue 7 https://doi.org/10.1021/acs.energyfuels.7b00805	journal	June 2017
Guide for Atomic Force Microscopy Image Analysis To Discriminate Heteroatoms in Aromatic Molecules Zahl, Percy; Zhang, Yunlong Energy & Fuels, Vol. 33, Issue 6 https://doi.org/10.1021/acs.energyfuels.9b00165	journal	April 2019
Elucidating the Geometric Substitution of Petroporphyrins by Spectroscopic Analysis and Atomic Force Microscopy Molecular Imaging Zhang, Yunlong; Schulz, Fabian; Rytting, B. McKay Energy & Fuels, Vol. 33, Issue 7 https://doi.org/10.1021/acs.energyfuels.9b00816	journal	June 2019
Understanding the Effects of Sample Preparation on the Chemical Structures of Petroleum Imaged with Noncontact Atomic Force Microscopy Zhang, Yunlong; Schuler, Bruno; Fatayer, Shadi Industrial & Engineering Chemistry Research, Vol. 57, Issue 46 https://doi.org/10.1021/acs.iecr.8b03962	journal	October 2018
Automated Tip Conditioning for Scanning Tunneling Spectroscopy Wang, Shenkai; Zhu, Junmian; Blackwell, Raymond The Journal of Physical Chemistry A, Vol. 125, Issue 6 https://doi.org/10.1021/acs.jpca.0c10731	journal	February 2021
Automated Searching and Identification of Self-Organized Nanostructures Gordon, Oliver M.; Hodgkinson, Jo E. A.; Farley, Steff M. Nano Letters, Vol. 20, Issue 10 https://doi.org/10.1021/acs.nanolett.0c03213	journal	August 2020
Molecular Identification from AFM Images Using the IUPAC Nomenclature and Attribute Multimodal Recurrent Neural Networks Carracedo-Cosme, Jaime; Romero-Muñiz, Carlos; Pou, Pablo ACS Applied Materials & Interfaces, Vol. 15, Issue 18 https://doi.org/10.1021/acsami.3c01550	journal	May 2023
Unraveling the Molecular Structures of Asphaltenes by Atomic Force Microscopy Schuler, Bruno; Meyer, Gerhard; Peña, Diego Journal of the American Chemical Society, Vol. 137, Issue 31 https://doi.org/10.1021/jacs.5b04056	journal	July 2015
Atomic Force Microscopy Identifying Fuel Pyrolysis Products and Directing the Synthesis of Analytical Standards Fatayer, Shadi; Poddar, Nimesh B.; Quiroga, Sabela Journal of the American Chemical Society, Vol. 140, Issue 26 https://doi.org/10.1021/jacs.8b02525	journal	June 2018
Direct Visualization of Individual Aromatic Compound Structures in Low Molecular Weight Marine Dissolved Organic Carbon Fatayer, Shadi; Coppola, Alysha I.; Schulz, Fabian Geophysical Research Letters, Vol. 45, Issue 11 https://doi.org/10.1029/2018GL077457	journal	June 2018
Organic structure determination using atomic-resolution scanning probe microscopy Gross, Leo; Mohn, Fabian; Moll, Nikolaj Nature Chemistry, Vol. 2, Issue 10 https://doi.org/10.1038/nchem.765	journal	August 2010
Synthesis and characterization of triangulene Pavliček, Niko; Mistry, Anish; Majzik, Zsolt Nature Nanotechnology, Vol. 12, Issue 4 https://doi.org/10.1038/nnano.2016.305	journal	February 2017
Artificial-intelligence-driven scanning probe microscopy Krull, A.; Hirsch, P.; Rother, C. Communications Physics, Vol. 3, Issue 1 https://doi.org/10.1038/s42005-020-0317-3	journal	March 2020
The flexible and modern open source scanning probe microscopy software package GXSM Zahl, Percy; Bierkandt, Markus; Schröder, Stefan Review of Scientific Instruments, Vol. 74, Issue 3 https://doi.org/10.1063/1.1540718	journal	March 2003
Scanning tunneling state recognition with multi-class neural network ensembles Gordon, O.; D’Hondt, P.; Knijff, L. Review of Scientific Instruments, Vol. 90, Issue 10 https://doi.org/10.1063/1.5099590	journal	October 2019
Embedding human heuristics in machine-learning-enabled probe microscopy Gordon, Oliver M.; Junqueira, Filipe L. Q.; Moriarty, Philip J. Machine Learning: Science and Technology, Vol. 1, Issue 1 https://doi.org/10.1088/2632-2153/ab42ec	journal	February 2020
Deep learning-guided surface characterization for autonomous hydrogen lithography Rashidi, Mohammad; Croshaw, Jeremiah; Mastel, Kieran Machine Learning: Science and Technology, Vol. 1, Issue 2 https://doi.org/10.1088/2632-2153/ab6d5e	journal	March 2020
Improving the segmentation of scanning probe microscope images using convolutional neural networks Farley, Steff; Hodgkinson, Jo E. A.; Gordon, Oliver M. Machine Learning: Science and Technology, Vol. 2, Issue 1 https://doi.org/10.1088/2632-2153/abc81c	journal	December 2020
Mechanism of high-resolution STM/AFM imaging with functionalized tips Hapala, Prokop; Kichin, Georgy; Wagner, Christian Physical Review B, Vol. 90, Issue 8 https://doi.org/10.1103/PhysRevB.90.085421	journal	August 2014
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Feature Pyramid Networks for Object Detection Lin, Tsung-Yi; Dollar, Piotr; Girshick, Ross 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.106	conference	July 2017
Mask R-CNN He, Kaiming; Gkioxari, Georgia; Dollar, Piotr 2017 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2017.322	conference	October 2017
Visualization and identification of single meteoritic organic molecules by atomic force microscopy Kaiser, Katharina; Schulz, Fabian; Maillard, Julien F. Meteoritics & Planetary Science, Vol. 57, Issue 3 https://doi.org/10.1111/maps.13784	journal	February 2022
Open source scanning probe microscopy control software package GXSM Zahl, Percy; Wagner, Thorsten; Möller, Rolf Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena, Vol. 28, Issue 3 https://doi.org/10.1116/1.3374719	journal	May 2010
The Chemical Structure of a Molecule Resolved by Atomic Force Microscopy Gross, L.; Mohn, F.; Moll, N. Science, Vol. 325, Issue 5944 https://doi.org/10.1126/science.1176210	journal	August 2009
Molecule graph reconstruction from atomic force microscope images with machine learning Oinonen, Niko; Kurki, Lauri; Ilin, Alexander MRS Bulletin, Vol. 47, Issue 9 https://doi.org/10.1557/s43577-022-00324-3	journal	July 2022
A Means of Assessing Deep Learning-Based Detection of ICOS Protein Expression in Colon Cancer Sarker, Md Mostafa Kamal; Makhlouf, Yasmine; Craig, Stephanie G. Cancers, Vol. 13, Issue 15 https://doi.org/10.3390/cancers13153825	journal	July 2021
Imaging Titan’s Organic Haze at Atomic Scale Schulz, Fabian; Maillard, Julien; Kaiser, Katharina The Astrophysical Journal Letters, Vol. 908, Issue 1 https://doi.org/10.3847/2041-8213/abd93e	journal	February 2021

Similar Records

Atomic force microscopy of biological samples

Journal Article · Thu Dec 31 23:00:00 EST 2009 · Wiley Interdisciplinary Reviews: Nanomedicine and Nanobiotechnology · OSTI ID:1045301

Autonomous Experiments in Scanning Probe Microscopy and Spectroscopy: Choosing Where to Explore Polarization Dynamics in Ferroelectrics

Journal Article · Mon Jul 05 20:00:00 EDT 2021 · ACS Nano · OSTI ID:1817554

Related Subjects

36 MATERIALS SCIENCE
mixtures
molecular imaging
molecular modeling
molecules
scanning probe microscopy

Autonomous Molecular Structure Imaging with High-Resolution Atomic Force Microscopy for Molecular Mixture Discovery

Citation Formats

References (38)

Similar Records

Related Subjects