An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models

Hattrick-Simpers, Jason R.; DeCost, Brian; Kusne, A. Gilad; Joress, Howie; Wong-Ng, Winnie; Kaiser, Debra L.; Zakutayev, Andriy; Phillips, Caleb; Sun, Shijing; Thapa, Janak; Yu, Heshan; Takeuchi, Ichiro; Buonassisi, Tonio

doi:10.1007/s40192-021-00213-8

Title: An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models

Abstract

Modern machine learning and autonomous experimentation schemes in materials science rely on accurate analysis of the data ingested by these models. Unfortunately, accurate analysis of the underlying data can be difficult, even for domain experts, complicating the training of the models intended to drive experiments. This is especially true when the goal is to identify the presence of weak signatures in diffraction or spectroscopic datasets. In this work, we examine a set of as-obtained diffraction data that track the phase transition from monoclinic to tetragonal in a Nb-doped VO2 film as a function of temperature and dopant concentration. We then task a set of domain experts and a set of machine learning experts with identifying which phase is present in each diffraction pattern manually and algorithmically, respectively; in both cases, the labels can vary dramatically, especially at the phase boundaries. We use the mode of the labels and the Shannon entropy as a method to capture, preserve and propagate consensus labels and their variance. Further we use the expert labels as a benchmark and demonstrate the use of Shannon entropy weighted scoring to test the performance of machine learning generated labels. Finally, we propose a material data challenge centered aroundmore »« less

Authors:

^[1]; DeCost, Brian ^[1]; Kusne, A. Gilad ^[1]; Joress, Howie ^[1]; Wong-Ng, Winnie ^[1]; Kaiser, Debra L. ^[1];

^[2];

^[2]; Sun, Shijing ^[3]; Thapa, Janak ^[3]; Yu, Heshan ^[4]; Takeuchi, Ichiro ^[4]; Buonassisi, Tonio ^[3]

National Inst. of Standards and Technology (NIST), Gaithersburg, MD (United States)
National Renewable Energy Lab. (NREL), Golden, CO (United States)
Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)
Univ. of Maryland, College Park, MD (United States)

Publication Date:: Wed Jun 09 00:00:00 EDT 2021

Research Org.:: National Renewable Energy Laboratory (NREL), Golden, CO (United States)

Sponsoring Org.:: USDOE National Renewable Energy Laboratory (NREL), Laboratory Directed Research and Development (LDRD) Program

OSTI Identifier:: 1798722

Report Number(s):: NREL/JA-5K00-78444
Journal ID: ISSN 2193-9764; MainId:32361;UUID:5f3ffc91-f636-4657-aa90-7f0c4826215d;MainAdminID:25675

Grant/Contract Number:: AC36-08GO28308

Resource Type:: Accepted Manuscript

Journal Name:: Integrating Materials and Manufacturing Innovation

Additional Journal Information:: Journal Volume: 10; Journal Issue: 2; Journal ID: ISSN 2193-9764

Publisher:: Springer

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; combinatorial; diffraction; machine learning; VO2

Citation Formats


                    Hattrick-Simpers, Jason R., DeCost, Brian, Kusne, A. Gilad, Joress, Howie, Wong-Ng, Winnie, Kaiser, Debra L., Zakutayev, Andriy, Phillips, Caleb, Sun, Shijing, Thapa, Janak, Yu, Heshan, Takeuchi, Ichiro, and Buonassisi, Tonio. An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models.  United States: N. p., 2021. 
Web.  doi:10.1007/s40192-021-00213-8.

Copy to clipboard


                    Hattrick-Simpers, Jason R., DeCost, Brian, Kusne, A. Gilad, Joress, Howie, Wong-Ng, Winnie, Kaiser, Debra L., Zakutayev, Andriy, Phillips, Caleb, Sun, Shijing, Thapa, Janak, Yu, Heshan, Takeuchi, Ichiro, & Buonassisi, Tonio. An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models.  United States.  https://doi.org/10.1007/s40192-021-00213-8

Copy to clipboard


                    Hattrick-Simpers, Jason R., DeCost, Brian, Kusne, A. Gilad, Joress, Howie, Wong-Ng, Winnie, Kaiser, Debra L., Zakutayev, Andriy, Phillips, Caleb, Sun, Shijing, Thapa, Janak, Yu, Heshan, Takeuchi, Ichiro, and Buonassisi, Tonio. Wed .  
"An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models".  United States.  https://doi.org/10.1007/s40192-021-00213-8.  https://www.osti.gov/servlets/purl/1798722.

Copy to clipboard


                    
@article{osti_1798722,

  title        = {An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models},

  author       = {Hattrick-Simpers, Jason R. and DeCost, Brian and Kusne, A. Gilad and Joress, Howie and Wong-Ng, Winnie and Kaiser, Debra L. and Zakutayev, Andriy and Phillips, Caleb and Sun, Shijing and Thapa, Janak and Yu, Heshan and Takeuchi, Ichiro and Buonassisi, Tonio},

  abstractNote = {Modern machine learning and autonomous experimentation schemes in materials science rely on accurate analysis of the data ingested by these models. Unfortunately, accurate analysis of the underlying data can be difficult, even for domain experts, complicating the training of the models intended to drive experiments. This is especially true when the goal is to identify the presence of weak signatures in diffraction or spectroscopic datasets. In this work, we examine a set of as-obtained diffraction data that track the phase transition from monoclinic to tetragonal in a Nb-doped VO2 film as a function of temperature and dopant concentration. We then task a set of domain experts and a set of machine learning experts with identifying which phase is present in each diffraction pattern manually and algorithmically, respectively; in both cases, the labels can vary dramatically, especially at the phase boundaries. We use the mode of the labels and the Shannon entropy as a method to capture, preserve and propagate consensus labels and their variance. Further we use the expert labels as a benchmark and demonstrate the use of Shannon entropy weighted scoring to test the performance of machine learning generated labels. Finally, we propose a material data challenge centered around generating improved labeling algorithms. This real-world dataset curated with expert labels can act as test bed for new algorithms. The raw data, annotations and code used in this study are all available online at data.gov and the interested reader is encouraged to replicate and improve the existing models},

  doi          = {10.1007/s40192-021-00213-8},

  journal      = {Integrating Materials and Manufacturing Innovation},

  number       = 2,

  volume       = 10,

  place        = {United States},

  year         = {Wed Jun 09 00:00:00 EDT 2021},

  month        = {Wed Jun 09 00:00:00 EDT 2021}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1007/s40192-021-00213-8

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Pauling's model not universally accepted
journal, January 1986

Cahn, John W.; Gratias, Denis; Shechtman, Dan
Nature, Vol. 319, Issue 6049
DOI: 10.1038/319102a0

Adjustment of thermal hysteresis in epitaxial VO2 films by doping metal ions
journal, January 2011

Nishikawa, Masami; Nakajima, Tomohiko; Kumagai, Toshiya
Journal of the Ceramic Society of Japan, Vol. 119, Issue 1391
DOI: 10.2109/jcersj2.119.577

Correlation between thermal hysteresis width and broadening of metal–insulator transition in Cr- and Nb-doped VO ₂ films
journal, June 2014

Miyazaki, Kenichi; Shibuya, Keisuke; Suzuki, Megumi
Japanese Journal of Applied Physics, Vol. 53, Issue 7
DOI: 10.7567/JJAP.53.071102

A Kriging-Based Approach to Autonomous Experimentation with Applications to X-Ray Scattering
journal, August 2019

Noack, Marcus M.; Yager, Kevin G.; Fukuto, Masafumi
Scientific Reports, Vol. 9, Issue 1
DOI: 10.1038/s41598-019-48114-3

Measurement of the hysteretic thermal properties of W-doped and undoped nanocrystalline powders of VO2
journal, October 2019

Gomez-Heredia, C. L.; Ramirez-Rincon, J. A.; Bhardwaj, D.
Scientific Reports, Vol. 9, Issue 1
DOI: 10.1038/s41598-019-51162-4

Get another label? improving data quality and data mining using multiple, noisy labelers
conference, January 2008

Sheng, Victor S.; Provost, Foster; Ipeirotis, Panagiotis G.
Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD 08
DOI: 10.1145/1401890.1401965

On-the-fly machine-learning for high-throughput experiments: search for rare-earth-free permanent magnets
journal, September 2014

Kusne, Aaron Gilad; Gao, Tieren; Mehta, Apurva
Scientific Reports, Vol. 4, Issue 1
DOI: 10.1038/srep06367

High-Throughput Measurements of Thermochromic Behavior in V _{1– x} Nb _x O ₂ Combinatorial Thin Film Libraries
journal, September 2014

Barron, S. C.; Gorham, J. M.; Patel, M. P.
ACS Combinatorial Science, Vol. 16, Issue 10
DOI: 10.1021/co500064p

Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2
journal, February 2019

Maksov, Artem; Dyck, Ondrej; Wang, Kai
npj Computational Materials, Vol. 5, Issue 1
DOI: 10.1038/s41524-019-0152-9

Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy
journal, August 2018

Krause, Jonathan; Gulshan, Varun; Rahimy, Ehsan
Ophthalmology, Vol. 125, Issue 8
DOI: 10.1016/j.ophtha.2018.01.034

Structural Characterization of Atomic Layer Deposited Vanadium Dioxide
journal, August 2017

Kozen, Alexander C.; Joress, Howie; Currie, Marc
The Journal of Physical Chemistry C, Vol. 121, Issue 35
DOI: 10.1021/acs.jpcc.7b04682

Recent advances and applications of machine learning in solid-state materials science
journal, August 2019

Schmidt, Jonathan; Marques, Mário R. G.; Botti, Silvana
npj Computational Materials, Vol. 5, Issue 1
DOI: 10.1038/s41524-019-0221-0

Automated defect analysis in electron microscopic images
journal, July 2018

Li, Wei; Field, Kevin G.; Morgan, Dane
npj Computational Materials, Vol. 4, Issue 1
DOI: 10.1038/s41524-018-0093-8

Self-driving laboratory for accelerated discovery of thin-film materials
journal, May 2020

MacLeod, B. P.; Parlane, F. G. L.; Morrissey, T. D.
Science Advances, Vol. 6, Issue 20
DOI: 10.1126/sciadv.aaz8867

Model, prediction, and experimental verification of composition and thickness in continuous spread thin film combinatorial libraries grown by pulsed laser deposition
journal, July 2007

Bassim, N. D.; Schenck, P. K.; Otani, M.
Review of Scientific Instruments, Vol. 78, Issue 7
DOI: 10.1063/1.2755783

A High-Throughput Structural and Electrochemical Study of Metallic Glass Formation in Ni–Ti–Al
journal, June 2020

Joress, Howie; DeCost, Brian L.; Sarker, Suchismita
ACS Combinatorial Science, Vol. 22, Issue 7
DOI: 10.1021/acscombsci.9b00215

Autonomy in materials research: a case study in carbon nanotube growth
journal, October 2016

Nikolaev, Pavel; Hooper, Daylond; Webber, Frederick
npj Computational Materials, Vol. 2, Issue 1
DOI: 10.1038/npjcompumats.2016.31

How Water’s Properties Are Encoded in Its Molecular Structure and Energies
journal, September 2017

Brini, Emiliano; Fennell, Christopher J.; Fernandez-Serra, Marivi
Chemical Reviews, Vol. 117, Issue 19
DOI: 10.1021/acs.chemrev.7b00259

The war over supercooled water
journal, August 2018

Smart, Ashley G.
Physics Today
DOI: 10.1063/pt.6.1.20180822a

A Bayesian experimental autonomous researcher for mechanical design
journal, April 2020

Gongora, Aldair E.; Xu, Bowen; Perry, Wyatt
Science Advances, Vol. 6, Issue 15
DOI: 10.1126/sciadv.aaz1708

Radial Basis Functions
book, January 2009

Buhmann, Martin D.
Cambridge University Press
DOI: 10.1017/CBO9780511543241

Tuning the hysteresis of a metal-insulator transition via lattice compatibility
journal, July 2020

Liang, Y. G.; Lee, S.; Yu, H. S.
Nature Communications, Vol. 11, Issue 1
DOI: 10.1038/s41467-020-17351-w

Active learning of uniformly accurate interatomic potentials for materials simulation
journal, February 2019

Zhang, Linfeng; Lin, De-Ye; Wang, Han
Physical Review Materials, Vol. 3, Issue 2
DOI: 10.1103/PhysRevMaterials.3.023804

Similar Records in DOE PAGES and OSTI.GOV collections:

ClimateNet: an expert-labeled open dataset and deep learning architecture for enabling high-precision analyses of extreme weather

Journal Article Prabhat ; Kashinath, Karthik ; Mudigonda, Mayur ; ... - Geoscientific Model Development (Online)

Abstract. Identifying, detecting, and localizing extreme weather events is a crucial first step in understanding how they may vary under different climate change scenarios. Pattern recognition tasks such as classification, object detection, and segmentation (i.e., pixel-level classification) have remained challenging problems in the weather and climate sciences. While there exist many empirical heuristics for detecting extreme events, the disparities between the output of these different methods even for a single event are large and often difficult to reconcile. Given the success of deep learning (DL) in tackling similar problems in computer vision, we advocate a DL-based approach. DL, however, worksmore »« less
https://doi.org/10.5194/gmd-14-107-2021
Tympana - Machine Learning Assisted Data Annotation

Technical Report Yacci, Paul M. ; Yacci, Allison ; Soni, Siddhant

A key component to making data Artificial Intelligence (AI)-ready is to provide machine-readable labels for Machine Learning (ML) algorithms. This mapping of input data to outcome provides the core foundation that supervised Machine Learning algorithms utilize. Without a proper mapping or sufficient volume of quality data, supervised learning becomes difficult if not impossible. In scientific domains this expertise comes from highly trained individuals who can properly interpret this data to provide the correct output mapping. As Deep Learning type algorithms expand, the appetite for larger and larger high-quality datasets becomes insatiable. In scientific domains it can be difficult to identifymore »« less
Comparison of Machine Learning and Deep Learning for View Identification from Cardiac Magnetic Resonance Images

Journal Article Chauhan, Daksh ; Anyanwu, Emeka C. ; Goes, Jacob P ; ... - Clinical Imaging

Background: Artificial intelligence is increasingly utilized to aid in the interpretation of cardiac magnetic resonance (CMR) studies. One of the first steps is the identification of the imaging plane depicted, which can be achieved by both deep learning (DL) and classical machine learning (ML) techniques without user input. We aimed to compare the accuracy of ML and DL for CMR view classification and to identify potential pitfalls during training and testing of the algorithms. Methods: To train our DL and ML algorithms, we first established datasets by retrospectively selecting 200 CMR cases. The models were trained using two different cohortsmore »« less
https://doi.org/10.1016/j.clinimag.2021.11.013
Accounting for Training Data Error in Machine Learning Applied to Earth Observations

Journal Article Elmes, Arthur ; Alemohammad, Hamed ; Avery, Ryan ; ... - Remote Sensing

Remote sensing, or Earth Observation (EO), is increasingly used to understand Earth system dynamics and create continuous and categorical maps of biophysical properties and land cover, especially based on recent advances in machine learning (ML). ML models typically require large, spatially explicit training datasets to make accurate predictions. Training data (TD) are typically generated by digitizing polygons on high spatial-resolution imagery, by collecting in situ data, or by using pre-existing datasets. TD are often assumed to accurately represent the truth, but in practice almost always have error, stemming from (1) sample design, and (2) sample collection errors. The latter ismore »« less
Cited by 25
https://doi.org/10.3390/rs12061034

Full Text Available
Highly transferable atomistic machine-learning potentials from curated and compact datasets across the periodic table

Journal Article Andolina, Christopher M. ; Saidi, Wissam A. - Digital Discovery

Machine learning atomistic potentials trained using density functional theory (DFT) datasets allow for the modeling of complex material properties with near-DFT accuracy while imposing a fraction of its computational cost. The curation of the DFT datasets can be extensive in size and time-consuming to train and refine. In this study, we focus on addressing these barriers by developing minimalistic and flexible datasets for many elements in the periodic table regardless of their mass, electronic configuration, and ground state lattice. These DFT datasets have on average, ~4000 different structures and 27 atoms per structure, which we found sufficient to maintain themore »« less
https://doi.org/10.1039/d3dd00046j

Full Text Available

Similar Records

Title: An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models

Abstract

Citation Formats

Pauling's model not universally accepted journal, January 1986

Adjustment of thermal hysteresis in epitaxial VO2 films by doping metal ions journal, January 2011

Correlation between thermal hysteresis width and broadening of metal–insulator transition in Cr- and Nb-doped VO 2 films journal, June 2014

A Kriging-Based Approach to Autonomous Experimentation with Applications to X-Ray Scattering journal, August 2019

Measurement of the hysteretic thermal properties of W-doped and undoped nanocrystalline powders of VO2 journal, October 2019

Get another label? improving data quality and data mining using multiple, noisy labelers conference, January 2008

On-the-fly machine-learning for high-throughput experiments: search for rare-earth-free permanent magnets journal, September 2014

High-Throughput Measurements of Thermochromic Behavior in V 1– x Nb x O 2 Combinatorial Thin Film Libraries journal, September 2014

Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2 journal, February 2019

Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy journal, August 2018

Structural Characterization of Atomic Layer Deposited Vanadium Dioxide journal, August 2017

Recent advances and applications of machine learning in solid-state materials science journal, August 2019

Automated defect analysis in electron microscopic images journal, July 2018

Self-driving laboratory for accelerated discovery of thin-film materials journal, May 2020

Model, prediction, and experimental verification of composition and thickness in continuous spread thin film combinatorial libraries grown by pulsed laser deposition journal, July 2007

A High-Throughput Structural and Electrochemical Study of Metallic Glass Formation in Ni–Ti–Al journal, June 2020

Autonomy in materials research: a case study in carbon nanotube growth journal, October 2016

How Water’s Properties Are Encoded in Its Molecular Structure and Energies journal, September 2017

The war over supercooled water journal, August 2018

A Bayesian experimental autonomous researcher for mechanical design journal, April 2020

Radial Basis Functions book, January 2009

Tuning the hysteresis of a metal-insulator transition via lattice compatibility journal, July 2020

Active learning of uniformly accurate interatomic potentials for materials simulation journal, February 2019

Pauling's model not universally accepted
journal, January 1986

Adjustment of thermal hysteresis in epitaxial VO2 films by doping metal ions
journal, January 2011

Correlation between thermal hysteresis width and broadening of metal–insulator transition in Cr- and Nb-doped VO ₂ films
journal, June 2014

A Kriging-Based Approach to Autonomous Experimentation with Applications to X-Ray Scattering
journal, August 2019

Measurement of the hysteretic thermal properties of W-doped and undoped nanocrystalline powders of VO2
journal, October 2019

Get another label? improving data quality and data mining using multiple, noisy labelers
conference, January 2008

On-the-fly machine-learning for high-throughput experiments: search for rare-earth-free permanent magnets
journal, September 2014

High-Throughput Measurements of Thermochromic Behavior in V _{1– x} Nb _x O ₂ Combinatorial Thin Film Libraries
journal, September 2014

Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2
journal, February 2019

Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy
journal, August 2018

Structural Characterization of Atomic Layer Deposited Vanadium Dioxide
journal, August 2017

Recent advances and applications of machine learning in solid-state materials science
journal, August 2019

Automated defect analysis in electron microscopic images
journal, July 2018

Self-driving laboratory for accelerated discovery of thin-film materials
journal, May 2020

Model, prediction, and experimental verification of composition and thickness in continuous spread thin film combinatorial libraries grown by pulsed laser deposition
journal, July 2007

A High-Throughput Structural and Electrochemical Study of Metallic Glass Formation in Ni–Ti–Al
journal, June 2020

Autonomy in materials research: a case study in carbon nanotube growth
journal, October 2016

How Water’s Properties Are Encoded in Its Molecular Structure and Energies
journal, September 2017

The war over supercooled water
journal, August 2018

A Bayesian experimental autonomous researcher for mechanical design
journal, April 2020

Radial Basis Functions
book, January 2009

Tuning the hysteresis of a metal-insulator transition via lattice compatibility
journal, July 2020

Active learning of uniformly accurate interatomic potentials for materials simulation
journal, February 2019