Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Optimizing training trajectories in variational autoencoders via latent Bayesian optimization approach*

Journal Article · · Machine Learning: Science and Technology
 [1];  [1];  [1];  [2]
  1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Center for Nanophase Materials Sciences (CNMS)
  2. Univ. of Tennessee, Knoxville, TN (United States)
+ Show Author Affiliations
Unsupervised and semi-supervised ML methods such as variational autoencoders (VAE) have become widely adopted across multiple areas of physics, chemistry, and materials sciences due to their capability in disentangling representations and ability to find latent manifolds for classification and/or regression of complex experimental data. Like other ML problems, VAEs require hyperparameter tuning, e.g. balancing the Kullback–Leibler and reconstruction terms. However, the training process and resulting manifold topology and connectivity depend not only on hyperparameters, but also their evolution during training. Because of the inefficiency of exhaustive search in a high-dimensional hyperparameter space for the expensive-to-train models, here we have explored a latent Bayesian optimization (zBO) approach for the hyperparameter trajectory optimization for the unsupervised and semi-supervised ML and demonstrated for joint-VAE with rotational invariances. We have demonstrated an application of this method for finding joint discrete and continuous rotationally invariant representations for modified national institute of standards and technology database (MNIST) and experimental data of a plasmonic nanoparticles material system. The performance of the proposed approach has been discussed extensively, where it allows for any high dimensional hyperparameter trajectory optimization of other ML models.
Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Energy Frontier Research Centers (EFRC) (United States). Center for the Science of Synthesis Across Scales (CSSAS)
Sponsoring Organization:
USDOE Office of Science (SC), Basic Energy Sciences (BES)
Grant/Contract Number:
AC05-00OR22725; SC0019288
OSTI ID:
1923196
Alternate ID(s):
OSTI ID: 1960555
Journal Information:
Machine Learning: Science and Technology, Journal Name: Machine Learning: Science and Technology Journal Issue: 1 Vol. 4; ISSN 2632-2153
Publisher:
IOP PublishingCopyright Statement
Country of Publication:
United States
Language:
English

References (42)

Disentangling Ferroelectric Wall Dynamics and Identification of Pinning Mechanisms via Deep Learning journal September 2021
Good practices for Bayesian optimization of high dimensional structured spaces journal May 2021
Machine Learning Method Reveals Hidden Strong Metal‐Support Interaction in Microscopy Datasets journal February 2021
Remarks on the Method of Paired Comparisons: I. The Least Squares Solution Assuming Equal Standard Deviations and Equal Correlations book January 2007
Using Gaussian Processes to Optimize Expensive Functions book January 2008
Sequential Model-Based Optimization for General Algorithm Configuration book January 2011
The Role of the Nugget Term in the Gaussian Process Method book January 2010
A Bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot journal August 2009
High-dimensional Bayesian optimization with projections using quantile Gaussian processes journal May 2019
Machine Learning: Algorithms, Real-World Applications and Research Directions journal March 2021
The effect of the nugget on Gaussian process emulators of computer models journal December 2012
Comparison of Gaussian process modeling software journal April 2018
Embedding high-dimensional Bayesian optimization via generative modeling: Parameter personalization of cardiac electrophysiological models journal May 2020
Deep learning analysis on microscopic imaging in materials science journal August 2020
On hyperparameter optimization of machine learning algorithms: Theory and practice journal November 2020
Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules journal January 2018
Efficient Global Optimization of Expensive Black-Box Functions journal January 1998
A Taxonomy of Global Optimization Methods Based on Response Surfaces journal December 2001
A law of comparative judgment. journal January 1927
Optimization of physical quantities in the autoencoder latent space journal May 2022
Machine learning in scanning transmission electron microscopy journal March 2022
Bayesian optimization in continuous spaces via virtual process embeddings journal January 2022
Principal component and spatial correlation analysis of spectroscopic-imaging data in scanning probe microscopy journal February 2009
Disentangling ferroelectric domain wall geometries and pathways in dynamic piezoresponse force microscopy via unsupervised machine learning journal November 2021
Towards automating structural discovery in scanning transmission electron microscopy * journal February 2022
Bayesian Optimization for Adaptive Experimental Design: A Review journal January 2020
Balancing Reconstruction Error and Kullback-Leibler Divergence in Variational Autoencoders journal January 2020
A statistical method for global optimization conference January 1992
Taking the Human Out of the Loop: A Review of Bayesian Optimization journal January 2016
The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] journal October 2012
A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise journal March 1964
Globally Approximate Gaussian Processes for Big Data With Application to Data-Driven Metamaterials Design journal September 2019
Shared-Gaussian Process: Learning Interpretable Shared Hidden Structure Across Data Spaces for Design Space Analysis and Exploration journal March 2020
An Approach to Bayesian Optimization for Design Feasibility Check on Discontinuous Black-Box Functions journal February 2021
Machine learning–enabled identification of material phase transitions based on experimental data: Exploring collective dynamics in ferroelectric relaxors journal March 2018
Exploring order parameters and dynamic processes in disordered systems via variational autoencoders journal April 2021
Optimizing deep learning hyper-parameters through an evolutionary algorithm
  • Young, Steven R.; Rose, Derek C.; Karnowski, Thomas P.
  • Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments - MLHPC '15 https://doi.org/10.1145/2834892.2834896
conference January 2015
Bayesian auxiliary variable models for binary and multinomial regression journal March 2006
Model-based active learning in hierarchical policies text January 2008
An Introduction to Variational Autoencoders journal January 2019
Bayesian Optimization in a Billion Dimensions via Random Embeddings journal January 2016
On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation conference January 2019

Similar Records

Physics and chemistry from parsimonious representations: image analysis via invariant variational autoencoders
Journal Article · Wed Aug 14 00:00:00 EDT 2024 · npj Computational Materials · OSTI ID:2439897

Related Subjects

42 ENGINEERING
Bayesian optimization
high-dimensional problem
latent space
unsupervised learning
variation auto-encoder