Autoencoder node saliency: Selecting relevant latent representations

Fan, Ya Ju

doi:10.1016/j.patcog.2018.12.015

Title: Autoencoder node saliency: Selecting relevant latent representations

Abstract

The autoencoder is an artificial neural network that performs nonlinear dimension reduction and learns hidden representations of unlabeled data. With a linear transfer function it is similar to the principal component analysis (PCA). While both methods use weight vectors for linear transformations, the autoencoder does not come with any indication similar to the eigenvalues in PCA that are paired with eigenvectors. In this work, we propose a novel autoencoder node saliency method that examines whether the features constructed by autoencoders exhibit properties related to known class labels. The supervised node saliency ranks the nodes based on their capability of performing a learning task. It is coupled with the normalized entropy difference (NED). We establish a property for NED values to verify classifying behaviors among the top ranked nodes. Lastly, by applying our methods to real datasets, we demonstrate their ability to provide indications on the performing nodes and explain the learned tasks in autoencoders.

Authors:

^[1]

Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Publication Date:: Mon Dec 17 00:00:00 EST 2018

Research Org.:: Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1491664

Alternate Identifier(s):: OSTI ID: 1636754

Report Number(s):: LLNL-JRNL-741590
Journal ID: ISSN 0031-3203; 896098

Grant/Contract Number:: AC52-07NA27344

Resource Type:: Accepted Manuscript

Journal Name:: Pattern Recognition

Additional Journal Information:: Journal Volume: 88; Journal Issue: C; Journal ID: ISSN 0031-3203

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Autoencoder; Latent representations; Unsupervised learning; Neural networks; Node selection; Model interpretation

Citation Formats


                    Fan, Ya Ju. Autoencoder node saliency: Selecting relevant latent representations.  United States: N. p., 2018. 
Web.  doi:10.1016/j.patcog.2018.12.015.

Copy to clipboard


                    Fan, Ya Ju. Autoencoder node saliency: Selecting relevant latent representations.  United States.  https://doi.org/10.1016/j.patcog.2018.12.015

Copy to clipboard


                    Fan, Ya Ju. Mon .  
"Autoencoder node saliency: Selecting relevant latent representations".  United States.  https://doi.org/10.1016/j.patcog.2018.12.015.  https://www.osti.gov/servlets/purl/1491664.

Copy to clipboard


                    
@article{osti_1491664,

  title        = {Autoencoder node saliency: Selecting relevant latent representations},

  author       = {Fan, Ya Ju},

  abstractNote = {The autoencoder is an artificial neural network that performs nonlinear dimension reduction and learns hidden representations of unlabeled data. With a linear transfer function it is similar to the principal component analysis (PCA). While both methods use weight vectors for linear transformations, the autoencoder does not come with any indication similar to the eigenvalues in PCA that are paired with eigenvectors. In this work, we propose a novel autoencoder node saliency method that examines whether the features constructed by autoencoders exhibit properties related to known class labels. The supervised node saliency ranks the nodes based on their capability of performing a learning task. It is coupled with the normalized entropy difference (NED). We establish a property for NED values to verify classifying behaviors among the top ranked nodes. Lastly, by applying our methods to real datasets, we demonstrate their ability to provide indications on the performing nodes and explain the learned tasks in autoencoders.},

  doi          = {10.1016/j.patcog.2018.12.015},

  journal      = {Pattern Recognition},

  number       = C,

  volume       = 88,

  place        = {United States},

  year         = {Mon Dec 17 00:00:00 EST 2018},

  month        = {Mon Dec 17 00:00:00 EST 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.patcog.2018.12.015

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 18 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Unsupervised feature extraction with autoencoder trees
journal, October 2017

İrsoy, Ozan; Alpaydın, Ethem
Neurocomputing, Vol. 258
DOI: 10.1016/j.neucom.2017.02.075

Reducing the Dimensionality of Data with Neural Networks
journal, July 2006

Hinton, G. E.
Science, Vol. 313, Issue 5786
DOI: 10.1126/science.1127647

A Connection Between Score Matching and Denoising Autoencoders
journal, July 2011

Vincent, Pascal
Neural Computation, Vol. 23, Issue 7
DOI: 10.1162/NECO_a_00142

Multi-Site Diagnostic Classification of Schizophrenia Using Discriminant Deep Learning with Functional Connectivity MRI
journal, April 2018

Zeng, Ling-Li; Wang, Huaning; Hu, Panpan
EBioMedicine, Vol. 30
DOI: 10.1016/j.ebiom.2018.03.017

Neural networks and principal component analysis: Learning from examples without local minima
journal, January 1989

Baldi, Pierre; Hornik, Kurt
Neural Networks, Vol. 2, Issue 1
DOI: 10.1016/0893-6080(89)90014-2

Consensus self-organized models for fault detection (COSMO)
journal, August 2011

Byttner, S.; Rögnvaldsson, T.; Svensson, M.
Engineering Applications of Artificial Intelligence, Vol. 24, Issue 5
DOI: 10.1016/j.engappai.2011.03.002

Self-monitoring for maintenance of vehicle fleets
journal, August 2017

Rögnvaldsson, Thorsteinn; Nowaczyk, Sławomir; Byttner, Stefan
Data Mining and Knowledge Discovery, Vol. 32, Issue 2
DOI: 10.1007/s10618-017-0538-6

A survey on feature selection methods
journal, January 2014

Chandrashekar, Girish; Sahin, Ferat
Computers & Electrical Engineering, Vol. 40, Issue 1
DOI: 10.1016/j.compeleceng.2013.11.024

A Survey on Feature Selection
journal, January 2016

Miao, Jianyu; Niu, Lingfeng
Procedia Computer Science, Vol. 91
DOI: 10.1016/j.procs.2016.07.111

An efficient semi-supervised representatives feature selection algorithm based on information theory
journal, January 2017

Wang, Yintong; Wang, Jiandong; Liao, Hao
Pattern Recognition, Vol. 61
DOI: 10.1016/j.patcog.2016.08.011

Connectionist learning procedures
journal, September 1989

Hinton, Geoffrey E.
Artificial Intelligence, Vol. 40, Issue 1-3
DOI: 10.1016/0004-3702(89)90049-0

Gradient-based learning applied to document recognition
journal, January 1998

Lecun, Y.; Bottou, L.; Bengio, Y.
Proceedings of the IEEE, Vol. 86, Issue 11
DOI: 10.1109/5.726791

A trainable feature extractor for handwritten digit recognition
journal, June 2007

Lauer, Fabien; Suen, Ching Y.; Bloch, Gérard
Pattern Recognition, Vol. 40, Issue 6
DOI: 10.1016/j.patcog.2006.10.011

A novel hybrid CNN–SVM classifier for recognizing handwritten digits
journal, April 2012

Niu, Xiao-Xiao; Suen, Ching Y.
Pattern Recognition, Vol. 45, Issue 4
DOI: 10.1016/j.patcog.2011.09.021

The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups
journal, April 2012

Curtis, Christina; Shah, Sohrab P.; Chin, Suet-Feung
Nature, Vol. 486, Issue 7403
DOI: 10.1038/nature10983

Extremely randomized trees
journal, March 2006

Geurts, Pierre; Ernst, Damien; Wehenkel, Louis
Machine Learning, Vol. 63, Issue 1
DOI: 10.1007/s10994-006-6226-1

Identification of bursts in spike trains
journal, January 1992

Cocatre-Zilgien, J. H.; Delcomyn, F.
Journal of Neuroscience Methods, Vol. 41, Issue 1
DOI: 10.1016/0165-0270(92)90120-3

Works referencing / citing this record:

Assessment of Autoencoder Architectures for Data Representation
book, October 2019

Pawar, Karishma; Attar, Vahida Z.; Pedrycz, Witold
Deep Learning: Concepts and Architectures, p. 101-132
DOI: 10.1007/978-3-030-31756-0_4

A Robust Deep Learning Approach for Spatiotemporal Estimation of Satellite AOD and PM2.5
journal, January 2020

Li, Lianfa
Remote Sensing, Vol. 12, Issue 2
DOI: 10.3390/rs12020264

Discriminative stacked autoencoder for feature representation and classification
journal, January 2020

Gao, Yiping; Li, Xinyu; Gao, Liang
Science China Information Sciences, Vol. 63, Issue 2
DOI: 10.1007/s11432-019-2722-3

Similar Records in DOE PAGES and OSTI.GOV collections:

Latent Representation Learning for Structural Characterization of Catalysts

Journal Article Routh, Prahlad K. ; Liu, Yang ; Marcella, Nicholas ; ... - Journal of Physical Chemistry Letters

Supervised machine learning-enabled mapping of the X-ray absorption near edge structure (XANES) spectra to local structural descriptors offers new methods for understanding the structure and function of working nanocatalysts. We briefly summarize a status of XANES analysis approaches by supervised machine learning methods. We present an example of an autoencoder-based, unsupervised machine learning approach for latent representation learning of XANES spectra. This new approach produces a lower-dimensional latent representation, which retains a spectrum–structure relationship that can be eventually mapped to physicochemical properties. Furthermore, the latent space of the autoencoder also provides a pathway to interpret the information content “hidden” inmore »« less
https://doi.org/10.1021/acs.jpclett.0c03792

Full Text Available
Saliency-driven system models for cell analysis with deep learning

Journal Article Ferreira, Daniel S. ; Ramalho, Geraldo L. B. ; Torres, Débora ; ... - Computer Methods and Programs in BioMedicine

Saliency refers to the visual perception quality that makes objects in a scene to stand out from others and attract attention. While computational saliency models can simulate the expert's visual attention, there is little evidence about how these models perform when used to predict the cytopathologist's eye fixations. Saliency models may be the key to instrumenting fast object detection on large Pap smear slides under real noisy conditions, artifacts, and cell occlusions. This paper describes how our computational schemes retrieve regions of interest (ROI) of clinical relevance using visual attention models. We also compare the performance of different computed saliencymore »« less
Cited by 1
https://doi.org/10.1016/j.cmpb.2019.105053
Deep structural clustering for single-cell RNA-seq data jointly through autoencoder and graph neural network

Journal Article Gan, Yanglan ; Huang, Xingyu ; Zou, Guobing ; ... - Briefings in Bioinformatics

Abstract Single-cell RNA sequencing (scRNA-seq) permits researchers to study the complex mechanisms of cell heterogeneity and diversity. Unsupervised clustering is of central importance for the analysis of the scRNA-seq data, as it can be used to identify putative cell types. However, due to noise impacts, high dimensionality and pervasive dropout events, clustering analysis of scRNA-seq data remains a computational challenge. Here, we propose a new deep structural clustering method for scRNA-seq data, named scDSC, which integrate the structural information into deep clustering of single cells. The proposed scDSC consists of a Zero-Inflated Negative Binomial (ZINB) model-based autoencoder, a graph neuralmore »« less
https://doi.org/10.1093/bib/bbac018
Reification of latent microstructures: On supervised unsupervised and semi-supervised deep learning applications for microstructures in materials informatics

Technical Report Tran, Anh ; Rodgers, Theron ; Wildey, Timothy Michael

Machine learning (ML), including deep learning (DL), has become increasingly popular in the last few years due to its continually outstanding performance. In this context, we apply machine learning techniques to "learn" the microstructure using both supervised and unsupervised DL techniques. In particular, we focus (1) on the localization problem bridging (micro)structure (localized) property using supervised DL and (2) on the microstructure reconstruction problem in latent space using unsupervised DL. The goal of supervised and semi-supervised DL is to replace crystal plasticity finite element model (CPFEM) that maps from (micro)structure (localized) property, and implicitly the (micro)structure (homogenized) property relationships, whilemore »« less
https://doi.org/10.2172/1673174

Full Text Available
Trade-offs in the latent representation of microstructure evolution

Journal Article Desai, Saaketh ; Shrivastava, Ankit ; D’Elia, Marta ; ... - Acta Materialia

Characterizing and quantifying microstructure evolution is critical to forming quantitative relationships between material processing conditions, resulting microstructure, and observed properties. Machine-learning methods are increasingly accelerating the development of these relationships by treating microstructure evolution as a pattern recognition problem, discovering relationships explicitly or implicitly. These methods often rely on identifying low-dimensional microstructural fingerprints as latent variables. However, using inappropriate latent variables can lead to challenges in learning meaningful relationships. In this work, we survey and discuss the ability of various linear and nonlinear dimensionality reduction methods including principal component analysis, autoencoders, and diffusion maps to quantify and characterize the learnedmore »« less
https://doi.org/10.1016/j.actamat.2023.119514

Similar Records

Title: Autoencoder node saliency: Selecting relevant latent representations

Abstract

Citation Formats

Unsupervised feature extraction with autoencoder trees journal, October 2017

Reducing the Dimensionality of Data with Neural Networks journal, July 2006

A Connection Between Score Matching and Denoising Autoencoders journal, July 2011

Multi-Site Diagnostic Classification of Schizophrenia Using Discriminant Deep Learning with Functional Connectivity MRI journal, April 2018

Neural networks and principal component analysis: Learning from examples without local minima journal, January 1989

Consensus self-organized models for fault detection (COSMO) journal, August 2011

Self-monitoring for maintenance of vehicle fleets journal, August 2017

A survey on feature selection methods journal, January 2014

A Survey on Feature Selection journal, January 2016

An efficient semi-supervised representatives feature selection algorithm based on information theory journal, January 2017

Connectionist learning procedures journal, September 1989

Gradient-based learning applied to document recognition journal, January 1998

A trainable feature extractor for handwritten digit recognition journal, June 2007

A novel hybrid CNN–SVM classifier for recognizing handwritten digits journal, April 2012

The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups journal, April 2012

Extremely randomized trees journal, March 2006

Identification of bursts in spike trains journal, January 1992

Assessment of Autoencoder Architectures for Data Representation book, October 2019

A Robust Deep Learning Approach for Spatiotemporal Estimation of Satellite AOD and PM2.5 journal, January 2020

Discriminative stacked autoencoder for feature representation and classification journal, January 2020

Unsupervised feature extraction with autoencoder trees
journal, October 2017

Reducing the Dimensionality of Data with Neural Networks
journal, July 2006

A Connection Between Score Matching and Denoising Autoencoders
journal, July 2011

Multi-Site Diagnostic Classification of Schizophrenia Using Discriminant Deep Learning with Functional Connectivity MRI
journal, April 2018

Neural networks and principal component analysis: Learning from examples without local minima
journal, January 1989

Consensus self-organized models for fault detection (COSMO)
journal, August 2011

Self-monitoring for maintenance of vehicle fleets
journal, August 2017

A survey on feature selection methods
journal, January 2014

A Survey on Feature Selection
journal, January 2016

An efficient semi-supervised representatives feature selection algorithm based on information theory
journal, January 2017

Connectionist learning procedures
journal, September 1989

Gradient-based learning applied to document recognition
journal, January 1998

A trainable feature extractor for handwritten digit recognition
journal, June 2007

A novel hybrid CNN–SVM classifier for recognizing handwritten digits
journal, April 2012

The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups
journal, April 2012

Extremely randomized trees
journal, March 2006

Identification of bursts in spike trains
journal, January 1992

Assessment of Autoencoder Architectures for Data Representation
book, October 2019

A Robust Deep Learning Approach for Spatiotemporal Estimation of Satellite AOD and PM2.5
journal, January 2020

Discriminative stacked autoencoder for feature representation and classification
journal, January 2020