skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: A Study of Complex Deep Learning Networks on High-Performance, Neuromorphic, and Quantum Computers

Abstract

Current deep learning approaches have been very successful using convolutional neural networks trained on large graphical-processing-unit-based computers. Three limitations of this approach are that (1) they are based on a simple layered network topology, i.e., highly connected layers, without intra-layer connections; (2) the networks are manually configured to achieve optimal results, and (3) the implementation of the network model is expensive in both cost and power. In this paper, we evaluate deep learning models using three different computing architectures to address these problems: quantum computing to train complex topologies, high performance computing to automatically determine network topology, and neuromorphic computing for a low-power hardware implementation. We use the MNIST dataset for our experiment, due to input size limitations of current quantum computers. Our results show the feasibility of using the three architectures in tandem to address the above deep learning limitations. Finally, we show that a quantum computer can find high quality values of intra-layer connection weights in a tractable time as the complexity of the network increases, a high performance computer can find optimal layer-based topologies, and a neuromorphic computer can represent the complex topology and weights derived from the other architectures in low power memristive hardware.

Authors:
 [1];  [1];  [1];  [1];  [2];  [2];  [2];  [3];  [3]
  1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Computational Data Analytics Group
  2. Univ. of Southern California, Marina del Rey, CA (United States). Information Sciences Inst.
  3. Univ. of Tennessee, Knoxville, TN (United States). Dept. of Electrical Engineering and Computer Science
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1474723
Grant/Contract Number:  
AC05-00OR22725
Resource Type:
Accepted Manuscript
Journal Name:
ACM Journal on Emerging Technologies in Computing Systems
Additional Journal Information:
Journal Volume: 14; Journal Issue: 2; Journal ID: ISSN 1550-4832
Publisher:
Association for Computing Machinery
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; deep learning; quantum computing; neuromorphic computing; high-performance computing

Citation Formats

Potok, Thomas E., Schuman, Catherine, Young, Steven, Patton, Robert, Spedalieri, Federico, Liu, Jeremy, Yao, Ke-Thia, Rose, Garrett, and Chakma, Gangotree. A Study of Complex Deep Learning Networks on High-Performance, Neuromorphic, and Quantum Computers. United States: N. p., 2018. Web. doi:10.1145/3178454.
Potok, Thomas E., Schuman, Catherine, Young, Steven, Patton, Robert, Spedalieri, Federico, Liu, Jeremy, Yao, Ke-Thia, Rose, Garrett, & Chakma, Gangotree. A Study of Complex Deep Learning Networks on High-Performance, Neuromorphic, and Quantum Computers. United States. doi:https://doi.org/10.1145/3178454
Potok, Thomas E., Schuman, Catherine, Young, Steven, Patton, Robert, Spedalieri, Federico, Liu, Jeremy, Yao, Ke-Thia, Rose, Garrett, and Chakma, Gangotree. Fri . "A Study of Complex Deep Learning Networks on High-Performance, Neuromorphic, and Quantum Computers". United States. doi:https://doi.org/10.1145/3178454. https://www.osti.gov/servlets/purl/1474723.
@article{osti_1474723,
title = {A Study of Complex Deep Learning Networks on High-Performance, Neuromorphic, and Quantum Computers},
author = {Potok, Thomas E. and Schuman, Catherine and Young, Steven and Patton, Robert and Spedalieri, Federico and Liu, Jeremy and Yao, Ke-Thia and Rose, Garrett and Chakma, Gangotree},
abstractNote = {Current deep learning approaches have been very successful using convolutional neural networks trained on large graphical-processing-unit-based computers. Three limitations of this approach are that (1) they are based on a simple layered network topology, i.e., highly connected layers, without intra-layer connections; (2) the networks are manually configured to achieve optimal results, and (3) the implementation of the network model is expensive in both cost and power. In this paper, we evaluate deep learning models using three different computing architectures to address these problems: quantum computing to train complex topologies, high performance computing to automatically determine network topology, and neuromorphic computing for a low-power hardware implementation. We use the MNIST dataset for our experiment, due to input size limitations of current quantum computers. Our results show the feasibility of using the three architectures in tandem to address the above deep learning limitations. Finally, we show that a quantum computer can find high quality values of intra-layer connection weights in a tractable time as the complexity of the network increases, a high performance computer can find optimal layer-based topologies, and a neuromorphic computer can represent the complex topology and weights derived from the other architectures in low power memristive hardware.},
doi = {10.1145/3178454},
journal = {ACM Journal on Emerging Technologies in Computing Systems},
number = 2,
volume = 14,
place = {United States},
year = {2018},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 1 work
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

A Learning Algorithm for Boltzmann Machines*
journal, January 1985

  • Ackley, David H.; Hinton, Geoffrey E.; Sejnowski, Terrence J.
  • Cognitive Science, Vol. 9, Issue 1
  • DOI: 10.1207/s15516709cog0901_7

What is the best multi-stage architecture for object recognition?
conference, September 2009

  • Jarrett, Kevin; Kavukcuoglu, Koray; Ranzato, Marc' Aurelio
  • 2009 IEEE 12th International Conference on Computer Vision (ICCV)
  • DOI: 10.1109/ICCV.2009.5459469

High switching endurance in TaOx memristive devices
journal, December 2010

  • Yang, J. Joshua; Zhang, M. -X.; Strachan, John Paul
  • Applied Physics Letters, Vol. 97, Issue 23
  • DOI: 10.1063/1.3524521

Harmonica: A Framework of Heterogeneous Computing Systems With Memristor-Based Neuromorphic Computing Accelerators
journal, May 2016

  • Liu, Xiaoxiao; Mao, Mengjie; Liu, Beiye
  • IEEE Transactions on Circuits and Systems I: Regular Papers, Vol. 63, Issue 5
  • DOI: 10.1109/TCSI.2016.2529279

Neuromorphic architectures for spiking deep neural networks
conference, December 2015

  • Indiveri, Giacomo; Corradi, Federico; Qiao, Ning
  • 2015 IEEE International Electron Devices Meeting (IEDM)
  • DOI: 10.1109/IEDM.2015.7409623

Spatiotemporal Classification Using Neuroscience-Inspired Dynamic Architectures
journal, January 2014


Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores
conference, August 2013

  • Cassidy, Andrew S.; Merolla, Paul; Arthur, John V.
  • 2013 International Joint Conference on Neural Networks (IJCNN 2013 - Dallas), The 2013 International Joint Conference on Neural Networks (IJCNN)
  • DOI: 10.1109/IJCNN.2013.6707077

Quantum annealing with manufactured spins
journal, May 2011

  • Johnson, M. W.; Amin, M. H. S.; Gildert, S.
  • Nature, Vol. 473, Issue 7346
  • DOI: 10.1038/nature10012

Training and operation of an integrated neuromorphic network based on metal-oxide memristors
journal, May 2015

  • Prezioso, M.; Merrikh-Bayat, F.; Hoskins, B. D.
  • Nature, Vol. 521, Issue 7550
  • DOI: 10.1038/nature14441

Extending SpikeProp
conference, January 2004

  • Schrauwen, B.; Van Campenhout, J.
  • 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541)
  • DOI: 10.1109/IJCNN.2004.1379954

Simulating physics with computers
journal, June 1982

  • Feynman, Richard P.
  • International Journal of Theoretical Physics, Vol. 21, Issue 6-7
  • DOI: 10.1007/BF02650179

Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing
conference, July 2015

  • Diehl, Peter U.; Neil, Daniel; Binas, Jonathan
  • 2015 International Joint Conference on Neural Networks (IJCNN)
  • DOI: 10.1109/IJCNN.2015.7280696

ImageNet Large Scale Visual Recognition Challenge
journal, April 2015

  • Russakovsky, Olga; Deng, Jia; Su, Hao
  • International Journal of Computer Vision, Vol. 115, Issue 3
  • DOI: 10.1007/s11263-015-0816-y

Darwin: a neuromorphic hardware co-processor based on Spiking Neural Networks
journal, December 2015


Nanoscale Memristor Device as Synapse in Neuromorphic Systems
journal, April 2010

  • Jo, Sung Hyun; Chang, Ting; Ebong, Idongesit
  • Nano Letters, Vol. 10, Issue 4, p. 1297-1301
  • DOI: 10.1021/nl904092h

Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer
journal, October 1997


Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
conference, December 2015

  • He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing
  • 2015 IEEE International Conference on Computer Vision (ICCV)
  • DOI: 10.1109/ICCV.2015.123

A Fast Learning Algorithm for Deep Belief Nets
journal, July 2006


Error-backpropagation in temporally encoded networks of spiking neurons
journal, October 2002


A Functional Hybrid Memristor Crossbar-Array/CMOS System for Data Storage and Neuromorphic Applications
journal, December 2011

  • Kim, Kuk-Hwan; Gaba, Siddharth; Wheeler, Dana
  • Nano Letters, Vol. 12, Issue 1
  • DOI: 10.1021/nl203687n

Reducing the Dimensionality of Data with Neural Networks
journal, July 2006


DANNA: A neuromorphic software ecosystem
journal, July 2016

  • Disney, Adam; Reynolds, John; Schuman, Catherine D.
  • Biologically Inspired Cognitive Architectures, Vol. 17
  • DOI: 10.1016/j.bica.2016.07.007

Circuit Techniques for Online Learning of Memristive Synapses in CMOS-Memristor Neuromorphic Systems
conference, January 2017

  • Sayyaparaju, Sagarvarma; Chakma, Gangotree; Amer, Sherif
  • Proceedings of the on Great Lakes Symposium on VLSI 2017 - GLSVLSI '17
  • DOI: 10.1145/3060403.3060418

Optimizing deep learning hyper-parameters through an evolutionary algorithm
conference, January 2015

  • Young, Steven R.; Rose, Derek C.; Karnowski, Thomas P.
  • Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments - MLHPC '15
  • DOI: 10.1145/2834892.2834896

FaceNet: A unified embedding for face recognition and clustering
conference, June 2015

  • Schroff, Florian; Kalenichenko, Dmitry; Philbin, James
  • 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • DOI: 10.1109/CVPR.2015.7298682

How We Found The Missing Memristor
journal, December 2008


Estimation of effective temperatures in quantum annealers for sampling applications: A case study with possible applications in deep learning
journal, August 2016


Building block of a programmable neuromorphic substrate: A digital neurosynaptic core
conference, June 2012

  • Arthur, John V.; Merolla, Paul A.; Akopyan, Filipp
  • 2012 International Joint Conference on Neural Networks (IJCNN 2012 - Brisbane), The 2012 International Joint Conference on Neural Networks (IJCNN)
  • DOI: 10.1109/IJCNN.2012.6252637

A Memristor Crossbar Based Computing Engine Optimized for High Speed and Accuracy
conference, July 2016

  • Liu, Chenchen; Yang, Qing; Yan, Bonan
  • 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI)
  • DOI: 10.1109/ISVLSI.2016.46

Long Short-Term Memory
journal, November 1997


Gradient-based learning applied to document recognition
journal, January 1998

  • Lecun, Y.; Bottou, L.; Bengio, Y.
  • Proceedings of the IEEE, Vol. 86, Issue 11
  • DOI: 10.1109/5.726791

Lognormal switching times for titanium dioxide bipolar memristors: origin and resolution
journal, January 2011


Competitive Hebbian learning through spike-timing-dependent synaptic plasticity
journal, September 2000

  • Song, Sen; Miller, Kenneth D.; Abbott, L. F.
  • Nature Neuroscience, Vol. 3, Issue 9
  • DOI: 10.1038/78829

    Works referencing / citing this record:

    Intelligent video surveillance: a review through deep learning techniques for crowd analysis
    journal, June 2019


    Quantum machine learning with D‐wave quantum computer
    journal, June 2019

    • Hu, Feng; Wang, Ban‐Nan; Wang, Ning
    • Quantum Engineering, Vol. 1, Issue 2
    • DOI: 10.1002/que2.12

    Performance analysis and optimization for scalable deployment of deep learning models for country‐scale settlement mapping on Titan supercomputer
    journal, May 2019

    • Kurte, Kuldeep; Sanyal, Jibonananda; Berres, Anne
    • Concurrency and Computation: Practice and Experience, Vol. 31, Issue 20
    • DOI: 10.1002/cpe.5305

    Reliability of analog resistive switching memory for neuromorphic computing
    journal, March 2020

    • Zhao, Meiran; Gao, Bin; Tang, Jianshi
    • Applied Physics Reviews, Vol. 7, Issue 1
    • DOI: 10.1063/1.5124915