Modular Spiking Neural Circuits for Mapping Long Short-Term Memory on a Neurosynaptic Processor

Shrestha, Amar; Ahmed, Khadeer; Wang, Yanzhi; Widemann, David P.; Moody, Adam T.; Van Essen, Brian C.; Qiu, Qinru

doi:10.1109/JETCAS.2018.2856117

Title: Modular Spiking Neural Circuits for Mapping Long Short-Term Memory on a Neurosynaptic Processor

Abstract

Due to the distributed and asynchronous nature of neural computation through low-energy spikes, brain-inspired hardware systems offer high energy efficiency and massive parallelism. One such platform is the IBM TrueNorth neurosynaptic system. Recently, TrueNorth compatible representation learning algorithms have emerged, achieving close to the state-of-the-art performance in various data sets. However, its application in temporal sequence processing models, such as recurrent neural networks (RNNs), is still only at the proof of concept level. There is an inherent difficulty in capturing temporal dynamics of an RNN using spiking neurons, which is only exasperated by the hardware constraints in connectivity and synaptic weight resolution. This paper presents a design flow that overcomes these difficulties and maps a special case of recurrent networks called long short-term memory (LSTM) onto a spike-based platform. The framework utilizes various approximation techniques, such as activation discretization, weight quantization, and scaling and rounding, spiking neural circuits that implement the complex gating mechanisms, and a store-and-release technique to enable neuron synchronization and faithful storage. While the presented techniques can be applied to map LSTM to any spiking neural network (SNN) simulator/emulator, here we choose the TrueNorth chip as the target platform by adhering to its hardware constraints. Three LSTMmore »« less

Authors:

^[1]; Ahmed, Khadeer ^[1]; Wang, Yanzhi ^[1]; Widemann, David P. ^[2]; Moody, Adam T. ^[2];

^[2]; Qiu, Qinru ^[1]

Syracuse Univ., NY (United States)
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Publication Date:: Fri Jul 13 00:00:00 EDT 2018

Research Org.:: Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA); National Science Foundation (NSF)

OSTI Identifier:: 1634300

Report Number(s):: LLNL-JRNL-810559
Journal ID: ISSN 2156-3357; 1017254

Grant/Contract Number:: AC52-07NA27344

Resource Type:: Accepted Manuscript

Journal Name:: IEEE Journal on Emerging and Selected Topics in Circuits and Systems

Additional Journal Information:: Journal Volume: 8; Journal Issue: 4; Journal ID: ISSN 2156-3357

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; mathematics and computing; neural networks; neuromorphics; logic gates; computer architecture; recurrent neural networks; biological neural networks; synchronization; spiking neural networks; long short-term memory; neuromorphic hardware

Citation Formats


                    Shrestha, Amar, Ahmed, Khadeer, Wang, Yanzhi, Widemann, David P., Moody, Adam T., Van Essen, Brian C., and Qiu, Qinru. Modular Spiking Neural Circuits for Mapping Long Short-Term Memory on a Neurosynaptic Processor.  United States: N. p., 2018. 
Web.  doi:10.1109/JETCAS.2018.2856117.

Copy to clipboard


                    Shrestha, Amar, Ahmed, Khadeer, Wang, Yanzhi, Widemann, David P., Moody, Adam T., Van Essen, Brian C., & Qiu, Qinru. Modular Spiking Neural Circuits for Mapping Long Short-Term Memory on a Neurosynaptic Processor.  United States.  https://doi.org/10.1109/JETCAS.2018.2856117

Copy to clipboard


                    Shrestha, Amar, Ahmed, Khadeer, Wang, Yanzhi, Widemann, David P., Moody, Adam T., Van Essen, Brian C., and Qiu, Qinru. Fri .  
"Modular Spiking Neural Circuits for Mapping Long Short-Term Memory on a Neurosynaptic Processor".  United States.  https://doi.org/10.1109/JETCAS.2018.2856117.  https://www.osti.gov/servlets/purl/1634300.

Copy to clipboard


                    
@article{osti_1634300,

  title        = {Modular Spiking Neural Circuits for Mapping Long Short-Term Memory on a Neurosynaptic Processor},

  author       = {Shrestha, Amar and Ahmed, Khadeer and Wang, Yanzhi and Widemann, David P. and Moody, Adam T. and Van Essen, Brian C. and Qiu, Qinru},

  abstractNote = {Due to the distributed and asynchronous nature of neural computation through low-energy spikes, brain-inspired hardware systems offer high energy efficiency and massive parallelism. One such platform is the IBM TrueNorth neurosynaptic system. Recently, TrueNorth compatible representation learning algorithms have emerged, achieving close to the state-of-the-art performance in various data sets. However, its application in temporal sequence processing models, such as recurrent neural networks (RNNs), is still only at the proof of concept level. There is an inherent difficulty in capturing temporal dynamics of an RNN using spiking neurons, which is only exasperated by the hardware constraints in connectivity and synaptic weight resolution. This paper presents a design flow that overcomes these difficulties and maps a special case of recurrent networks called long short-term memory (LSTM) onto a spike-based platform. The framework utilizes various approximation techniques, such as activation discretization, weight quantization, and scaling and rounding, spiking neural circuits that implement the complex gating mechanisms, and a store-and-release technique to enable neuron synchronization and faithful storage. While the presented techniques can be applied to map LSTM to any spiking neural network (SNN) simulator/emulator, here we choose the TrueNorth chip as the target platform by adhering to its hardware constraints. Three LSTM applications, parity check, extended Reber grammar, and question classification, are evaluated. The tradeoffs among accuracy, performance, and energy tradeoffs achieved on TrueNorth are demonstrated. Finally, this is compared with the performance on an SNN platform without hardware constraints, which represents the upper bound of the achievable accuracy.},

  doi          = {10.1109/JETCAS.2018.2856117},

  journal      = {IEEE Journal on Emerging and Selected Topics in Circuits and Systems},

  number       = 4,

  volume       = 8,

  place        = {United States},

  year         = {Fri Jul 13 00:00:00 EDT 2018},

  month        = {Fri Jul 13 00:00:00 EDT 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/JETCAS.2018.2856117

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 5 works

Citation information provided by
Web of Science

Figures / Tables:

Figure 1: (a)Unrolled RNN (b)Standard LSTM (c) LSTM equations

All figures and tables (18 total)

Save / Share:

Export Metadata

Save to My Library

Figures / Tables found in this record:

Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.

Similar Records in DOE PAGES and OSTI.GOV collections:

DFSynthesizer: Dataflow-based Synthesis of Spiking Neural Networks to Neuromorphic Hardware

Journal Article Song, Shihao ; Chong, Harry ; Balaji, Adarsha ; ... - ACM Transactions on Embedded Computing Systems

Spiking Neural Networks (SNNs) are an emerging computation model that uses event-driven activation and bio-inspired learning algorithms. SNN-based machine learning programs are typically executed on tile-based neuromorphic hardware platforms, where each tile consists of a computation unit called a crossbar, which maps neurons and synapses of the program. However, synthesizing such programs on an off-the-shelf neuromorphic hardware is challenging. This is because of the inherent resource and latency limitations of the hardware, which impact both model performance, e.g., accuracy, and hardware performance, e.g., throughput. We propose DFSynthesizer, an end-to-end framework for synthesizing SNN-based machine learning programs to neuromorphic hardware. Themore »« less
https://doi.org/10.1145/3479156
Skip-Connected Self-Recurrent Spiking Neural Networks with Joint Intrinsic Parameter and Synaptic Weight Training

Journal Article Zhang, Wenrui ; Li, Peng - Neural Computation

As an important class of spiking neural networks (SNNs), recurrent spiking neural networks (RSNNs) possess great computational power and have been widely used for processing sequential data like audio and text. However, most RSNNs suffer from two problems. 1. Due to the lack of architectural guidance, random recurrent connectivity is often adopted, which does not guarantee good performance. 2. Training of RSNNs is in general challenging, bottlenecking achievable model accuracy. To address these problems, we propose a new type of RSNNs called Skip-Connected Self-Recurrent SNNs (ScSr-SNNs). Recurrence in ScSr-SNNs is introduced in a stereotyped manner by adding self-recurrent connections tomore »« less
https://doi.org/10.1162/neco_a_01393

Full Text Available
Training Spiking Neural Networks with Synaptic Plasticity under Integer Representation

Conference Kulkarni, Shruti ; Parsa, Maryam ; Mitchell, Parker ; ...

Neuromorphic computing is emerging as a promising Beyond Moore computing paradigm that employs event-triggered computation and non-von Neumann hardware. Spike Timing Dependent Plasticity (STDP) is a well-known bio-inspired learning rule that relies on activities of locally connected neurons to adjust the weights of their respective synapses. In this work, we analyze a basic STDP rule and its sensitivity on the different hyperparameters for training spiking neural networks (SNNs) with supervision, customized for a neuromorphic hardware implementation with integer weights. We compare the classification performance on four UCI datasets (iris, wine, breast cancer and digits) that depict varying levels of complexity.more »« less
https://doi.org/10.1145/3477145.3477152

Full Text Available
Energy Efficient Stochastic-Based Deep Spiking Neural Networks for Sparse Datasets

Conference Alawad, Mohammed ; Yoon, Hong-Jun ; Tourassi, Georgia

With large deep neural networks (DNNs) neces- sary to solve complex and data-intensive problems, energy efficiency is a key bottleneck for effectively deploying DL in the real world. Deep spiking NNs have gained much research at- tention recently due to the interest in building biological neural networks and the availability of neuromorphic platforms, which can be orders of magnitude more energy efficient compared to CPUs and GPUs. Although spiking NNs have proven to be an efficient technique for solving many machine learning and computer vision problems, to the best of our knowledge, this is the first attempt to adapt spikingmore »« less
https://doi.org/10.1109/BigData.2017.8257939

Full Text Available
Automated Generation of Integrated Digital and Spiking Neuromorphic Machine Learning Accelerators

Conference Curzel, Serena ; Bohm Agostini, Nicolas ; Song, Shihao ; ...

The growing numbers of application areas for artificial intelligence (AI) methods have led to an explosion of domain-specific accelerators that could support every new machine learning (ML) algorithm advancement, clearly highlighting the need for a capability to quickly and automatically transition from algorithm definition to hardware implementation and explore design space along a variety of SWaP (size, weight and Power). The software defined architectures (SODA) synthesizer implements a compiler-based modular infrastructure for the end-to-end generation of machine learning accelerators from high-level frameworks to hardware description language. At the same time, neuromorphic computing, by mimicking how the brain operates, promises tomore »« less
https://doi.org/10.1109/ICCAD51958.2021.9643474

Similar Records