Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Rethinking skip connections in Spiking Neural Networks with Time-To-First-Spike coding

Journal Article · · Frontiers in Neuroscience (Online)
 [1];  [2];  [1];  [1];  [3];  [3];  [1]
  1. Yale Univ., New Haven, CT (United States)
  2. Brown Univ., Providence, RI (United States)
  3. Brown Univ., Providence, RI (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
+ Show Author Affiliations

Time-To-First-Spike (TTFS) coding in Spiking Neural Networks (SNNs) offers significant advantages in terms of energy efficiency, closely mimicking the behavior of biological neurons. In this work, we delve into the role of skip connections, a widely used concept in Artificial Neural Networks (ANNs), within the domain of SNNs with TTFS coding. Our focus is on two distinct types of skip connection architectures: (1) addition-based skip connections, and (2) concatenation-based skip connections. We find that addition-based skip connections introduce an additional delay in terms of spike timing. On the other hand, concatenation-based skip connections circumvent this delay but produce time gaps between after-convolution and skip connection paths, thereby restricting the effective mixing of information from these two paths. To mitigate these issues, we propose a novel approach involving a learnable delay for skip connections in the concatenation-based skip connection architecture. This approach successfully bridges the time gap between the convolutional and skip branches, facilitating improved information mixing. We conduct experiments on public datasets including MNIST and Fashion-MNIST, illustrating the advantage of the skip connection in TTFS coding architectures. Additionally, we demonstrate the applicability of TTFS coding on beyond image recognition tasks and extend it to scientific machine-learning tasks, broadening the potential uses of SNNs.

Research Organization:
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
Grant/Contract Number:
AC05-76RL01830
OSTI ID:
2407016
Report Number(s):
PNNL-SA--193461
Journal Information:
Frontiers in Neuroscience (Online), Journal Name: Frontiers in Neuroscience (Online) Journal Issue: _ Vol. 18; ISSN 1662-453X
Publisher:
Frontiers Research FoundationCopyright Statement
Country of Publication:
United States
Language:
English

References (38)

BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learning journal November 2021
A novel and efficient classifier using spiking neural network journal May 2019
Spiking Deep Convolutional Neural Networks for Energy-Efficient Object Recognition journal November 2014
Time reversal with partial information for wave refocusing and scatterer identification journal March 2012
Phase-of-Firing Coding of Natural Visual Stimuli in Primary Visual Cortex journal March 2008
A physically-informed deep-learning model using time-reversal for locating a source from sparse and highly noisy sensors data journal December 2022
A supervised multi-spike learning algorithm based on gradient descent for spiking neural networks journal July 2013
A biologically plausible supervised learning method for spiking neural networks using the symmetric STDP rule journal January 2020
Towards spike-based machine intelligence with neuromorphic computing journal November 2019
2022 roadmap on neuromorphic computing and engineering journal May 2022
Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function conference May 2020
Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing conference July 2015
Conversion of analog to spiking neural networks using sparse temporal coding conference January 2018
1.1 Computing's energy problem (and what we can do about it)
  • Horowitz, Mark
  • 2014 IEEE International Solid- State Circuits Conference (ISSCC), 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) https://doi.org/10.1109/ISSCC.2014.6757323
conference February 2014
Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks journal November 2019
SIBoLS: Robust and Energy-efficient Learning for Spike-based Machine Intelligence in Information Bottleneck Framework journal January 2024
Highway Connection for Low-Latency and High-Accuracy Spiking Neural Networks journal December 2023
Robustness to Training Disturbances in SpikeProp Learning journal July 2018
Supervised Learning Based on Temporal Coding in Spiking Neural Networks journal January 2017
A Supervised Learning Algorithm for Multilayer Spiking Neural Networks Based on Temporal Coding Toward Energy-Efficient VLSI Processor Design journal January 2023
Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks journal May 2022
Spiking Deep Residual Networks journal January 2021
Effective Surrogate Gradient Learning With High-Order Information Bottleneck for Spike-Based Machine Intelligence journal January 2024
SNIB: Improving Spike-Based Machine Learning Using Nonlinear Information Bottleneck journal December 2023
A survey of sound source localization with deep learning methods journal July 2022
Temporal Backpropagation for Spiking Neural Networks with One Spike per Neuron journal May 2020
Direct Training for Spiking Neural Networks: Faster, Larger, Better journal July 2019
TDSNN: From Deep Neural Networks to Deep Spike Neural Networks with Temporal-Coding journal July 2019
Training Spiking Neural Networks with Accumulated Spiking Flow journal May 2021
Unsupervised learning of digit recognition using spike-timing-dependent plasticity journal August 2015
Training Deep Spiking Neural Networks Using Backpropagation journal November 2016
Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification journal December 2017
Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks journal May 2018
Going Deeper in Spiking Neural Networks: VGG and Residual Architectures journal March 2019
Enabling Spike-Based Backpropagation for Training Deep Neural Network Architectures journal February 2020
Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization journal June 2020
Neural Coding in Spiking Neural Networks: A Comparative Study for Robust Neuromorphic Systems journal March 2021
Rethinking the Role of Normalization and Residual Blocks for Spiking Neural Networks journal April 2022

Similar Records

Related Subjects

97 MATHEMATICS AND COMPUTING
Spiking Neural Networks
artificial intelligence
energy-efficient deep learning
event-based processing
image recognition
machine learning
signal processing
temporal coding