Neurogenesis Deep Learning: Extending deep networks to accommodate new classes

Technical Report · Thu Dec 01 04:00:00 EST 2016

DOI:https://doi.org/10.2172/1505351· OSTI ID:1505351

Draelos, Timothy John ^[1]; Miner, Nadine E. ^[1]; Lamb, Christopher C. ^[1]; Vineyard, Craig Michael ^[2]; Carlson, Kristofor David ^[1]; James, Conrad D. ^[2]; Aimone, James Bradley ^[3]

---

1. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
2. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States). Hardware Acceleration of Adaptive Neural Algorithms (HAANA) Project, Neuromorphic Computing
3. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States). Hardware Acceleration of Adaptive Neural Algorithms (HAANA) Project, Computational Neuroscience

Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing – data processing domains in which humans have long held clear advantages over conventional algorithms. In contrast to biological neural systems, which are capable of learning continuously, deep artificial networks have a limited ability for incorporating new information in an already trained network. As a result, methods for continuous learning are potentially highly impactful in enabling the application of deep networks to dynamic data sets. Here, inspired by the process of adult neurogenesis in the hippocampus, we explore the potential for adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations. Our results on the MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes lower and upper case letters and digits, demonstrate that neurogenesis is well suited for addressing the stability-plasticity dilemma that has long challenged adaptive machine learning algorithms.

• Research Organization: Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)
• Sponsoring Organization: USDOE National Nuclear Security Administration (NNSA)
• DOE Contract Number: AC04-94AL85000
• OSTI ID: 1505351
• Report Number(s): SAND--2016-12514R; 649839
• Country of Publication: United States
• Language: English

Related Subjects

59 BASIC BIOLOGICAL SCIENCES
autoencoder
class conditional sampling
deep learning
deep neural networks
hippocampus
replay