Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions
- Brown University, Providence, RI (United States)
- Harvard University, Cambridge, MA (United States)
Here we propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while keeping the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks. This is also empirically verified through a set of computational examples. Furthermore, under certain technical assumptions, we establish global convergence of gradient descent for KNNs. As a specific case, we propose the Rowdy activation function that is designed to get rid of any saturation region by injecting sinusoidal fluctuations, which include trainable parameters. The proposed Rowdy activation function can be employed in any neural network architecture like feed-forward neural networks, Recurrent neural networks, Convolutional neural networks etc. The effectiveness of KNNs with Rowdy activation is demonstrated through various computational experiments including function approximation using feed-forward neural networks, solution inference of partial differential equations using the physics-informed neural networks, and standard deep learning benchmark problems using convolutional and fully-connected neural networks.
- Research Organization:
- Brown Univ., Providence, RI (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC); US Air Force Office of Scientific Research (AFOSR)
- Grant/Contract Number:
- SC0019453; FA9550-20–1-0358
- OSTI ID:
- 1977480
- Alternate ID(s):
- OSTI ID: 1827465
- Journal Information:
- Neurocomputing, Vol. 468, Issue C; ISSN 0925-2312
- Publisher:
- ElsevierCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Similar Records
Scalable algorithms for physics-informed neural and graph networks
Multi-fidelity Bayesian neural networks: Algorithms and applications