SineKAN: Kolmogorov-Arnold Networks using sinusoidal activation functions

Journal Article · 15 January 2025 · Frontiers in Artificial Intelligence

DOI: https://doi.org/10.3389/frai.2024.1462952 · OSTI ID:2500918

Reinhardt, Eric; Ramakrishnan, Dinesh; Gleyzer, Sergei

Recent work has established an alternative to traditional multi-layer perceptron neural networks in the form of Kolmogorov-Arnold Networks (KAN). The general KAN framework uses learnable activation functions on the edges of the computational graph followed by summation on nodes. The learnable edge activation functions in the original implementation are basis spline functions (B-Spline). Here, we present a model in which learnable grids of B-Spline activation functions are replaced by grids of re-weighted sine functions (SineKAN). We evaluate numerical performance of our model on a benchmark vision task. We show that our model can perform better than or comparable to B-Spline KAN models and an alternative KAN implementation based on periodic cosine and sine functions representing a Fourier Series. Further, we show that SineKAN has numerical accuracy that could scale comparably to dense neural networks (DNNs). Compared to the two baseline KAN models, SineKAN achieves a substantial speed increase at all hidden layer sizes, batch sizes, and depths. Current advantage of DNNs due to hardware and software optimizations are discussed along with theoretical scaling. Additionally, properties of SineKAN compared to other KAN implementations and current limitations are also discussed.

• Sponsoring Organization: USDOE
• Grant/Contract Number: SC0012447
• OSTI ID: 2500918
• Journal Information: Frontiers in Artificial Intelligence, Journal Name: Frontiers in Artificial Intelligence Vol. 7; ISSN 2624-8212
• Publisher: Frontiers Media SACopyright Statement
• Country of Publication: Switzerland
• Language: English