SymbolNet: neural symbolic regression with adaptive dynamic pruning for compression

Journal Article · 28 January 2025 · Machine Learning: Science and Technology

DOI: https://doi.org/10.1088/2632-2153/adaad8 · OSTI ID:2507371

Tsoi, Ho Fung ; Loncar, Vladimir ; Dasu, Sridhara ; Harris, Philip

Abstract Compact symbolic expressions have been shown to be more efficient than neural network (NN) models in terms of resource consumption and inference speed when implemented on custom hardware such as field-programmable gate arrays (FPGAs), while maintaining comparable accuracy (Tsoi et al 2024 EPJ Web Conf. 295 09036). These capabilities are highly valuable in environments with stringent computational resource constraints, such as high-energy physics experiments at the CERN Large Hadron Collider. However, finding compact expressions for high-dimensional datasets remains challenging due to the inherent limitations of genetic programming (GP), the search algorithm of most symbolic regression (SR) methods. Contrary to GP, the NN approach to SR offers scalability to high-dimensional inputs and leverages gradient methods for faster equation searching. Common ways of constraining expression complexity often involve multistage pruning with fine-tuning, which can result in significant performance loss. In this work, we propose $\mathtt{S}\mathtt{y}\mathtt{m}\mathtt{b}\mathtt{o}\mathtt{l}\mathtt{N}\mathtt{e}\mathtt{t}$ , a NN approach to SR specifically designed as a model compression technique, aimed at enabling low-latency inference for high-dimensional inputs on custom hardware such as FPGAs. This framework allows dynamic pruning of model weights, input features, and mathematical operators in a single training process, where both training loss and expression complexity are optimized simultaneously. We introduce a sparsity regularization term for each pruning type, which can adaptively adjust its strength, leading to convergence at a target sparsity ratio. Unlike most existing SR methods that struggle with datasets containing more than $O\left(10\right)$ inputs, we demonstrate the effectiveness of our model on the LHC jet tagging task (16 inputs), MNIST (784 inputs), and SVHN (3072 inputs).

• Sponsoring Organization: USDOE; USDOE Office of Science (SC), High Energy Physics (HEP)
• Grant/Contract Number: SC0017647
• OSTI ID: 2507371
• Journal Information: Machine Learning: Science and Technology, Journal Name: Machine Learning: Science and Technology Journal Issue: 1 Vol. 6; ISSN 2632-2153
• Publisher: IOP PublishingCopyright Statement
• Country of Publication: United Kingdom
• Language: English