AutoReP: Automatic ReLU Replacement for Fast Private Network Inference

Conference · Sun Dec 31 04:00:00 EST 2023

DOI:https://doi.org/10.1109/ICCV51070.2023.00478· OSTI ID:2328507

Peng, Hongwu ^[1]; Huang, Shaoyi ^[1]; Zhou, Tong ^[2]; Luo, Yukui ^[2]; Wang, Chenghong ^[3]; Wang, Zigeng ^[1]; Zhao, Jiahui ^[1]; Xie, Xi ^[1]; Li, Ang ^[4]; Geng, Tong ^[5]; Mahmood, Kaleel ^[1]; Wen, Wujie ^[6]; Xu, Xiaolin ^[2]; Ding, Caiwen ^[1]

---

1. University of Connecticut
2. Northeastern University
3. Duke University
4. BATTELLE (PACIFIC NW LAB)
5. University of Rochester
6. Lehigh University

The proliferation of the Machine-Learning-As-A-Service (MLaaS) market has brought to light a number of clients’ data privacy and security concerns. One promising solution is private inference (PI) techniques using cryptographic primitives. These techniques often come with high computation and communication overhead associated with the non-linear operator such as ReLU. Several approaches have been developed in reducing the number of ReLU operations, however, they either require a heuristic threshold selection or introduce significant accuracy drop. This work presents AutoReP, a gradient-based framework for non-linear operators reduction that aims to mitigate these concerns from a systematic perspective. AutoReP automates the process of discrete selection of ReLU and polynomial functions on neurons to accelerate PI applications. We also introduce distribution-aware polynomial approximation (DaPa) to accurately approximate ReLUs under given distribution, preserving model expressivity. Our experimental results demonstrate significant accuracy improvements of 6.12% (94.31%, 12.9K ReLU budget, CIFAR-10), 8.39% (74.92%, 12.9K ReLU budget, CIFAR-100), and 9.45% (63.69%, 55K ReLU budget, Tiny-ImageNet) over current state-of-the-art methods, e.g., SNL. Morever, AutoReP is applied to EfficientNet-B2 on ImageNet dataset, and achieved 75.55% accuracy with 176.1 × ReLU budget reduction.

• Research Organization: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-76RL01830
• OSTI ID: 2328507
• Report Number(s): PNNL-SA-187458
• Country of Publication: United States
• Language: English