ICDARTS: Improving the Stability of Cyclic DARTS

Conference · 01 December 2022

DOI:https://doi.org/10.1109/ICMLA55696.2022.00175· OSTI ID:1976051

Herron, Emily ^[1]; Young, Steven ^[2]; Rose, Derek ^[2]

---

1. University of Tennessee, Knoxville (UTK)
2. ORNL

Cyclic DARTS (CDARTS) is a Differentiable Architecture Search (DARTS)-based approach to neural architecture search (NAS) that uses a cyclic feedback mechanism to train search and evaluation networks concurrently. This training protocol aims to optimize the search process and evaluate the deep evaluation network comprised of discretized candidate operations. However, this approach introduces a loss function for the evaluation network dependent on the search network. The dissimilarity between the evaluation network’s loss function used during the search and retraining phases results in a search network that is a sub-optimal proxy for the final evaluation network accessed during retraining. We present a revised approach that removes the dependency of the evaluation network weights upon those of the search network. In addition, we introduce a modified process for relaxing the search network’s zero operations that allows these operations to be retained in the final evaluation networks.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1976051
• Country of Publication: United States
• Language: English