Streamlining Ocean Dynamics Modeling with Fourier Neural Operators: A Multiobjective Hyperparameter and Architecture Optimization Approach

Sun, Yixuan; Sowunmi, Ololade; Egele, Romain; Narayanan, Sri Krishna; Van Roekel, Luke; Balaprakash, Prasanna

doi:10.3390/math12101483

Streamlining Ocean Dynamics Modeling with Fourier Neural Operators: A Multiobjective Hyperparameter and Architecture Optimization Approach

Journal Article · Fri May 10 00:00:00 EDT 2024 · Mathematics

DOI:https://doi.org/10.3390/math12101483· OSTI ID:2477212

^[1]; Sowunmi, Ololade ^[2]; ^[3]; ^[1]; ^[4]; ^[5]

Argonne National Laboratory (ANL), Argonne, IL (United States)
Florida State Univ., Tallahassee, FL (United States)
Argonne National Laboratory (ANL), Argonne, IL (United States); Univ. Paris-Saclay, Gif-sur-Yvette (France)
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Training an effective deep learning model to learn ocean processes involves careful choices of various hyperparameters. We leverage DeepHyper’s advanced search algorithms for multiobjective optimization, streamlining the development of neural networks tailored for ocean modeling. The focus is on optimizing Fourier neural operators (FNOs), a data-driven model capable of simulating complex ocean behaviors. Selecting the correct model and tuning the hyperparameters are challenging tasks, requiring much effort to ensure model accuracy. DeepHyper allows efficient exploration of hyperparameters associated with data preprocessing, FNO architecture-related hyperparameters, and various model training strategies. We aim to obtain an optimal set of hyperparameters leading to the most performant model. Moreover, on top of the commonly used mean squared error for model training, we propose adopting the negative anomaly correlation coefficient as the additional loss term to improve model performance and investigate the potential trade-off between the two terms. The numerical experiments show that the optimal set of hyperparameters enhanced model performance in single timestepping forecasting and greatly exceeded the baseline configuration in the autoregressive rollout for long-horizon forecasting up to 30 days. Utilizing DeepHyper, we demonstrate an approach to enhance the use of FNO in ocean dynamics forecasting, offering a scalable solution with improved precision.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR). Scientific Discovery through Advanced Computing (SciDAC); USDOE Office of Science (SC), Biological and Environmental Research (BER)

Grant/Contract Number:: 89233218CNA000001; AC02-06CH11357

OSTI ID:: 2477212

Journal Information:: Mathematics, Journal Name: Mathematics Journal Issue: 10 Vol. 12; ISSN 2227-7390

Publisher:: MDPICopyright Statement

Country of Publication:: United States

Language:: English

References (31)

Perceptual Losses for Real-Time Style Transfer and Super-Resolution Johnson, Justin; Alahi, Alexandre; Fei-Fei, Li Computer Vision – ECCV 2016 https://doi.org/10.1007/978-3-319-46475-6_43	book	September 2016
Practical Recommendations for Gradient-Based Training of Deep Architectures Bengio, Yoshua Lecture Notes in Computer Science https://doi.org/10.1007/978-3-642-35289-8_26	book	January 2012
Extremely randomized trees Geurts, Pierre; Ernst, Damien; Wehenkel, Louis Machine Learning, Vol. 63, Issue 1 https://doi.org/10.1007/s10994-006-6226-1	journal	March 2006
Using Machine Learning at scale in numerical simulations with SmartSim: An application to ocean climate modeling Partee, Sam; Ellis, Matthew; Rigazzi, Alessandro Journal of Computational Science, Vol. 62 https://doi.org/10.1016/j.jocs.2022.101707	journal	July 2022
Time series prediction of sea surface temperature based on BiLSTM model with attention mechanism Zrira, Nabila; Kamal-Idrissi, Assia; Farssi, Rahma Journal of Sea Research, Vol. 198 https://doi.org/10.1016/j.seares.2024.102472	journal	April 2024
Efficient Global Optimization of Expensive Black-Box Functions Jones, Donald R.; Schonlau, Matthias; Welch, William J. Journal of Global Optimization, Vol. 13, Issue 4, p. 455-492 https://doi.org/10.1023/A:1008306431147	journal	January 1998
Random Forests Breiman, Leo Machine Learning, Vol. 45, Issue 1, p. 5-32 https://doi.org/10.1023/A:1010933404324	journal	January 2001
Data‐Driven Equation Discovery of Ocean Mesoscale Closures Zanna, Laure; Bolton, Thomas Geophysical Research Letters, Vol. 47, Issue 17 https://doi.org/10.1029/2020GL088376	journal	August 2020
Stochastic‐Deep Learning Parameterization of Ocean Momentum Forcing Guillaumin, Arthur P.; Zanna, Laure Journal of Advances in Modeling Earth Systems, Vol. 13, Issue 9 https://doi.org/10.1029/2021MS002534	journal	September 2021
Training machine learning models on climate model output yields skillful interpretable seasonal precipitation forecasts Gibson, Peter B.; Chapman, William E.; Altinok, Alphan Communications Earth & Environment, Vol. 2, Issue 1 https://doi.org/10.1038/s43247-021-00225-4	journal	August 2021
Deep learning to represent subgrid processes in climate models Rasp, Stephan; Pritchard, Michael S.; Gentine, Pierre Proceedings of the National Academy of Sciences, Vol. 115, Issue 39 https://doi.org/10.1073/pnas.1810286115	journal	September 2018
Deep-learning-based information mining from ocean remote-sensing imagery Li, Xiaofeng; Liu, Bin; Zheng, Gang National Science Review, Vol. 7, Issue 10 https://doi.org/10.1093/nsr/nwaa047	journal	March 2020
Physics-informed deep-learning parameterization of ocean vertical mixing improves climate simulations Zhu, Yuchao; Zhang, Rong-Hua; Moum, James N. National Science Review, Vol. 9, Issue 8 https://doi.org/10.1093/nsr/nwac044	journal	March 2022
DeepOcean: A General Deep Learning Framework for Spatio-Temporal Ocean Sensing Data Prediction Gou, Yu; Zhang, Tong; Liu, Jun IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2020.2990939	journal	January 2020
Padding Module: Learning the Padding in Deep Neural Networks Alrasheedi, Fahad; Zhong, Xin; Huang, Pei-Chi IEEE Access, Vol. 11 https://doi.org/10.1109/ACCESS.2023.3238315	journal	January 2023
DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks Balaprakash, Prasanna; Salim, Michael; Uram, Thomas D. 2018 IEEE 25th International Conference on High Performance Computing (HiPC) https://doi.org/10.1109/HiPC.2018.00014	conference	December 2018
Deep Learning and the Oceans Radeta, Marko; Zuniga, Agustin; Motlagh, Naser Hossein Computer, Vol. 55, Issue 5 https://doi.org/10.1109/MC.2022.3143087	journal	May 2022
Exascale Deep Learning for Climate Analytics Kurth, Thorsten; Treichler, Sean; Romero, Joshua SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00054	conference	November 2018
Loss Functions for Image Restoration With Neural Networks Zhao, Hang; Gallo, Orazio; Frosio, Iuri IEEE Transactions on Computational Imaging, Vol. 3, Issue 1 https://doi.org/10.1109/TCI.2016.2644865	journal	March 2017
Rain Regime Segmentation of Sentinel-1 Observation Learning From NEXRAD Collocations With Convolution Neural Networks Colin, Aurélien; Tandeo, Pierre; Peureux, Charles IEEE Transactions on Geoscience and Remote Sensing, Vol. 62 https://doi.org/10.1109/TGRS.2024.3353311	journal	January 2024
Training a Task-Specific Image Reconstruction Loss Mustafa, Aamir; Mikhailiuk, Aliaksei; Iliescu, Dan Andrei 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) https://doi.org/10.1109/WACV51458.2022.00010	conference	January 2022
Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimization Egelé, Romain; Guyon, Isabelle; Vishwanath, Venkatram 2023 IEEE 19th International Conference on e-Science (e-Science) https://doi.org/10.1109/e-Science58273.2023.10254839	conference	October 2023
Improved estimates of ocean heat content from 1960 to 2015 Cheng, Lijing; Trenberth, Kevin E.; Fasullo, John Science Advances, Vol. 3, Issue 3 https://doi.org/10.1126/sciadv.1601545	journal	March 2017
Learning skillful medium-range global weather forecasting Lam, Remi; Sanchez-Gonzalez, Alvaro; Willson, Matthew Science, Vol. 382, Issue 6677 https://doi.org/10.1126/science.adi2336	journal	December 2023
Spectral Neural Operators Fanaskov, V. S.; Oseledets, I. V. Doklady Mathematics, Vol. 108, Issue S2 https://doi.org/10.1134/S1064562423701107	journal	December 2023
An Empirical Study of the Impact of Hyperparameter Tuning and Model Optimization on the Performance Properties of Deep Neural Networks Liao, Lizhi; Li, Heng; Shang, Weiyi ACM Transactions on Software Engineering and Methodology, Vol. 31, Issue 3 https://doi.org/10.1145/3506695	journal	April 2022
Physics-Informed Neural Operator for Learning Partial Differential Equations Li, Zongyi; Zheng, Hongkai; Kovachki, Nikola ACM / IMS Journal of Data Science, Vol. 1, Issue 3 https://doi.org/10.1145/3648506	journal	May 2024
Skill Scores and Correlation Coefficients in Model Verification Murphy, Allan H.; Epstein, Edward S. Monthly Weather Review, Vol. 117, Issue 3 https://doi.org/10.1175/1520-0493(1989)117<0572:SSACCI>2.0.CO;2	journal	March 1989
Diagnosing Isopycnal Diffusivity in an Eddying, Idealized Midlatitude Ocean Basin via Lagrangian, in Situ, Global, High-Performance Particle Tracking (LIGHT) Wolfram, Phillip J.; Ringler, Todd D.; Maltrud, Mathew E. Journal of Physical Oceanography, Vol. 45, Issue 8 https://doi.org/10.1175/JPO-D-14-0260.1	journal	August 2015
Learning to solve the elastic wave equation with Fourier neural operators Zhang, Tianze; Trad, Daniel; Innanen, Kristopher GEOPHYSICS, Vol. 88, Issue 3 https://doi.org/10.1190/geo2022-0268.1	journal	April 2023
Improving Ocean Forecasting Using Deep Learning and Numerical Model Integration Choi, Youngjin; Park, Youngmin; Hwang, Jaedong Journal of Marine Science and Engineering, Vol. 10, Issue 4 https://doi.org/10.3390/jmse10040450	journal	March 2022

Similar Records

Multiobjective Hyperparameter Optimization for Deep Learning Interatomic Potential Training Using NSGA-II

Conference · Tue Aug 01 00:00:00 EDT 2023 · OSTI ID:1996670

DeepHyper: A Python Package for Massively Parallel Hyperparameter Optimization in Machine Learning

Journal Article · Sun May 18 20:00:00 EDT 2025 · Journal of Open Source Software · OSTI ID:3002304

DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks

Conference · Sun Dec 31 23:00:00 EST 2017 · OSTI ID:1772592

Related Subjects

97 MATHEMATICS AND COMPUTING
Hyperparameter optimization
Ocean modeling
Operator learning

Streamlining Ocean Dynamics Modeling with Fourier Neural Operators: A Multiobjective Hyperparameter and Architecture Optimization Approach

Citation Formats

References (31)

Similar Records

Related Subjects