A Co-design Framework for Online Data Analysis and Reduction

Mehta, Kshitij; Allen, Bryce; Wolf, Matthew; Logan, Jeremy; Suchyta, Eric; Singhal, Swati; Choi, Jong Youl; Takahashi, Keichi; Huck, Kevin; Yakushin, Igor; Sussman, Alan; Munson, Todd; Foster, Ian; Klasky, Scott

doi:10.1002/cpe.6519

Title: A Co-design Framework for Online Data Analysis and Reduction

Journal Article · Thu Aug 26 00:00:00 EDT 2021 · Concurrency and Computation. Practice and Experience

DOI:https://doi.org/10.1002/cpe.6519· OSTI ID:1817542

^[1]; Allen, Bryce ^[2];

^[1]; Logan, Jeremy ^[1];

^[1]; Singhal, Swati ^[3];

^[1]; Takahashi, Keichi ^[4]; Huck, Kevin ^[5]; Yakushin, Igor ^[2]; Sussman, Alan ^[3]; Munson, Todd ^[2]; Foster, Ian ^[6];

^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Argonne National Lab. (ANL), Argonne, IL (United States)
Univ. of Maryland, College Park, MD (United States)
Nara Inst. of Science and Technology (Japan)
Univ. of Oregon, Eugene, OR (United States)
Argonne National Lab. (ANL), Argonne, IL (United States); Univ. of Chicago, IL (United States)

Science applications preparing for the exascale era are increasingly exploring in situ computations comprising of simulation-analysis-reduction pipelines coupled in-memory. Efficient composition and execution of such complex pipelines for a target platform is a codesign process that evaluates the impact and tradeoffs of various application- and system-specific parameters. In this article, we describe a toolset for automating performance studies of composed HPC applications that perform online data reduction and analysis. We describe Cheetah, a new framework for composing parametric studies on coupled applications, and Savanna, a runtime engine for orchestrating and executing campaigns of codesign experiments. Furthermore, this toolset facilitates understanding the impact of various factors such as process placement, synchronicity of algorithms, and storage versus compute requirements for online analysis of large data. Ultimately, we aim to create a catalog of performance results that can help scientists understand tradeoffs when designing next-generation simulations that make use of online processing techniques. We illustrate the design of Cheetah and Savanna, and present application examples that use this framework to conduct codesign studies on small clusters as well as leadership class supercomputers.

View Accepted Manuscript (DOE)

View Accepted Manuscript (Publisher)

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC05-00OR22725; AC02-05CH11231; AC02-06CH11357

OSTI ID:: 1817542

Alternate ID(s):: OSTI ID: 1869488; OSTI ID: 1887662

Journal Information:: Concurrency and Computation. Practice and Experience, Vol. 34, Issue 14; ISSN 1532-0626

Publisher:: WileyCopyright Statement

Country of Publication:: United States

Language:: English

References (27)

A Vision for Managing Extreme-Scale Data Hoards Logan, Jeremy; Mehta, Kshitij; Heber, Gerd 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS) https://doi.org/10.1109/ICDCS.2019.00179	conference	July 2019
Optimal scheduling of in-situ analysis for large-scale scientific simulations Malakar, Preeti; Vishwanath, Venkatram; Munson, Todd Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807656	conference	January 2015
Fault-aware, utility-based job scheduling on Blue, Gene/P systems Tang, Wei; Lan, Zhiling; Desai, Narayan 2009 IEEE International Conference on Cluster Computing and Workshops https://doi.org/10.1109/CLUSTR.2009.5289206	conference	August 2009
Expertus: A Generator Approach to Automate Performance Testing in IaaS Clouds Jayasinghe, Deepal; Swint, Galen; Malkowski, Simon 2012 IEEE 5th International Conference on Cloud Computing (CLOUD), 2012 IEEE Fifth International Conference on Cloud Computing https://doi.org/10.1109/CLOUD.2012.98	conference	June 2012
Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems Deelman, Ewa; Singh, Gurmeet; Su, Mei-Hui Scientific Programming, Vol. 13, Issue 3 https://doi.org/10.1155/2005/128026	journal	January 2005
Harnessing the Power of Many: Extensible Toolkit for Scalable Ensemble Applications Balasubramanian, Vivek; Turilli, Matteo; Hu, Weiming 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2018.00063	conference	May 2018
Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks: HELLO ADIOS Liu, Qing; Logan, Jeremy; Tian, Yuan Concurrency and Computation: Practice and Experience, Vol. 26, Issue 7 https://doi.org/10.1002/cpe.3125	journal	August 2013
Feature-preserving Lossy Compression for In Situ Data Analysis Yakushin, Igor; Mehta, Kshitij; Chen, Jieyang ICPP Workshops '20: Workshops, 49th International Conference on Parallel Processing - ICPP : Workshops https://doi.org/10.1145/3409390.3409400	conference	August 2020
Zoo: a desktop experiment management environment Ioannidis, Yannis E.; Livny, Miron; Ailamaki, Anastassia ACM SIGMOD Record, Vol. 26, Issue 2 https://doi.org/10.1145/253262.253415	journal	June 1997
Experiment Management and Analysis with perfbase Worringen, Joachim 2005 IEEE International Conference on Cluster Computing https://doi.org/10.1109/CLUSTR.2005.347052	conference	September 2005
Optimal Execution of Co-analysis for Large-Scale Molecular Dynamics Simulations Malakar, Preeti; Vishwanath, Venkatram; Knight, Christopher SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.59	conference	November 2016
Swift: A language for distributed parallel scripting Wilde, Michael; Hategan, Mihael; Wozniak, Justin M. Parallel Computing, Vol. 37, Issue 9 https://doi.org/10.1016/j.parco.2011.05.005	journal	September 2011
The Exascale Computing Project Messina, Paul Computing in Science & Engineering, Vol. 19, Issue 3 https://doi.org/10.1109/MCSE.2017.57	journal	May 2017
Fast Error-Bounded Lossy HPC Data Compression with SZ Di, Sheng; Cappello, Franck 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.11	conference	May 2016
Flux: A Next-Generation Resource Management Framework for Large HPC Centers Ahn, Dong H.; Garlick, Jim; Grondona, Mark 2014 43nd International Conference on Parallel Processing Workshops (ICCPW), 2014 43rd International Conference on Parallel Processing Workshops https://doi.org/10.1109/ICPPW.2014.15	conference	September 2014
Z-checker: A framework for assessing lossy compression of scientific data Tao, Dingwen; Di, Sheng; Guo, Hanqi The International Journal of High Performance Computing Applications, Vol. 33, Issue 2 https://doi.org/10.1177/1094342017737147	journal	November 2017
Fixed-Rate Compressed Floating-Point Arrays Lindstrom, Peter IEEE Transactions on Visualization and Computer Graphics, Vol. 20, Issue 12 https://doi.org/10.1109/TVCG.2014.2346458	journal	December 2014
The Tau Parallel Performance System Shende, Sameer S.; Malony, Allen D. The International Journal of High Performance Computing Applications, Vol. 20, Issue 2 https://doi.org/10.1177/1094342006064482	journal	May 2006
Multilevel Techniques for Compression and Reduction of Scientific Data---The Unstructured Case Ainsworth, Mark; Tugluk, Ozan; Whitney, Ben SIAM Journal on Scientific Computing, Vol. 42, Issue 2 https://doi.org/10.1137/19M1267878	journal	January 2020
Complex Patterns in a Simple System Pearson, J. E. Science, Vol. 261, Issue 5118 https://doi.org/10.1126/science.261.5118.189	journal	July 1993
The Exascale Framework for High Fidelity coupled Simulations (EFFIS): Enabling whole device modeling in fusion science Suchyta, Eric; Klasky, Scott; Podhorszki, Norbert The International Journal of High Performance Computing Applications https://doi.org/10.1177/10943420211019119	journal	May 2021
FTK: A Simplicial Spacetime Meshing Framework for Robust and Scalable Feature Tracking Guo, Hanqi; Lenz, David; Xu, Jiayi IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2021.3073399	journal	January 2021
SLURM: Simple Linux Utility for Resource Management Yoo, Andy B.; Jette, Morris A.; Grondona, Mark Job Scheduling Strategies for Parallel Processing https://doi.org/10.1007/10968987_3	book	January 2003
Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales Foster, Ian 2017 IEEE 24th International Conference on High Performance Computing (HiPC) https://doi.org/10.1109/HiPC.2017.00042	conference	December 2017
Coupling Exascale Multiphysics Applications: Methods and Lessons Learned Choi, Jong Youl; Chang, Choong-Seock; Dominski, Julien 2018 IEEE 14th International Conference on e-Science (e-Science) https://doi.org/10.1109/eScience.2018.00133	conference	October 2018
Parsl: Pervasive Parallel Programming in Python Babuji, Yadu; Foster, Ian; Wilde, Michael Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '19 https://doi.org/10.1145/3307681.3325400	conference	January 2019
A Codesign Framework for Online Data Analysis and Reduction Mehta, Kshitij; Foster, Ian; Klasky, Scott 2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS) https://doi.org/10.1109/WORKS49585.2019.00007	conference	November 2019

Similar Records

A Co-design Framework for Online Data Analysis and Reduction

Conference · Tue Jan 01 00:00:00 EST 2019 · OSTI ID:1817542

Mehta, Kshitij; Allen, Bryce; Wolf, Matthew; +11 more

A Codesign Framework for Online Data Analysis and Reduction

Conference · Fri Nov 01 00:00:00 EDT 2019 · OSTI ID:1817542

Mehta, Kshitij; Allen, Bryce; Wolf, Matthew; +8 more

A Codesign Framework for Online Data Analysis and Reduction at the Exascale

Conference · Tue Jan 01 00:00:00 EST 2019 · OSTI ID:1817542

Mehta, Kshitij; Allen, Bryce; Wolf, Matthew; +8 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Exascale
Cheetah
Savanna
CODAR
workflows
in situ
online
reduction
co-design

Title: A Co-design Framework for Online Data Analysis and Reduction

Citation Formats

References (27)

Similar Records

Related Subjects