Online data analysis and reduction: An important co-design motif for extreme-scale computers

Foster, Ian; Ainsworth, Mark; Bessac, Julie; Cappello, Franck; Choi, Jong; Di, Sheng; Di, Zichao; Gok, Ali Murat; Guo, Hanqi; Huck, Kevin A.; Peterka, Tom; Shu, Tong; Wozniak, Justin M.; Yakushin, Igor; Munson, Todd

doi:10.1177/10943420211023549

Online data analysis and reduction: An important co-design motif for extreme-scale computers

Journal Article · 01 November 2021 · International Journal of High Performance Computing Applications

DOI:https://doi.org/10.1177/10943420211023549· OSTI ID:1873528

Foster, Ian; Ainsworth, Mark; Bessac, Julie; Cappello, Franck; Choi, Jong; Di, Sheng; Di, Zichao; Gok, Ali Murat; Guo, Hanqi; Huck, Kevin A.; Peterka, Tom; Shu, Tong; Wozniak, Justin M.; Yakushin, Igor; Munson, Todd

A growing disparity between supercomputer computation speeds and I/O rates means that it is rapidly becoming infeasible to analyze supercomputer application output only after that output has been written to a file system. Instead, data-generating applications must run concurrently with data reduction and/or analysis operations, with which they exchange information via high-speed methods such as interprocess communications. The resulting parallel computing motif, online data analysis and reduction (ODAR), has important implications for both application and HPC systems design. Here we introduce the ODAR motif and its co-design concerns, describe a co-design process for identifying and addressing those concerns, present tools that assist in the co-design process, and present case studies to illustrate the use of the process and tools in practical settings.

Research Organization:: Argonne National Lab. (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Exascale Computing Project (ECP); USDOE Office of Science - Office of Basic Energy Sciences - Scientific User Facilities Division

DOE Contract Number:: AC02-06CH11357

OSTI ID:: 1873528

Journal Information:: International Journal of High Performance Computing Applications, Vol. 35, Issue 6

Country of Publication:: United States

Language:: English

References (57)

Nimrod: a tool for performing parametrised simulations using distributed workstations No authors listed Proceedings of the Fourth IEEE International Symposium on High Performance Distributed Computing https://doi.org/10.1109/HPDC.1995.518701		January 1995
Multilevel Techniques for Compression and Reduction of Scientific Data---The Unstructured Case No authors listed SIAM Journal on Scientific Computing https://doi.org/10.1137/19M1267878		January 2020
Multilevel Techniques for Compression and Reduction of Scientific Data---The Multivariate Case No authors listed SIAM Journal on Scientific Computing https://doi.org/10.1137/18M1166651		January 2019
Exascale applications: skin in the game No authors listed Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences https://doi.org/10.1098/rsta.2019.0056		January 2020
Performance Analysis, Design Considerations, and Applications of Extreme-Scale In Situ Infrastructures No authors listed SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.78		November 2016
The ITER design No authors listed Plasma Physics and Controlled Fusion https://doi.org/10.1088/0741-3335/44/5/304		April 2002
Autotuning in High-Performance Computing Applications No authors listed Proceedings of the IEEE https://doi.org/10.1109/JPROC.2018.2841200		November 2018
In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms No authors listed Computer Graphics Forum https://doi.org/10.1111/cgf.12930		June 2016
Lightweight computational steering of very large scale molecular dynamics simulations No authors listed Proceedings of the 1996 ACM/IEEE conference on Supercomputing (CDROM) - Supercomputing '96 https://doi.org/10.1145/369028.369136		January 1996
Linda in context No authors listed Communications of the ACM https://doi.org/10.1145/63334.63337		April 1989
A terminology for in situ visualization and analysis systems No authors listed The International Journal of High Performance Computing Applications https://doi.org/10.1177/1094342020935991		August 2020
Coupling Exascale Multiphysics Applications: Methods and Lessons Learned No authors listed 2018 IEEE 14th International Conference on e-Science (e-Science) https://doi.org/10.1109/eScience.2018.00133		October 2018
A Co-Design Study Of Fusion Whole Device Modeling Using Code Coupling No authors listed 2019 IEEE/ACM 5th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-5) https://doi.org/10.1109/DRBSD-549595.2019.00011		November 2019
TGE: Machine Learning Based Task Graph Embedding for Large-Scale Topology Mapping No authors listed 2017 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2017.67		September 2017
A field study of the software design process for large systems No authors listed Communications of the ACM https://doi.org/10.1145/50087.50089		November 1988
Hardware/software co-design No authors listed Proceedings of the IEEE https://doi.org/10.1109/5.558708		March 1997
Fast Error-Bounded Lossy HPC Data Compression with SZ No authors listed 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.11		May 2016
Error Analysis of ZFP Compression for Floating-Point Data No authors listed SIAM Journal on Scientific Computing https://doi.org/10.1137/18M1168832		January 2019
DataSpaces: an interaction and coordination framework for coupled simulation workflows No authors listed Cluster Computing https://doi.org/10.1007/s10586-011-0162-y		February 2011
A tight-coupling scheme sharing minimum information across a spatial interface between gyrokinetic turbulence codes No authors listed Physics of Plasmas https://doi.org/10.1063/1.5044707		July 2018
Active Learning in Performance Analysis No authors listed 2016 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2016.63		September 2016
Weighted random sampling with a reservoir No authors listed Information Processing Letters https://doi.org/10.1016/j.ipl.2005.11.003		March 2006
Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales No authors listed 2017 IEEE 24th International Conference on High Performance Computing (HiPC) https://doi.org/10.1109/HiPC.2017.00042		December 2017
Distance visualization: data exploration on the grid No authors listed Computer https://doi.org/10.1109/2.809249		January 1999
Scaling System-Level Science: Scientific Exploration and IT Implications No authors listed Computer https://doi.org/10.1109/MC.2006.397		January 2006
ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management No authors listed SoftwareX https://doi.org/10.1016/j.softx.2020.100561		July 2020
Rules of thumb in data engineering No authors listed Proceedings of 16th International Conference on Data Engineering (Cat. No.00CB37073) https://doi.org/10.1109/ICDE.2000.839382		January 2000
FTK: A Simplicial Spacetime Meshing Framework for Robust and Scalable Feature Tracking No authors listed IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2021.3073399		January 2021
Performance modeling for systematic performance tuning No authors listed State of the Practice Reports on - SC '11 https://doi.org/10.1145/2063348.2063356		January 2011
Runtime Visualization of the Human Arterial Tree No authors listed IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2007.1017		July 2007
Interactive simulation and visualization No authors listed Computer https://doi.org/10.1109/2.809252		January 1999
The ALPINE In Situ Infrastructure: Ascending from the Ashes of Strawman No authors listed Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization - ISAV'17 https://doi.org/10.1145/3144769.3144778		January 2017
DeepDriveMD: Deep-Learning Driven Adaptive Molecular Simulations for Protein Folding No authors listed 2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS) https://doi.org/10.1109/DLS49591.2019.00007		November 2019
Error-Controlled Lossy Compression Optimized for High Compression Ratios of Scientific Datasets No authors listed 2018 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2018.8622520		December 2018
In-situ processing and visualization for ultrascale simulations No authors listed Journal of Physics: Conference Series https://doi.org/10.1088/1742-6596/78/1/012043		July 2007
Topology-Aware Space-Shared Co-Analysis of Large-Scale Molecular Dynamics Simulations No authors listed SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00027		November 2018
Optimal Execution of Co-analysis for Large-Scale Molecular Dynamics Simulations No authors listed SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.59		November 2016
Optimal scheduling of in-situ analysis for large-scale scientific simulations No authors listed Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807656		January 2015
A Codesign Framework for Online Data Analysis and Reduction No authors listed 2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS) https://doi.org/10.1109/WORKS49585.2019.00007		November 2019
From desktop to Large-Scale Model Exploration with Swift/T No authors listed 2016 Winter Simulation Conference (WSC) https://doi.org/10.1109/WSC.2016.7822090		December 2016
Prescriptive provenance for streaming analysis of workflows at scale No authors listed 2018 New York Scientific Data Summit (NYSDS) https://doi.org/10.1109/NYSDS.2018.8538951		August 2018
Fast Lossless Compression of Scientific Floating-Point Data No authors listed Data Compression Conference (DCC'06) https://doi.org/10.1109/DCC.2006.35		January 2006
Programming with BIG Data in R: Scaling Analytics from One to Thousands of Nodes No authors listed Big Data Research https://doi.org/10.1016/j.bdr.2016.10.002		July 2017
The Tau Parallel Performance System No authors listed The International Journal of High Performance Computing Applications https://doi.org/10.1177/1094342006064482		May 2006
Exploiting task and data parallelism on a multicomputer No authors listed Proceedings of the fourth ACM SIGPLAN symposium on Principles and practice of parallel programming - PPOPP '93 https://doi.org/10.1145/155332.155334		January 1993
The Exascale Framework for High Fidelity coupled Simulations (EFFIS): Enabling whole device modeling in fusion science No authors listed The International Journal of High Performance Computing Applications https://doi.org/10.1177/10943420211019119		May 2021
Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization No authors listed 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2017.115		May 2017
Z-checker: A framework for assessing lossy compression of scientific data No authors listed The International Journal of High Performance Computing Applications https://doi.org/10.1177/1094342017737147		November 2017
NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations No authors listed Computer Physics Communications https://doi.org/10.1016/j.cpc.2010.04.018		September 2010
Scaling Deep Learning for Cancer with Advanced Workflow Storage Integration No authors listed 2018 IEEE/ACM Machine Learning in HPC Environments (MLHPC) https://doi.org/10.1109/MLHPC.2018.8638629		November 2018
MPI jobs within MPI jobs: A practical way of enabling task-level fault-tolerance in HPC workflows No authors listed Future Generation Computer Systems https://doi.org/10.1016/j.future.2019.05.020		December 2019
CANDLE/Supervisor: a workflow framework for machine learning applied to cancer research No authors listed BMC Bioinformatics https://doi.org/10.1186/s12859-018-2508-4		December 2018
A Visual Analytics Framework for the Detection of Anomalous Call Stack Trees in High Performance Computing Applications No authors listed IEEE Transactions on Visualization and Computer Graphics https://doi.org/10.1109/TVCG.2018.2865026		January 2019
Feature-preserving Lossy Compression for In Situ Data Analysis No authors listed ICPP Workshops '20: Workshops, 49th International Conference on Parallel Processing - ICPP : Workshops https://doi.org/10.1145/3409390.3409400		August 2020
Streaming spectral clustering No authors listed 2016 IEEE 32nd International Conference on Data Engineering (ICDE) https://doi.org/10.1109/ICDE.2016.7498277		May 2016
Streaming Classical Multidimensional Scaling No authors listed 2018 New York Scientific Data Summit (NYSDS) https://doi.org/10.1109/NYSDS.2018.8538942		August 2018
Significantly Improving Lossy Compression for HPC Datasets with Second-Order Prediction and Parameter Optimization No authors listed HPDC '20: The 29th International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/3369583.3392688		June 2020

Similar Records

Online data analysis and reduction: An important Co-design motif for extreme-scale computers

Journal Article · 2021 · International Journal of High Performance Computing Applications · OSTI ID:1788100

Foster, Ian; Ainsworth, Mark; Bessac, Julie; +25 more

...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats

Conference · 2009 · OSTI ID:982187

Klasky, Scott A; Lofstead, J.; Bent, John; +5 more

A Co-design Framework for Online Data Analysis and Reduction

Journal Article · 2021 · Concurrency and Computation. Practice and Experience · OSTI ID:1817542

Mehta, Kshitij; Allen, Bryce; Wolf, Matthew; +11 more

Related Subjects

Data Analysis
exascale computing
in situ
online data analysis and reduction

Online data analysis and reduction: An important co-design motif for extreme-scale computers

Citation Formats

References (57)

Similar Records

Related Subjects