skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Damaris: Addressing performance variability in data management for post-petascale simulations

Abstract

With exascale computing on the horizon, reducing performance variability in data management tasks (storage, visualization, analysis, etc.) is becoming a key challenge in sustaining high performance. Here, this variability significantly impacts the overall application performance at scale and its predictability over time. In this article, we present Damaris, a system that leverages dedicated cores in multicore nodes to offload data management tasks, including I/O, data compression, scheduling of data movements, in situ analysis, and visualization. We evaluate Damaris with the CM1 atmospheric simulation and the Nek5000 computational fluid dynamic simulation on four platforms, including NICS’s Kraken and NCSA’s Blue Waters. Our results show that (1) Damaris fully hides the I/O variability as well as all I/O-related costs, thus making simulation performance predictable; (2) it increases the sustained write throughput by a factor of up to 15 compared with standard I/O approaches; (3) it allows almost perfect scalability of the simulation up to over 9,000 cores, as opposed to state-of-the-art approaches that fail to scale; and (4) it enables a seamless connection to the VisIt visualization software to perform in situ analysis and visualization in a way that impacts neither the performance of the simulation nor its variability. In addition, wemore » extended our implementation of Damaris to also support the use of dedicated nodes and conducted a thorough comparison of the two approaches—dedicated cores and dedicated nodes—for I/O tasks with the aforementioned applications.« less

Authors:
 [1];  [2];  [1];  [1];  [3];  [2];  [2];  [1];  [4]
  1. Argonne National Lab. (ANL), Argonne, IL (United States)
  2. Inria, Rennes - Bretagne Atlantique Research Centre (France)
  3. Univ. of Illinois at Urbana-Champaign, Urbana, IL (United States)
  4. Univ. of Wisconsin, Madison, WI (United States)
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Basic Energy Sciences (BES); Central Michigan University; National Center for Atmospheric Research
OSTI Identifier:
1346736
Grant/Contract Number:  
AC02-06CH11357
Resource Type:
Accepted Manuscript
Journal Name:
ACM Transactions on Parallel Computing
Additional Journal Information:
Journal Volume: 3; Journal Issue: 3; Journal ID: ISSN 2329-4949
Publisher:
Association for Computing Machinery
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Damaris; Dedicated Cores; Dedicated Nodes; Design; Exascale Computing; Experimentation; I/O; In Situ Visualization; Performance

Citation Formats

Dorier, Matthieu, Antoniu, Gabriel, Cappello, Franck, Snir, Marc, Sisneros, Robert, Yildiz, Orcun, Ibrahim, Shadi, Peterka, Tom, and Orf, Leigh. Damaris: Addressing performance variability in data management for post-petascale simulations. United States: N. p., 2016. Web. doi:10.1145/2987371.
Dorier, Matthieu, Antoniu, Gabriel, Cappello, Franck, Snir, Marc, Sisneros, Robert, Yildiz, Orcun, Ibrahim, Shadi, Peterka, Tom, & Orf, Leigh. Damaris: Addressing performance variability in data management for post-petascale simulations. United States. doi:10.1145/2987371.
Dorier, Matthieu, Antoniu, Gabriel, Cappello, Franck, Snir, Marc, Sisneros, Robert, Yildiz, Orcun, Ibrahim, Shadi, Peterka, Tom, and Orf, Leigh. Sat . "Damaris: Addressing performance variability in data management for post-petascale simulations". United States. doi:10.1145/2987371. https://www.osti.gov/servlets/purl/1346736.
@article{osti_1346736,
title = {Damaris: Addressing performance variability in data management for post-petascale simulations},
author = {Dorier, Matthieu and Antoniu, Gabriel and Cappello, Franck and Snir, Marc and Sisneros, Robert and Yildiz, Orcun and Ibrahim, Shadi and Peterka, Tom and Orf, Leigh},
abstractNote = {With exascale computing on the horizon, reducing performance variability in data management tasks (storage, visualization, analysis, etc.) is becoming a key challenge in sustaining high performance. Here, this variability significantly impacts the overall application performance at scale and its predictability over time. In this article, we present Damaris, a system that leverages dedicated cores in multicore nodes to offload data management tasks, including I/O, data compression, scheduling of data movements, in situ analysis, and visualization. We evaluate Damaris with the CM1 atmospheric simulation and the Nek5000 computational fluid dynamic simulation on four platforms, including NICS’s Kraken and NCSA’s Blue Waters. Our results show that (1) Damaris fully hides the I/O variability as well as all I/O-related costs, thus making simulation performance predictable; (2) it increases the sustained write throughput by a factor of up to 15 compared with standard I/O approaches; (3) it allows almost perfect scalability of the simulation up to over 9,000 cores, as opposed to state-of-the-art approaches that fail to scale; and (4) it enables a seamless connection to the VisIt visualization software to perform in situ analysis and visualization in a way that impacts neither the performance of the simulation nor its variability. In addition, we extended our implementation of Damaris to also support the use of dedicated nodes and conducted a thorough comparison of the two approaches—dedicated cores and dedicated nodes—for I/O tasks with the aforementioned applications.},
doi = {10.1145/2987371},
journal = {ACM Transactions on Parallel Computing},
number = 3,
volume = 3,
place = {United States},
year = {2016},
month = {10}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Understanding the causes of performance variability in HPC workloads
conference, January 2005

  • Skinner, D.; Kramer, W.
  • IEEE International. 2005 IEEE Workload Characterization Symposium, 2005., IEEE International. 2005 Proceedings of the IEEE Workload Characterization Symposium, 2005.
  • DOI: 10.1109/IISWC.2005.1526010

Functional Partitioning to Optimize End-to-End Performance on Many-core Architectures
conference, November 2010

  • Li, Min; Vazhkudai, Sudharshan S.; Butt, Ali R.
  • 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2010.28

Parallel I/O performance: From events to ensembles
conference, April 2010

  • Uselton, Andrew; Howison, Mark; Wright, Nicholas J.
  • 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS)
  • DOI: 10.1109/IPDPS.2010.5470424

A Flexible Framework for Asynchronous in Situ and in Transit Analytics for Scientific Simulations
conference, May 2014

  • Dreher, Matthieu; Raffin, Bruno
  • 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid)
  • DOI: 10.1109/CCGrid.2014.92

High end scientific codes with computational I/O pipelines: improving their end-to-end performance
conference, January 2011

  • Zheng, Fang; Cao, Jianting; Dayal, Jai
  • Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities - PDAC '11
  • DOI: 10.1145/2110205.2110210

Scalable I/O forwarding framework for high-performance computing systems
conference, August 2009

  • Ali, Nawab; Carns, Philip; Iskra, Kamil
  • 2009 IEEE International Conference on Cluster Computing and Workshops
  • DOI: 10.1109/CLUSTR.2009.5289188

Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)
conference, January 2008

  • Lofstead, Jay F.; Klasky, Scott; Schwan, Karsten
  • Proceedings of the 6th international workshop on Challenges of large applications in distributed environments - CLADE '08
  • DOI: 10.1145/1383529.1383533

In-situ processing and visualization for ultrascale simulations
journal, July 2007


On implementing MPI-IO portably and with high performance
conference, January 1999

  • Thakur, Rajeev; Gropp, William; Lusk, Ewing
  • Proceedings of the sixth workshop on I/O in parallel and distributed systems - IOPADS '99
  • DOI: 10.1145/301816.301826

Electronic poster: co-visualization of full data and in situ data extracts from unstructured grid cfd at 160k cores
conference, January 2011

  • Rasquin, Michel; Sahni, Onkar; Fu, Jing
  • Proceedings of the 2011 companion on High Performance Computing Networking, Storage and Analysis Companion - SC '11 Companion
  • DOI: 10.1145/2148600.2148653

Enabling high-speed asynchronous data extraction and transfer using DART
journal, January 2010

  • Docan, Ciprian; Parashar, Manish; Klasky, Scott
  • Concurrency and Computation: Practice and Experience
  • DOI: 10.1002/cpe.1567

Enabling In-situ Execution of Coupled Scientific Workflow on Multi-core Platform
conference, May 2012

  • Zhang, Fan; Docan, Ciprian; Parashar, Manish
  • 2012 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), 2012 IEEE 26th International Parallel and Distributed Processing Symposium
  • DOI: 10.1109/IPDPS.2012.122

Scaling parallel I/O performance through I/O delegate and caching system
conference, November 2008

  • Nisar, Arifa; Liao, Wei-keng; Choudhary, Alok
  • 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2008.5214358

On the role of burst buffers in leadership-class storage systems
conference, April 2012

  • Liu, Ning; Cope, Jason; Carns, Philip
  • 2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST)
  • DOI: 10.1109/MSST.2012.6232369

DataStager: scalable data staging services for petascale applications
conference, January 2009

  • Abbasi, Hasan; Wolf, Matthew; Eisenhauer, Greg
  • Proceedings of the 18th ACM international symposium on High performance distributed computing - HPDC '09
  • DOI: 10.1145/1551609.1551618

In-situ I/O processing: a case for location flexibility
conference, January 2011

  • Zheng, Fang; Abbasi, Hasan; Cao, Jianting
  • Proceedings of the sixth workshop on Parallel Data Storage - PDSW '11
  • DOI: 10.1145/2159352.2159362

Managing Variability in the IO Performance of Petascale Storage Systems
conference, November 2010

  • Lofstead, Jay; Zheng, Fang; Liu, Qing
  • 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2010.32

Provisioning a Multi-tiered Data Staging Area for Extreme-Scale Machines
conference, June 2011

  • Prabhakar, Ramya; Vazhkudai, Sudharshan S.; Kim, Youngjae
  • 2011 31st International Conference on Distributed Computing Systems (ICDCS)
  • DOI: 10.1109/ICDCS.2011.33

Data sieving and collective I/O in ROMIO
conference, January 1999

  • Thakur, R.; Gropp, W.; Lusk, E.
  • Proceedings. Frontiers '99. Seventh Symposium on the Frontiers of Massively Parallel Computation
  • DOI: 10.1109/FMPC.1999.750599

pClock: an arrival curve based approach for QoS guarantees in shared storage systems
conference, January 2007

  • Gulati, Ajay; Merchant, Arif; Varman, Peter J.
  • Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems - SIGMETRICS '07
  • DOI: 10.1145/1254882.1254885

Design, Modeling, and Evaluation of a Scalable Multi-level Checkpointing System
conference, November 2010

  • Moody, Adam; Bronevetsky, Greg; Mohror, Kathryn
  • 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2010.18

PreDatA – preparatory data analytics on peta-scale machines
conference, April 2010

  • Zheng, Fang; Abbasi, Hasan; Docan, Ciprian
  • 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS)
  • DOI: 10.1109/IPDPS.2010.5470454

QoS support for end users of I/O-intensive applications using shared storage systems
conference, January 2011

  • Zhang, Xuechen; Davis, Kei; Jiang, Song
  • Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11
  • DOI: 10.1145/2063384.2063408

I/O threads to reduce checkpoint blocking for an electromagnetics solver on Blue Gene/P and Cray XK6
conference, January 2012

  • Fu, Jing; Latham, Robert; Min, Misun
  • Proceedings of the 2nd International Workshop on Runtime and Operating Systems for Supercomputers - ROSS '12
  • DOI: 10.1145/2318916.2318919

In-situ Feature-Based Objects Tracking for Large-Scale Scientific Simulations
conference, November 2012

  • Zhang, Fan; Lasluisa, Solomon; Jin, Tong
  • 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis
  • DOI: 10.1109/SC.Companion.2012.100

Extreme Scaling of Production Visualization Software on Diverse Architectures
journal, May 2010

  • Childs, Hank; Pugmire, David; Ahern, Sean
  • IEEE Computer Graphics and Applications, Vol. 30, Issue 3
  • DOI: 10.1109/MCG.2010.51

Examples of in transit visualization
conference, January 2011

  • Moreland, Kenneth; Hereld, Mark; Papka, Michael E.
  • Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities - PDAC '11
  • DOI: 10.1145/2110205.2110207

In Situ Visualization at Extreme Scale: Challenges and Opportunities
journal, November 2009


In Situ Visualization for Large-Scale Combustion Simulations
journal, May 2010

  • Hongfeng Yu, ; Grout, Ray W.
  • IEEE Computer Graphics and Applications, Vol. 30, Issue 3
  • DOI: 10.1109/MCG.2010.55

Concurrent Visualization in a Production Supercomputing Environment
journal, September 2006

  • Ellsworth, D.; Green, B.; Henze, C.
  • IEEE Transactions on Visualization and Computer Graphics, Vol. 12, Issue 5
  • DOI: 10.1109/TVCG.2006.128

Scalable parallel building blocks for custom data analysis
conference, October 2011

  • Peterka, Tom; Ross, Robert; Gyulassy, Attila
  • 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV)
  • DOI: 10.1109/LDAV.2011.6092324

CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination
conference, May 2014

  • Dorier, Matthieu; Antoniu, Gabriel; Ross, Rob
  • 2014 IEEE International Parallel & Distributed Processing Symposium (IPDPS), 2014 IEEE 28th International Parallel and Distributed Processing Symposium
  • DOI: 10.1109/IPDPS.2014.27

The ParaView Coprocessing Library: A scalable, general purpose in situ visualization library
conference, October 2011

  • Fabian, Nathan; Moreland, Kenneth; Thompson, David
  • 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV)
  • DOI: 10.1109/LDAV.2011.6092322