On the memory attribution problem: A solution and case study using MPI

Gutiérrez, Samuel Keith; Arnold, Dorian C.; Davis, Kei Marion; McCormick, Patrick Sean

doi:10.1002/cpe.5159

Title: On the memory attribution problem: A solution and case study using MPI

Abstract

As parallel applications running on large–scale computing systems become increasingly memory constrained, the ability to attribute memory usage to the various components of the application is becoming increasingly important. We present the design and implementation of memnesia, a novel memory usage profiler for parallel and distributed message–passing applications. Our approach captures both application– and message–passing library–specific memory usage statistics from unmodified binaries dynamically linked to a message–passing communication library. Using microbenchmarks and proxy applications, we evaluated our profiler across three Message Passing Interface (MPI) implementations and two hardware platforms. Furthermore, the results show that our approach and the corresponding implementation can accurately quantify memory resource usage as a function of time, scale, communication workload, and software or hardware system architecture, clearly distinguishing between application and MPI library memory usage at a per–process level. With this new capability, we show that job size, communication workload, and hardware/software architecture influence peak runtime memory usage. In practice, this tool provides a potentially valuable source of information for application developers seeking to measure and optimize memory usage.

Authors:

^[1]; Arnold, Dorian C. ^[2];

^[3];

^[3]

Los Alamos National Lab. (LANL), Los Alamos, NM (United States); Univ. of New Mexico, Albuquerque, NM (United States)
Emory Univ., Atlanta, GA (United States)
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Publication Date:: Mon Feb 04 00:00:00 EST 2019

Research Org.:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1495167

Alternate Identifier(s):: OSTI ID: 1493495

Report Number(s):: LA-UR-18-30292
Journal ID: ISSN 1532-0626

Grant/Contract Number:: 89233218CNA000001; AC52‐06NA25396

Resource Type:: Accepted Manuscript

Journal Name:: Concurrency and Computation. Practice and Experience

Additional Journal Information:: Journal Volume: 32; Journal Issue: 3; Journal ID: ISSN 1532-0626

Publisher:: Wiley

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Computer Science

Citation Formats


                    Gutiérrez, Samuel Keith, Arnold, Dorian C., Davis, Kei Marion, and McCormick, Patrick Sean. On the memory attribution problem: A solution and case study using MPI.  United States: N. p., 2019. 
Web.  doi:10.1002/cpe.5159.

Copy to clipboard


                    Gutiérrez, Samuel Keith, Arnold, Dorian C., Davis, Kei Marion, & McCormick, Patrick Sean. On the memory attribution problem: A solution and case study using MPI.  United States.  https://doi.org/10.1002/cpe.5159

Copy to clipboard


                    Gutiérrez, Samuel Keith, Arnold, Dorian C., Davis, Kei Marion, and McCormick, Patrick Sean. Mon .  
"On the memory attribution problem: A solution and case study using MPI".  United States.  https://doi.org/10.1002/cpe.5159.  https://www.osti.gov/servlets/purl/1495167.

Copy to clipboard


                    
@article{osti_1495167,

  title        = {On the memory attribution problem: A solution and case study using MPI},

  author       = {Gutiérrez, Samuel Keith and Arnold, Dorian C. and Davis, Kei Marion and McCormick, Patrick Sean},

  abstractNote = {As parallel applications running on large–scale computing systems become increasingly memory constrained, the ability to attribute memory usage to the various components of the application is becoming increasingly important. We present the design and implementation of memnesia, a novel memory usage profiler for parallel and distributed message–passing applications. Our approach captures both application– and message–passing library–specific memory usage statistics from unmodified binaries dynamically linked to a message–passing communication library. Using microbenchmarks and proxy applications, we evaluated our profiler across three Message Passing Interface (MPI) implementations and two hardware platforms. Furthermore, the results show that our approach and the corresponding implementation can accurately quantify memory resource usage as a function of time, scale, communication workload, and software or hardware system architecture, clearly distinguishing between application and MPI library memory usage at a per–process level. With this new capability, we show that job size, communication workload, and hardware/software architecture influence peak runtime memory usage. In practice, this tool provides a potentially valuable source of information for application developers seeking to measure and optimize memory usage.},

  doi          = {10.1002/cpe.5159},

  journal      = {Concurrency and Computation. Practice and Experience},

  number       = 3,

  volume       = 32,

  place        = {United States},

  year         = {Mon Feb 04 00:00:00 EST 2019},

  month        = {Mon Feb 04 00:00:00 EST 2019}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1002/cpe.5159

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 2 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Valgrind: a framework for heavyweight dynamic binary instrumentation
journal, June 2007

Nethercote, Nicholas; Seward, Julian
ACM SIGPLAN Notices, Vol. 42, Issue 6
DOI: 10.1145/1273442.1250746

Memory registration caching correctness
conference, January 2005

Wyckoff, P.; Wu, J.
CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005.
DOI: 10.1109/CCGRID.2005.1558671

Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation
book, January 2004

Gabriel, Edgar; Fagg, Graham E.; Bosilca, George
Recent Advances in Parallel Virtual Machine and Message Passing Interface
DOI: 10.1007/978-3-540-30218-6_19

A uGNI-Based MPICH2 Nemesis Network Module for the Cray XE
book, January 2011

Pritchard, Howard; Gorodetsky, Igor; Buntinas, Darius
Recent Advances in the Message Passing Interface
DOI: 10.1007/978-3-642-24449-0_14

DAS User Manual
journal, January 2008

,
Communications in Asteroseismology, Vol. 155
DOI: 10.1553/cia155s7

Technology-Driven, Highly-Scalable Dragonfly Topology
conference, June 2008

Kim, John; Dally, Wiliam J.; Scott, Steve
2008 35th International Symposium on Computer Architecture (ISCA), 2008 International Symposium on Computer Architecture
DOI: 10.1109/ISCA.2008.19

A high-performance, portable implementation of the MPI message passing interface standard
journal, September 1996

Gropp, William; Lusk, Ewing; Doss, Nathan
Parallel Computing, Vol. 22, Issue 6
DOI: 10.1016/0167-8191(96)00024-5

MRNet: A Software-Based Multicast/Reduction Network for Scalable Tools
conference, January 2003

Roth, Philip C.; Arnold, Dorian C.; Miller, Barton P.
Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03
DOI: 10.1145/1048935.1050172

Improving the reliability of commodity operating systems
journal, February 2005

Swift, Michael M.; Bershad, Brian N.; Levy, Henry M.
ACM Transactions on Computer Systems, Vol. 23, Issue 1
DOI: 10.1145/1047915.1047919

Technology-Driven, Highly-Scalable Dragonfly Topology
journal, June 2008

Kim, John; Dally, Wiliam J.; Scott, Steve
ACM SIGARCH Computer Architecture News, Vol. 36, Issue 3
DOI: 10.1145/1394608.1382129

Improving the reliability of commodity operating systems
conference, January 2003

Swift, Michael M.; Bershad, Brian N.; Levy, Henry M.
Proceedings of the nineteenth ACM symposium on Operating systems principles - SOSP '03
DOI: 10.1145/945445.945466

Similar Records in DOE PAGES and OSTI.GOV collections:

Performance Characterization of a Hierarchical MPI Implementations on Large-scale Distributed-memory Platforms

Conference Alam, Sadaf R ; Barrett, Richard F ; Kuehn, Jeffery A ; ...

The building blocks of emerging Petascale massively parallel processing (MPP) systems are multi-core processors with four or more cores as a single processing element and a customized network interface. The resulting memory and communication hierarchy of these platforms are now exposed to application developers and end users by creating a hierarchical or multi-core aware message-passing (MPI) programming interface and by providing a handful of runtime, tunable parameters that allows mapping and control of MPI tasks and message handling. We characterize performance of MPI communication patterns and present strategies for optimizing applications performance on Cray XT series systems that are composedmore »« less
Compiled MPI: Cost-Effective Exascale Applications Development

Technical Report Bronevetsky, G ; Quinlan, D ; Lumsdaine, A ; ...

The complexity of petascale and exascale machines makes it increasingly difficult to develop applications that can take advantage of them. Future systems are expected to feature billion-way parallelism, complex heterogeneous compute nodes and poor availability of memory (Peter Kogge, 2008). This new challenge for application development is motivating a significant amount of research and development on new programming models and runtime systems designed to simplify large-scale application development. Unfortunately, DoE has significant multi-decadal investment in a large family of mission-critical scientific applications. Scaling these applications to exascale machines will require a significant investment that will dwarf the costs of hardwaremore »« less
https://doi.org/10.2172/1043665

Full Text Available
Characterizing MPI matching via trace-based simulation

Journal Article Ferreira, Kurt Brian ; Levy, Scott Larson Nicoll ; Pedretti, Kevin ; ... - Parallel Computing

With the increased scale expected on future leadership-class systems, detailed information about the resource usage and performance of MPI message matching provides important insights into how to maintain application performance on next-generation systems. However, obtaining MPI message matching performance data is often not possible without significant effort. A common approach is to instrument an MPI implementation to collect relevant statistics. While this approach can provide important data, collecting matching data at runtime perturbs the application's execution, including its matching performance, and is highly dependent on the MPI library's matchlist implementation. In this paper, we introduce a trace-based simulation approach tomore »« less
https://doi.org/10.1145/3127024.3127040

Full Text Available
Programming future architectures : dusty decks, memory walls, and the speed of light.

Conference Rodrigues, Arun F

Due to advances in CMOS fabrication technology, high performance computing capabilities have continually grown. More capable hardware has allowed a range of complex scientific applications to be developed. However, these applications present a bottleneck to future performance. Entrenched 'legacy' codes - 'Dusty Decks' - demand that new hardware must remain compatible with existing software. Additionally, conventional architectures faces increasing challenges. Many of these challenges revolve around the growing disparity between processor and memory speed - the 'Memory Wall' - and difficulties scaling to large numbers of parallel processors. To a large extent, these limitations are inherent to the traditional computermore »« less
Production Application Performance Data Streaming for System Monitoring

Journal Article Izadpanah, Ramin ; Allan, Benjamin A. ; Dechev, Damian ; ... - ACM Transactions on Modeling and Performance Evaluation of Computing Systems

In this article, we present an approach to streaming collection of application performance data. Practical application performance tuning and troubleshooting in production high-performance computing (HPC) environments requires an understanding of how applications interact with the platform, including (but not limited to) parallel programming libraries such as Message Passing Interface (MPI). Several profiling and tracing tools exist that collect heavy runtime data traces either in memory (released only at application exit) or on a file system (imposing an I/O load that may interfere with the performance being measured). Although these approaches are beneficial in development stages and post-run analysis, a systemwidemore »« less
https://doi.org/10.1145/3319498

Full Text Available

Similar Records

Title: On the memory attribution problem: A solution and case study using MPI

Abstract

Citation Formats

Valgrind: a framework for heavyweight dynamic binary instrumentation journal, June 2007

Memory registration caching correctness conference, January 2005

Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation book, January 2004

A uGNI-Based MPICH2 Nemesis Network Module for the Cray XE book, January 2011

DAS User Manual journal, January 2008

Technology-Driven, Highly-Scalable Dragonfly Topology conference, June 2008

A high-performance, portable implementation of the MPI message passing interface standard journal, September 1996

MRNet: A Software-Based Multicast/Reduction Network for Scalable Tools conference, January 2003

Improving the reliability of commodity operating systems journal, February 2005

Technology-Driven, Highly-Scalable Dragonfly Topology journal, June 2008

Improving the reliability of commodity operating systems conference, January 2003

Valgrind: a framework for heavyweight dynamic binary instrumentation
journal, June 2007

Memory registration caching correctness
conference, January 2005

Open MPI: Goals, Concept, and Design of a Next Generation MPI Implementation
book, January 2004

A uGNI-Based MPICH2 Nemesis Network Module for the Cray XE
book, January 2011

DAS User Manual
journal, January 2008

Technology-Driven, Highly-Scalable Dragonfly Topology
conference, June 2008

A high-performance, portable implementation of the MPI message passing interface standard
journal, September 1996

MRNet: A Software-Based Multicast/Reduction Network for Scalable Tools
conference, January 2003

Improving the reliability of commodity operating systems
journal, February 2005

Technology-Driven, Highly-Scalable Dragonfly Topology
journal, June 2008

Improving the reliability of commodity operating systems
conference, January 2003