Hardware MPI message matching: Insights into MPI matching behavior to inform design: Hardware MPI message matching

Ferreira, Kurt; Grant, Ryan E.; Levenhagen, Michael J.; Levy, Scott; Groves, Taylor

doi:10.1002/cpe.5150

Title: Hardware MPI message matching: Insights into MPI matching behavior to inform design: Hardware MPI message matching

Abstract

Here, this paper explores key differences of MPI match lists for several important United States Department of Energy (DOE) applications and proxy applications. This understanding is critical in determining the most promising hardware matching design for any given high-speed network. The results of MPI match list studies for the major open-source MPI implementations, MPICH and Open MPI, are presented, and we modify an MPI simulator, LogGOPSim, to provide match list statistics. These results are discussed in the context of several different potential design approaches to MPI matching–capable hardware. The data illustrate the requirements for different hardware designs in terms of performance and memory capacity. Finally, this paper's contributions are the collection and analysis of data to help inform hardware designers of common MPI requirements and highlight the difficulties in determining these requirements by only examining a single MPI implementation.

Authors:

Ferreira, Kurt ^[1]; Grant, Ryan E. ^[1]; Levenhagen, Michael J. ^[1];

^[1]; Groves, Taylor ^[2]

Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Publication Date:: Wed Feb 27 00:00:00 EST 2019

Research Org.:: Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1501630

Report Number(s):: SAND-2019-0943J
Journal ID: ISSN 1532-0626; 671923

Grant/Contract Number:: AC04-94AL85000; NA0003525; AC02‐05CH11231

Resource Type:: Accepted Manuscript

Journal Name:: Concurrency and Computation. Practice and Experience

Additional Journal Information:: Journal Volume: 32; Journal Issue: 3; Journal ID: ISSN 1532-0626

Publisher:: Wiley

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; hardware matching; MPI; MPI matching

Citation Formats


                    Ferreira, Kurt, Grant, Ryan E., Levenhagen, Michael J., Levy, Scott, and Groves, Taylor. Hardware MPI message matching: Insights into MPI matching behavior to inform design: Hardware MPI message matching.  United States: N. p., 2019. 
Web.  doi:10.1002/cpe.5150.

Copy to clipboard


                    Ferreira, Kurt, Grant, Ryan E., Levenhagen, Michael J., Levy, Scott, & Groves, Taylor. Hardware MPI message matching: Insights into MPI matching behavior to inform design: Hardware MPI message matching.  United States.  https://doi.org/10.1002/cpe.5150

Copy to clipboard


                    Ferreira, Kurt, Grant, Ryan E., Levenhagen, Michael J., Levy, Scott, and Groves, Taylor. Wed .  
"Hardware MPI message matching: Insights into MPI matching behavior to inform design: Hardware MPI message matching".  United States.  https://doi.org/10.1002/cpe.5150.  https://www.osti.gov/servlets/purl/1501630.

Copy to clipboard


                    
@article{osti_1501630,

  title        = {Hardware MPI message matching: Insights into MPI matching behavior to inform design: Hardware MPI message matching},

  author       = {Ferreira, Kurt and Grant, Ryan E. and Levenhagen, Michael J. and Levy, Scott and Groves, Taylor},

  abstractNote = {Here, this paper explores key differences of MPI match lists for several important United States Department of Energy (DOE) applications and proxy applications. This understanding is critical in determining the most promising hardware matching design for any given high-speed network. The results of MPI match list studies for the major open-source MPI implementations, MPICH and Open MPI, are presented, and we modify an MPI simulator, LogGOPSim, to provide match list statistics. These results are discussed in the context of several different potential design approaches to MPI matching–capable hardware. The data illustrate the requirements for different hardware designs in terms of performance and memory capacity. Finally, this paper's contributions are the collection and analysis of data to help inform hardware designers of common MPI requirements and highlight the difficulties in determining these requirements by only examining a single MPI implementation.},

  doi          = {10.1002/cpe.5150},

  journal      = {Concurrency and Computation. Practice and Experience},

  number       = 3,

  volume       = 32,

  place        = {United States},

  year         = {Wed Feb 27 00:00:00 EST 2019},

  month        = {Wed Feb 27 00:00:00 EST 2019}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1002/cpe.5150

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 6 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

An architecture to perform NIC based MPI matching
conference, September 2007

Hemmert, K. Scott; Underwood, Keith D.; Rodrigues, Arun
2007 IEEE International Conference on Cluster Computing (CLUSTER)
DOI: 10.1109/CLUSTR.2007.4629234

A Dedicated Message Matching Mechanism for Collective Communications
conference, January 2018

Ghazimirsaeed, S. Mahdieh; Grant, Ryan E.; Afsahi, Ahmad
Proceedings of the 47th International Conference on Parallel Processing Companion - ICPP '18
DOI: 10.1145/3229710.3229712

Improving MPI Multi-threaded RMA Communication Performance
conference, January 2018

Hjelm, Nathan; Dosanjh, Matthew G. F.; Grant, Ryan E.
Proceedings of the 47th International Conference on Parallel Processing - ICPP 2018
DOI: 10.1145/3225058.3225114

The Case for Semi-Permanent Cache Occupancy: Understanding the Impact of Data Locality on Network Processing
conference, January 2018

Dosanjh, Matthew G. F.; Ghazimirsaeed, S. Mahdieh; Grant, Ryan E.
Proceedings of the 47th International Conference on Parallel Processing - ICPP 2018
DOI: 10.1145/3225058.3225130

Early Experiences Co-Scheduling Work and Communication Tasks for Hybrid MPI+X Applications
conference, November 2014

Stark, Dylan T.; Barrett, Richard F.; Grant, Ryan E.
2014 Workshop on Exascale MPI at Supercomputing Conference (ExaMPI)
DOI: 10.1109/ExaMPI.2014.6

An evaluation of MPI message rate on hybrid-core processors
journal, November 2014

Barrett, Brian W.; Brightwell, Ron; Grant, Ryan
The International Journal of High Performance Computing Applications, Vol. 28, Issue 4
DOI: 10.1177/1094342014552085

Re-evaluating Network Onload vs. Offload for the Many-Core Era
conference, September 2015

Dosanjh, Matthew G. F.; Grant, Ryan E.; Bridges, Patrick G.
2015 IEEE International Conference on Cluster Computing (CLUSTER)
DOI: 10.1109/CLUSTER.2015.55

Myrinet: a gigabit-per-second local area network
journal, January 1995

Boden, N. J.; Cohen, D.; Felderman, R. E.
IEEE Micro, Vol. 15, Issue 1
DOI: 10.1109/40.342015

Performance of particle in cell methods on highly concurrent computational architectures
journal, July 2007

Adams, M. F.; Ethier, S.; Wichmann, N.
Journal of Physics: Conference Series, Vol. 78
DOI: 10.1088/1742-6596/78/1/012001

Eliminating contention bottlenecks in multithreaded MPI
journal, November 2017

Dang, Hoang-Vu; Snir, Marc; Gropp, William
Parallel Computing, Vol. 69
DOI: 10.1016/j.parco.2017.08.003

Enabling communication concurrency through flexible MPI endpoints
journal, September 2014

Dinan, James; Grant, Ryan E.; Balaji, Pavan
The International Journal of High Performance Computing Applications, Vol. 28, Issue 4
DOI: 10.1177/1094342014548772

Characterizing MPI matching via trace-based simulation
conference, January 2017

Ferreira, Kurt B.; Levy, Scott; Pedretti, Kevin
Proceedings of the 24th European MPI Users' Group Meeting on - EuroMPI '17
DOI: 10.1145/3127024.3127040

Understanding Performance Interference in Next-Generation HPC Systems
conference, November 2016

Mondragon, Oscar H.; Bridges, Patrick G.; Levy, Scott
SC16: International Conference for High Performance Computing, Networking, Storage and Analysis
DOI: 10.1109/SC.2016.32

LogGOPSim: simulating large-scale applications in the LogGOPS model
conference, January 2010

Hoefler, Torsten; Schneider, Timo; Lumsdaine, Andrew
Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10
DOI: 10.1145/1851476.1851564

Fast Parallel Algorithms for Short-Range Molecular Dynamics
journal, March 1995

Plimpton, Steve
Journal of Computational Physics, Vol. 117, Issue 1
DOI: 10.1006/jcph.1995.1039

Characterizing MPI matching via trace-based simulation
journal, September 2018

Ferreira, Kurt B.; Levy, Scott; Pedretti, Kevin
Parallel Computing, Vol. 77
DOI: 10.1016/j.parco.2018.05.005

A high-performance, portable implementation of the MPI message passing interface standard
journal, September 1996

Gropp, William; Lusk, Ewing; Doss, Nathan
Parallel Computing, Vol. 22, Issue 6
DOI: 10.1016/0167-8191(96)00024-5

Why is MPI so slow?: analyzing the fundamental limits in implementing MPI-3.1
conference, January 2017

Raffenetti, Ken; Blocksome, Michael; Si, Min
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17
DOI: 10.1145/3126908.3126963

Characterizing the Influence of System Noise on Large-Scale Applications by Simulation
conference, November 2010

Hoefler, Torsten; Schneider, Timo; Lumsdaine, Andrew
2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
DOI: 10.1109/SC.2010.12

The BXI Interconnect Architecture
conference, August 2015

Derradji, Said; Palfer-Sollier, Thibaut; Panziera, Jean-Pierre
2015 IEEE 23rd Annual Symposium on High-Performance Interconnects (HOTI)
DOI: 10.1109/HOTI.2015.15

Instrumentation and Analysis of MPI Queue Times on the SeaStar High-Performance Network
conference, August 2008

Brightwell, R.; Pedretti, K.; Ferreira, K.
17th International Conference on Computer Communications and Networks 2008, 2008 Proceedings of 17th International Conference on Computer Communications and Networks
DOI: 10.1109/ICCCN.2008.ECP.116

Preparing for exascale: modeling MPI for many-core systems using fine-grain queues
conference, January 2015

Bridges, Patrick G.; Dosanjh, Matthew G. F.; Grant, Ryan
Proceedings of the 3rd Workshop on Exascale MPI - ExaMPI '15
DOI: 10.1145/2831129.2831134

The impact of MPI queue usage on message latency
conference, January 2004

Underwood, K. D.; Brightwell, R.
International Conference on Parallel Processing, 2004. ICPP 2004.
DOI: 10.1109/ICPP.2004.1327915

SeaStar Interconnect: Balanced Bandwidth for Scalable Performance
journal, May 2006

Brightwell, R.; Pedretti, K. T.; Underwood, K. D.
IEEE Micro, Vol. 26, Issue 3
DOI: 10.1109/MM.2006.65

How I Learned to Stop Worrying and Love In Situ Analytics: Leveraging Latent Synchronization in MPI Collective Algorithms
conference, January 2016

Levy, Scott; Ferreira, Kurt B.; Widener, Patrick
Proceedings of the 23rd European MPI Users' Group Meeting on - EuroMPI 2016
DOI: 10.1145/2966884.2966920

The Quadrics network: high-performance clustering technology
journal, January 2002

Petrini, F.; Hoisie, A.
IEEE Micro, Vol. 22, Issue 1
DOI: 10.1109/40.988689

A fast and resource-conscious MPI message queue mechanism for large-scale jobs
journal, January 2014

Zounmevo, Judicael A.; Afsahi, Ahmad
Future Generation Computer Systems, Vol. 30
DOI: 10.1016/j.future.2013.07.003

Protocols for Fully Offloaded Collective Operations on Accelerated Network Adapters
conference, October 2013

Schneider, Timo; Hoefler, Torsten; Grant, Ryan E.
2013 42nd International Conference on Parallel Processing (ICPP)
DOI: 10.1109/ICPP.2013.73

Toward an evolutionary task parallel integrated MPI + X programming model
conference, January 2015

Barrett, Richard F.; Stark, Dylan T.; Vaughan, Courtenay T.
Proceedings of the Sixth International Workshop on Programming Models and Applications for Multicores and Manycores - PMAM '15
DOI: 10.1145/2712386.2712388

sPIN: High-performance streaming Processing In the Network
conference, November 2017

Hoefler, Torsten; Di Girolamo, Salvatore; Taranov, Konstantin
SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
DOI: 10.1145/3126908.3126970

Works referencing / citing this record:

Foreword to the Special Issue of the Workshop on Exascale MPI (ExaMPI 2017)
journal, July 2019

Skjellum, Anthony; Bangalore, Purushotham V.; Grant, Ryan E.
Concurrency and Computation: Practice and Experience, Vol. 32, Issue 3
DOI: 10.1002/cpe.5459

Performance drop at executing communication-intensive parallel algorithms
journal, January 2020

Moríñigo, José A.; García-Muller, Pablo; Rubio-Montero, Antonio J.
The Journal of Supercomputing, Vol. 76, Issue 9
DOI: 10.1007/s11227-019-03142-8

Similar Records in DOE PAGES and OSTI.GOV collections:

Using Simulation to Examine the Effect of MPI Message Matching Costs on Application Performance

Journal Article Levy, Scott ; Ferreira, Kurt B. ; Schonbein, Whit ; ... - Parallel Computing

Attaining high performance with MPI applications requires efficient message matching to minimize message processing overheads and the latency these overheads introduce into application communication. We use a validated simulation-based approach to examine the relationship between MPI message matching performance and application time-to-solution. Specifically, we examine how the performance of several important HPC workloads is affected by the time required for matching. Our analysis yields several important contributions: (i) the performance of current workloads is unlikely to be significantly affected by MPI matching unless match queue operations get much slower or match queues get much longer; (ii) match queue designs thatmore »« less
Cited by 1
https://doi.org/10.1016/j.parco.2019.02.008

Full Text Available
A high-performance, portable implementation of the MPI message passing interface standard.

Journal Article Gropp, W ; Lusk, E ; Doss, N ; ... - Parallel Comput.

MPI (Message Passing Interface) is a specification for a standard library for message passing that was defined by the MPI Forum, a broadly based group of parallel computer vendors, library writers, and applications specialists. Multiple implementations of MPI have been developed. In this paper, we describe MPICH, unique among existing implementations in its design goal of combining portability with high performance. We document its portability and performance and describe the architecture by which these features are simultaneously achieved. We also discuss the set of tools that accompany the free distribution of MPICH, which constitute the beginnings of a portable parallelmore »« less
https://doi.org/10.1016/0167-8191(96)00024-5
A grid-enabled MPI : message passing in heterogeneous distributed computing systems.

Conference Foster, I ; Karonis, N T

Application development for high-performance distributed computing systems, or computational grids as they are sometimes called, requires grid-enabled tools that hide mundate aspects of the heterogeneous grid environment without compromising performance. As part of an investigation of these issues, they have developed MPICH-G, a grid-enabled implementation of the Message Passing Interface (MPI) that allows a user to run MPI programs across multiple computers at different sites using the same commands that would be used on a parallel computer. This library extends the Argonne MPICH implementation of MPI to use services provided by the globus grid toolkit. In this paper, they describemore »« less
Full Text Available
Characterizing MPI matching via trace-based simulation

Journal Article Ferreira, Kurt Brian ; Levy, Scott Larson Nicoll ; Pedretti, Kevin ; ... - Parallel Computing

With the increased scale expected on future leadership-class systems, detailed information about the resource usage and performance of MPI message matching provides important insights into how to maintain application performance on next-generation systems. However, obtaining MPI message matching performance data is often not possible without significant effort. A common approach is to instrument an MPI implementation to collect relevant statistics. While this approach can provide important data, collecting matching data at runtime perturbs the application's execution, including its matching performance, and is highly dependent on the MPI library's matchlist implementation. In this paper, we introduce a trace-based simulation approach tomore »« less
https://doi.org/10.1145/3127024.3127040

Full Text Available
A standard interface for debugger access to message queue information in MPI.

Conference Cownie, J ; Gropp, W

This paper discusses the design and implementation of an interface that allows a debugger to obtain the information necessary to display the contents of the MPI message queues. The design has been implemented in the TotalView debugger, and dynamic libraries that conform to the interface exist for MPICH, as well as the proprietary MPI implementations from Compaq, IBM, and SGI.
Full Text Available

Similar Records

Title: Hardware MPI message matching: Insights into MPI matching behavior to inform design: Hardware MPI message matching

Abstract

Citation Formats

An architecture to perform NIC based MPI matching conference, September 2007

A Dedicated Message Matching Mechanism for Collective Communications conference, January 2018

Improving MPI Multi-threaded RMA Communication Performance conference, January 2018

The Case for Semi-Permanent Cache Occupancy: Understanding the Impact of Data Locality on Network Processing conference, January 2018

Early Experiences Co-Scheduling Work and Communication Tasks for Hybrid MPI+X Applications conference, November 2014

An evaluation of MPI message rate on hybrid-core processors journal, November 2014

Re-evaluating Network Onload vs. Offload for the Many-Core Era conference, September 2015

Myrinet: a gigabit-per-second local area network journal, January 1995

Performance of particle in cell methods on highly concurrent computational architectures journal, July 2007

Eliminating contention bottlenecks in multithreaded MPI journal, November 2017

Enabling communication concurrency through flexible MPI endpoints journal, September 2014

Characterizing MPI matching via trace-based simulation conference, January 2017

Understanding Performance Interference in Next-Generation HPC Systems conference, November 2016

LogGOPSim: simulating large-scale applications in the LogGOPS model conference, January 2010

Fast Parallel Algorithms for Short-Range Molecular Dynamics journal, March 1995

Characterizing MPI matching via trace-based simulation journal, September 2018

A high-performance, portable implementation of the MPI message passing interface standard journal, September 1996

Why is MPI so slow?: analyzing the fundamental limits in implementing MPI-3.1 conference, January 2017

Characterizing the Influence of System Noise on Large-Scale Applications by Simulation conference, November 2010

The BXI Interconnect Architecture conference, August 2015

Instrumentation and Analysis of MPI Queue Times on the SeaStar High-Performance Network conference, August 2008

Preparing for exascale: modeling MPI for many-core systems using fine-grain queues conference, January 2015

The impact of MPI queue usage on message latency conference, January 2004

SeaStar Interconnect: Balanced Bandwidth for Scalable Performance journal, May 2006

How I Learned to Stop Worrying and Love In Situ Analytics: Leveraging Latent Synchronization in MPI Collective Algorithms conference, January 2016

The Quadrics network: high-performance clustering technology journal, January 2002

A fast and resource-conscious MPI message queue mechanism for large-scale jobs journal, January 2014

Protocols for Fully Offloaded Collective Operations on Accelerated Network Adapters conference, October 2013

Toward an evolutionary task parallel integrated MPI + X programming model conference, January 2015

sPIN: High-performance streaming Processing In the Network conference, November 2017

Foreword to the Special Issue of the Workshop on Exascale MPI (ExaMPI 2017) journal, July 2019

Performance drop at executing communication-intensive parallel algorithms journal, January 2020

An architecture to perform NIC based MPI matching
conference, September 2007

A Dedicated Message Matching Mechanism for Collective Communications
conference, January 2018

Improving MPI Multi-threaded RMA Communication Performance
conference, January 2018

The Case for Semi-Permanent Cache Occupancy: Understanding the Impact of Data Locality on Network Processing
conference, January 2018

Early Experiences Co-Scheduling Work and Communication Tasks for Hybrid MPI+X Applications
conference, November 2014

An evaluation of MPI message rate on hybrid-core processors
journal, November 2014

Re-evaluating Network Onload vs. Offload for the Many-Core Era
conference, September 2015

Myrinet: a gigabit-per-second local area network
journal, January 1995

Performance of particle in cell methods on highly concurrent computational architectures
journal, July 2007

Eliminating contention bottlenecks in multithreaded MPI
journal, November 2017

Enabling communication concurrency through flexible MPI endpoints
journal, September 2014

Characterizing MPI matching via trace-based simulation
conference, January 2017

Understanding Performance Interference in Next-Generation HPC Systems
conference, November 2016

LogGOPSim: simulating large-scale applications in the LogGOPS model
conference, January 2010

Fast Parallel Algorithms for Short-Range Molecular Dynamics
journal, March 1995

Characterizing MPI matching via trace-based simulation
journal, September 2018

A high-performance, portable implementation of the MPI message passing interface standard
journal, September 1996

Why is MPI so slow?: analyzing the fundamental limits in implementing MPI-3.1
conference, January 2017

Characterizing the Influence of System Noise on Large-Scale Applications by Simulation
conference, November 2010

The BXI Interconnect Architecture
conference, August 2015

Instrumentation and Analysis of MPI Queue Times on the SeaStar High-Performance Network
conference, August 2008

Preparing for exascale: modeling MPI for many-core systems using fine-grain queues
conference, January 2015

The impact of MPI queue usage on message latency
conference, January 2004

SeaStar Interconnect: Balanced Bandwidth for Scalable Performance
journal, May 2006

How I Learned to Stop Worrying and Love In Situ Analytics: Leveraging Latent Synchronization in MPI Collective Algorithms
conference, January 2016

The Quadrics network: high-performance clustering technology
journal, January 2002

A fast and resource-conscious MPI message queue mechanism for large-scale jobs
journal, January 2014

Protocols for Fully Offloaded Collective Operations on Accelerated Network Adapters
conference, October 2013

Toward an evolutionary task parallel integrated MPI + X programming model
conference, January 2015

sPIN: High-performance streaming Processing In the Network
conference, November 2017

Foreword to the Special Issue of the Workshop on Exascale MPI (ExaMPI 2017)
journal, July 2019

Performance drop at executing communication-intensive parallel algorithms
journal, January 2020