Large-Scale System Monitoring Experiences and Recommendations
MPI benchmarks are used for analyzing or tuning the performance of MPI libraries. Generally, every MPI library should be adjusted to the given parallel machine, especially on supercomputers. System operators can define which algorithm should be selected for a specific MPI operation, and this decision which algorithm to select is usually made after analyzing bench-mark results. The problem is that the latency of communication operations in MPI is very sensitive to the chosen data acquisition and data processing method. For that reason, depending on how the performance is measured, system operators may end up with a completely different MPI library setup. In the present work, we focus on the problem of precisely measuring the latency of collective operations, in particular, for small payloads, where external experimental factors play a significant role. We present a novel clock synchronization algorithm, which exploits the hierarchical architecture of compute clusters, and we show that it outperforms previous approaches, both in run-time and in precision. We also propose a different scheme to obtain precise MPI run-time measurements (called Round-Time), which is based on given, fixed time slices, as opposed to the traditional way of measuring for a predefined number of repetitions. We also highlight that the use of MPI_Barrier has a significant effect on experimentally determined latency values of MPI collectives. We argue that MPI_Barrier should be avoided if the average run-time of the barrier function is in the same order of magnitude as the run-time of the MPI function to be measured.
- Research Organization:
- Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Univ. of California, Oakland, CA (United States); Argonne National Lab. (ANL), Argonne, IL (United States); UT-Battelle LLC/ORNL, Oak Ridge, TN (Unted States); Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
- Sponsoring Organization:
- USDOE Office of Science; USDOE
- DOE Contract Number:
- AC02-05CH11231; AC02-06CH11357; AC05-00OR22725; NA0003525
- OSTI ID:
- 1567486
- Journal Information:
- 2018 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING (CLUSTER), Journal Name: 2018 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING (CLUSTER); ISSN 1552-5244
- Country of Publication:
- United States
- Language:
- English
Similar Records
A framework for characterizing overlap of communication and computation in parallel applications.
Characterizing Computation-Communication Overlap in Message-Passing Systems