An evaluation of the state of time synchronization on leadership class supercomputers

Jones, Terry; Ostrouchov, George; Koenig, Gregory A.; Mondragon, Oscar H.; Bridges, Patrick G.

doi:10.1002/cpe.4341

An evaluation of the state of time synchronization on leadership class supercomputers

Journal Article · Mon Oct 09 00:00:00 EDT 2017 · Concurrency and Computation. Practice and Experience

DOI:https://doi.org/10.1002/cpe.4341· OSTI ID:1432152

^[1]; Ostrouchov, George ^[1]; Koenig, Gregory A. ^[1]; Mondragon, Oscar H. ^[2]; Bridges, Patrick G. ^[3]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Computer Science and Mathematics Division
Univ. Autonoma de Occidente, Cali (Colombia)
Univ. of New Mexico, Albuquerque, NM (United States). Dept. of Computer Science

We present a detailed examination of time agreement characteristics for nodes within extreme-scale parallel computers. Using a software tool we introduce in this paper, we quantify attributes of clock skew among nodes in three representative high-performance computers sited at three national laboratories. Our measurements detail the statistical properties of time agreement among nodes and how time agreement drifts over typical application execution durations. We discuss the implications of our measurements, why the current state of the field is inadequate, and propose strategies to address observed shortcomings.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

Grant/Contract Number:: AC05-00OR22725; AC02-06CH11357; AC02-05CH11231

OSTI ID:: 1432152

Journal Information:: Concurrency and Computation. Practice and Experience, Journal Name: Concurrency and Computation. Practice and Experience Journal Issue: 4 Vol. 30; ISSN 1532-0626

Publisher:: WileyCopyright Statement

Country of Publication:: United States

Language:: English

References (41)

Clock synchronization in high-end computing environments: a strategy for minimizing clock variance at runtime: CLOCK SYNCHRONIZATION IN HIGH-END COMPUTING ENVIRONMENTS Jones, Terry; Koenig, Gregory A. Concurrency and Computation: Practice and Experience, Vol. 25, Issue 6 https://doi.org/10.1002/cpe.2868	journal	July 2012
Time synchronization: pivotal element in cloud forensics: Time synchronization: pivotal element in cloud forensics Marangos, Nikolaos; Rizomiliotis, Panagiotis; Mitrou, Lilian Security and Communication Networks, Vol. 9, Issue 6 https://doi.org/10.1002/sec.1056	journal	June 2014
On Efficiently Implementing Global Time for Performance Evaluation on Multiprocessor Systems Maillet, E.; Tron, C. Journal of Parallel and Distributed Computing, Vol. 28, Issue 1 https://doi.org/10.1006/jpdc.1995.1090	journal	July 1995
Internal Timer Synchronization for Parallel Event Tracing Doleschal, Jens; Knüpfer, Andreas; Müller, Matthias S. Recent Advances in Parallel Virtual Machine and Message Passing Interface https://doi.org/10.1007/978-3-540-87475-1_29	book	January 2008
Clockscalpel: Understanding Root Causes of Internet Clock Synchronization Inaccuracy Hong, Chi-Yao; Lin, Chia-Chi; Caesar, Matthew Passive and Active Measurement https://doi.org/10.1007/978-3-642-19260-9_21	book	January 2011
Practical uses of synchronized clocks in distributed systems Liskov, Barbara Distributed Computing, Vol. 6, Issue 4 https://doi.org/10.1007/BF02242709	journal	July 1993
Gang scheduling performance benefits for fine-grain synchronization Feitelson, Dror G.; Rudolph, Larry Journal of Parallel and Distributed Computing, Vol. 16, Issue 4 https://doi.org/10.1016/0743-7315(92)90014-E	journal	December 1992
Extreme scale computing: Modeling the impact of system noise in multi-core clustered systems Seelam, Seetharami; Fong, Liana; Tantawi, Asser Journal of Parallel and Distributed Computing, Vol. 73, Issue 7 https://doi.org/10.1016/j.jpdc.2013.01.016	journal	July 2013
Transition due to base roughness in a dense granular flow down an inclined plane Kumaran, V.; Maheshwari, S. Physics of Fluids, Vol. 24, Issue 5 https://doi.org/10.1063/1.4710543	journal	May 2012
Logical time in distributed computing systems Fidge, C. Computer, Vol. 24, Issue 8 https://doi.org/10.1109/2.84874	journal	August 1991
Internet time synchronization: the network time protocol Mills, D. L. IEEE Transactions on Communications, Vol. 39, Issue 10 https://doi.org/10.1109/26.103043	journal	January 1991
Scheduling In-Situ Analytics in Next-Generation Applications Mondragon, Oscar H.; Bridges, Patrick G.; Levy, Scott 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) https://doi.org/10.1109/CCGrid.2016.42	conference	May 2016
Identifying sources of Operating System Jitter through fine-grained kernel instrumentation De, Pradipta; Kothari, Ravi; Mann, Vijay 2007 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTR.2007.4629247	conference	September 2007
Using correlated surprise to infer shared influence Oliner, Adam J.; Kulkarni, Ashutosh V.; Aiken, Alex 2010 IEEE/IFIP International Conference on Dependable Systems & Networks (DSN) https://doi.org/10.1109/DSN.2010.5544921	conference	June 2010
The Gemini System Interconnect Alverson, Robert; Roweth, Duncan; Kaplan, Larry 2010 IEEE 18th Annual Symposium on High-Performance Interconnects (HOTI), 2010 18th IEEE Symposium on High Performance Interconnects https://doi.org/10.1109/HOTI.2010.23	conference	August 2010
Replay-Based Synchronization of Timestamps in Event Traces of Massively Parallel Applications Becker, Daniel; Linford, John C.; Rabenseifner, Rolf 2008 International Conference on Parallel Processing Workshops (ICPP-W), 2008 International Conference on Parallel Processing - Workshops https://doi.org/10.1109/ICPP-W.2008.17	conference	September 2008
Serving time in the cloud: Why time-as-a-service? Mizrahi, Tal; Moses, Yoram IEEE INFOCOM 2016 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), 2016 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) https://doi.org/10.1109/INFCOMW.2016.7562052	conference	April 2016
Early Experience on the Blue Gene/Q Supercomputing System Morozov, Vitali; Kumaran, Kalyan; Vishwanath, Venkatram 2013 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), 2013 IEEE 27th International Symposium on Parallel and Distributed Processing https://doi.org/10.1109/IPDPS.2013.106	conference	May 2013
A Paradigm Change: From Performance Monitoring to Performance Analysis DeRose, Luiz; Poxon, Heidi 2009 21st International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) https://doi.org/10.1109/SBAC-PAD.2009.28	conference	October 2009
A Clock Synchronization Strategy for Minimizing Clock Variance at Runtime in High-End Computing Environments Jones, Terry; Koenig, Gregory A. 2010 22nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) https://doi.org/10.1109/SBAC-PAD.2010.33	conference	October 2010
Characterizing application sensitivity to OS interference using kernel-level noise injection Ferreira, Kurt B.; Bridges, Patrick; Brightwell, Ron 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2008.5219920	conference	November 2008
Characterizing the Influence of System Noise on Large-Scale Applications by Simulation Hoefler, Torsten; Schneider, Timo; Lumsdaine, Andrew 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.12	conference	November 2010
Cray Cascade: A scalable HPC system based on a Dragonfly network Faanes, Greg; Bataineh, Abdulla; Roweth, Duncan 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.39	conference	November 2012
Understanding Performance Interference in Next-Generation HPC Systems Mondragon, Oscar H.; Bridges, Patrick G.; Levy, Scott SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.32	conference	November 2016
On Maximum-Likelihood Estimation of Clock Offset Jeske, D. R. IEEE Transactions on Communications, Vol. 53, Issue 1 https://doi.org/10.1109/TCOMM.2004.840668	journal	January 2005
Network classless time protocol based on clock offset optimization Gurewitz, O.; Cidon, I.; Sidi, M. IEEE/ACM Transactions on Networking, Vol. 14, Issue 4 https://doi.org/10.1109/TNET.2006.880181	journal	August 2006
Robust Synchronization of Absolute and Difference Clocks Over Networks Veitch, D.; Ridoux, J.; Korada, S. B. IEEE/ACM Transactions on Networking, Vol. 17, Issue 2 https://doi.org/10.1109/TNET.2008.926505	journal	April 2009
The Case for Feed-Forward Clock Synchronization Ridoux, Julien; Veitch, Darryl; Broomhead, Timothy IEEE/ACM Transactions on Networking, Vol. 20, Issue 1 https://doi.org/10.1109/TNET.2011.2158443	journal	February 2012
Improving the Scalability of Parallel Jobs by adding Parallel Awareness to the Operating System Jones, Terry; Tomlinson, Paul; Roberts, Mark Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03 https://doi.org/10.1145/1048935.1050161	conference	January 2003
Technology-Driven, Highly-Scalable Dragonfly Topology Kim, John; Dally, Wiliam J.; Scott, Steve ACM SIGARCH Computer Architecture News, Vol. 36, Issue 3 https://doi.org/10.1145/1394608.1382129	journal	June 2008
Hobbes: composition and virtualization as the foundations of an extreme-scale OS/R Brightwell, Ron; Oldfield, Ron; Maccabe, Arthur B. Proceedings of the 3rd International Workshop on Runtime and Operating Systems for Supercomputers - ROSS '13 https://doi.org/10.1145/2491661.2481427	conference	January 2013
Spanner: Google’s Globally Distributed Database Corbett, James C.; Hochschild, Peter; Hsieh, Wilson ACM Transactions on Computer Systems, Vol. 31, Issue 3 https://doi.org/10.1145/2518037.2491245	journal	August 2013
Noise-Tolerant Explicit Stencil Computations for Nonuniform Process Execution Rates Hammouda, Adam; Siegel, Andrew R.; Siegel, Stephen F. ACM Transactions on Parallel Computing, Vol. 2, Issue 1 https://doi.org/10.1145/2742351	journal	April 2015
Quantifying Scheduling Challenges for Exascale System Software Mondragon, Oscar H.; Bridges, Patrick G.; Jones, Terry Proceedings of the 5th International Workshop on Runtime and Operating Systems for Supercomputers - ROSS '15 https://doi.org/10.1145/2768405.2768413	conference	January 2015
How I Learned to Stop Worrying and Love In Situ Analytics: Leveraging Latent Synchronization in MPI Collective Algorithms Levy, Scott; Ferreira, Kurt B.; Widener, Patrick Proceedings of the 23rd European MPI Users' Group Meeting on - EuroMPI 2016 https://doi.org/10.1145/2966884.2966920	conference	January 2016
Time, clocks, and the ordering of events in a distributed system Lamport, Leslie Communications of the ACM, Vol. 21, Issue 7 https://doi.org/10.1145/359545.359563	journal	July 1978
A bridging model for parallel computation Valiant, Leslie G. Communications of the ACM, Vol. 33, Issue 8 https://doi.org/10.1145/79173.79181	journal	August 1990
Fine-grained network time synchronization using reference broadcasts Elson, Jeremy; Girod, Lewis; Estrin, Deborah ACM SIGOPS Operating Systems Review, Vol. 36, Issue SI https://doi.org/10.1145/844128.844143	journal	December 2002
On the Accuracy and Stablility of Clocks Synchronized by the Network Time Protocol in the Internet System Mills, David L. ACM SIGCOMM Computer Communication Review, Vol. 20, Issue 1 https://doi.org/10.1145/86587.86591	journal	December 1989
Linux kernel co-scheduling and bulk synchronous parallelism Jones, Terry The International Journal of High Performance Computing Applications, Vol. 26, Issue 2 https://doi.org/10.1177/1094342011433523	journal	January 2012
Understanding and isolating the noise in the Linux kernel Akkan, Hakan; Lang, Michael; Liebrock, Lorie The International Journal of High Performance Computing Applications, Vol. 27, Issue 2 https://doi.org/10.1177/1094342013477892	journal	February 2013

Similar Records

A Clock Synchronization Strategy for Minimizing Clock Variance at Runtime in High-end Computing Environments

Conference · Thu Dec 31 23:00:00 EST 2009 · OSTI ID:1056896

Clock Synchronization in High-end Computing Environments: A Strategy for Minimizing Clock Variance at Runtime

Journal Article · Mon Dec 31 23:00:00 EST 2012 · Concurrency and Computation. Practice and Experience · OSTI ID:1105917

Clock Agreement Among Parallel Supercomputer Nodes

Dataset · Wed Apr 30 00:00:00 EDT 2014 · OSTI ID:1130048

Related Subjects

97 MATHEMATICS AND COMPUTING
clock synchronization
large-scale systems
system software
time service

An evaluation of the state of time synchronization on leadership class supercomputers

Citation Formats

References (41)

Similar Records

Related Subjects