Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale

Park, Byung; Hukerikar, Saurabh; Adamson, Ryan; Engelmann, Christian

doi:10.1109/CLUSTER.2017.113

Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale

Conference · Fri Sep 01 00:00:00 EDT 2017

DOI:https://doi.org/10.1109/CLUSTER.2017.113· OSTI ID:1460236

^[1]; ^[1]; ^[1]; ^[1]

ORNL

Today's high-performance computing (HPC) systems are heavily instrumented, generating logs containing information about abnormal events, such as critical conditions, faults, errors and failures, system resource utilization, and about the resource usage of user applications. These logs, once fully analyzed and correlated, can produce detailed information about the system health, root causes of failures, and analyze an application's interactions with the system, providing valuable insights to domain scientists and system administrators. However, processing HPC logs requires a deep understanding of hardware and software components at multiple layers of the system stack. Moreover, most log data is unstructured and voluminous, making it more difficult for system users and administrators to manually inspect the data. With rapid increases in the scale and complexity of HPC systems, log data processing is becoming a big data challenge. This paper introduces a HPC log data analytics framework that is based on a distributed NoSQL database technology, which provides scalability and high availability, and the Apache Spark framework for rapid in-memory processing of the log data. The analytics framework enables the extraction of a range of information about the system so that system administrators and end users alike can obtain necessary insights for their specific needs. We describe our experience with using this framework to glean insights from the log data about system behavior from the Titan supercomputer at the Oak Ridge National Laboratory.

View Conference

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1460236

Country of Publication:: United States

Language:: English

References (14)

Correlation clustering Bansal, N.; Blum, A.; Chawla, S. The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings. https://doi.org/10.1109/SFCS.2002.1181947	conference	November 2002
Feng shui of supercomputer memory: positional effects in DRAM and SRAM faults Sridharan, Vilas; Stearley, Jon; DeBardeleben, Nathan Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13 https://doi.org/10.1145/2503210.2503257	conference	January 2013
OVIS: a tool for intelligent, real-time monitoring of computational clusters Brandt, J. M.; Gentile, A. C.; Hale, D. J. Proceedings 20th IEEE International Parallel & Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2006.1639698	conference	January 2006
What Supercomputers Say: A Study of Five System Logs Oliner, Adam; Stearley, Jon 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07) https://doi.org/10.1109/DSN.2007.103	conference	June 2007
Cosmic rays don't strike twice: understanding the nature of DRAM errors and the implications for system design Hwang, Andy A.; Stefanovici, Ioan A.; Schroeder, Bianca Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS '12 https://doi.org/10.1145/2150976.2150989	conference	January 2012
Improving Log-based Field Failure Data Analysis of multi-node computing systems Pecchia, Antonio; Cotroneo, Domenico; Kalbarczyk, Zbigniew 2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks (DSN) https://doi.org/10.1109/DSN.2011.5958210	conference	June 2011
Fault prediction under the microscope: A closer look into HPC systems Gainaru, Ana; Cappello, Franck; Snir, Marc 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.57	conference	November 2012
Assessing time coalescence techniques for the analysis of supercomputer logs Di Martino, Catello; Cinque, Marcello; Cotroneo, Domenico IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2012) https://doi.org/10.1109/DSN.2012.6263946	conference	June 2012
Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility Tiwari, Devesh; Gupta, Saurabh; Gallarno, George Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807666	conference	January 2015
A Large-Scale Study of Failures in High-Performance Computing Systems Schroeder, Bianca; Gibson, Garth A. IEEE Transactions on Dependable and Secure Computing, Vol. 7, Issue 4 https://doi.org/10.1109/TDSC.2009.4	journal	October 2010
DRAM errors in the wild: a large-scale field study Schroeder, Bianca; Pinheiro, Eduardo; Weber, Wolf-Dietrich Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems - SIGMETRICS '09 https://doi.org/10.1145/1555349.1555372	conference	January 2009
The ganglia distributed monitoring system: design, implementation, and experience Massie, Matthew L.; Chun, Brent N.; Culler, David E. Parallel Computing, Vol. 30, Issue 7 https://doi.org/10.1016/j.parco.2004.04.001	journal	July 2004
Induction of decision trees Quinlan, J. R. Machine Learning, Vol. 1, Issue 1, p. 81-106 https://doi.org/10.1007/BF00116251	journal	March 1986
Lessons Learned from the Analysis of System Failures at Petascale: The Case of Blue Waters Martino, Catello Di; Kalbarczyk, Zbigniew; Iyer, Ravishankar K. 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) https://doi.org/10.1109/DSN.2014.62	conference	June 2014

Similar Records

A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log

Conference · Sat Sep 01 00:00:00 EDT 2018 · 2018 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING (CLUSTER) · OSTI ID:1567487

A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log

Conference · Thu Nov 01 00:00:00 EDT 2018 · OSTI ID:1570137

Deactivation and decommissioning web log analysis using big data technology - 15710

Conference · Wed Jul 01 00:00:00 EDT 2015 · OSTI ID:22824525

Big Data Meets HPC Log Analytics: Scalable Approach to Understanding Systems at Extreme Scale

Citation Formats

References (14)

Similar Records

Related Subjects