GUIDE: A Scalable Information Directory Service to Collect, Federate, and Analyze Logs for Operational Insights into a Leadership HPC Facility

Conference · Wed Nov 01 04:00:00 EDT 2017

DOI:https://doi.org/10.1145/3126908.3126946· OSTI ID:1400206

Vazhkudai, Sudharshan ^[1]; Miller, Ross ^[1]; Tiwari, Devesh ^[2]; Zimmer, Christopher ^[1]; Wang, Feiyi ^[1]; Oral, Sarp ^[1]; Gunasekaran, Raghul ^[1]; Steinert, Deryl A. ^[1]

---

1. ORNL
2. Northeastern University, Boston

In this paper, we describe the GUIDE framework to collect, federate, and analyze log data from the Oak Ridge Leadership Computing Facility's (OLCF), and how we use it derive insights into center operations. We collectsystem logs and extract monitoring data at every level of the various OLCF subsystems, and have developed a suite of pre-processing tools to make the raw data consumable. The cleansed logs are then ingested and federated into a central, scalable data warehouse, Splunk, that offers storage, indexing, querying, and visualization capabilities. We have further developed and deployed a set of analytics tools to analyze these multiple disparate log streams in concert, and derive operational insights. We describe our experience from developing and deploying the GUIDE infrastructure, and deriving valuable insights on the various subsystems, based on two years of operations in the production OLCF environment.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1400206
• Country of Publication: United States
• Language: English