skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: The NetLogger Toolkit V2.0

Abstract

The NetLogger Toolkit is designed to monitor, under actual operating conditions, the behavior of all the elements of the application-to-application communication path in order to determine exactly where time is spent within a complex system Using NetLogger, distnbuted application components are modified to produce timestamped logs of "interesting" events at all the critical points of the distributed system Events from each component are correlated, which allov^ one to characterize the performance of all aspects of the system and network in detail. The NetLogger Toolkit itself consists of four components an API and library of functions to simplify the generation of application-level event logs, a set of tools for collecting and sorting log files, an event archive system, and a tool for visualization and analysis of the log files In order to instrument an application to produce event logs, the application developer inserts calls to the NetLogger API at all the critical points in the code, then links the application with the NetLogger library All the tools in the NetLogger Toolkit share a common log format, and assume the existence of accurate and synchronized system clocks NetLogger messages can be logged using an easy-to-read text based format based on the lETF-proposedmore » ULM format, or a binary format that can still be used through the same API but that is several times faster and smaller, with performance comparable or better than binary message formats such as MPI, XDR, SDDF-Binary, and PBIO. The NetLogger binary format is both highly efficient and self-describing, thus optimized for the dynamic message construction and parsing of application instrumentation. NetLogger includes an "activation" API that allows NetLogger logging to be turned on, off, or modified by changing an external file This IS useful for activating logging in daemons/services (e g GndFTP server). The NetLogger reliability API provides the ability to specify backup logging locations and penodically try to reconnect broken TCP pipe. A typical use for this is to store data on local disk while net is down. An event archiver can log one or more incoming NetLogger streams to a local disk file (netlogd) or to a mySQL database (netarchd). We have found exploratory, visual analysis of the log event data to be the most useful means of determining the causes of performance anomalies The NetLogger Visualization tool, niv, has been developed to provide a flexible and interactive graphical representation of system-level and application-level events.« less

Authors:
; ; ;
Publication Date:
Research Org.:
Lawrence Berkeley National Laboratory
Sponsoring Org.:
USDOE
OSTI Identifier:
1230986
Report Number(s):
NetLoggerV2.0; 002086MLTPL00
R&D Project: B&R #KJ0102000Project ID K52014; CR-1503 NetLogger
DOE Contract Number:  
AC03-76SF00098
Resource Type:
Software
Software Revision:
00
Software Package Number:
002086
Software Package Contents:
OPEN SOURCE SOFTWARE AVAILABLE FROM LAWRENCE BERKELEY NATIONAL LABORATORY AT: http://www-didc.lbl.gov/NetLogger/
Software CPU:
MLTPL
Open Source:
Yes
Source Code Available:
No
Related Software:
Tcl, Tk, BLT (for NLV); Python
Country of Publication:
United States

Citation Formats

Gunter, Dan, Lee, Jason, Stoufer, Martin, and Tierney, Brian. The NetLogger Toolkit V2.0. Computer software. https://www.osti.gov//servlets/purl/1230986. Vers. 00. USDOE. 28 Mar. 2003. Web.
Gunter, Dan, Lee, Jason, Stoufer, Martin, & Tierney, Brian. (2003, March 28). The NetLogger Toolkit V2.0 (Version 00) [Computer software]. https://www.osti.gov//servlets/purl/1230986.
Gunter, Dan, Lee, Jason, Stoufer, Martin, and Tierney, Brian. The NetLogger Toolkit V2.0. Computer software. Version 00. March 28, 2003. https://www.osti.gov//servlets/purl/1230986.
@misc{osti_1230986,
title = {The NetLogger Toolkit V2.0, Version 00},
author = {Gunter, Dan and Lee, Jason and Stoufer, Martin and Tierney, Brian},
abstractNote = {The NetLogger Toolkit is designed to monitor, under actual operating conditions, the behavior of all the elements of the application-to-application communication path in order to determine exactly where time is spent within a complex system Using NetLogger, distnbuted application components are modified to produce timestamped logs of "interesting" events at all the critical points of the distributed system Events from each component are correlated, which allov^ one to characterize the performance of all aspects of the system and network in detail. The NetLogger Toolkit itself consists of four components an API and library of functions to simplify the generation of application-level event logs, a set of tools for collecting and sorting log files, an event archive system, and a tool for visualization and analysis of the log files In order to instrument an application to produce event logs, the application developer inserts calls to the NetLogger API at all the critical points in the code, then links the application with the NetLogger library All the tools in the NetLogger Toolkit share a common log format, and assume the existence of accurate and synchronized system clocks NetLogger messages can be logged using an easy-to-read text based format based on the lETF-proposed ULM format, or a binary format that can still be used through the same API but that is several times faster and smaller, with performance comparable or better than binary message formats such as MPI, XDR, SDDF-Binary, and PBIO. The NetLogger binary format is both highly efficient and self-describing, thus optimized for the dynamic message construction and parsing of application instrumentation. NetLogger includes an "activation" API that allows NetLogger logging to be turned on, off, or modified by changing an external file This IS useful for activating logging in daemons/services (e g GndFTP server). The NetLogger reliability API provides the ability to specify backup logging locations and penodically try to reconnect broken TCP pipe. A typical use for this is to store data on local disk while net is down. An event archiver can log one or more incoming NetLogger streams to a local disk file (netlogd) or to a mySQL database (netarchd). We have found exploratory, visual analysis of the log event data to be the most useful means of determining the causes of performance anomalies The NetLogger Visualization tool, niv, has been developed to provide a flexible and interactive graphical representation of system-level and application-level events.},
url = {https://www.osti.gov//servlets/purl/1230986},
doi = {},
year = {2003},
month = {3},
note =
}

Software:
To order this software, request consultation services, or receive further information, please fill out the following request.

Save / Share:

To receive further information, fill out the request form below. OSTI staff will begin to process an order for scientific and technical software once the signed site license agreement is received. You may also reach us by email at: .

Software Request

(required)
(required)
(required)
(required)
(required)
(required)
(required)
(required)