When Workflow Management Systems and Logging Systems Meet: Analyzing Large-Scale Execution Traces

Conference · Thu Jul 31 04:00:00 EDT 2008

OSTI ID:1050631

Gunter, Daniel

This poster shows the benefits of integrating a workflow management system with logging and log mining capabilities. By combing two existing, mature technologies: Pegasus-WMS and Netlogger, we are able to efficiently process execution logs of earthquake science workflows consisting of hundreds of thousands to one million tasks. In particular we show results of processing logs of CyberShake, a workflow application running on the TeraGrid. Client-side tools allow scientists to quickly gather statistics about a workflow run and find out which tasks executed, where they were executed, what was their runtime, etc. These statistics can be used to understand the performance characteristics of a workflow and help tune the execution parameters of the workflow management system. This poster shows the scalability of the system presenting results of uploading task execution records into the system and by showing results of querying the system for overall workflow performance information.

• Research Organization: Ernest Orlando Lawrence Berkeley National Laboratory, Berkeley, CA (US)
• Sponsoring Organization: Computational Research Division
• DOE Contract Number: AC02-05CH11231
• OSTI ID: 1050631
• Report Number(s): LBNL-4777E
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
DATA BASE MANAGEMENT
DATA-FLOW PROCESSING
MANAGEMENT
PERFORMANCE
PROCESSING
STATISTICS
workflow logging netlogger pegasus