skip to main content


Title: Tigres Workflow Library: Supporting Scientific Pipelines on HPC Systems

The growth in scientific data volumes has resulted in the need for new tools that enable users to operate on and analyze data on large-scale resources. In the last decade, a number of scientific workflow tools have emerged. These tools often target distributed environments, and often need expert help to compose and execute the workflows. Data-intensive workflows are often ad-hoc, they involve an iterative development process that includes users composing and testing their workflows on desktops, and scaling up to larger systems. In this paper, we present the design and implementation of Tigres, a workflow library that supports the iterative workflow development cycle of data-intensive workflows. Tigres provides an application programming interface to a set of programming templates i.e., sequence, parallel, split, merge, that can be used to compose and execute computational and data pipelines. We discuss the results of our evaluation of scientific and synthetic workflows showing Tigres performs with minimal template overheads (mean of 13 seconds over all experiments). We also discuss various factors (e.g., I/O performance, execution mechanisms) that affect the performance of scientific workflows on HPC systems.
 [1] ;  [1] ;  [1] ;  [1]
  1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Publication Date:
Grant/Contract Number:
Accepted Manuscript
Journal Name:
Proceedings - 2016 16th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2016
Additional Journal Information:
Conference: 2016 16. IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2016, Cartagena (Colombia), 16-19 May 2016
Research Org:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Country of Publication:
United States
97 MATHEMATICS AND COMPUTING; Monitoring; Programming; Libraries; Syntactics; Arrays; Pipelines; Collaboration; Data Analysis; Scientific Workflows; High Performance Computing
OSTI Identifier: