skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Template Interfaces for Agile Parallel Data-Intensive Science

Abstract

Tigres provides a programming library to compose and execute large-scale data-intensive scientific workflows from desktops to supercomputers. DOE User Facilities and large science collaborations are increasingly generating large enough data sets that it is no longer practical to download them to a desktop to operate on them. They are instead stored at centralized compute and storage resources such as high performance computing (HPC) centers. Analysis of this data requires an ability to run on these facilities, but with current technologies, scaling an analysis to an HPC center and to a large data set is difficult even for experts. Tigres is addressing the challenge of enabling collaborative analysis of DOE Science data through a new concept of reusable "templates" that enable scientists to easily compose, run and manage collaborative computational tasks. These templates define common computation patterns used in analyzing a data set.

Authors:
; ; ; ; ; ; ; ;
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE
Contributing Org.:
Lawrence Berkeley National Laboratory
OSTI Identifier:
1353431
Report Number(s):
Tigres; 005260MLTPL00
R&D Project: KJ0404000; 2016-147
DOE Contract Number:
AC02-05CH11231
Resource Type:
Software
Software Revision:
00
Software Package Number:
005260
Software CPU:
MLTPL
Open Source:
Yes
Source Code Available:
Yes
Country of Publication:
United States

Citation Formats

Ramakrishnan, Lavanya, Gunter, Daniel, Pastorello, Gilerto Z., Hendrix, Valerie, Fox, James, Rodrigo Alvarez, Gonzalo P, Kushner, Gary, Rodriguez, Ryan, and Agarwal, Deborah A. Template Interfaces for Agile Parallel Data-Intensive Science. Computer software. https://www.osti.gov//servlets/purl/1353431. Vers. 00. USDOE. 28 Apr. 2017. Web.
Ramakrishnan, Lavanya, Gunter, Daniel, Pastorello, Gilerto Z., Hendrix, Valerie, Fox, James, Rodrigo Alvarez, Gonzalo P, Kushner, Gary, Rodriguez, Ryan, & Agarwal, Deborah A. (2017, April 28). Template Interfaces for Agile Parallel Data-Intensive Science (Version 00) [Computer software]. https://www.osti.gov//servlets/purl/1353431.
Ramakrishnan, Lavanya, Gunter, Daniel, Pastorello, Gilerto Z., Hendrix, Valerie, Fox, James, Rodrigo Alvarez, Gonzalo P, Kushner, Gary, Rodriguez, Ryan, and Agarwal, Deborah A. Template Interfaces for Agile Parallel Data-Intensive Science. Computer software. Version 00. April 28, 2017. https://www.osti.gov//servlets/purl/1353431.
@misc{osti_1353431,
title = {Template Interfaces for Agile Parallel Data-Intensive Science, Version 00},
author = {Ramakrishnan, Lavanya and Gunter, Daniel and Pastorello, Gilerto Z. and Hendrix, Valerie and Fox, James and Rodrigo Alvarez, Gonzalo P and Kushner, Gary and Rodriguez, Ryan and Agarwal, Deborah A.},
abstractNote = {Tigres provides a programming library to compose and execute large-scale data-intensive scientific workflows from desktops to supercomputers. DOE User Facilities and large science collaborations are increasingly generating large enough data sets that it is no longer practical to download them to a desktop to operate on them. They are instead stored at centralized compute and storage resources such as high performance computing (HPC) centers. Analysis of this data requires an ability to run on these facilities, but with current technologies, scaling an analysis to an HPC center and to a large data set is difficult even for experts. Tigres is addressing the challenge of enabling collaborative analysis of DOE Science data through a new concept of reusable "templates" that enable scientists to easily compose, run and manage collaborative computational tasks. These templates define common computation patterns used in analyzing a data set.},
url = {https://www.osti.gov//servlets/purl/1353431},
doi = {},
year = {Fri Apr 28 00:00:00 EDT 2017},
month = {Fri Apr 28 00:00:00 EDT 2017},
note =
}

Software:
To order this software, request consultation services, or receive further information, please fill out the following request.

Save / Share:
  • Software lifecycles are becoming an increasingly important issue for computational science and engineering (CSE) software. The process by which a piece of CSE software begins life as a set of research requirements and then matures into a trusted high-quality capability is both commonplace and extremely challenging. Although an implicit lifecycle is obviously being used in any effort, the challenges of this process - respecting the competing needs of research vs. production - cannot be overstated. Here we describe a proposal for a well-defined software lifecycle process based on modern Lean/Agile software engineering principles. What we propose is appropriate for manymore » CSE software projects that are initially heavily focused on research but also are expected to eventually produce usable high-quality capabilities. The model is related to TriBITS, a build, integration and testing system, which serves as a strong foundation for this lifecycle model, and aspects of this lifecycle model are ingrained in the TriBITS system. Here, we advocate three to four phases or maturity levels that address the appropriate handling of many issues associated with the transition from research to production software. The goals of this lifecycle model are to better communicate maturity levels with customers and to help to identify and promote Software Engineering (SE) practices that will help to improve productivity and produce better software. An important collection of software in this domain is Trilinos, which is used as the motivation and the initial target for this lifecycle model. However, many other related and similar CSE (and non-CSE) software projects can also make good use of this lifecycle model, especially those that use the TriBITS system. Indeed this lifecycle process, if followed, will enable large-scale sustainable integration of many complex CSE software efforts across several institutions.« less
  • Software lifecycles are becoming an increasingly important issue for computational science & engineering (CSE) software. The process by which a piece of CSE software begins life as a set of research requirements and then matures into a trusted high-quality capability is both commonplace and extremely challenging. Although an implicit lifecycle is obviously being used in any effort, the challenges of this process--respecting the competing needs of research vs. production--cannot be overstated. Here we describe a proposal for a well-defined software lifecycle process based on modern Lean/Agile software engineering principles. What we propose is appropriate for many CSE software projects thatmore » are initially heavily focused on research but also are expected to eventually produce usable high-quality capabilities. The model is related to TriBITS, a build, integration and testing system, which serves as a strong foundation for this lifecycle model, and aspects of this lifecycle model are ingrained in the TriBITS system. Indeed this lifecycle process, if followed, will enable large-scale sustainable integration of many complex CSE software efforts across several institutions.« less
  • The advancement in computing technology has enabled scientists to collect massive amounts of data, taking us a step closer to solving complex problems such as global climate change and uncovering the secrets hidden in genes. The exponential growth in the amount of data collected from experiments, measurements and observations, however, has created an urgent technical challenge. A talented group of computational scientists are leading the effort at PNNL to tackle the challenge through a major initiative on high-performance and data-intensive computing. PNNL’s data-intensive computing initiative will attempt to accelerate the creation of computational solutions to support the study of problemsmore » of national scope involving large amounts of data from very complex systems.« less
  • This article coincides with the release of "Data-Intensive Science," co-edited by Dr. Kerstin Kleese van Dam. In the piece, Dr. Kleese van Dam explains how data-intensive science has the potential to transform not only how we do science but how quickly we can translate scientific progress into complete solutions, policies, decisions and, ultimately, economic success. In the article, she states it is clear that nations that can most effectively transform tons of scientific data into actionable knowledge are going to be the leaders in the future of science and commerce and how creating the required new insights for complex challengesmore » cannot be done without effective collaboration. Because many science domains already are unable to explore all of the data they collect (or which is relevant to their research), progress in collaborative, data-intensive science is crucial toward unlocking the potential of big data.« less

To initiate an order for this software, request consultation services, or receive further information, fill out the request form below. You may also reach us by email at: .

OSTI staff will begin to process an order for scientific and technical software once the payment and signed site license agreement are received. If the forms are not in order, OSTI will contact you. No further action will be taken until all required information and/or payment is received. Orders are usually processed within three to five business days.

Software Request

(required)
(required)
(required)
(required)
(required)
(required)
(required)
(required)