skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Build and Execute Environment

Abstract

At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system.more » Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.« less

Authors:
 [1]
  1. LANL
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
Contributing Org.:
Los Alamos National Laboratory (LANL)
OSTI Identifier:
1356983
Report Number(s):
BEE; 005280WKSTN00
C17056
DOE Contract Number:
AC52-06NA25396
Resource Type:
Software
Software Revision:
00
Software Package Number:
005280
Software CPU:
WKSTN
Open Source:
Yes
Open Source under the BSD License.
Source Code Available:
Yes
Related Software:
KVM or QEMU
Country of Publication:
United States

Citation Formats

Guan, Qiang. Build and Execute Environment. Computer software. https://www.osti.gov//servlets/purl/1356983. Vers. 00. USDOE. 21 Apr. 2017. Web.
Guan, Qiang. (2017, April 21). Build and Execute Environment (Version 00) [Computer software]. https://www.osti.gov//servlets/purl/1356983.
Guan, Qiang. Build and Execute Environment. Computer software. Version 00. April 21, 2017. https://www.osti.gov//servlets/purl/1356983.
@misc{osti_1356983,
title = {Build and Execute Environment, Version 00},
author = {Guan, Qiang},
abstractNote = {At exascale, the challenge becomes to develop applications that run at scale and use exascale platforms reliably, efficiently, and flexibly. Workflows become much more complex because they must seamlessly integrate simulation and data analytics. They must include down-sampling, post-processing, feature extraction, and visualization. Power and data transfer limitations require these analysis tasks to be run in-situ or in-transit. We expect successful workflows will comprise multiple linked simulations along with tens of analysis routines. Users will have limited development time at scale and, therefore, must have rich tools to develop, debug, test, and deploy applications. At this scale, successful workflows will compose linked computations from an assortment of reliable, well-defined computation elements, ones that can come and go as required, based on the needs of the workflow over time. We propose a novel framework that utilizes both virtual machines (VMs) and software containers to create a workflow system that establishes a uniform build and execution environment (BEE) beyond the capabilities of current systems. In this environment, applications will run reliably and repeatably across heterogeneous hardware and software. Containers, both commercial (Docker and Rocket) and open-source (LXC and LXD), define a runtime that isolates all software dependencies from the machine operating system. Workflows may contain multiple containers that run different operating systems, different software, and even different versions of the same software. We will run containers in open-source virtual machines (KVM) and emulators (QEMU) so that workflows run on any machine entirely in user-space. On this platform of containers and virtual machines, we will deliver workflow software that provides services, including repeatable execution, provenance, checkpointing, and future proofing. We will capture provenance about how containers were launched and how they interact to annotate workflows for repeatable and partial re-execution. We will coordinate the physical snapshots of virtual machines with parallel programming constructs, such as barriers, to automate checkpoint and restart. We will also integrate with HPC-specific container runtimes to gain access to accelerators and other specialized hardware to preserve native performance. Containers will link development to continuous integration. When application developers check code in, it will automatically be tested on a suite of different software and hardware architectures.},
url = {https://www.osti.gov//servlets/purl/1356983},
doi = {},
year = {Fri Apr 21 00:00:00 EDT 2017},
month = {Fri Apr 21 00:00:00 EDT 2017},
note =
}

Software:
To order this software, request consultation services, or receive further information, please fill out the following request.

Save / Share:
  • We describe our experience in using a software bus architectureto build a fault location system from reusable software parts. The system consists of plug compatible'' software components connected by a software bus. This architecture makes it possible to assemble the system either as a collection of cooperating distributed processes, or as a single program running on a single computer. Each component is useful in its own right, making reuse probable, and inviting commercialization. Development effort is significantly higher using this technique. However, the benefits (in our experience) far outweigh the costs. Experience in using the system is described in amore » poster session in this conference.« less
  • The buzz-phrase of the 90s in the petroleum industry has become {open_quotes}Buy, don`t Build.{close_quotes} For an end user in an oil company, this generally means acquiring application software rather than developing it internally. For a software developer, either within an oil company or with a software vendor, the concept of {open_quotes}buy don`t build{close_quotes} can apply to software toolkit components and can expedite the development of an application as well as reduce future support requirements. This paper presents several software tools and the process by which they were evaluated for use in a commercial petroleum engineering application (DeskTop VIP). It highlightsmore » the tendency in the software development process to underestimate the complexity of the development process as well as to underestimate the value of the services provided by a software tool. Ultimately, the decision of {open_quotes}buy don`t build{close_quotes} should be an economic decision. As a slogan, it reminds us that whenever one considers building or developing new software, one should also consider the possibility that buying off-the-shelf software could cost less in the long run and bring a product to completion quicker.« less
  • The buzz phrase of the 1990`s for the petroleum software industry has become ``buy, don`t build.`` For an end user in an oil company, this generally means acquiring application software rather than developing it internally. The concept of buy, don`t build can also apply for a software developer. Purchasing software toolkit components can expedite the development of an application as well as reduce future support requirements.
  • Modern scientific software is daunting in its diversity and complexity. From massively parallel simulations running on the world’s largest supercomputers, to visualizations and user support environments that manage ever growing complex data collections, the challenges for software engineers are plentiful. While high performance simulators are necessarily specialized codes to maximize performance on specific supercomputer architectures, we argue the vast majority of supporting infrastructure, data management and analysis tools can leverage commodity open source and component-based technologies. This approach can significantly drive down the effort and costs of building complex, collaborative scientific user environments, as well as increase their reliability andmore » extensibility. In this paper we describe our experiences in creating an initial user environment for scientists involved in modeling the detailed effects of climate change on the environment of selected geographical regions. Our approach composes the user environment using the Velo scientific knowledge management platform and the MeDICi Integration Framework for scientific workflows. These established platforms leverage component-based technologies and extend commodity open source platforms with abstractions and capabilities that make them amenable for broad use in science. Using this approach we were able to deliver an operational user environment capable of running thousands of simulations in a 7 month period, and achieve significant software reuse.« less
  • Embody (Environment Modules Build) is a software build tool with integrated support for the environment-modules package. The tool eases and automates the task of building and installing software packages from source or binary distributions, as well as the management of associated modulefiles. An administrator or software pool maintainer has to write a brief script for the installation process. These steps are usually described for manual execution, in a package's README or INSTALL file. It would be up to the site administrator to work out a procedure to capture the steps taken. This tool: streamlines and codifies the installation tasks inmore » a common framework; it provides a self-documenting and unified way for maintaining package installations; uses bash shell variables and functions for portability; keeps log files of the proceedings. It is similar in intent and function to other existing tools, such as RPM, but has several novel features tailored for High Performance Computing (HPC) software deployments. The design goal is simplicity and decoupling from RPM's dependencies and its database, which enables coexistence of several builds. Useful on HPC systems, new builds can be deployed centrally to shared file systems and without affecting running jobs.« less

To initiate an order for this software, request consultation services, or receive further information, fill out the request form below. You may also reach us by email at: .

OSTI staff will begin to process an order for scientific and technical software once the payment and signed site license agreement are received. If the forms are not in order, OSTI will contact you. No further action will be taken until all required information and/or payment is received. Orders are usually processed within three to five business days.

Software Request

(required)
(required)
(required)
(required)
(required)
(required)
(required)
(required)