Exploring Two Approaches for an End-to-End Scientific Analysis Workflow

Dodelson, Scott; Kent, Steve; Kowalkowski, Jim; Paterno, Marc; Sehrish, Saba

doi:10.1088/1742-6596/664/6/062058

Title: Exploring Two Approaches for an End-to-End Scientific Analysis Workflow

Full Record
Other Related Research

Abstract

The advance of the scientific discovery process is accomplished by the integration of independently-developed programs run on disparate computing facilities into coherent workflows usable by scientists who are not experts in computing. For such advancement, we need a system which scientists can use to formulate analysis workflows, to integrate new components to these workflows, and to execute different components on resources that are best suited to run those components. In addition, we need to monitor the status of the workflow as components get scheduled and executed, and to access the intermediate and final output for visual exploration and analysis. Finally, it is important for scientists to be able to share their workflows with collaborators. Moreover we have explored two approaches for such an analysis framework for the Large Synoptic Survey Telescope (LSST) Dark Energy Science Collaboration (DESC), the first one is based on the use and extension of Galaxy, a web-based portal for biomedical research, and the second one is based on a programming language, Python. In our paper, we present a brief description of the two approaches, describe the kinds of extensions to the Galaxy system we have found necessary in order to support the wide variety of scientificmore »« less

Authors:

Dodelson, Scott ^[1]; Kent, Steve ^[2]; Kowalkowski, Jim ^[2]; Paterno, Marc ^[2]; Sehrish, Saba ^[2]

Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Univ. of Chicago, IL (United States)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Publication Date:: Wed Dec 23 00:00:00 EST 2015

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP)

OSTI Identifier:: 1260268

Report Number(s):: FERMILAB-PUB-15-608-CD
Journal ID: ISSN 1742-6588; 1413975

Grant/Contract Number:: AC02-07CH11359

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Physics. Conference Series

Additional Journal Information:: Journal Volume: 664; Journal Issue: 6; Journal ID: ISSN 1742-6588

Publisher:: IOP Publishing

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Dodelson, Scott, Kent, Steve, Kowalkowski, Jim, Paterno, Marc, and Sehrish, Saba. Exploring Two Approaches for an End-to-End Scientific Analysis Workflow.  United States: N. p., 2015. 
Web.  doi:10.1088/1742-6596/664/6/062058.

Copy to clipboard


                    Dodelson, Scott, Kent, Steve, Kowalkowski, Jim, Paterno, Marc, & Sehrish, Saba. Exploring Two Approaches for an End-to-End Scientific Analysis Workflow.  United States.  https://doi.org/10.1088/1742-6596/664/6/062058

Copy to clipboard


                    Dodelson, Scott, Kent, Steve, Kowalkowski, Jim, Paterno, Marc, and Sehrish, Saba. Wed .  
"Exploring Two Approaches for an End-to-End Scientific Analysis Workflow".  United States.  https://doi.org/10.1088/1742-6596/664/6/062058.  https://www.osti.gov/servlets/purl/1260268.

Copy to clipboard


                    
@article{osti_1260268,

  title        = {Exploring Two Approaches for an End-to-End Scientific Analysis Workflow},

  author       = {Dodelson, Scott and Kent, Steve and Kowalkowski, Jim and Paterno, Marc and Sehrish, Saba},

  abstractNote = {The advance of the scientific discovery process is accomplished by the integration of independently-developed programs run on disparate computing facilities into coherent workflows usable by scientists who are not experts in computing. For such advancement, we need a system which scientists can use to formulate analysis workflows, to integrate new components to these workflows, and to execute different components on resources that are best suited to run those components. In addition, we need to monitor the status of the workflow as components get scheduled and executed, and to access the intermediate and final output for visual exploration and analysis. Finally, it is important for scientists to be able to share their workflows with collaborators. Moreover we have explored two approaches for such an analysis framework for the Large Synoptic Survey Telescope (LSST) Dark Energy Science Collaboration (DESC), the first one is based on the use and extension of Galaxy, a web-based portal for biomedical research, and the second one is based on a programming language, Python. In our paper, we present a brief description of the two approaches, describe the kinds of extensions to the Galaxy system we have found necessary in order to support the wide variety of scientific analysis in the cosmology community, and discuss how similar efforts might be of benefit to the HEP community.},

  doi          = {10.1088/1742-6596/664/6/062058},

  journal      = {Journal of Physics. Conference Series},

  number       = 6,

  volume       = 664,

  place        = {United States},

  year         = {Wed Dec 23 00:00:00 EST 2015},

  month        = {Wed Dec 23 00:00:00 EST 2015}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1088/1742-6596/664/6/062058

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

High Energy Physics and Nuclear Physics Network Requirements

Technical Report Dart, Eli ; Bauerdick, Lothar ; Bell, Greg ; ...

The Energy Sciences Network (ESnet) is the primary provider of network connectivity for the U.S. Department of Energy (DOE) Office of Science (SC), the single largest supporter of basic research in the physical sciences in the United States. In support of SC programs, ESnet regularly updates and refreshes its understanding of the networking requirements needed by instruments, facilities, scientists, and science programs that it serves. This focus has helped ESnet to be a highly successful enabler of scientific discovery for over 25 years. In August 2013, ESnet and the DOE SC Offices of High Energy Physics (HEP) and Nuclear Physicsmore »« less
https://doi.org/10.2172/1134243

Full Text Available
Numerical Codes for the DESC-LSST Analysis Pipeline: Core Cosmology Library Standard Modules and Beyond wCDM Modules (Final Technical Report)

Technical Report Ishak-Boushaki, Mustapah

The overall objective of the project is to investigate and develop specific software modules and analysis components for the software pipeline of LSST Dark Energy Science Collaboration (DESC). Following the key projects of DESC Science Roadmap (SRM), we will write and test computer codes for the Core Cosmology Library (CCL) in order to complete its modules, functionalities, and interface to work with the analysis pipelines from the five science probes of DESC (parts of SRM deliverables CX4.2TJP, CX6.2CS). We will also code modules for CCL to test models beyond w-Cold-Dark-Matter (wCDM) and modification to gravity (MG). Interfaces for MG modelsmore »« less
https://doi.org/10.2172/1797575

Full Text Available
Probing the Physics of Dark Energy with LSST: Large Scale Structure and Supernovae

Technical Report Gawiser, Eric ; Jha, Saurabh

The Large Synoptic Survey Telescope (LSST) will make a huge leap forward in understanding the mysterious “dark energy” that drives the accelerating Universe. Its unprecedented combination of sky area and depth allows for multiple, independent probes to address fundamental questions: Is dark energy consistent with Einstein’s cosmological constant? Does it change with space or time? Does modified gravity provide a better explanation of cosmic acceleration? The LSST design is advanced but leaves significant freedom in the cadence of when a given field on the sky will be observed in each filter as well as the specific dithering pattern used tomore »« less
https://doi.org/10.2172/1580563

Full Text Available
Towards a Scalable and Adaptive Application Support Platform for Large-Scale Distributed E-Sciences in High-Performance Network Environments

Technical Report Wu, Chase Qishi ; Zhu, Michelle Mengxia

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models featuremore »« less
https://doi.org/10.2172/1255487

Full Text Available
Scientific Data Management Center for Enabling Technologies

Technical Report Vouk, Mladen A.

Managing scientific data has been identified by the scientific community as one of the most important emerging needs because of the sheer volume and increasing complexity of data being collected. Effectively generating, managing, and analyzing this information requires a comprehensive, end-to-end approach to data management that encompasses all of the stages from the initial data acquisition to the final analysis of the data. Fortunately, the data management problems encountered by most scientific domains are common enough to be addressed through shared technology solutions. Based on community input, we have identified three significant requirements. First, more efficient access to storage systemsmore »« less
https://doi.org/10.2172/1095275

Full Text Available

Similar Records