FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework

Ghoshal, Devarshi; Hendrix, Valerie; Fox, William; Balasubhramanian, Sowmya; Ramakrishnan, Lavanya

doi:10.21105/joss.00164

Title: FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework

Abstract

Scientific applications are increasingly using cloud resources for their data analysis workflows. However, managing data effectively and efficiently over these cloud resources is challenging due to the myriad storage choices with different performance, cost trade-offs, complex application choices and complexity associated with elasticity, failure rates in these environments. The different data access patterns for data-intensive scientific applications require a more flexible and robust data management solution than the ones currently in existence. FRIEDA is a Flexible Robust Intelligent Elastic Data Management framework that employs a range of data management strategies in cloud environments. FRIEDA can manage storage and data lifecycle of applications in cloud environments. There are four different stages in the data management lifecycle of FRIEDA – (i) storage planning, (ii) provisioning and preparation, (iii) data placement, and (iv) execution. FRIEDA defines a data control plane and an execution plane. The data control plane defines the data partition and distribution strategy, whereas the execution plane manages the execution of the application using a master-worker paradigm. FRIEDA also provides different data management strategies, either to partition the data in real-time, or predetermine the data partitions prior to application execution.

Authors:

^[1];

^[1]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Publication Date:: Wed Feb 01 00:00:00 EST 2017

Research Org.:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1435078

Grant/Contract Number:: AC02-05CH11231

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Open Source Software

Additional Journal Information:: Journal Volume: 2; Journal Issue: 10; Journal ID: ISSN 2475-9066

Publisher:: Open Source Initiative - NumFOCUS

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Ghoshal, Devarshi, Hendrix, Valerie, Fox, William, Balasubhramanian, Sowmya, and Ramakrishnan, Lavanya. FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework.  United States: N. p., 2017. 
Web.  doi:10.21105/joss.00164.

Copy to clipboard


                    Ghoshal, Devarshi, Hendrix, Valerie, Fox, William, Balasubhramanian, Sowmya, & Ramakrishnan, Lavanya. FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework.  United States.  https://doi.org/10.21105/joss.00164

Copy to clipboard


                    Ghoshal, Devarshi, Hendrix, Valerie, Fox, William, Balasubhramanian, Sowmya, and Ramakrishnan, Lavanya. Wed .  
"FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework".  United States.  https://doi.org/10.21105/joss.00164.  https://www.osti.gov/servlets/purl/1435078.

Copy to clipboard


                    
@article{osti_1435078,

  title        = {FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework},

  author       = {Ghoshal, Devarshi and Hendrix, Valerie and Fox, William and Balasubhramanian, Sowmya and Ramakrishnan, Lavanya},

  abstractNote = {Scientific applications are increasingly using cloud resources for their data analysis workflows. However, managing data effectively and efficiently over these cloud resources is challenging due to the myriad storage choices with different performance, cost trade-offs, complex application choices and complexity associated with elasticity, failure rates in these environments. The different data access patterns for data-intensive scientific applications require a more flexible and robust data management solution than the ones currently in existence. FRIEDA is a Flexible Robust Intelligent Elastic Data Management framework that employs a range of data management strategies in cloud environments. FRIEDA can manage storage and data lifecycle of applications in cloud environments. There are four different stages in the data management lifecycle of FRIEDA – (i) storage planning, (ii) provisioning and preparation, (iii) data placement, and (iv) execution. FRIEDA defines a data control plane and an execution plane. The data control plane defines the data partition and distribution strategy, whereas the execution plane manages the execution of the application using a master-worker paradigm. FRIEDA also provides different data management strategies, either to partition the data in real-time, or predetermine the data partitions prior to application execution.},

  doi          = {10.21105/joss.00164},

  journal      = {Journal of Open Source Software},

  number       = 10,

  volume       = 2,

  place        = {United States},

  year         = {Wed Feb 01 00:00:00 EST 2017},

  month        = {Wed Feb 01 00:00:00 EST 2017}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.21105/joss.00164

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Storage and Data Life Cycle Management in Cloud Environments with FRIEDA
book, January 2014

Ramakrishnan, Lavanya; Ghoshal, Devarshi; Hendrix, Valerie
Cloud Computing for Data-Intensive Applications
DOI: 10.1007/978-1-4939-1905-5_15

Provisioning, Placement and Pipelining Strategies for Data-Intensive Applications in Cloud Environments
conference, March 2014

Ghoshal, Devarshi; Ramakrishnan, Lavanya
2014 IEEE International Conference on Cloud Engineering (IC2E)
DOI: 10.1109/IC2E.2014.66

FRIEDA: Flexible Robust Intelligent Elastic Data Management in Cloud Environments
conference, November 2012

Ghoshal, Devarshi; Ramakrishnan, Lavanya
2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis
DOI: 10.1109/SC.Companion.2012.132

Similar Records in DOE PAGES and OSTI.GOV collections:

Structured Composition of Dataflow and Control-Flow for Reusable and Robust Scientific Workflows

Conference Bowers, S ; Ludaescher, B ; Ngu, A ; ...

Data-centric scientific workflows are often modeled as dataflow process networks. The simplicity of the dataflow framework facilitates workflow design, analysis, and optimization. However, some workflow tasks are particularly ''control-flow intensive'', e.g., procedures to make workflows more fault-tolerant and adaptive in an unreliable, distributed computing environment. Modeling complex control-flow directly within a dataflow framework often leads to overly complicated workflows that are hard to comprehend, reuse, schedule, and maintain. In this paper, we develop a framework that allows a structured embedding of control-flow intensive subtasks within dataflow process networks. In this way, we can seamlessly handle complex control-flows without sacrificing themore »« less
Full Text Available
Orchestration of materials science workflows for heterogeneous resources at large scale

Journal Article Zhou, Naweiluo ; Scorzelli, Giorgio ; Luettgau, Jakob ; ... - International Journal of High Performance Computing Applications

In the era of big data, materials science workflows need to handle large-scale data distribution, storage, and computation. Any of these areas can become a performance bottleneck. We present a framework for analyzing internal material structures (e.g., cracks) to mitigate these bottlenecks. We demonstrate the effectiveness of our framework for a workflow performing synchrotron X-ray computed tomography reconstruction and segmentation of a silica-based structure. Our framework provides a cloud-based, cutting-edge solution to challenges such as growing intermediate and output data and heavy resource demands during image reconstruction and segmentation. Specifically, our framework efficiently manages data storage, scaling up compute resourcesmore »« less
https://doi.org/10.1177/10943420231167800

Full Text Available
Towards a Scalable and Adaptive Application Support Platform for Large-Scale Distributed E-Sciences in High-Performance Network Environments

Technical Report Wu, Chase Qishi ; Zhu, Michelle Mengxia

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models featuremore »« less
https://doi.org/10.2172/1255487

Full Text Available
CMS Workflow Execution using Intelligent Job Scheduling and Data Access Strategies

Journal Article Hasham, Khawar ; Delgado Peris, Antonio ; Anjum, Ashiq ; ... - IEEE Trans.Nucl.Sci.

Complex scientific workflows can process large amounts of data using thousands of tasks. The turnaround times of these workflows are often affected by various latencies such as the resource discovery, scheduling and data access latencies for the individual workflow processes or actors. Minimizing these latencies will improve the overall execution time of a workflow and thus lead to a more efficient and robust processing environment. In this paper, we propose a pilot job based infrastructure that has intelligent data reuse and job execution strategies to minimize the scheduling, queuing, execution and data access latencies. The results have shown that significantmore »« less
https://doi.org/10.1109/TNS.2011.2146276

Full Text Available
Review of Enabling Technologies to Facilitate Secure Compute Customization

Technical Report Aderholdt, Ferrol ; Caldwell, Blake A ; Hicks, Susan Elaine ; ...

High performance computing environments are often used for a wide variety of workloads ranging from simulation, data transformation and analysis, and complex workflows to name just a few. These systems may process data for a variety of users, often requiring strong separation between job allocations. There are many challenges to establishing these secure enclaves within the shared infrastructure of high-performance computing (HPC) environments. The isolation mechanisms in the system software are the basic building blocks for enabling secure compute enclaves. There are a variety of approaches and the focus of this report is to review the different virtualization technologies thatmore »« less
https://doi.org/10.2172/1195817

Full Text Available

Similar Records

Title: FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework

Abstract

Citation Formats

Storage and Data Life Cycle Management in Cloud Environments with FRIEDA book, January 2014

Provisioning, Placement and Pipelining Strategies for Data-Intensive Applications in Cloud Environments conference, March 2014

FRIEDA: Flexible Robust Intelligent Elastic Data Management in Cloud Environments conference, November 2012

Storage and Data Life Cycle Management in Cloud Environments with FRIEDA
book, January 2014

Provisioning, Placement and Pipelining Strategies for Data-Intensive Applications in Cloud Environments
conference, March 2014

FRIEDA: Flexible Robust Intelligent Elastic Data Management in Cloud Environments
conference, November 2012