Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Mochi: Composing Data Services for High-Performance Computing Environments

Abstract

Technology enhancements and the growing breadth of application workflows running on high-performance computing (HPC) platforms drive the development of new data services that provide high performance on these new platforms, provide capable and productive interfaces and abstractions for a variety of applications, and are readily adapted when new technologies are deployed. The Mochi framework enables composition of specialized distributed data services from a collection of connectable modules and subservices. Rather than forcing all applications to use a one-size-fits-all data staging and I/O software configuration, Mochi allows each application to use a data service specialized to its needs and access patterns. This paper introduces the Mochi framework and methodology. The Mochi core components and microservices are described. Examples of the application of the Mochi methodology to the development of four specialized services are detailed. Finally, a performance evaluation of a Mochi core component, a Mochi microservice, and a composed service providing an object model is performed. The paper concludes by positioning Mochi relative to related work in the HPC space and indicating directions for future work.

Authors:
 [1];  [2];  [1];  [2];  [1];  [1];  [2];  [3];  [4];  [1];  [4];  [5];  [4];  [4];  [1];  [5];  [2]
+ Show Author Affiliations
  1. Argonne National Lab. (ANL), Lemont, IL (United States)
  2. Carnegie Mellon Univ., Pittsburgh, PA (United States)
  3. Vector Inst. for Artificial Intelligence, Toronto (Canada)
  4. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  5. HDF Group Champaign, IL (United States)
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1596688
Grant/Contract Number:  
AC02-06CH11357; AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
Journal of Computer Science and Technology
Additional Journal Information:
Journal Volume: 35; Journal Issue: 1; Journal ID: ISSN 1000-9000
Publisher:
Springer Nature
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; data-intensive computing; distributed services; high-performance computing; storage and I/O

Citation Formats

Ross, Robert B., Amvrosiadis, George, Carns, Philip, Cranor, Charles D., Dorier, Matthieu, Harms, Kevin, Ganger, Greg, Gibson, Garth, Gutierrez, Samuel K., Latham, Robert, Robey, Bob, Robinson, Dana, Settlemyer, Bradley, Shipman, Galen, Snyder, Shane, Soumagne, Jerome, and Zheng, Qing. Mochi: Composing Data Services for High-Performance Computing Environments. United States: N. p., 2020. Web. doi:10.1007/s11390-020-9802-0.
Copy to clipboard
Ross, Robert B., Amvrosiadis, George, Carns, Philip, Cranor, Charles D., Dorier, Matthieu, Harms, Kevin, Ganger, Greg, Gibson, Garth, Gutierrez, Samuel K., Latham, Robert, Robey, Bob, Robinson, Dana, Settlemyer, Bradley, Shipman, Galen, Snyder, Shane, Soumagne, Jerome, & Zheng, Qing. Mochi: Composing Data Services for High-Performance Computing Environments. United States. https://doi.org/10.1007/s11390-020-9802-0
Copy to clipboard
Ross, Robert B., Amvrosiadis, George, Carns, Philip, Cranor, Charles D., Dorier, Matthieu, Harms, Kevin, Ganger, Greg, Gibson, Garth, Gutierrez, Samuel K., Latham, Robert, Robey, Bob, Robinson, Dana, Settlemyer, Bradley, Shipman, Galen, Snyder, Shane, Soumagne, Jerome, and Zheng, Qing. Fri . "Mochi: Composing Data Services for High-Performance Computing Environments". United States. https://doi.org/10.1007/s11390-020-9802-0. https://www.osti.gov/servlets/purl/1596688.
Copy to clipboard
@article{osti_1596688,
title = {Mochi: Composing Data Services for High-Performance Computing Environments},
author = {Ross, Robert B. and Amvrosiadis, George and Carns, Philip and Cranor, Charles D. and Dorier, Matthieu and Harms, Kevin and Ganger, Greg and Gibson, Garth and Gutierrez, Samuel K. and Latham, Robert and Robey, Bob and Robinson, Dana and Settlemyer, Bradley and Shipman, Galen and Snyder, Shane and Soumagne, Jerome and Zheng, Qing},
abstractNote = {Technology enhancements and the growing breadth of application workflows running on high-performance computing (HPC) platforms drive the development of new data services that provide high performance on these new platforms, provide capable and productive interfaces and abstractions for a variety of applications, and are readily adapted when new technologies are deployed. The Mochi framework enables composition of specialized distributed data services from a collection of connectable modules and subservices. Rather than forcing all applications to use a one-size-fits-all data staging and I/O software configuration, Mochi allows each application to use a data service specialized to its needs and access patterns. This paper introduces the Mochi framework and methodology. The Mochi core components and microservices are described. Examples of the application of the Mochi methodology to the development of four specialized services are detailed. Finally, a performance evaluation of a Mochi core component, a Mochi microservice, and a composed service providing an object model is performed. The paper concludes by positioning Mochi relative to related work in the HPC space and indicating directions for future work.},
doi = {10.1007/s11390-020-9802-0},
journal = {Journal of Computer Science and Technology},
number = 1,
volume = 35,
place = {United States},
year = {Fri Jan 17 00:00:00 EST 2020},
month = {Fri Jan 17 00:00:00 EST 2020}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1007/s11390-020-9802-0
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 17 works
Citation information provided by
Web of Science
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

Scaling Embedded In-Situ Indexing with DeltaFS
conference, November 2018

  • Zheng, Qing; Cranor, Charles D.; Guo, Danhao
  • SC18: International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2018.00006

Evaluation of HPC Application I/O on Object Storage Systems
conference, November 2018

  • Liu, Jialin; Koziol, Quincey; Butler, Gregory F.
  • 2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems (PDSW-DISCS)
  • DOI: 10.1109/PDSW-DISCS.2018.00005

BESPOKV: Application Tailored Scale-Out Key-Value Stores
conference, November 2018

  • Anwar, Ali; Cheng, Yue; Huang, Hai
  • SC18: International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2018.00005

Toward Scalable and Asynchronous Object-Centric Data Management for HPC
conference, May 2018

  • Tang, Houjun; Byna, Suren; Tessier, Francois
  • 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)
  • DOI: 10.1109/CCGRID.2018.00026

Dragonfly+: Low Cost Topology for Scaling Datacenters
conference, February 2017

  • Shpiner, Alexander; Haramaty, Zachy; Eliad, Saar
  • 2017 IEEE 3rd International Workshop on High-Performance Interconnection Networks in the Exascale and Big-Data Era (HiPINEB)
  • DOI: 10.1109/HiPINEB.2017.11

Mercury: Enabling remote procedure call for high-performance computing
conference, September 2013

  • Soumagne, Jerome; Kimpe, Dries; Zounmevo, Judicael
  • 2013 IEEE International Conference on Cluster Computing (CLUSTER)
  • DOI: 10.1109/CLUSTER.2013.6702617

LABIOS: A Distributed Label-Based I/O System
conference, January 2019

  • Kougkas, Anthony; Devarajan, Hariharan; Lofstead, Jay
  • Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '19
  • DOI: 10.1145/3307681.3325405

Overview of 3D NAND Technologies and Outlook Invited Paper
conference, October 2018

Slim Fly: A Cost Effective Low-Diameter Network Topology
conference, November 2014

  • Besta, Maciej; Hoefler, Torsten
  • SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2014.34

FusionFS: Toward supporting data-intensive scientific applications on extreme-scale high-performance computing systems
conference, October 2014

ROOT — An object oriented data analysis framework
journal, April 1997

  • Brun, Rene; Rademakers, Fons
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 389, Issue 1-2
  • DOI: 10.1016/S0168-9002(97)00048-X

Malacology: A Programmable Storage System
conference, January 2017

  • Sevilla, Michael A.; Watkins, Noah; Jimenez, Ivo
  • Proceedings of the Twelfth European Conference on Computer Systems - EuroSys '17
  • DOI: 10.1145/3064176.3064208

Long-Time Dynamics through Parallel Trajectory Splicing
journal, December 2015

  • Perez, Danny; Cubuk, Ekin D.; Waterland, Amos
  • Journal of Chemical Theory and Computation, Vol. 12, Issue 1
  • DOI: 10.1021/acs.jctc.5b00916

DataSpaces: an interaction and coordination framework for coupled simulation workflows
journal, February 2011

Technology-Driven, Highly-Scalable Dragonfly Topology
journal, June 2008

  • Kim, John; Dally, Wiliam J.; Scott, Steve
  • ACM SIGARCH Computer Architecture News, Vol. 36, Issue 3
  • DOI: 10.1145/1394608.1382129

PapyrusKV: a high-performance parallel key-value store for distributed NVM architectures
conference, January 2017

  • Kim, Jungwon; Lee, Seyong; Vetter, Jeffrey S.
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17
  • DOI: 10.1145/3126908.3126943

The design and implementation of a log-structured file system
journal, February 1992

  • Rosenblum, Mendel; Ousterhout, John K.
  • ACM Transactions on Computer Systems, Vol. 10, Issue 1, p. 26-52
  • DOI: 10.1145/146941.146943

Argobots: A Lightweight Low-Level Threading and Tasking Framework
journal, March 2018

  • Seo, Sangmin; Amer, Abdelhalim; Balaji, Pavan
  • IEEE Transactions on Parallel and Distributed Systems, Vol. 29, Issue 3
  • DOI: 10.1109/TPDS.2017.2766062

Methodology for the Rapid Development of Scalable HPC Data Services
conference, November 2018

  • Dorier, Matthieu; Settlemyer, Brad; Shipman, Galen
  • 2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems (PDSW-DISCS)
  • DOI: 10.1109/PDSW-DISCS.2018.00013

Programmable Caches with a Data Management Language and Policy Engine
conference, May 2018

  • Sevilla, Michael A.; Maltzahn, Carlos; Alvaro, Peter
  • 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)
  • DOI: 10.1109/CCGRID.2018.00035

RADOS: a scalable, reliable storage service for petabyte-scale storage clusters
conference, January 2007

  • Weil, Sage A.; Leung, Andrew W.; Brandt, Scott A.
  • Proceedings of the 2nd international workshop on Petascale data storage held in conjunction with Supercomputing '07 - PDSW '07
  • DOI: 10.1145/1374596.1374606

Platform Storage Performance With 3D XPoint Technology
journal, September 2017

GekkoFS - A Temporary Distributed File System for HPC Applications
conference, September 2018

  • Vef, Marc-Andre; Moti, Nafiseh; SuB, Tim
  • 2018 IEEE International Conference on Cluster Computing (CLUSTER)
  • DOI: 10.1109/CLUSTER.2018.00049

Massively parallel loading
conference, January 2013

  • Frings, Wolfgang; Ahn, Dong H.; LeGendre, Matthew
  • Proceedings of the 27th international ACM conference on International conference on supercomputing - ICS '13
  • DOI: 10.1145/2464996.2465020

The NumPy Array: A Structure for Efficient Numerical Computation
journal, March 2011

  • van der Walt, Stéfan; Colbert, S. Chris; Varoquaux, Gaël
  • Computing in Science & Engineering, Vol. 13, Issue 2
  • DOI: 10.1109/MCSE.2011.37

An Ephemeral Burst-Buffer File System for Scientific Applications
conference, November 2016

  • Wang, Teng; Mohror, Kathryn; Moody, Adam
  • SC16: International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2016.68
    Sort by:
    [ × clear filter / sort ]

    Similar Records in DOE PAGES and OSTI.GOV collections: