skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Portable SHMEMCache: A High-Performance Key-Value Store on OpenSHMEM and MPI

Abstract

The integration of Big Data frameworks and HPC capabilities has drawn enormous interests in recent years. SHMEMCache is a distributed key-value store built on the OpenSHMEM global address space. It has solved several practical issues in leveraging OpenSHMEM’s one-sided operations for a distributed key-value store and providing efficient key-value operations on both commodity machines and supercomputers. However, being based solely on OpenSHMEM, SHMEMCache cannot leverage one-sided operations from a variety of software packages. This results in several limitations for SHMEMCache. First, we cannot make SHMEMCache available to a wider range of platforms. Second, an opportunity for potential performance improvement is missed. Third, there is a lack of deep understanding about how different one-sided operations can fit in with SHMEMCache and other distributed key-values in general. For example, the one-sided operations in OpenSHMEM and MPI have many differences in their interfaces, memory semantics and synchronization methods, all of which can have distinct implications and also increase the complexity in supporting both OpenSHMEM and MPI for SHMEMCache. Therefore, we have taken on an effort on leveraging different one-sided operations for SHMEMCache and proposed a design of portable SHMEMCache. Based on this new framework, we have supported both OpenSHMEM and MPI for SHMEMCache.more » We have also conducted an extensive set of experiments to compare the performance of the two versions on both commodity machines and the Titan supercomputer.« less

Authors:
 [1];  [2];  [2];  [1]
  1. Florida State Univ., Tallahassee, FL (United States)
  2. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); UT-Battelle LLC/ORNL, Oak Ridge, TN (Unted States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1567475
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Journal Name:
OPENSHMEM AND RELATED TECHNOLOGIES: BIG COMPUTE AND BIG DATA CONVERGENCE, OPENSHMEM 2017
Additional Journal Information:
Journal Volume: 10679; Conference: OpenSHMEM 2017: OpenSHMEM and Related Technologies. Big Compute and Big Data Convergence, Annapolis, Maryland, August 7-9, 2017.
Country of Publication:
United States
Language:
English
Subject:
Computer Science

Citation Formats

Fu, Huansong, Venkata, Manjunath Gorentla, Imam, Neena, and Yu, Weikuan. Portable SHMEMCache: A High-Performance Key-Value Store on OpenSHMEM and MPI. United States: N. p., 2018. Web. doi:10.1007/978-3-319-73814-7_8.
Fu, Huansong, Venkata, Manjunath Gorentla, Imam, Neena, & Yu, Weikuan. Portable SHMEMCache: A High-Performance Key-Value Store on OpenSHMEM and MPI. United States. doi:10.1007/978-3-319-73814-7_8.
Fu, Huansong, Venkata, Manjunath Gorentla, Imam, Neena, and Yu, Weikuan. Wed . "Portable SHMEMCache: A High-Performance Key-Value Store on OpenSHMEM and MPI". United States. doi:10.1007/978-3-319-73814-7_8.
@article{osti_1567475,
title = {Portable SHMEMCache: A High-Performance Key-Value Store on OpenSHMEM and MPI},
author = {Fu, Huansong and Venkata, Manjunath Gorentla and Imam, Neena and Yu, Weikuan},
abstractNote = {The integration of Big Data frameworks and HPC capabilities has drawn enormous interests in recent years. SHMEMCache is a distributed key-value store built on the OpenSHMEM global address space. It has solved several practical issues in leveraging OpenSHMEM’s one-sided operations for a distributed key-value store and providing efficient key-value operations on both commodity machines and supercomputers. However, being based solely on OpenSHMEM, SHMEMCache cannot leverage one-sided operations from a variety of software packages. This results in several limitations for SHMEMCache. First, we cannot make SHMEMCache available to a wider range of platforms. Second, an opportunity for potential performance improvement is missed. Third, there is a lack of deep understanding about how different one-sided operations can fit in with SHMEMCache and other distributed key-values in general. For example, the one-sided operations in OpenSHMEM and MPI have many differences in their interfaces, memory semantics and synchronization methods, all of which can have distinct implications and also increase the complexity in supporting both OpenSHMEM and MPI for SHMEMCache. Therefore, we have taken on an effort on leveraging different one-sided operations for SHMEMCache and proposed a design of portable SHMEMCache. Based on this new framework, we have supported both OpenSHMEM and MPI for SHMEMCache. We have also conducted an extensive set of experiments to compare the performance of the two versions on both commodity machines and the Titan supercomputer.},
doi = {10.1007/978-3-319-73814-7_8},
journal = {OPENSHMEM AND RELATED TECHNOLOGIES: BIG COMPUTE AND BIG DATA CONVERGENCE, OPENSHMEM 2017},
number = ,
volume = 10679,
place = {United States},
year = {2018},
month = {8}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Works referenced in this record:

An implementation and evaluation of the MPI 3.0 one-sided communication interface: MPI 3.0 RMA IMPLEMENTATION AND EVALUATION
journal, January 2016

  • Dinan, James; Balaji, Pavan; Buntinas, Darius
  • Concurrency and Computation: Practice and Experience, Vol. 28, Issue 17
  • DOI: 10.1002/cpe.3758

High Performance RDMA-Based MPI Implementation over InfiniBand
journal, June 2004


Advances, Applications and Performance of the Global Arrays Shared Memory Programming Toolkit
journal, May 2006

  • Nieplocha, Jarek; Palmer, Bruce; Tipparaju, Vinod
  • The International Journal of High Performance Computing Applications, Vol. 20, Issue 2
  • DOI: 10.1177/1094342006064503

Co-array Fortran for parallel programming
journal, August 1998