skip to main content

DOE PAGESDOE PAGES

Title: MarFS, a Near-POSIX Interface to Cloud Objects

The engineering forces driving development of “cloud” storage have produced resilient, cost-effective storage systems that can scale to 100s of petabytes, with good parallel access and bandwidth. These features would make a good match for the vast storage needs of High-Performance Computing datacenters, but cloud storage gains some of its capability from its use of HTTP-style Representational State Transfer (REST) semantics, whereas most large datacenters have legacy applications that rely on POSIX file-system semantics. MarFS is an open-source project at Los Alamos National Laboratory that allows us to present cloud-style object-storage as a scalable near-POSIX file system. We have also developed a new storage architecture to improve bandwidth and scalability beyond what’s available in commodity object stores, while retaining their resilience and economy. Additionally, we present a scheme for scaling the POSIX interface to allow billions of files in a single directory and trillions of files in total.
Authors:
 [1] ;  [1] ;  [1] ;  [1]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Report Number(s):
LA-UR-16-28720; LA-UR-16-28952
Journal ID: ISSN 1044-6397
Grant/Contract Number:
AC52-06NA25396; FC02-06ER25750
Type:
Accepted Manuscript
Journal Name:
;Login
Additional Journal Information:
Journal Volume: 42; Journal Issue: 1; Related Information: Spring 2017 issue; Journal ID: ISSN 1044-6397
Publisher:
USENIX Association
Research Org:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org:
USDOE
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Computer Hardware; Computer Science; Information Science; MarFS; Campaign; scalability; storage; objects; POSIX
OSTI Identifier:
1351232