DOE Patents title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Scalable filesystem enumeration and metadata operations

Abstract

Systems, apparatus, and methods are disclosed for performing scalable operations in a file system, including POSIX-like file systems. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, while retaining standard functionality transparently to clients. Other enumeration operations include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.

Inventors:
Issue Date:
Research Org.:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1805635
Patent Number(s):
10949387
Application Number:
15/713,492
Assignee:
Triad National Security, LLC (Los Alamos, NM)
Patent Classifications (CPCs):
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
DOE Contract Number:  
AC52-06NA25396
Resource Type:
Patent
Resource Relation:
Patent File Date: 09/22/2017
Country of Publication:
United States
Language:
English

Citation Formats

Grider, Gary A. Scalable filesystem enumeration and metadata operations. United States: N. p., 2021. Web.
Grider, Gary A. Scalable filesystem enumeration and metadata operations. United States.
Grider, Gary A. Tue . "Scalable filesystem enumeration and metadata operations". United States. https://www.osti.gov/servlets/purl/1805635.
@article{osti_1805635,
title = {Scalable filesystem enumeration and metadata operations},
author = {Grider, Gary A.},
abstractNote = {Systems, apparatus, and methods are disclosed for performing scalable operations in a file system, including POSIX-like file systems. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, while retaining standard functionality transparently to clients. Other enumeration operations include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2021},
month = {3}
}

Works referenced in this record:

Method for restoring files from a continuous recovery system
patent-application, February 2017


Small file aggregation in a parallel computing system
patent, September 2014


Optimized Data Placement for Individual File Accesses on Deduplication-enabled Sequential Storage Systems
patent-application, January 2014


Replacing file system processors by hot swapping
patent-application, December 2002


System and method for optimizing data migration in a partitioned database
patent-application, November 2017


System and methods for processing a multidimensional database
patent, October 2007


Fault tolerant multi-stage data replication with relaxed coherency guarantees
patent, January 2010


Versioning Schemes for Compute-Centric Object Stores
patent-application, September 2014


Data Syncing in a Distributed System
patent-application, August 2015


Media file management on a media storage and playback device
patent-application, July 2005


Conditional writes at distributed storage services
patent, February 2017


Pattern-Aware Prefetching Using Parallel Log-Structured File System
patent-application, April 2020


Enhancing file system performance
patent-application, October 2002


Computing Erasure Metadata and Data Layout Prior to Storage Using A Processing Platform
patent-application, December 2016


Storing files in a parallel computing system based on user-specified parser function
patent, October 2014


Oversubscribed storage extents with on-demand page allocation
patent, October 2017


Scalable file management for a shared file system
patent, December 2012


Estimating Data Reduction in Storage Systems
patent-application, August 2013


Parallel Serialization of Request Processing
patent-application, August 2012


Parallel file system with metadata distributed across partitioned key-value store
patent, September 2017


Enumerating Metadata in File System Directories
patent-application, May 2010


Distributed File System Consistency Check
patent-application, November 2017