DOE Patents title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Scalable augmented enumeration and metadata operations for large filesystems

Abstract

Systems, apparatus, and methods are disclosed for performing scalable operations in a file system. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An augmented enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, transparently to clients. Exemplary augmentation features can include filtering and sorting. Augmentation features can be executed concurrently with enumeration, prior to enumeration, after enumeration, or as a combination of these, and can utilize pre-built index structures or holding structures for intermediate results. Augmented enumeration operations can also include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.

Inventors:
Issue Date:
Research Org.:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1860146
Patent Number(s):
11194763
Application Number:
15/713,495
Assignee:
Triad National Security, LLC (Los Alamos, NM)
Patent Classifications (CPCs):
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
DOE Contract Number:  
AC52-06NA25396
Resource Type:
Patent
Resource Relation:
Patent File Date: 09/22/2017
Country of Publication:
United States
Language:
English

Citation Formats

Grider, Gary A. Scalable augmented enumeration and metadata operations for large filesystems. United States: N. p., 2021. Web.
Grider, Gary A. Scalable augmented enumeration and metadata operations for large filesystems. United States.
Grider, Gary A. Tue . "Scalable augmented enumeration and metadata operations for large filesystems". United States. https://www.osti.gov/servlets/purl/1860146.
@article{osti_1860146,
title = {Scalable augmented enumeration and metadata operations for large filesystems},
author = {Grider, Gary A.},
abstractNote = {Systems, apparatus, and methods are disclosed for performing scalable operations in a file system. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An augmented enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, transparently to clients. Exemplary augmentation features can include filtering and sorting. Augmentation features can be executed concurrently with enumeration, prior to enumeration, after enumeration, or as a combination of these, and can utilize pre-built index structures or holding structures for intermediate results. Augmented enumeration operations can also include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Dec 07 00:00:00 EST 2021},
month = {Tue Dec 07 00:00:00 EST 2021}
}

Works referenced in this record:

Method and Apparatus for Fault-Tolerant Memory Management
patent-application, August 2010


Method for restoring files from a continuous recovery system
patent-application, February 2017


Method and apparatus for organizing data sources
patent, May 2009


Small file aggregation in a parallel computing system
patent, September 2014


Optimized Data Placement for Individual File Accesses on Deduplication-enabled Sequential Storage Systems
patent-application, January 2014


Replacing file system processors by hot swapping
patent-application, December 2002


System and method for optimizing data migration in a partitioned database
patent-application, November 2017


System and methods for processing a multidimensional database
patent, October 2007


Fault tolerant multi-stage data replication with relaxed coherency guarantees
patent, January 2010


Versioning Schemes for Compute-Centric Object Stores
patent-application, September 2014


MTREEINI: Intermediate Nodes and Indexes
patent-application,  


Data Syncing in a Distributed System
patent-application, August 2015


Media file management on a media storage and playback device
patent-application, July 2005


Conditional writes at distributed storage services
patent, February 2017


Pattern-Aware Prefetching Using Parallel Log-Structured File System
patent-application, April 2020


Enhancing file system performance
patent-application, October 2002


Computing Erasure Metadata and Data Layout Prior to Storage Using A Processing Platform
patent-application, December 2016


Storing files in a parallel computing system based on user-specified parser function
patent, October 2014


Oversubscribed storage extents with on-demand page allocation
patent, October 2017


Scalable file management for a shared file system
patent, December 2012


Estimating Data Reduction in Storage Systems
patent-application, August 2013


Parallel Serialization of Request Processing
patent-application, August 2012


Parallel file system with metadata distributed across partitioned key-value store
patent, September 2017


Enumerating Metadata in File System Directories
patent-application, May 2010


Distributed File System Consistency Check
patent-application, November 2017