Scalable augmented enumeration and metadata operations for large filesystems
Abstract
Systems, apparatus, and methods are disclosed for performing scalable operations in a file system. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An augmented enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, transparently to clients. Exemplary augmentation features can include filtering and sorting. Augmentation features can be executed concurrently with enumeration, prior to enumeration, after enumeration, or as a combination of these, and can utilize pre-built index structures or holding structures for intermediate results. Augmented enumeration operations can also include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.
- Inventors:
- Issue Date:
- Research Org.:
- Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
- Sponsoring Org.:
- USDOE
- OSTI Identifier:
- 1860146
- Patent Number(s):
- 11194763
- Application Number:
- 15/713,495
- Assignee:
- Triad National Security, LLC (Los Alamos, NM)
- Patent Classifications (CPCs):
-
G - PHYSICS G06 - COMPUTING G06F - ELECTRIC DIGITAL DATA PROCESSING
- DOE Contract Number:
- AC52-06NA25396
- Resource Type:
- Patent
- Resource Relation:
- Patent File Date: 09/22/2017
- Country of Publication:
- United States
- Language:
- English
Citation Formats
Grider, Gary A. Scalable augmented enumeration and metadata operations for large filesystems. United States: N. p., 2021.
Web.
Grider, Gary A. Scalable augmented enumeration and metadata operations for large filesystems. United States.
Grider, Gary A. Tue .
"Scalable augmented enumeration and metadata operations for large filesystems". United States. https://www.osti.gov/servlets/purl/1860146.
@article{osti_1860146,
title = {Scalable augmented enumeration and metadata operations for large filesystems},
author = {Grider, Gary A.},
abstractNote = {Systems, apparatus, and methods are disclosed for performing scalable operations in a file system. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An augmented enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, transparently to clients. Exemplary augmentation features can include filtering and sorting. Augmentation features can be executed concurrently with enumeration, prior to enumeration, after enumeration, or as a combination of these, and can utilize pre-built index structures or holding structures for intermediate results. Augmented enumeration operations can also include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Dec 07 00:00:00 EST 2021},
month = {Tue Dec 07 00:00:00 EST 2021}
}
Works referenced in this record:
Method and Apparatus for Fault-Tolerant Memory Management
patent-application, August 2010
- Thomas, Brandon; Sherrill, Jeremy Glenn
- US Patent Application 12/638923; 20100198849
Directory aggregation for files distributed over a plurality of servers in a switched file system
patent-application, July 2004
- Miloushev, Vladimir; Nickolov, Peter
- 10/336833; 20040133606
Method for restoring files from a continuous recovery system
patent-application, February 2017
- Kedem, Oded; Ben-Or, Tomer; Barash, Gil
- 14/813169; 20170031783
Methods and apparatus for multi-resolution replication of files in a parallel computing system using semantic information
patent, October 2015
- Faibish, Sorin; Bent, John M.; Tzelnic, Percy
- US Patent Document 9,165,014
Parallel log structured file system collective buffering to achieve a compact representation of scientific and/or dimensional data
patent, September 2015
- Grider, Gary; Poole, Stephen W.
- US Patent Document 9,122,700
Method and apparatus for organizing data sources
patent, May 2009
- Chang, Yuan-Chi; Lim, Lipyeow; Wang, Min
- US Patent Document 7,529,740
Small file aggregation in a parallel computing system
patent, September 2014
- Faibish, Sorin; Bent, John M.; Tzelnic, Percy
- US Patent Document 8,825,652
Distributed Data Storage System Using a Common Manifest for Storing and accessing Versions of an Object
patent-application, May 2018
- Bestler, Caitlin
- US Patent Application 15/358412; 20180145983
Apparatus and method for realizing big data into a big object and non-transitory tangible machine-readable medium thereof
patent, January 2013
- Hseush, Wenwey; Hsu, Shih-Chang; Huang, Yi-Cheng
- US Patent Document 8,364,723
Optimized Data Placement for Individual File Accesses on Deduplication-enabled Sequential Storage Systems
patent-application, January 2014
- Constantinescu, Mihail C.; Gharaibeh, Abdullah; Lu, Maohua
- 13/537851; 20140006363
Replacing file system processors by hot swapping
patent-application, December 2002
- Ulrich, Thomas R.; Schweitzer, James R.; Bolstad, Gregory D.
- 10/060908; 20020194523
System and method for optimizing data migration in a partitioned database
patent-application, November 2017
- Horowitz, Eliot; Merriman, Dwight
- 15/654590; 20170322996
Cloud object store for archive storage of high performance computing data using decoupling middleware
patent, June 2015
- Bent, John M.; Faibish, Sorin; Grider, Gary
- US Patent Document 9,069,778
System and methods for processing a multidimensional database
patent, October 2007
- Narayanaswamy, Arun; Banci, Ronald R.
- US Patent Document 7,284,011
Fault tolerant multi-stage data replication with relaxed coherency guarantees
patent, January 2010
- Shelat, Radha; Ranade, Dilip Madhusudan; Kabra, Navin
- US Patent Document 7,653,668
Versioning Schemes for Compute-Centric Object Stores
patent-application, September 2014
- Cavage, Mark; Pacheco, David; Cantrill, Bryan
- US Patent Application 13/838150; 20140282615
MTREEINI: Intermediate Nodes and Indexes
patent-application,
- Pettovello, Primo M.
- US Patent Application 11/624510; 20070174309
Data Syncing in a Distributed System
patent-application, August 2015
- Cantwell, Jared; Minckler, Bill; Roback, Joe
- 14/186847; 20150244795
Cloud object store for checkpoints of high performance computing applications using decoupling middleware
patent, April 2016
- Bent, John M.; Faibish, Sorin; Grider, Gary
- US Patent Document 9,317,521
Media file management on a media storage and playback device
patent-application, July 2005
- Guzak, Chris J.; Tubbs, Ken; Lee, Oliver
- 10/751680; 20050149213
Scalable filesystem enumeration and metadata operations
patent, March 2021
- Grider, Gary
- US Patent Document 10,949,387
Parallel sort with a ranged, partitioned key-value store in a high perfomance computing environment
patent, January 2016
- Bent, John M.; Faibish, Sorin; Grider, Gary
- US Patent Document 9,245,048
Conditional writes at distributed storage services
patent, February 2017
- Strauss, Jacob A.; Vincent, Pradeep; Frasca, Michael Robert
- US Patent Document 9,569,459
Pattern-Aware Prefetching Using Parallel Log-Structured File System
patent-application, April 2020
- Bent, John M.; Faibish, Sorin; Grider, Gary
- 16/716972; 20200125493
Enhancing file system performance
patent-application, October 2002
- Ulrich, Thomas R.; Schweitzer, James R.; Bolstad, Gregory D.
- 10/060918; 20020156891
System, method, and computer program product for utilizing a data structure including event relationships to detect unwanted activity
patent-application, September 2013
- Spurlock, Joel R.
- 12/177601; 20130247190
Cooperative storage of shared files in a parallel computing system with dynamic block size
patent, November 2015
- Bent, John M.; Faibish, Sorin; Grider, Gary
- US Patent Document 9,183,211
Computing Erasure Metadata and Data Layout Prior to Storage Using A Processing Platform
patent-application, December 2016
- Bent, John M.; Faibish, Sorin; Ting, Dennis P. J.
- 14/751372; 20160381138
Distributed metadata servers in a file system with separate metadata servers for file metadata and directory metadata
patent, November 2018
- Bent, John M.; Faibish, Sorin; Grider, Gary
- US Patent Document 10,140,304
Object Storage System with Local Transaction Logs, a Distributed Namespace, and Optimized Support for User Directories
patent-application, December 2015
- Aizman, Alexander; Bestler, Caitlin; Novak, Robert E.
- 14/820471; 20150347553
Allocating data based on memory device performance in a dispersed storage network
patent, January 2020
- Dhuse, Greg; Resch, Jason K.
- US Patent Document 10,540,230
Storing files in a parallel computing system based on user-specified parser function
patent, October 2014
- Faibish, Sorin; Bent, John M.; Tzelnic, Percy
- US Patent Document 8,868,576
Storing files in a parallel computing system based on user or application specification
patent, March 2016
- Faibish, Sorin; Bent, John M.; Nick, Jeffrey M.
- US Patent Document 9,298,733
Storing files in a parallel computing system using list-based index to identify replica files
patent, July 2015
- Faibish, Sorin; Bent, John M.; Tzelnic, Percy
- US Patent Document 9,087,075
Oversubscribed storage extents with on-demand page allocation
patent, October 2017
- Oikarinen, Matti Juhani; Vincent, Pradeep; Frigo, Matteo
- US Patent Document 9,779,015
Provisioning System-Permissions Using Attribute-Based Access Control Policies
patent-application, March 2016
- Martinelli, Andres
- 14/546018; 20160072814
Scalable file management for a shared file system
patent, December 2012
- Kaplan, Marc A.; Sawdon, Wayne A.
- US Patent Document 8,341,130
Estimating Data Reduction in Storage Systems
patent-application, August 2013
- Chambliss, David D.; Constantinescu, Mihail C.; Glider, Joseph S.
- 13/360628; 20130198148
Parallel Serialization of Request Processing
patent-application, August 2012
- Nightingale, Edmund B.; Elson, Jeremy E.
- 13/017193; 20120197958
Parallel file system with metadata distributed across partitioned key-value store
patent, September 2017
- Bent, John M.; Faibish, Sorin; Grider, Gary
- US Patent Document 9,767,107
Methods and apparatus for capture and storage of semantic information with sub-files in a parallel computing system
patent, February 2015
- Faibish, Sorin; Bent, John M.; Tzelnic, Percy
- US Patent Document 8,949,255
Enumerating Metadata in File System Directories
patent-application, May 2010
- Callahan, Michael J.; Revitch, Samuel
- 12/261071; 20100115011
Distributed File System Consistency Check
patent-application, November 2017
- Wang, Wenguang; Lu, Yunshan
- 15/163471; 20170344597