Scalable augmented enumeration and metadata operations for large filesystems

Patent · Tue Dec 07 04:00:00 EST 2021

OSTI ID:1860146

Grider, Gary A.

Systems, apparatus, and methods are disclosed for performing scalable operations in a file system. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An augmented enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, transparently to clients. Exemplary augmentation features can include filtering and sorting. Augmentation features can be executed concurrently with enumeration, prior to enumeration, after enumeration, or as a combination of these, and can utilize pre-built index structures or holding structures for intermediate results. Augmented enumeration operations can also include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.

• Research Organization: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC52-06NA25396
• Assignee: Triad National Security, LLC (Los Alamos, NM)
• Patent Number(s): 11,194,763
• Application Number: 15/713,495
• OSTI ID: 1860146
• Country of Publication: United States
• Language: English