Map reduce using coordination namespace hardware acceleration

Patent · Tue Mar 08 04:00:00 EST 2022

OSTI ID:1892689

Jacob, Philip; Strenski, Philip Neil; Johns, Charles

A system and method for supporting data MapReduce operations in a tuple space/coordinated namespace (CNS) extended memory storage architecture. The system-wide CNS provides an efficient means for storing and communicating data generated by local processes running at the nodes, and coordinated to provide MapReduce operations in a multi-nodal system. A hardware accelerated mechanism supports map reduce sorting/shuffle operations and reduce operations according to an aggregate function. Local processes running at a node generate a tuple corresponding to data generated by a process, each tuple having a tuple name and tuple data value corresponding to the generated data. Each tuple is processed and stored at the node or another node, dependent upon its tuple name. Tuple records associated with a tuple name are accumulated at one or more nodes according to a linked list structure at each that is accessible via a hash table index pointer at the node.

• Research Organization: International Business Machines Corp., Armonk, NY (United States)
• Sponsoring Organization: USDOE
• Assignee: International Business Machines Corporation (Armonk, NY)
• Patent Number(s): 11,269,687
• Application Number: 16/437,320
• OSTI ID: 1892689
• Country of Publication: United States
• Language: English