Data shuffling with hierarchical tuple spaces

Patent · Tue Jan 12 04:00:00 EST 2021

OSTI ID:1805382

Kayi, Abdullah; Andrade Costa, Carlos Henrique; Park, Yoonho; Johns, Charles Ray

Methods and systems for shuffling data to generate a dataset are described. A first map module may generate first pair data, and a second map module may generate second pair data, from source data. The first map module may insert the first pair data into a first local tuple space accessible to the first map module. The second map module may insert the second pair data into a second local tuple space accessible to the second map module. A shuffle module may request pair data that includes a particular key. The first and second pair data may be inserted into a global tuple space accessible by the first and second map modules. The shuffle module may identify the requested pair data in the global tuple space, and may fetch the identified pair data from a memory. The shuffle module may shuffle the fetched pair data to generate the dataset.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC02-05CH11231
• Assignee: International Business Machines Corporation (Armonk, NY)
• Patent Number(s): 10,891,274
• Application Number: 15/851,511
• OSTI ID: 1805382
• Country of Publication: United States
• Language: English