3D Coded SUMMA: Communication-Efficient and Robust Parallel Matrix Multiplication

Conference · Sat Aug 01 04:00:00 EDT 2020

DOI:https://doi.org/10.1007/978-3-030-57675-2_25· OSTI ID:1651320

Jeong, Haewon ^[1]; Yang, Yaoqing ^[2]; Gupta, Vipul ^[2]; Engelmann, Christian ^[3]; Meng Low, Tze ^[1]; Cadambe, Viveck ^[4]; Ramchandran, Kannan ^[2]; Grover, Pulkit ^[1]

---

1. Carnegie Mellon Univ., Pittsburgh, PA (United States)
2. Univ. of California, Berkeley, CA (United States)
3. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
4. Pennsylvania State Univ., University Park, PA (United States)

In this paper, we propose a novel fault-tolerant parallel matrix multiplication algorithm called 3D Coded SUMMA that achieves higher failure-tolerance than replication-based schemes for the same amount of redundancy. This work bridges the gap between recent developments in coded computing and fault-tolerance in high-performance computing (HPC). The core idea of coded computing is the same as algorithm-based fault-tolerance (ABFT), which is weaving redundancy in the computation using error-correcting codes. In particular, we show that MatDot codes, an innovative code construction for parallel matrix multiplications, can be integrated into three-dimensional SUMMA (Scalable Universal Matrix Multiplication Algorithm [30]) in a communication-avoiding manner. To tolerate any two node failures, the proposed 3D Coded SUMMA requires ~50% less redundancy than replication, while the overhead in execution time is only about 5–10%.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1651320
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Algorithm-based fault tolerance
Coded computing
Communication-efficient algorithms
Error detection and correction
Fault-tolerant algorithms
Parallel Matrix Multiplication