Exploring the Performance of Spark for a Scientific Use Case

Conference · Fri Jan 01 04:00:00 EST 2016

OSTI ID:1250827

Sehrish, Saba ^[1]; Kowalkowski, Jim ^[1]; Paterno, Marc ^[1]

---

1. Fermilab

We present an evaluation of the performance of a Spark implementation of a classification algorithm in the domain of High Energy Physics (HEP). Spark is a general engine for in-memory, large-scale data processing, and is designed for applications where similar repeated analysis is performed on the same large data sets. Classification problems are one of the most common and critical data processing tasks across many domains. Many of these data processing tasks are both computation- and data-intensive, involving complex numerical computations employing extremely large data sets. We evaluated the performance of the Spark implementation on Cori, a NERSC resource, and compared the results to an untuned MPI implementation of the same algorithm. While the Spark implementation scaled well, it is not competitive in speed to our MPI implementation, even when using significantly greater computational resources.

• Research Organization: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
• Sponsoring Organization: USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
• DOE Contract Number: AC02-07CH11359
• OSTI ID: 1250827
• Report Number(s): FERMILAB-CONF-16-072-CD; 1442301
• Country of Publication: United States
• Language: English