Data-parallel Python for High Energy Physics Analyses

Conference · Fri Oct 26 04:00:00 EDT 2018

OSTI ID:1490837

Paterno, Marc ^[1]; Green, C. ^[1]; Kowalski, J. ^[1]; Sehrish, S. ^[1]

---

1. Fermilab

In this paper, we explore features available in Python which are useful for data reduction tasks in High Energy Physics (HEP). Highlevel abstractions in Python are convenient for implementing data reduction tasks. However, in order for such abstractions to be practical, the efficiency of their performance must also be high. Because the data sets we process are typically large, we care about both I/O performance and in-memory processing speed. In particular, we evaluate the use of data-parallel programming, using MPI and numpy, to process a large experimental data set (42 TiB) stored in an HDF5 file. We measure the speed of processing of the data, distinguishing between the time spent reading data and the time spent processing the data in memory, and demonstrate the scalability of both, using up to 1200 KNL nodes (76800 cores) on Cori at NERSC

• Research Organization: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
• Sponsoring Organization: USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
• DOE Contract Number: AC02-07CH11359
• OSTI ID: 1490837
• Report Number(s): FERMILAB-CONF-18-577-CD; 1712348
• Country of Publication: United States
• Language: English