Delta-Rice: A HDF5 Compression Plugin optimized for Digitized Detector Data

Journal Article · 20 June 2024 · Journal of Open Source Software

DOI: https://doi.org/10.21105/joss.06598 · OSTI ID:2428068

Mathews, David G.  ^[1]; Crawford, C. B.  ^[2]; Baeßler, Stefan  ^[3]; Birge, N.  ^[4]; Broussard, Leah J.  ^[5]; Gonzalez, Frank  ^[5]; Hayen, L.  ^[6]; Jezghani, A.  ^[7]; Li, H.  ^[8]; Mammei, R.  ^[9]; Mendelsohn, A.  ^[10]; Randall, G.  ^[11]; Riley, G. V.  ^[12]; Schaper, D. C.  ^[13]

---

1. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Univ. of Kentucky, Lexington, KY (United States)
2. Univ. of Kentucky, Lexington, KY (United States)
3. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Univ. of Virginia, Charlottesville, VA (United States)
4. Univ. of Tennessee, Knoxville, TN (United States)
5. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
6. North Carolina State University, Raleigh, NC (United States); Triangle Universities Nuclear Laboratory (TUNL), Durham, NC (United States); University of Rouen Normandy (France)
7. Univ. of Kentucky, Lexington, KY (United States); Georgia Institute of Technology, Atlanta, GA (United States)
8. Univ. of Virginia, Charlottesville, VA (United States)
9. Univ. of Manitoba, Winnipeg, MB (Canada); University of Winnipeg, Winnipeg, MB (Canada)
10. Univ. of Manitoba, Winnipeg, MB (Canada)
11. Arizona State Univ., Tempe, AZ (United States)
12. Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
13. Univ. of Kentucky, Lexington, KY (United States); Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Delta-Rice is an HDF5 (The HDF Group et al., 2020) filter plugin that was developed to compress digitized detector signals recorded by the Nab experiment (Fry et al., 2019), a fundamental neutron physics experiment. This is a two-step process where incoming data is passed through a pre-processing filter and then compressed with Rice coding. A routine for determining the optimal pre-processing filter for a dataset is provided along with an example GPU deployment. When applied to data collected by the Nab data acquisition system, this method produced output files 29% their initial size, and was able to do so with an average read/write throughput in excess of 2 GB/s on a single CPU. Compared to the widely used Gzip compression routine, Delta-Rice reduces the file size by 33% more with over an order of magnitude increase in read/write throughput. Delta-Rice is available on CPU to users through the HDF5 library.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Nuclear Physics (NP); USDOE Office of Science (SC), Office of Workforce Development for Teachers & Scientists (WDTS)
• Grant/Contract Number: AC05-00OR22725; FG02-03ER41258; SC0014622
• OSTI ID: 2428068
• Journal Information: Journal of Open Source Software, Journal Name: Journal of Open Source Software Journal Issue: 98 Vol. 9; ISSN 2475-9066
• Publisher: Open Source Initiative - NumFOCUSCopyright Statement
• Country of Publication: United States
• Language: English