Automatic Generation of Algorithms for High-Speed Reliable Lossy Data Compression (Final Report)

Technical Report · Wed Dec 24 00:00:00 EST 2025

DOI:https://doi.org/10.2172/3010315· OSTI ID:3010315

Burtscher, Martin ^[1]

---

1. Texas State Univ., San Marcos, TX (United States)

Fast reliable data compression is urgently needed for many leading-edge scientific instruments and for exascale high-performance computing applications because they produce vast amounts of data at extremely high rates. The goal of this project has been to develop a framework named LC that is able to automatically generate high-speed lossless and reliable lossy compression and decompression algorithms that can be customized for different kinds of data. The resulting LC framework is freely available on GitHub. To achieve high-speed operation, LC outputs optimized and parallelized CPU and GPU implementations of the generated algorithms. To ensure the quality of lossily compressed data, LC guarantees the user-provided error bound. To be able to customize the compression algorithm to various use cases, LC can synthesize millions of different algorithms and automatically search for the one that works best for the given data. We have already employed LC to create state-of-the-art lossless and lossy compressors for scientific data as well as leading lossless compressors for images. We hope that LC and the customized, fast, reliable, and CPU/GPU-compatible compression algorithms that it can generate will greatly benefit the many scientific applications that need not only high trustworthiness but also high performance.

• Research Organization: Texas State Univ., San Marcos, TX (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
• DOE Contract Number: SC0022223
• OSTI ID: 3010315
• Report Number(s): DOE-TXST--0022223
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING