Title: Evaluating lossy data compression on climate simulation data within a large ensemble

Abstract

High-resolution Earth system model simulations generate enormous data volumes, and retaining the data from these simulations often strains institutional storage resources. Further, these exceedingly large storage requirements negatively impact science objectives, for example, by forcing reductions in data output frequency, simulation length, or ensemble size. To lessen data volumes from the Community Earth System Model (CESM), we advocate the use of lossy data compression techniques. While lossy data compression does not exactly preserve the original data (as lossless compression does), lossy techniques have an advantage in terms of smaller storage requirements. To preserve the integrity of the scientific simulation data, the effects of lossy data compression on the original data should, at a minimum, not be statistically distinguishable from the natural variability of the climate system, and previous preliminary work with data from CESM has shown this goal to be attainable. However, to ultimately convince climate scientists that it is acceptable to use lossy data compression, we provide climate scientists with access to publicly available climate data that have undergone lossy data compression. In particular, we report on the results of a lossy data compression experiment with output from the CESM Large Ensemble (CESM-LE) Community Project, in which we challengemore » climate scientists to examine features of the data relevant to their interests, and attempt to identify which of the ensemble members have been compressed and reconstructed. We find that while detecting distinguishing features is certainly possible, the compression effects noticeable in these features are often unimportant or disappear in post-processing analyses. In addition, we perform several analyses that directly compare the original data to the reconstructed data to investigate the preservation, or lack thereof, of specific features critical to climate science. Overall, we conclude that applying lossy data compression to climate simulation data is both advantageous in terms of data reduction and generally acceptable in terms of effects on scientific results.« less

Authors:

Baker, Allison H. ^[1]; Hammerling, Dorit M.  ^[1];

• Search DOE PAGES for author "Hammerling, Dorit M."
• Search DOE PAGES for ORCID "0000-0003-3583-3611"

Mickelson, Sheri A. ^[1]; Xu, Haiying ^[1]; Stolpe, Martin B. ^[2]; Naveau, Phillipe ^[3]; Sanderson, Ben ^[1]; Ebert-Uphoff, Imme  ^[4];

• Search DOE PAGES for author "Ebert-Uphoff, Imme"
• Search DOE PAGES for ORCID "0000-0001-6470-1947"

Samarasinghe, Savini ^[4]; De Simone, Francesco ^[5]; Carbone, Francesco ^[5]; Gencarelli, Christian N.  ^[5];

• Search DOE PAGES for author "Gencarelli, Christian N."
• Search DOE PAGES for ORCID "0000-0002-2366-661X"

Dennis, John M. ^[1]; Kay, Jennifer E. ^[6]; Lindstrom, Peter  ^[7]

• Search DOE PAGES for author "Lindstrom, Peter"
• Search DOE PAGES for ORCID "0000-0003-3817-4199"

---

+ Show Author Affiliations

1. National Center for Atmospheric Research, Boulder, CO (United States)
2. ETH Zurich (Switzerland). Institute for Atmospheric and Climate Science
3. Laboratoire des Sciences du Climat et l Environnement, Gif-sur-Yvette (France)
4. Colorado State Univ., Fort Collins, CO (United States). Department of Electrical and Computer Engineering
5. CNR-Institute of Atmospheric Pollution Research, Rende (Italy). Division of Rende, UNICAL-Polifunzional
6. Univ. of Colorado, Boulder, CO (United States). Department of Oceanic and Atmospheric Sciences
7. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States). Center for Applied Scientific Computing

Publication Date:

Wed Dec 07 00:00:00 EST 2016

Research Org.:

Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Sponsoring Org.:

USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:

1389988

Report Number(s):

LLNL-JRNL-691060
Journal ID: ISSN 1991-9603

Grant/Contract Number:  

AC52-07NA27344

Resource Type:

Accepted Manuscript

Journal Name:

Geoscientific Model Development (Online)

Additional Journal Information:

Journal Name: Geoscientific Model Development (Online); Journal Volume: 9; Journal Issue: 12; Journal ID: ISSN 1991-9603

Publisher:

European Geosciences Union

Country of Publication:

United States

Language:

English

Subject:

58 GEOSCIENCES; 97 MATHEMATICS AND COMPUTING; 54 ENVIRONMENTAL SCIENCES