AmeriFlux BASE data pipeline to support network growth and data sharing

Chu, Housen; Christianson, Danielle S.; Cheah, You-Wei; Pastorello, Gilberto; O’Brien, Fianna; Geden, Joshua; Ngo, Sy-Toan; Hollowgrass, Rachel; Leibowitz, Karla; Beekwilder, Norman F.; Sandesh, Megha; Dengel, Sigrid; Chan, Stephen W.; Santos, André; Delwiche, Kyle; Yi, Koong; Buechner, Christin; Baldocchi, Dennis; Papale, Dario; Keenan, Trevor F.; Biraud, Sébastien C.; Agarwal, Deborah A.; Torn, Margaret S.

doi:10.1038/s41597-023-02531-2

Title: AmeriFlux BASE data pipeline to support network growth and data sharing

Journal Article · 11 September 2023 · Scientific Data

DOI: https://doi.org/10.1038/s41597-023-02531-2 · OSTI ID:1999441

^[1];

^[1]; Cheah, You-Wei ^[1];

^[1];

^[1]; Geden, Joshua ^[1]; Ngo, Sy-Toan ^[1]; Hollowgrass, Rachel ^[2]; Leibowitz, Karla ^[3]; Beekwilder, Norman F. ^[4]; Sandesh, Megha ^[1]; Dengel, Sigrid ^[1]; Chan, Stephen W. ^[1]; Santos, André ^[1]; Delwiche, Kyle ^[2];

^[1]; Buechner, Christin ^[1]; Baldocchi, Dennis ^[2];

^[5];

^[6] more »

Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
University of California, Berkeley, CA (United States)
HyperArts, Inc, Oakland, CA (United States)
University of Virginia, Charlottesville, VA (United States)
University of Tuscia, Viterbo (Italy); Euro-Mediterranean Center on Climate Change (CMCC), Viterbo (Italy)
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); University of California, Berkeley, CA (United States)

AmeriFlux is a network of research sites that measure carbon, water, and energy fluxes between ecosystems and the atmosphere using the eddy covariance technique to study a variety of Earth science questions. AmeriFlux’s diversity of ecosystems, instruments, and data-processing routines create challenges for data standardization, quality assurance, and sharing across the network. To address these challenges, the AmeriFlux Management Project (AMP) designed and implemented the BASE data-processing pipeline. The pipeline begins with data uploaded by the site teams, followed by the AMP team’s quality assurance and quality control (QA/QC), ingestion of site metadata, and publication of the BASE data product. The semi-automated pipeline enables us to keep pace with the rapid growth of the network. As of 2022, the AmeriFlux BASE data product contains 3,130 site years of data from 444 sites, with standardized units and variable names of more than 60 common variables, representing the largest long-term data repository for flux-met data in the world. The standardized, quality-ensured data product facilitates multisite comparisons, model evaluations, and data syntheses.

View Journal Article

Cite

Export

Save

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER). Earth & Environmental Systems Science (EESS); USDOE

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1999441

Alternate ID(s):: OSTI ID: 2281234

Journal Information:: Scientific Data, Vol. 10, Issue 1; ISSN 2052-4463

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United States

Language:: English

References (64)

Annual carbon dioxide exchange in irrigated and rainfed maize-based agroecosystems Verma, Shashi B.; Dobermann, Achim; Cassman, Kenneth G. Agricultural and Forest Meteorology, Vol. 131, Issue 1-2 https://doi.org/10.1016/j.agrformet.2005.05.003	journal	July 2005
FLUXNET-CH ₄ Synthesis Activity: Objectives, Observations, and Future Directions Knox, Sara H.; Jackson, Robert B.; Poulter, Benjamin Bulletin of the American Meteorological Society https://doi.org/10.1175/BAMS-D-18-0268.1	journal	July 2019
Impact of post-field data processing on eddy covariance flux estimates and energy balance closure [Impact of post-field data processing on eddy covariance flux estimates and energy balance closure] Mauder, Matthias; Foken, Thomas Meteorologische Zeitschrift, Vol. 15, Issue 6 https://doi.org/10.1127/0941-2948/2006/0167	journal	December 2006
Simulating surface energy fluxes using the variable-resolution Community Earth System Model (VR-CESM) Burakowski, Elizabeth A.; Tawfik, Ahmed; Ouimette, Andrew Theoretical and Applied Climatology, Vol. 138, Issue 1-2 https://doi.org/10.1007/s00704-019-02785-0	journal	February 2019
A Global Terrestrial Monitoring Network Integrating Tower Fluxes, Flask Sampling, Ecosystem Modeling and EOS Satellite Data Running, S. W.; Baldocchi, D. D.; Turner, D. P. Remote Sensing of Environment, Vol. 70, Issue 1 https://doi.org/10.1016/S0034-4257(99)00061-9	journal	October 1999
Relative humidity effects on water vapour fluxes measured with closed-path eddy-covariance systems with short sampling lines Fratini, Gerardo; Ibrom, Andreas; Arriga, Nicola Agricultural and Forest Meteorology, Vol. 165 https://doi.org/10.1016/j.agrformet.2012.05.018	journal	November 2012
Assessing the carbon and climate benefit of restoring degraded agricultural peat soils to managed wetlands Hemes, Kyle S.; Chamberlain, Samuel D.; Eichelmann, Elke Agricultural and Forest Meteorology, Vol. 268 https://doi.org/10.1016/j.agrformet.2019.01.017	journal	April 2019
Seasonal variation in methane emission from a temperate Phragmites‐dominated marsh: effect of growth stage and plant‐mediated transport Kim, JooN.; Verma, Shashi B.; Billesbach, David P. Global Change Biology, Vol. 5, Issue 4 https://doi.org/10.1046/j.1365-2486.1999.00237.x	journal	April 1999
Evaluation of remote sensing based terrestrial productivity from MODIS using regional tower eddy flux network observations Heinsch, F. A.; Running, S. W. IEEE Transactions on Geoscience and Remote Sensing, Vol. 44, Issue 7 https://doi.org/10.1109/TGRS.2005.853936	journal	July 2006
Challenging problems of quality assurance and quality control (QA/QC) of meteorological time series data Faybishenko, B.; Versteeg, R.; Pastorello, G. Stochastic Environmental Research and Risk Assessment, Vol. 36, Issue 4 https://doi.org/10.1007/s00477-021-02106-w	journal	November 2021
Dynamics of isotopic exchange of carbon dioxide in a Tennessee deciduous forest Bowling, David R.; Baldocchi, Dennis D.; Monson, Russell K. Global Biogeochemical Cycles, Vol. 13, Issue 4 https://doi.org/10.1029/1999GB900072	journal	December 1999
Observational Data Patterns for Time Series Data Quality Assessment Pastorello, Gilberto; Agarwal, Deb; Papale, Dario 2014 IEEE 10th International Conference on e-Science (e-Science) https://doi.org/10.1109/eScience.2014.45	conference	October 2014
Assessment of the Long-Term High-Spatial-Resolution Global LAnd Surface Satellite (GLASS) Surface Longwave Radiation Product Using Ground Measurements Zeng, Qi; Cheng, Jie; Dong, Lixin IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 13 https://doi.org/10.1109/JSTARS.2020.2992472	journal	January 2020
The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation Collier, Nathan; Hoffman, Forrest M.; Lawrence, David M. Journal of Advances in Modeling Earth Systems, Vol. 10, Issue 11 https://doi.org/10.1029/2018MS001354	journal	November 2018
FLUXNET-CH₄: a global, multi-ecosystem dataset and analysis of methane seasonality from freshwater wetlands Delwiche, Kyle B.; Knox, Sara Helen; Malhotra, Avni Earth System Science Data, Vol. 13, Issue 7 https://doi.org/10.5194/essd-13-3607-2021	journal	January 2021
Informing Nature‐based Climate Solutions for the United States with the best‐available science Novick, Kimberly A.; Metzger, Stefan; Anderegg, William R. L. Global Change Biology, Vol. 28, Issue 12 https://doi.org/10.1111/gcb.16156	journal	April 2022
CO ₂ exchange and evapotranspiration across dryland ecosystems of southwestern North America Biederman, Joel A.; Scott, Russell L.; Bell, Tom W. Global Change Biology, Vol. 23, Issue 10 https://doi.org/10.1111/gcb.13686	journal	April 2017
Carbon budget of the Harvard Forest Long‐Term Ecological Research site: pattern, process, and response to global change Finzi, Adrien C.; Giasson, Marc‐André; Barker Plotkin, Audrey A. Ecological Monographs, Vol. 90, Issue 4 https://doi.org/10.1002/ecm.1423	journal	October 2020
Pairing FLUXNET sites to validate model representations of land-use/land-cover change Chen, Liang; Dirmeyer, Paul A.; Guo, Zhichang Hydrology and Earth System Sciences, Vol. 22, Issue 1 https://doi.org/10.5194/hess-22-111-2018	journal	January 2018
The Integrated Carbon Observation System in Europe Heiskanen, Jouni; Brümmer, Christian; Buchmann, Nina Bulletin of the American Meteorological Society, Vol. 103, Issue 3 https://doi.org/10.1175/BAMS-D-19-0364.1	journal	March 2022
The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data Pastorello, Gilberto; Trotta, Carlo; Canfora, Eleonora Scientific Data, Vol. 7, Issue 1 https://doi.org/10.1038/s41597-020-0534-3	journal	July 2020
Improving the performance of remote sensing models for capturing intra- and inter-annual variations in daily GPP: An analysis using global FLUXNET tower data Verma, M.; Friedl, M. A.; Law, B. E. Agricultural and Forest Meteorology, Vol. 214-215 https://doi.org/10.1016/j.agrformet.2015.09.005	journal	December 2015
Effects of land-use change and drought on decadal evapotranspiration and water balance of natural and managed forested wetlands along the southeastern US lower coastal plain Aguilos, Maricar; Sun, Ge; Noormets, Asko Agricultural and Forest Meteorology, Vol. 303 https://doi.org/10.1016/j.agrformet.2021.108381	journal	June 2021
Towards long-term standardised carbon and greenhouse gas observations for monitoring Europe’s terrestrial ecosystems: a review Franz, Daniela; Acosta, Manuel; Altimir, Núria International Agrophysics, Vol. 32, Issue 4 https://doi.org/10.1515/intag-2017-0039	journal	December 2018
Environmental Informatics Using Research Infrastructures and their Data: Fall 2020 Edition Duffy, Katharyn; Seyednasrollah, Bijan; Read, Jordan S. Zenodo https://doi.org/10.5281/zenodo.4576496	interactiveresource	January 2021
Climate control of terrestrial carbon exchange across biomes and continents Yi, Chuixiang; Ricciuto, Daniel; Li, Runze Environmental Research Letters, Vol. 5, Issue 3 https://doi.org/10.1088/1748-9326/5/3/034007	journal	July 2010
Agricultural peatland restoration: effects of land-use change on greenhouse gas (CO ₂ and CH ₄ ) fluxes in the Sacramento-San Joaquin Delta Knox, Sara Helen; Sturtevant, Cove; Matthes, Jaclyn Hatala Global Change Biology, Vol. 21, Issue 2 https://doi.org/10.1111/gcb.12745	journal	October 2014
An eddy covariance mesonet to measure the effect of forest age on land?atmosphere exchange Goulden, Michael L.; Winston, Gregory C.; McMILLAN, Andrew M. S. Global Change Biology, Vol. 12, Issue 11 https://doi.org/10.1111/j.1365-2486.2006.01251.x	journal	November 2006
From NEON Field Sites to Data Portal: A Community Resource for Surface–Atmosphere Research Comes Online Metzger, Stefan; Ayres, Edward; Durden, David Bulletin of the American Meteorological Society, Vol. 100, Issue 11 https://doi.org/10.1175/BAMS-D-17-0307.1	journal	November 2019
Disentangling the Relative Drivers of Seasonal Evapotranspiration Across a Continental‐Scale Aridity Gradient Young, Adam M.; Friedl, Mark A.; Novick, Kimberly Journal of Geophysical Research: Biogeosciences, Vol. 127, Issue 8 https://doi.org/10.1029/2022JG006916	journal	August 2022
MODIS-Based Estimation of Terrestrial Latent Heat Flux over North America Using Three Machine Learning Algorithms Wang, Xuanyu; Yao, Yunjun; Zhao, Shaohua Remote Sensing, Vol. 9, Issue 12 https://doi.org/10.3390/rs9121326	journal	December 2017
OzFlux data: network integration from collection to curation Isaac, Peter; Cleverly, James; McHugh, Ian Biogeosciences, Vol. 14, Issue 12 https://doi.org/10.5194/bg-14-2903-2017	journal	January 2017
A robust data cleaning procedure for eddy covariance flux measurements Vitale, Domenico; Fratini, Gerardo; Bilancia, Massimo Biogeosciences, Vol. 17, Issue 6 https://doi.org/10.5194/bg-17-1367-2020	journal	January 2020
A process approach to quality management doubles NEON sensor data quality Sturtevant, Cove; DeRego, Elizabeth; Metzger, Stefan Methods in Ecology and Evolution, Vol. 13, Issue 9 https://doi.org/10.1111/2041-210X.13943	journal	July 2022
Eddy covariance raw data processing for CO2 and energy fluxes calculation at ICOS ecosystem stations Sabbatini, Simone; Mammarella, Ivan; Arriga, Nicola International Agrophysics, Vol. 32, Issue 4 https://doi.org/10.1515/intag-2017-0043	journal	December 2018
eddy4R 0.2.0: a DevOps model for community-extensible processing and analysis of eddy-covariance data based on R, Git, Docker, and HDF5 Metzger, Stefan; Durden, David; Sturtevant, Cove Geoscientific Model Development, Vol. 10, Issue 9 https://doi.org/10.5194/gmd-10-3189-2017	journal	January 2017
Rectification of the Bias in the Wavelet Power Spectrum Liu, Yonggang; San Liang, X.; Weisberg, Robert H. Journal of Atmospheric and Oceanic Technology, Vol. 24, Issue 12 https://doi.org/10.1175/2007JTECHO511.1	journal	December 2007
Drivers of Decadal Carbon Fluxes Across Temperate Ecosystems Desai, Ankur R.; Murphy, Bailey A.; Wiesner, Susanne Journal of Geophysical Research: Biogeosciences, Vol. 127, Issue 12 https://doi.org/10.1029/2022JG007014	journal	December 2022
ICOS eddy covariance flux-station site setup: a review Rebmann, Corinna; Aubinet, Marc; Schmid, HaPe International Agrophysics, Vol. 32, Issue 4 https://doi.org/10.1515/intag-2017-0044	journal	December 2018
Growth and opportunities in networked synthesis through AmeriFlux Keenan, Trevor F.; Moore, David J. P.; Desai, Ankur New Phytologist, Vol. 222, Issue 4 https://doi.org/10.1111/nph.15835	journal	May 2019
Combined measurement and modeling of the hydrological impact of hydraulic redistribution using CLM4.5 at eight AmeriFlux sites Fu, Congsheng; Wang, Guiling; Goulden, Michael L. Hydrology and Earth System Sciences, Vol. 20, Issue 5 https://doi.org/10.5194/hess-20-2001-2016	journal	January 2016
Data-driven diagnostics of terrestrial carbon dynamics over North America Xiao, Jingfeng; Ollinger, Scott V.; Frolking, Steve Agricultural and Forest Meteorology, Vol. 197 https://doi.org/10.1016/j.agrformet.2014.06.013	journal	October 2014
Modification of surface energy balance during springtime: The relative importance of biophysical and meteorological changes Moon, Minkyu; Li, Dan; Liao, Weilin Agricultural and Forest Meteorology, Vol. 284 https://doi.org/10.1016/j.agrformet.2020.107905	journal	April 2020
Ideas and perspectives: enhancing the impact of the FLUXNET network of eddy covariance sites Papale, Dario Biogeosciences, Vol. 17, Issue 22 https://doi.org/10.5194/bg-17-5587-2020	journal	January 2020
Warm spring reduced carbon cycle impact of the 2012 US summer drought Wolf, Sebastian; Keenan, Trevor F.; Fisher, Joshua B. Proceedings of the National Academy of Sciences, Vol. 113, Issue 21 https://doi.org/10.1073/pnas.1519620113	journal	April 2016
Terrestrial carbon balance in a drier world: the effects of water availability in southwestern North America Biederman, Joel A.; Scott, Russell L.; Goulden, Michael L. Global Change Biology, Vol. 22, Issue 5 https://doi.org/10.1111/gcb.13222	journal	February 2016
Improved dryland carbon flux predictions with explicit consideration of water-carbon coupling Barnes, Mallory L.; Farella, Martha M.; Scott, Russell L. Communications Earth & Environment, Vol. 2, Issue 1 https://doi.org/10.1038/s43247-021-00308-2	journal	December 2021
ECOSTRESS: NASA's Next Generation Mission to Measure Evapotranspiration From the International Space Station Fisher, Joshua B.; Lee, Brian; Purdy, Adam J. Water Resources Research, Vol. 56, Issue 4 https://doi.org/10.1029/2019WR026058	journal	April 2020
Shrubland carbon sink depends upon winter water availability in the warm deserts of North America Biederman, Joel A.; Scott, Russell L.; Arnone III, John A. Agricultural and Forest Meteorology, Vol. 249 https://doi.org/10.1016/j.agrformet.2017.11.005	journal	February 2018
Multi‐Decadal Carbon Cycle Measurements Indicate Resistance to External Drivers of Change at the Howland Forest AmeriFlux Site Hollinger, D. Y.; Davidson, E. A.; Fraver, S. Journal of Geophysical Research: Biogeosciences, Vol. 126, Issue 8 https://doi.org/10.1029/2021JG006276	journal	August 2021
Net Exchange of CO2 in a Mid-Latitude Forest Wofsy, S. C.; Goulden, M. L.; Munger, J. W. Science, Vol. 260, Issue 5112 https://doi.org/10.1126/science.260.5112.1314	journal	May 1993
Tidal Wetland Gross Primary Production Across the Continental United States, 2000–2019 Feagin, R. A.; Forbrich, I.; Huff, T. P. Global Biogeochemical Cycles, Vol. 34, Issue 2 https://doi.org/10.1029/2019GB006349	journal	February 2020
Inter-annual variability of net and gross ecosystem carbon fluxes: A review Baldocchi, Dennis; Chu, Housen; Reichstein, Markus Agricultural and Forest Meteorology, Vol. 249 https://doi.org/10.1016/j.agrformet.2017.05.015	journal	February 2018
Comparing laser-based open- and closed-path gas analyzers to measure methane fluxes using the eddy covariance method Detto, Matteo; Verfaillie, Joseph; Anderson, Frank Agricultural and Forest Meteorology, Vol. 151, Issue 10 https://doi.org/10.1016/j.agrformet.2011.05.014	journal	October 2011
FLUXNET: A New Tool to Study the Temporal and Spatial Variability of Ecosystem–Scale Carbon Dioxide, Water Vapor, and Energy Flux Densities Baldocchi, Dennis; Falge, Eva; Gu, Lianhong Bulletin of the American Meteorological Society, Vol. 82, Issue 11 https://doi.org/10.1175/1520-0477(2001)082<2415:FANTTS>2.3.CO;2	journal	November 2001
The three major axes of terrestrial ecosystem function Migliavacca, Mirco; Musavi, Talie; Mahecha, Miguel D. Nature, Vol. 598, Issue 7881 https://doi.org/10.1038/s41586-021-03939-9	journal	September 2021
How close are we to the temperature tipping point of the terrestrial biosphere? Duffy, Katharyn A.; Schwalm, Christopher R.; Arcus, Vickery L. Science Advances, Vol. 7, Issue 3 https://doi.org/10.1126/sciadv.aay1052	journal	January 2021
Quantifying the uncertainty of eddy covariance fluxes due to the use of different software packages and combinations of processing steps in two contrasting ecosystems Mammarella, Ivan; Peltola, Olli; Nordbo, Annika Atmospheric Measurement Techniques, Vol. 9, Issue 10 https://doi.org/10.5194/amt-9-4915-2016	journal	January 2016
Evaluating the Spatial Representativeness of the MODerate Resolution Image Spectroradiometer Albedo Product (MCD43) at AmeriFlux Sites Zhou, Hongmin; Liang, Shunlin; He, Tao Remote Sensing, Vol. 11, Issue 5 https://doi.org/10.3390/rs11050547	journal	March 2019
Canopy near-infrared reflectance and terrestrial photosynthesis Badgley, Grayson; Field, Christopher B.; Berry, Joseph A. Science Advances, Vol. 3, Issue 3 https://doi.org/10.1126/sciadv.1602244	journal	March 2017
Howland Forest, ME, USA: Multi-Gas Flux (CO2, CH4, N2O) Social Cost Product Underscores Limited Carbon Proxies Marino, Bruno D. V.; Bautista, Nahuel; Rousseaux, Brandt Land, Vol. 10, Issue 4 https://doi.org/10.3390/land10040436	journal	April 2021
The AmeriFlux network: A coalition of the willing Novick, K. A.; Biederman, J. A.; Desai, A. R. Agricultural and Forest Meteorology, Vol. 249 https://doi.org/10.1016/j.agrformet.2017.10.009	journal	February 2018
Environmental Informatics Using Research Infrastructures and their Data: Fall 2020 Edition Duffy, Katharyn; Seyednasrollah, Bijan; Read, Jordan S. Zenodo https://doi.org/10.5281/zenodo.4576496	interactiveresource	January 2021
ECOSTRESS: NASA’s next generation mission to measure evapotranspiration from the International Space Station Fisher, Joshua B. Root https://doi.org/10.48577/jpl.cyxicv	dataset	January 2023

Similar Records

AmeriFlux Measurement Network: Science Team Research

Technical Report · 2012 · OSTI ID:1057202

Law, B E

The AmeriFlux Data Activity and Data System: An Evolving Collection of Data Management Techniques, Tools, Products and Services

Journal Article · 2013 · Geoscientific Instrumentation, Methods and Data Systems (Online) · OSTI ID:1086638

Boden, Thomas A.; Krassovski, Misha B.; Yang, Bai

QA/QC-ed Groundwater Level Time Series in PLM-1 and PLM-6 Monitoring Wells, East River, Colorado (2016-2022)

Dataset · 2023 · OSTI ID:1866836

Faybishenko, Boris; Versteeg, Roelof; Williams, Kenneth; +4 more

Related Subjects

29 ENERGY PLANNING, POLICY, AND ECONOMY

Title: AmeriFlux BASE data pipeline to support network growth and data sharing

Citation Formats

References (64)

Similar Records

Related Subjects