A Performant, Scalable Processing Pipeline for High‐Quality and FAIR Environmental Sensor Data

Pennington, Stephanie C.; Bond‐Lamberty, Ben; Peixoto, Roberta Bittencourt; Chen, Xingyuan; Cheng, Selina L.; Machado‐Silva, Fausto; Maier, Kurt; Phillips, Evan; Regier, Peter J.; Stearns, Alice; Ward, Nicholas D.; Weintraub, Michael N.; Wilson, Stephanie J.; Bailey, Vanessa; Rich, Roy

doi:10.1029/2025JG008807

A Performant, Scalable Processing Pipeline for High‐Quality and FAIR Environmental Sensor Data

Journal Article · Fri Nov 07 19:00:00 EST 2025 · Journal of Geophysical Research. Biogeosciences

DOI:https://doi.org/10.1029/2025JG008807· OSTI ID:3008947

^[1]; ^[1]; Peixoto, Roberta Bittencourt ^[2]; ^[3]; ^[4]; ^[2]; ^[3]; Phillips, Evan ^[4]; ^[3]; ^[4]; ^[3]; ^[5]; ^[4]; ^[5]; ^[4]

Pacific Northwest National Laboratory (PNNL), College Park, MD (United States). Joint Global Change Research Institute
Univ. of Toledo, OH (United States)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
Smithsonian Environmental Research Center, Edgewater, MD (United States)
Univ. of Toledo, OH (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

High-resolution environmental monitoring is necessary to record, understand, and predict biogeochemical and ecological changes particularly in coastal systems but brings significant challenges in processing and making rapidly available the resulting data. The COMPASS-FME project established a network of coastal observational sites across the Chesapeake Bay and western Lake Erie regions extensively instrumented with soil, vegetation, and weather sensors logging data every 15 min. Our data processing framework, written in R and completely open source, prioritizes rapid model-experiment iteration and makes biogeochemical data rapidly available for quality assurance/quality control, analysis, and model ingestion. This pipeline is distinguished by a standardized and modular approach to data curation, extensive metadata and documentation, and its high performance. These attributes combine to make biogeochemical data rapidly accessible across COMPASS-FME and the broader community. Flexible, powerful, and reproducible approaches to handling high-volume environmental data are crucial for accelerating biogeosciences research.

View Accepted Manuscript (DOE)

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER). Earth & Environmental Systems Science (EESS)

Grant/Contract Number:: AC05-76RL01830

OSTI ID:: 3008947

Report Number(s):: PNNL-SA--205989

Journal Information:: Journal of Geophysical Research. Biogeosciences, Journal Name: Journal of Geophysical Research. Biogeosciences Journal Issue: 11 Vol. 130; ISSN 2169-8953; ISSN 2169-8961

Publisher:: American Geophysical UnionCopyright Statement

Country of Publication:: United States

Language:: English

References (35)

Open science, reproducibility, and transparency in ecology Powers, Stephen M.; Hampton, Stephanie E. Ecological Applications, Vol. 29, Issue 1 https://doi.org/10.1002/eap.1822	journal	November 2018
There and back again: Outlier detection between statistical reasoning and data mining algorithms Zimek, Arthur; Filzmoser, Peter WIREs Data Mining and Knowledge Discovery, Vol. 8, Issue 6 https://doi.org/10.1002/widm.1280	journal	August 2018
ggplot2 Wickham, Hadley Springer New York, NY https://doi.org/10.1007/978-0-387-98141-3	book	January 2009
Multiple models and experiments underscore large uncertainty in soil carbon dynamics Sulman, Benjamin N.; Moore, Jessica A. M.; Abramoff, Rose Biogeochemistry, Vol. 141, Issue 2 https://doi.org/10.1007/s10533‐018‐0509‐z	journal	October 2018
Gap-filling eddy covariance methane fluxes: Comparison of machine learning model predictions and uncertainties at FLUXNET-CH4 wetlands Irvin, Jeremy; Zhou, Sharon; McNicol, Gavin Agricultural and Forest Meteorology, Vol. 308-309 https://doi.org/10.1016/j.agrformet.2021.108528	journal	October 2021
An R package for identification of outliers in environmental time series data Čampulová, Martina; Čampula, Roman; Holešovský, Jan Environmental Modelling & Software, Vol. 155 https://doi.org/10.1016/j.envsoft.2022.105435	journal	September 2022
Data pipeline for managing field experiments Liu, Jian; Cichota, Rogerio; Langer, Stephanie MethodsX, Vol. 10 https://doi.org/10.1016/j.mex.2023.102031	journal	January 2023
Salinization, inundation and tree mortality interact to affect greenhouse gas emissions from stressed coastal forests Seyfried, Georgia S.; Chow, Alex T.; O'Halloran, Thomas L. Soil Biology and Biochemistry, Vol. 184 https://doi.org/10.1016/j.soilbio.2023.109101	journal	September 2023
Ecoinformatics: supporting ecology as a data-intensive science Michener, William K.; Jones, Matthew B. Trends in Ecology & Evolution, Vol. 27, Issue 2 https://doi.org/10.1016/j.tree.2011.11.016	journal	February 2012
Short-Term Groundwater Level Fluctuations Drive Subsurface Redox Variability Machado-Silva, Fausto; Weintraub, Michael N.; Ward, Nicholas D. Environmental Science & Technology, Vol. 58, Issue 33 https://doi.org/10.1021/acs.est.4c01115	journal	August 2024
Launching an Accessible Archive of Environmental Data Varadharajan, Charuleka; Cholia, Shreyas; Snavely, Cory Eos, Vol. 100 https://doi.org/10.1029/2019EO111263	journal	January 2019
From Open Data to Open Science Ramachandran, Rahul; Bugbee, Kaylin; Murphy, Kevin Earth and Space Science, Vol. 8, Issue 5 https://doi.org/10.1029/2020ea001562	journal	May 2021
Sea Level‐Driven Marsh Migration Results in Rapid Net Loss of Carbon Smith, Alexander J.; Kirwan, Matthew L. Geophysical Research Letters, Vol. 48, Issue 13 https://doi.org/10.1029/2021gl092420	journal	July 2021
Design and Assessment of a Novel Approach for Ecosystem Warming Experiments in High‐Energy Tidal Wetlands Rich, Roy L.; Mueller, Peter; Fuß, Miriam Journal of Geophysical Research: Biogeosciences, Vol. 128, Issue 11 https://doi.org/10.1029/2023jg007550	journal	October 2023
Challenges Facing Scientific Publishing in the Field of Earth & Space Sciences No authors listed AGU Advances, Vol. 5, Issue 4 https://doi.org/10.1029/2024AV001334	journal	July 2024
A Synoptic System for Capturing Ecosystem Control Points Across Terrestrial‐Aquatic Interfaces Ward, Nicholas D.; Megonigal, J. Patrick; Weintraub, Michael N. Journal of Geophysical Research: Biogeosciences, Vol. 130, Issue 10 https://doi.org/10.1029/2025JG009335	journal	October 2025
Tidal wetland stability in the face of human impacts and sea-level rise Kirwan, Matthew L.; Megonigal, J. Patrick Nature, Vol. 504, Issue 7478 https://doi.org/10.1038/nature12856	journal	December 2013
The changing carbon cycle of the coastal ocean Bauer, James E.; Cai, Wei-Jun; Raymond, Peter A. Nature, Vol. 504, Issue 7478 https://doi.org/10.1038/nature12857	journal	December 2013
The FAIR Guiding Principles for scientific data management and stewardship Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan Scientific Data, Vol. 3, Issue 1 https://doi.org/10.1038/sdata.2016.18	journal	March 2016
Situating Ecology as a Big-Data Science: Current Advances, Challenges, and Solutions Farley, Scott S.; Dawson, Andria; Goring, Simon J. BioScience, Vol. 68, Issue 8 https://doi.org/10.1093/biosci/biy068	journal	July 2018
The role of sustained observations in tracking impacts of environmental change on marine biodiversity and ecosystems Mieszkowska, N.; Sugden, H.; Firth, L. B. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 372, Issue 2025 https://doi.org/10.1098/rsta.2013.0339	journal	September 2014
Design and performance of combined infrared canopy and belowground warming in the B4WarmED (Boreal Forest Warming at an Ecotone in Danger) experiment Rich, Roy L.; Stefanski, Artur; Montgomery, Rebecca A. Global Change Biology, Vol. 21, Issue 6 https://doi.org/10.1111/gcb.12855	journal	March 2015
Advancing global change biology through experimental manipulations: Where have we been and where might we go? Hanson, Paul J.; Walker, Anthony P. Global Change Biology, Vol. 26, Issue 1 https://doi.org/10.1111/gcb.14894	journal	November 2019
Nature 4.0: A networked sensor system for integrated biodiversity monitoring Zeuss, Dirk; Bald, Lisa; Gottwald, Jannis Global Change Biology, Vol. 30, Issue 1 https://doi.org/10.1111/gcb.17056	journal	December 2023
Challenges and Opportunities of Open Data in Ecology Reichman, O. J.; Jones, M. B.; Schildhauer, M. P. Science, Vol. 331, Issue 6018 https://doi.org/10.1126/science.1197962	journal	February 2011
Performance analysis of MD5 Touch, Joseph D. ACM SIGCOMM Computer Communication Review, Vol. 25, Issue 4 https://doi.org/10.1145/217391.217414	journal	October 1995
Best Practices for Scientific Computing Wilson, Greg; Aruliah, D. A.; Brown, C. Titus PLoS Biology, Vol. 12, Issue 1 https://doi.org/10.1371/journal.pbio.1001745	journal	January 2014
Ten simple rules for writing Dockerfiles for reproducible data science Nüst, Daniel; Sochat, Vanessa; Marwick, Ben PLOS Computational Biology, Vol. 16, Issue 11 https://doi.org/10.1371/journal.pcbi.1008316	journal	November 2020
COMPASS-FME Synoptic Sites Level 1 Sensor Data v2-0 Pennington, Stephanie; Bond-Lamberty, Ben; Rich, Roy COMPASS-FME https://doi.org/10.15485/2588616	dataset	January 2025
COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Level 1 Sensor Data v2-0 Pennington, Stephanie; Bond-Lamberty, Ben; Rich, Roy COMPASS-FME https://doi.org/10.15485/2588618	dataset	January 2025
Dates and Times Made Easy with lubridate Grolemund, Garrett; Wickham, Hadley Journal of Statistical Software, Vol. 40, Issue 3 https://doi.org/10.18637/JSS.V040.I03	journal	January 2011
zoo : S3 Infrastructure for Regular and Irregular Time Series Zeileis, Achim; Grothendieck, Gabor Journal of Statistical Software, Vol. 14, Issue 6 https://doi.org/10.18637/jss.v014.i06	journal	January 2005
The impacts of climate change on ecosystem structure and function Grimm, Nancy B.; Chapin, F. Stuart; Bierwagen, Britta Frontiers in Ecology and the Environment, Vol. 11, Issue 9 https://doi.org/10.1890/120282	journal	November 2013
NEON: a hierarchically designed national ecological network Schimel, David; Hargrove, William; Hoffman, Forrest Frontiers in Ecology and the Environment, Vol. 5, Issue 2, 59 https://doi.org/10.1890/1540‐9295(2007)5[59:nahdne]2.0.co;2	journal	March 2007
A Pore Water Conductivity Sensor Hilhorst, M. A. Soil Science Society of America Journal, Vol. 64, Issue 6 https://doi.org/10.2136/sssaj2000.6461922x	journal	November 2000

Similar Records

COMPASS-FME Synoptic Sites Level 1 Sensor Data v1-2

Dataset · Sun Dec 31 23:00:00 EST 2023 · OSTI ID:2439400

COMPASS-FME Synoptic Sites Level 1 Sensor Data v2-0

Dataset · Tue Dec 31 23:00:00 EST 2024 · OSTI ID:2588616

COMPASS-FME Synoptic Site Characterization

Dataset · Tue Dec 31 23:00:00 EST 2024 · OSTI ID:2583150

A Performant, Scalable Processing Pipeline for High‐Quality and FAIR Environmental Sensor Data

Citation Formats

References (35)

Similar Records

Related Subjects