Big Data Analytics for Long-Term Meteorological Observations at Hanford Site

Zhou, Huifen; Ren, Huiying; Royer, Patrick; Hou, Hongfei; Yu, Xiao-Ying

doi:10.3390/atmos13010136

Title: Big Data Analytics for Long-Term Meteorological Observations at Hanford Site

Journal Article · 14 January 2022 · Atmosphere (Basel)

DOI: https://doi.org/10.3390/atmos13010136 · OSTI ID:1840080

Zhou, Huifen;

; Royer, Patrick;

;

A growing number of physical objects with embedded sensors with typically high volume and frequently updated data sets has accentuated the need to develop methodologies to extract useful information from big data for supporting decision making. This study applies a suite of data analytics and core principles of data science to characterize near real-time meteorological data with a focus on extreme weather events. To highlight the applicability of this work and make it more accessible from a risk management perspective, a foundation for a software platform with an intuitive Graphical User Interface (GUI) was developed to access and analyze data from a decommissioned nuclear production complex operated by the U.S. Department of Energy (DOE, Richland, USA). Exploratory data analysis (EDA), involving classical non-parametric statistics, and machine learning (ML) techniques, were used to develop statistical summaries and learn characteristic features of key weather patterns and signatures. The new approach and GUI provide key insights into using big data and ML to assist site operation related to safety management strategies for extreme weather events. Specifically, this work offers a practical guide to analyzing long-term meteorological data and highlights the integration of ML and classical statistics to applied risk and decision science.

View Journal Article

Cite

Export

Save

Sponsoring Organization:: USDOE

OSTI ID:: 1840080

Journal Information:: Atmosphere (Basel), Journal Name: Atmosphere (Basel) Journal Issue: 1 Vol. 13; ISSN 2073-4433; ISSN ATMOCZ

Publisher:: MDPI AGCopyright Statement

Country of Publication:: Switzerland

Language:: English

References (44)

Regional Extreme Precipitation Events: Robust Inference From Credibly Simulated GCM Variables Farnham, David J.; Doss-Gollin, James; Lall, Upmanu Water Resources Research, Vol. 54, Issue 6 https://doi.org/10.1002/2017WR021318	journal	June 2018
Trend analysis in Turkish precipitation data Partal, Turgay; Kahya, Ercan Hydrological Processes, Vol. 20, Issue 9 https://doi.org/10.1002/hyp.5993	journal	January 2006
Trends of precipitation in Beijiang River Basin, Guangdong Province, China Luo, Yan; Liu, Shen; Fu, ShengLei Hydrological Processes, Vol. 22, Issue 13 https://doi.org/10.1002/hyp.6801	journal	January 2008
Recent changes in rainfall and rainy days in Ethiopia Seleshi, Yilma; Zanke, Ulrich International Journal of Climatology, Vol. 24, Issue 8 https://doi.org/10.1002/joc.1052	journal	June 2004
Modelling nonlinear trend for developing non-stationary rainfall intensity-duration-frequency curve: MODELLING NONLINEAR TREND FOR DEVELOPING NON-STATIONARY IDF CURVE Agilan, V.; Umamahesh, N. V. International Journal of Climatology, Vol. 37, Issue 3 https://doi.org/10.1002/joc.4774	journal	May 2016
Diurnal pressure variation: the atmospheric tide Le Blancq, Frank Weather, Vol. 66, Issue 11 https://doi.org/10.1002/wea.857	journal	October 2011
An empirical comparison of selection measures for decision-tree induction Mingers, John Machine Learning, Vol. 3, Issue 4 https://doi.org/10.1007/BF00116837	journal	March 1989
Extreme events in a changing climate: Variability is more important than averages Katz, Richard W.; Brown, Barbara G. Climatic Change, Vol. 21, Issue 3 https://doi.org/10.1007/BF00139728	journal	July 1992
Simulation of tropical cyclone impacts to the U.S. power system under climate change scenarios Staid, Andrea; Guikema, Seth D.; Nateghi, Roshanak Climatic Change, Vol. 127, Issue 3-4 https://doi.org/10.1007/s10584-014-1272-3	journal	October 2014
Attributing high-impact extreme events across timescales—a case study of four different types of events Otto, Friederike E. L.; Philip, Sjoukje; Kew, Sarah Climatic Change, Vol. 149, Issue 3-4 https://doi.org/10.1007/s10584-018-2258-3	journal	August 2018
Rainfall and river flow trends using Mann–Kendall and Sen’s slope estimator statistical tests in the Cobres River basin da Silva, Richarde Marques; Santos, Celso A. G.; Moreira, Madalena Natural Hazards, Vol. 77, Issue 2 https://doi.org/10.1007/s11069-015-1644-7	journal	February 2015
Exploratory data analysis: A comparison of statistical methods with artificial neural networks Joseph, B.; Wang, F. H.; Shieh, D. S. -S. Computers & Chemical Engineering, Vol. 16, Issue 4 https://doi.org/10.1016/0098-1354(92)80057-G	journal	April 1992
Stochastic gradient boosting Friedman, Jerome H. Computational Statistics & Data Analysis, Vol. 38, Issue 4 https://doi.org/10.1016/S0167-9473(01)00065-2	journal	February 2002
A research progress review on regional extreme events Ren, Fu-Min; Trewin, Blair; Brunet, Manola Advances in Climate Change Research, Vol. 9, Issue 3 https://doi.org/10.1016/j.accre.2018.08.001	journal	September 2018
Spatial and temporal trends of mean and extreme rainfall and temperature for the 33 urban centers of the arid and semi-arid state of Rajasthan, India Pingale, Santosh M.; Khare, Deepak; Jat, Mahesh K. Atmospheric Research, Vol. 138 https://doi.org/10.1016/j.atmosres.2013.10.024	journal	March 2014
An assessment of the effectiveness of a random forest classifier for land-cover classification Rodriguez-Galiano, V. F.; Ghimire, B.; Rogan, J. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 67 https://doi.org/10.1016/j.isprsjprs.2011.11.002	journal	January 2012
Comparing methods for estimating flow duration curves at ungauged sites Booker, D. J.; Snelder, T. H. Journal of Hydrology, Vol. 434-435 https://doi.org/10.1016/j.jhydrol.2012.02.031	journal	April 2012
The importance of weather variations in a quantitative risk analysis Marx, Jeffrey D.; Cornwell, John B. Journal of Loss Prevention in the Process Industries, Vol. 22, Issue 6 https://doi.org/10.1016/j.jlp.2009.08.009	journal	November 2009
A statistical analysis of causes and consequences of the release of hazardous materials from pipelines. The influence of layout Bubbico, Roberto Journal of Loss Prevention in the Process Industries, Vol. 56 https://doi.org/10.1016/j.jlp.2018.10.006	journal	November 2018
Random forest as a potential multivariate method for near-infrared (NIR) spectroscopic analysis of complex mixture samples: Gasoline and naphtha Lee, Sanguk; Choi, Hangseok; Cha, Kyungjoon Microchemical Journal, Vol. 110 https://doi.org/10.1016/j.microc.2013.08.007	journal	September 2013
A random forest partition model for predicting NO2 concentrations from traffic flow and meteorological conditions Kamińska, Joanna A. Science of The Total Environment, Vol. 651 https://doi.org/10.1016/j.scitotenv.2018.09.196	journal	February 2019
A gradient boosting method to improve travel time prediction Zhang, Yanru; Haghani, Ali Transportation Research Part C: Emerging Technologies, Vol. 58 https://doi.org/10.1016/j.trc.2015.02.019	journal	September 2015
Random Forests Breiman, Leo Machine Learning, Vol. 45, Issue 1, p. 5-32 https://doi.org/10.1023/A:1010933404324	journal	January 2001
A Survey of Outlier Detection Methodologies Hodge, Victoria; Austin, Jim Artificial Intelligence Review, Vol. 22, Issue 2 https://doi.org/10.1023/B:AIRE.0000045502.10941.a9	journal	October 2004
Using Machine Learning to Parameterize Moist Convection: Potential for Modeling of Climate, Climate Change, and Extreme Events O'Gorman, Paul A.; Dwyer, John G. Journal of Advances in Modeling Earth Systems, Vol. 10, Issue 10 https://doi.org/10.1029/2018MS001351	journal	October 2018
Techniques of trend analysis for monthly water quality data Hirsch, Robert M.; Slack, James R.; Smith, Richard A. Water Resources Research, Vol. 18, Issue 1 https://doi.org/10.1029/WR018i001p00107	journal	February 1982
Extreme Rainfall Nonstationarity Investigation and Intensity–Frequency–Duration Relationship Yilmaz, A. G.; Perera, B. J. C. Journal of Hydrologic Engineering, Vol. 19, Issue 6 https://doi.org/10.1061/(ASCE)HE.1943-5584.0000878	journal	June 2014
Random forest classifier for remote sensing classification Pal, M. International Journal of Remote Sensing, Vol. 26, Issue 1 https://doi.org/10.1080/01431160412331269698	journal	January 2005
Estimates of the Regression Coefficient Based on Kendall's Tau Sen, Pranab Kumar Journal of the American Statistical Association, Vol. 63, Issue 324 https://doi.org/10.1080/01621459.1968.10480934	journal	December 1968
Progress in Outlier Detection Techniques: A Survey Wang, Hongzhi; Bah, Mohamed Jaward; Hammad, Mohamed IEEE Access, Vol. 7 https://doi.org/10.1109/ACCESS.2019.2932769	journal	January 2019
An Adaptive Outlier Detection and Processing Approach Towards Time Series Sensor Data Zhang, Minghu; Li, Xin; Wang, Lili IEEE Access, Vol. 7 https://doi.org/10.1109/ACCESS.2019.2957602	journal	January 2019
Predictive Risk Analytics for Weather-Resilient Operation of Electric Power Systems Dehghanian, Payman; Zhang, Bei; Dokic, Tatjana IEEE Transactions on Sustainable Energy, Vol. 10, Issue 1 https://doi.org/10.1109/TSTE.2018.2825780	journal	January 2019
On the Probability of Extreme Rainfall Events Hershfield, David M. Bulletin of the American Meteorological Society, Vol. 54, Issue 10 https://doi.org/10.1175/1520-0477(1973)054<1013:OTPOER>2.0.CO;2	journal	October 1973
Classifications of Atmospheric Circulation Patterns Huth, Radan; Beck, Christoph; Philipp, Andreas Annals of the New York Academy of Sciences, Vol. 1146, Issue 1 https://doi.org/10.1196/annals.1446.019	journal	December 2008
machine. Friedman, Jerome H. The Annals of Statistics, Vol. 29, Issue 5 https://doi.org/10.1214/aos/1013203451	journal	October 2001
Synergies between urban heat island and heat waves in Seoul: The role of wind speed and land use characteristics Ngarambe, Jack; Nganyiyimana, Jacques; Kim, Inhan PLOS ONE, Vol. 15, Issue 12 https://doi.org/10.1371/journal.pone.0243571	journal	December 2020
edarf: Exploratory Data Analysis using Random Forests M. Jones, Zachary; J. Linder, Fridolin The Journal of Open Source Software, Vol. 1, Issue 6 https://doi.org/10.21105/joss.00092	journal	October 2016
Nonparametric Tests Against Trend Mann, Henry B. Econometrica, Vol. 13, Issue 3 https://doi.org/10.2307/1907187	journal	July 1945
Rank Correlation Methods. Kendall, M. G. Biometrika, Vol. 44, Issue 1/2 https://doi.org/10.2307/2333282	journal	June 1957
Null Hypothesis Testing: Problems, Prevalence, and an Alternative Anderson, David R.; Burnham, Kenneth P.; Thompson, William L. The Journal of Wildlife Management, Vol. 64, Issue 4 https://doi.org/10.2307/3803199	journal	October 2000
Machine Learning Analysis of Hydrologic Exchange Flows and Transit Time Distributions in a Large Regulated River Ren, Huiying; Song, Xuehang; Fang, Yilin Frontiers in Artificial Intelligence, Vol. 4 https://doi.org/10.3389/frai.2021.648071	journal	April 2021
Comparison between Random Forests, Artificial Neural Networks and Gradient Boosted Machines Methods of On-Line Vis-NIR Spectroscopy Measurements of Soil Total Nitrogen and Total Carbon Nawar, Said; Mouazen, Abdul Sensors, Vol. 17, Issue 10 https://doi.org/10.3390/s17102428	journal	October 2017
Impacts of Spatial Heterogeneity and Temporal Non-Stationarity on Intensity-Duration-Frequency Estimates—A Case Study in a Mountainous California-Nevada Watershed Ren, Huiying; Hou, Z. Jason; Wigmosta, Mark Water, Vol. 11, Issue 6 https://doi.org/10.3390/w11061296	journal	June 2019
Regionalization of patterns of flow intermittence from gauging station records Snelder, T. H.; Datry, T.; Lamouroux, N. Hydrology and Earth System Sciences, Vol. 17, Issue 7 https://doi.org/10.5194/hess-17-2685-2013	journal	January 2013

Similar Records

Data Mining and Machine Learning for Power System Monitoring, Understanding, and Impact Evaluation

Book · 2023 · OSTI ID:1963258

Ke, Xinda; Ren, Huiying; Huang, Qiuhua; +2 more

A Review of Recent and Emerging Machine Learning Applications for Climate Variability and Weather Phenomena

Journal Article · 2023 · Artificial Intelligence for the Earth Systems · OSTI ID:2000611

Molina, Maria J.; O’Brien, Travis A.; Anderson, Gemma; +6 more

An Inventory of AI-ready Benchmark Data for US Fires, Heatwaves, and Droughts

Dataset · 2023 · OSTI ID:2004956

Lin, Xinming; Hou, Zhangshuan

Title: Big Data Analytics for Long-Term Meteorological Observations at Hanford Site

Citation Formats

References (44)

Similar Records

Related Subjects