Visualisation and outlier detection for probability density function ensembles

Murph, Alexander C.; Strait, Justin D.; Moran, Kelly R.; Hyman, Jeffrey D.; Stauffer, Philip H.

doi:10.1002/sta4.662

Title: Visualisation and outlier detection for probability density function ensembles

Journal Article · 10 March 2024 · Stat

DOI: https://doi.org/10.1002/sta4.662 · OSTI ID:2323357

^[1]; Strait, Justin D. ^[1]; Moran, Kelly R. ^[1]; Hyman, Jeffrey D. ^[2]; Stauffer, Philip H. ^[2]

Statistical Sciences (CCS‐6), Computer, Computational, and Statistical Sciences Division Los Alamos National Laboratory Los Alamos New Mexico USA
Energy and Natural Resources Security (EES‐16), Earth and Environmental Sciences Division Los Alamos National Laboratory Los Alamos New Mexico USA

Abstract Exploratory data analysis (EDA) for functional data—data objects where observations are entire functions—is a difficult problem that has seen significant attention in recent literature. This surge in interest is motivated by the ubiquitous nature of functional data, which are prevalent in applications across fields such as meteorology, biology, medicine and engineering. Empirical probability density functions (PDFs) can be viewed as constrained functional data objects that must integrate to one and be nonnegative. They show up in contexts such as yearly income distributions, zooplankton size structure in oceanography and in connectivity patterns in the brain, among others. While PDF data are certainly common in modern research, little attention has been given to EDA specifically for PDFs. In this paper, we extend several methods for EDA on functional data for PDFs and compare them on simulated data that exhibit different types of variation, designed to mimic that seen in real‐world applications. We then use our new methods to perform EDA on the breakthrough curves observed in gas transport simulations for underground fracture networks.

View Journal Article

Cite

Export

Save

Sponsoring Organization:: USDOE

OSTI ID:: 2323357

Alternate ID(s):: OSTI ID: 2340908; OSTI ID: 2346235

Journal Information:: Stat, Journal Name: Stat Journal Issue: 2 Vol. 13; ISSN 2049-1573

Publisher:: Wiley Blackwell (John Wiley & Sons)Copyright Statement

Country of Publication:: United Kingdom

Language:: English

References (50)

Riemannian Analysis of Probability Density Functions with Applications in Vision Srivastava, Anuj; Jermyn, Ian; Joshi, Shantanu 2007 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2007.383188	conference	June 2007
From Fluid Flow to Coupled Processes in Fractured Rock: Recent Advances and New Frontiers Viswanathan, H. S.; Ajo‐Franklin, J.; Birkholzer, J. T. Reviews of Geophysics, Vol. 60, Issue 1 https://doi.org/10.1029/2021RG000744	journal	February 2022
Characterizing flow and transport in fractured geological media: A review Berkowitz, Brian Advances in Water Resources, Vol. 25, Issue 8-12 https://doi.org/10.1016/S0309-1708(02)00042-8	journal	August 2002
Fracture network flow prediction with uncertainty using physics-informed graph features Strait, Justin D.; Moran, Kelly R.; Hyman, Jeffrey D. Computational Geosciences, Vol. 27, Issue 6 https://doi.org/10.1007/s10596-023-10256-9	journal	October 2023
Bayes Hilbert Spaces van den Boogaart, Karl Gerald; Egozcue, Juan José; Pawlowsky‐Glahn, Vera Australian & New Zealand Journal of Statistics, Vol. 56, Issue 2 https://doi.org/10.1111/anzs.12074	journal	June 2014
Economic Applications of Quantile Regression Fitzenberger, Bernd; Koenker, Roger; Machado, José A. F. https://doi.org/10.1007/978-3-662-11592-3	book	January 2002
Functional Outlier Detection for Density-Valued Data with Application to Robustify Distribution-to-Distribution Regression Lei, Xinyi; Chen, Zhicheng; Li, Hui Technometrics, Vol. 65, Issue 3 https://doi.org/10.1080/00401706.2022.2164063	journal	January 2023
A Probabilistic Clustering Approach for Identifying Primary Subnetworks of Discrete Fracture Networks with Quantified Uncertainty Osthus, Dave; Hyman, Jeffrey D.; Karra, Satish SIAM/ASA Journal on Uncertainty Quantification, Vol. 8, Issue 2 https://doi.org/10.1137/19M1279265	journal	January 2020
Contour Boxplots: A Method for Characterizing Uncertainty in Feature Sets from Simulation Ensembles Whitaker, Ross T.; Mirzargar, Mahsa; Kirby, Robert M. IEEE Transactions on Visualization and Computer Graphics, Vol. 19, Issue 12 https://doi.org/10.1109/TVCG.2013.143	journal	December 2013
Functional Boxplots Sun, Ying; Genton, Marc G. Journal of Computational and Graphical Statistics, Vol. 20, Issue 2 https://doi.org/10.1198/jcgs.2011.09224	journal	January 2011
Scaling of fracture systems in geological media Bonnet, E.; Bour, O.; Odling, N. E. Reviews of Geophysics, Vol. 39, Issue 3 https://doi.org/10.1029/1999RG000074	journal	August 2001
Functional data analysis characterizes the shapes of the first COVID-19 epidemic wave in Italy Boschi, Tobia; Di Iorio, Jacopo; Testa, Lorenzo Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-021-95866-y	journal	August 2021
Introduction to Nonparametric Estimation Tsybakov, Alexandre B. Springer Series in Statistics https://doi.org/10.1007/b13794	book	January 2009
Efficient Monte Carlo With Graph‐Based Subsurface Flow and Transport Models O'Malley, D.; Karra, S.; Hyman, J. D. Water Resources Research, Vol. 54, Issue 5 https://doi.org/10.1029/2017WR022073	journal	May 2018
Comparison of alternative modelling approaches for groundwater flow in fractured rock Selroos, Jan-Olof; Walker, Douglas D.; Ström, Anders Journal of Hydrology, Vol. 257, Issue 1-4 https://doi.org/10.1016/S0022-1694(01)00551-0	journal	February 2002
Simplicial band depth for multivariate functional data López-Pintado, Sara; Sun, Ying; Lin, Juan K. Advances in Data Analysis and Classification, Vol. 8, Issue 3 https://doi.org/10.1007/s11634-014-0166-6	journal	March 2014
Fracture size and transmissivity correlations: Implications for transport simulations in sparse three-dimensional discrete fracture networks following a truncated power law distribution of fracture size: FRACTURE SIZE AND TRANSMISSIVITY CORRELATIONS Hyman, J. D.; Aldrich, G.; Viswanathan, H. Water Resources Research, Vol. 52, Issue 8 https://doi.org/10.1002/2016WR018806	journal	August 2016
Modeling Probability Density Functions as Data Objects Petersen, Alexander; Zhang, Chao; Kokoszka, Piotr Econometrics and Statistics, Vol. 21 https://doi.org/10.1016/j.ecosta.2021.04.004	journal	January 2022
Curve Boxplot: Generalization of Boxplot for Ensembles of Curves Mirzargar, Mahsa; Whitaker, Ross T.; Kirby, Robert M. IEEE Transactions on Visualization and Computer Graphics, Vol. 20, Issue 12 https://doi.org/10.1109/TVCG.2014.2346455	journal	December 2014
Understanding hydraulic fracturing: a multi-scale problem Hyman, J. D.; Jiménez-Martínez, J.; Viswanathan, H. S. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 374, Issue 2078 https://doi.org/10.1098/rsta.2015.0426	journal	October 2016
The state of the art in monitoring and verification—Ten years on Jenkins, Charles; Chadwick, Andy; Hovorka, Susan D. International Journal of Greenhouse Gas Control, Vol. 40 https://doi.org/10.1016/j.ijggc.2015.05.009	journal	September 2015
Visualization and Outlier Detection for Multivariate Elastic Curve Data Xie, Weiyi; Chkrebtii, Oksana; Kurtek, Sebastian IEEE Transactions on Visualization and Computer Graphics, Vol. 26, Issue 11 https://doi.org/10.1109/TVCG.2019.2921541	journal	November 2020
The Behavior of Dense, Nonaqueous Phase Liquids in Fractured Clay and Rock Kueper, Bernard H.; McWhorter, David B. Ground Water, Vol. 29, Issue 5 https://doi.org/10.1111/j.1745-6584.1991.tb00563.x	journal	September 1991
dfnWorks: A discrete fracture network framework for modeling subsurface flow and transport Hyman, Jeffrey D.; Karra, Satish; Makedonska, Nataliia Computers & Geosciences, Vol. 84 https://doi.org/10.1016/j.cageo.2015.08.001	journal	November 2015
Elastic Depths for Detecting Shape Anomalies in Functional Data Harris, Trevor; Tucker, J. Derek; Li, Bo Technometrics, Vol. 63, Issue 4 https://doi.org/10.1080/00401706.2020.1811156	journal	October 2020
LQD-RKHS-based distribution-to-distribution regression methodology for restoring the probability distributions of missing SHM data Chen, Zhicheng; Bao, Yuequan; Li, Hui Mechanical Systems and Signal Processing, Vol. 121 https://doi.org/10.1016/j.ymssp.2018.11.052	journal	April 2019
The shale gas revolution: Barriers, sustainability, and emerging opportunities Middleton, Richard S.; Gupta, Rajan; Hyman, Jeffrey D. Applied Energy, Vol. 199 https://doi.org/10.1016/j.apenergy.2017.04.034	journal	August 2017
Investigating Concurrency in Online Auctions Through Visualization Hyde, Valerie; Jank, Wolfgang; Shmueli, Galit The American Statistician, Vol. 60, Issue 3 https://doi.org/10.1198/000313006X124163	journal	August 2006
Trajectory functional boxplots Yao, Zonghui; Dai, Wenlin; G. Genton, Marc Stat, Vol. 9, Issue 1 https://doi.org/10.1002/sta4.289	journal	January 2020
Singular Value Decomposition and Its Visualization Zhang, Lingsong; Marron, J. S.; Shen, Haipeng Journal of Computational and Graphical Statistics, Vol. 16, Issue 4 https://doi.org/10.1198/106186007X256080	journal	December 2007
On the Concept of Depth for Functional Data López-Pintado, Sara; Romo, Juan Journal of the American Statistical Association, Vol. 104, Issue 486 https://doi.org/10.1198/jasa.2009.0108	journal	June 2009
Classifying densities using functional regression trees: Applications in oceanology Nerini, David; Ghattas, Badih Computational Statistics & Data Analysis, Vol. 51, Issue 10 https://doi.org/10.1016/j.csda.2006.09.028	journal	June 2007
Trends, prospects and challenges in quantifying flow and transport through fractured rocks Neuman, Shlomo P. Hydrogeology Journal, Vol. 13, Issue 1 https://doi.org/10.1007/s10040-004-0397-2	journal	February 2005
Hilbert Space of Probability Density Functions Based on Aitchison Geometry Egozcue, J. J.; Díaz–Barrero, J. L.; Pawlowsky–Glahn, V. Acta Mathematica Sinica, English Series, Vol. 22, Issue 4 https://doi.org/10.1007/s10114-005-0678-2	journal	January 2006
Sensitivity Analysis in the Presence of Intrinsic Stochasticity for Discrete Fracture Network Simulations Murph, A. C.; Strait, J. D.; Moran, K. R. Journal of Geophysical Research: Machine Learning and Computation, Vol. 1, Issue 3 https://doi.org/10.1029/2023JH000113	journal	August 2024
Functional Data Analysis Ramsay, James; Everitt, Brian S.; Howell, David C. Encyclopedia of Statistics in Behavioral Science https://doi.org/10.1002/0470013192.bsa239	book	January 2005
Dissolution of non-aqueous-phase liquids and aqueous-phase contaminant transport in discretely-fractured porous media VanderKwaak, J. E.; Sudicky, E. A. Journal of Contaminant Hydrology, Vol. 23, Issue 1-2 https://doi.org/10.1016/0169-7722(95)00087-9	journal	June 1996
Quantifying Transport Uncertainty in Unsaturated Rock using Monte Carlo Sampling of Retention Curves Stauffer, Philip H.; Lu, Zhiming Vadose Zone Journal, Vol. 11, Issue 4 https://doi.org/10.2136/vzj2011.0171	journal	October 2012
Functional data analysis for density functions by transformation to a Hilbert space Petersen, Alexander; Müller, Hans-Georg The Annals of Statistics, Vol. 44, Issue 1 https://doi.org/10.1214/15-AOS1363	journal	February 2016
Short-Term Spatio-Temporal Wind Power Forecast in Robust Look-ahead Power System Dispatch Xie, Le; Gu, Yingzhong; Zhu, Xinxin IEEE Transactions on Smart Grid, Vol. 5, Issue 1 https://doi.org/10.1109/TSG.2013.2282300	journal	January 2014
Forecasting of density functions with an application to cross-sectional and intraday returns Kokoszka, Piotr; Miao, Hong; Petersen, Alexander International Journal of Forecasting, Vol. 35, Issue 4 https://doi.org/10.1016/j.ijforecast.2019.05.007	journal	October 2019
A methodology to constrain the parameters of a hydrogeological discrete fracture network model for sparsely fractured crystalline rock, exemplified by data from the proposed high-level nuclear waste repository site at Forsmark, Sweden Follin, Sven; Hartley, Lee; Rhén, Ingvar Hydrogeology Journal, Vol. 22, Issue 2 https://doi.org/10.1007/s10040-013-1080-2	journal	December 2013
A Geometric Approach to Visualization of Variability in Functional Data Xie, Weiyi; Kurtek, Sebastian; Bharath, Karthik Journal of the American Statistical Association, Vol. 112, Issue 519 https://doi.org/10.1080/01621459.2016.1256813	journal	July 2016
Conforming Delaunay Triangulation of Stochastically Generated Three Dimensional Discrete Fracture Networks: A Feature Rejection Algorithm for Meshing Strategy Hyman, Jeffrey D.; Gable, Carl W.; Painter, Scott L. SIAM Journal on Scientific Computing, Vol. 36, Issue 4 https://doi.org/10.1137/130942541	journal	January 2014
Optimal Transport: Theory and Applications Ollivier, Yann; Pajot, Hervé; Villani, Cedric https://doi.org/10.1017/CBO9781107297296	book	August 2014
Functional data analysis of the dynamics of the monthly index of nondurable goods production Ramsay, James O.; Ramsey, James B. Journal of Econometrics, Vol. 107, Issue 1-2 https://doi.org/10.1016/S0304-4076(01)00127-0	journal	March 2002
Outlier detection in functional data by depth measures, with application to identify abnormal NO_x levels Febrero, Manuel; Galeano, Pedro; González‐Manteiga, Wenceslao Environmetrics, Vol. 19, Issue 4 https://doi.org/10.1002/env.878	journal	August 2007
A functional analysis of NOx levels: location and scale estimation and outlier detection Febrero, Manuel; Galeano, Pedro; González-Manteiga, Wenceslao Computational Statistics, Vol. 22, Issue 3 https://doi.org/10.1007/s00180-007-0048-x	journal	March 2007
Inference for Density Families Using Functional Principal Component Analysis Kneip, Alois; Utikal, Klaus J. Journal of the American Statistical Association, Vol. 96, Issue 454 https://doi.org/10.1198/016214501753168235	journal	June 2001
Rainbow Plots, Bagplots, and Boxplots for Functional Data Hyndman, Rob J.; Shang, Han Lin Journal of Computational and Graphical Statistics, Vol. 19, Issue 1 https://doi.org/10.1198/jcgs.2009.08158	journal	January 2010

Similar Records

WE-A-16A-01: International Medical Physics Symposium: Increasing Access to Medical Physics Education/Training and Research Excellence

Journal Article · 2014 · Medical Physics · OSTI ID:22407910

Bortfeld, T; Ngoma, T; Odedina, F; +4 more

Parametric description of intermittent probability distribution functions in solar wind and magnetohydrodynamic turbulence

Journal Article · 2024 · Monthly Notices of the Royal Astronomical Society · OSTI ID:2349547

Palacios, Juan C.; Perez, Jean C.; Bourouaine, Sofiane

Time-dependent probability density function analysis of H-mode transitions

Journal Article · 2023 · Europhysics Letters · OSTI ID:1984182

Farre-Kaga, Hiro J.; Andrew, Yasmin; Dunsmore, Jamie; +4 more

Title: Visualisation and outlier detection for probability density function ensembles

Citation Formats

References (50)

Similar Records

Related Subjects