skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments.

Abstract

We present a formula for the pairwise update of arbitrary-order centered statistical moments. This formula is of particular interest to compute such moments in parallel for large-scale, distributed data sets. As a corollary, we indicate a specialization of this formula for incremental updates, of particular interest to streaming implementations. Finally, we provide pairwise and incremental update formulas for the covariance. Centered statistical moments are one of the most widely used tools in descriptive statistics. It is therefore essential for statistical analysis packages that robust and efficient algorithms be devised and implemented. However, robustness and speed of execution, in this context as well as in others, tend to be orthogonal. For instance, it is well known1 that algorithms for calculating centered statistical moments that utilize sum of powers for the sake of execution speed (one-pass algorithms) lead to unacceptable numerical instability.

Authors:
Publication Date:
Research Org.:
Sandia National Laboratories
Sponsoring Org.:
USDOE
OSTI Identifier:
1028931
Report Number(s):
SAND2008-6212
TRN: US201201%%57
DOE Contract Number:  
AC04-94AL85000
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; ALGORITHMS; INSTABILITY; STATISTICS; VELOCITY; Algorithms.; Statistics.; Process control-Statistical methods.

Citation Formats

Pebay, Philippe Pierre. Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments.. United States: N. p., 2008. Web. doi:10.2172/1028931.
Pebay, Philippe Pierre. Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments.. United States. doi:10.2172/1028931.
Pebay, Philippe Pierre. Mon . "Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments.". United States. doi:10.2172/1028931. https://www.osti.gov/servlets/purl/1028931.
@article{osti_1028931,
title = {Formulas for robust, one-pass parallel computation of covariances and arbitrary-order statistical moments.},
author = {Pebay, Philippe Pierre},
abstractNote = {We present a formula for the pairwise update of arbitrary-order centered statistical moments. This formula is of particular interest to compute such moments in parallel for large-scale, distributed data sets. As a corollary, we indicate a specialization of this formula for incremental updates, of particular interest to streaming implementations. Finally, we provide pairwise and incremental update formulas for the covariance. Centered statistical moments are one of the most widely used tools in descriptive statistics. It is therefore essential for statistical analysis packages that robust and efficient algorithms be devised and implemented. However, robustness and speed of execution, in this context as well as in others, tend to be orthogonal. For instance, it is well known1 that algorithms for calculating centered statistical moments that utilize sum of powers for the sake of execution speed (one-pass algorithms) lead to unacceptable numerical instability.},
doi = {10.2172/1028931},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2008},
month = {9}
}

Technical Report:

Save / Share: