Automatic Kalman-filter-based wavelet shrinkage denoising of 1D stellar spectra

Gilda, Sankalp; Slepian, Zachary

doi:10.1093/mnras/stz2577

Title: Automatic Kalman-filter-based wavelet shrinkage denoising of 1D stellar spectra

Abstract

We propose a non-parametric method to denoise 1D stellar spectra based on wavelet shrinkage followed by adaptive Kalman thresholding. Wavelet shrinkage denoising involves applying the discrete wavelet transform (DWT) to the input signal, ‘shrinking’ certain frequency components in the transform domain, and then applying inverse DWT to the reduced components. The performance of this procedure is influenced by the choice of base wavelet, the number of decomposition levels, and the thresholding function. Typically, these parameters are chosen by ‘trial and error’, which can be strongly dependent on the properties of the data being denoised. We here introduce an adaptive Kalman-filter-based thresholding method that eliminates the need for choosing the number of decomposition levels. We use the ‘Haar’ wavelet basis, which we found to provide excellent filtering for 1D stellar spectra, at a low computational cost. We introduce various levels of Poisson noise into synthetic PHOENIX spectra, and test the performance of several common denoising methods against our own. It proves superior in terms of noise suppression and peak shape preservation. Finally, we expect it may also be of use in automatically and accurately filtering low signal-to-noise galaxy and quasar spectra obtained from surveys such as SDSS, Gaia, LSST, PESSTO, VANDELS,more » LEGA-C, and DESI.« less

Authors:

Gilda, Sankalp ^[1]; Slepian, Zachary ^[2]

Univ. of Florida, Gainesville, FL (United States)
Univ. of Florida, Gainesville, FL (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California, Berkeley, CA (United States)

Publication Date:: 2019-09-16

Research Org.:: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1603546

Grant/Contract Number:: AC02-05CH11231

Resource Type:: Accepted Manuscript

Journal Name:: Monthly Notices of the Royal Astronomical Society

Additional Journal Information:: Journal Volume: 490; Journal Issue: 4; Journal ID: ISSN 0035-8711

Publisher:: Royal Astronomical Society

Country of Publication:: United States

Language:: English

Subject:: 79 ASTRONOMY AND ASTROPHYSICS; data analysis; statistical; image processing; spectroscopic

Citation Formats


                    Gilda, Sankalp, and Slepian, Zachary. Automatic Kalman-filter-based wavelet shrinkage denoising of 1D stellar spectra.  United States: N. p., 2019. 
Web.  doi:10.1093/mnras/stz2577.

Copy to clipboard


                    Gilda, Sankalp, & Slepian, Zachary. Automatic Kalman-filter-based wavelet shrinkage denoising of 1D stellar spectra.  United States.  https://doi.org/10.1093/mnras/stz2577

Copy to clipboard


                    Gilda, Sankalp, and Slepian, Zachary. Mon .  
"Automatic Kalman-filter-based wavelet shrinkage denoising of 1D stellar spectra".  United States.  https://doi.org/10.1093/mnras/stz2577.  https://www.osti.gov/servlets/purl/1603546.

Copy to clipboard


                    
@article{osti_1603546,

  title        = {Automatic Kalman-filter-based wavelet shrinkage denoising of 1D stellar spectra},

  author       = {Gilda, Sankalp and Slepian, Zachary},

  abstractNote = {We propose a non-parametric method to denoise 1D stellar spectra based on wavelet shrinkage followed by adaptive Kalman thresholding. Wavelet shrinkage denoising involves applying the discrete wavelet transform (DWT) to the input signal, ‘shrinking’ certain frequency components in the transform domain, and then applying inverse DWT to the reduced components. The performance of this procedure is influenced by the choice of base wavelet, the number of decomposition levels, and the thresholding function. Typically, these parameters are chosen by ‘trial and error’, which can be strongly dependent on the properties of the data being denoised. We here introduce an adaptive Kalman-filter-based thresholding method that eliminates the need for choosing the number of decomposition levels. We use the ‘Haar’ wavelet basis, which we found to provide excellent filtering for 1D stellar spectra, at a low computational cost. We introduce various levels of Poisson noise into synthetic PHOENIX spectra, and test the performance of several common denoising methods against our own. It proves superior in terms of noise suppression and peak shape preservation. Finally, we expect it may also be of use in automatically and accurately filtering low signal-to-noise galaxy and quasar spectra obtained from surveys such as SDSS, Gaia, LSST, PESSTO, VANDELS, LEGA-C, and DESI.},

  doi          = {10.1093/mnras/stz2577},

  journal      = {Monthly Notices of the Royal Astronomical Society},

  number       = 4,

  volume       = 490,

  place        = {United States},

  year         = {2019},

  month        = {9}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1093/mnras/stz2577

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

A New Signal Denoising Algorithm from Wavelet Modulus Maxima
conference, August 2007

Liu, Yi; Cheng, Xu
Fourth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD 2007)
DOI: 10.1109/FSKD.2007.90

The Effect of AGN Heating on the Low-redshift Ly α Forest
journal, January 2017

Gurvich, Alex; Burkhart, Blakesley; Bird, Simeon
The Astrophysical Journal, Vol. 835, Issue 2
DOI: 10.3847/1538-4357/835/2/175

Bayesian unscented Kalman filter for state estimation of nonlinear and non-Gaussian systems
conference, August 2016

Liu, Zhong; Chan, Shing-Chow; Wu, Ho-Chun
2016 24th European Signal Processing Conference (EUSIPCO)
DOI: 10.1109/EUSIPCO.2016.7760287

Multiwavelets denoising using neighboring coefficients
journal, July 2003

Chen, G. Y.; Bui, T. D.
IEEE Signal Processing Letters, Vol. 10, Issue 7
DOI: 10.1109/LSP.2003.811586

Parameter estimation for leaky aquifers using the extended Kalman filter, and considering model and data measurement uncertainties
journal, February 2005

Yeh, H. D.; Huang, Y. C.
Journal of Hydrology, Vol. 302, Issue 1-4
DOI: 10.1016/j.jhydrol.2004.06.035

A theory for multiresolution signal decomposition: the wavelet representation
journal, July 1989

Mallat, S. G.
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 11, Issue 7
DOI: 10.1109/34.192463

A method of eliminating the signal-dependent random noise from the raw CMOS image sensor data based on Kalman filter
journal, November 2014

Zhang, Yu; Wang, Guangyi; Xu, Jiangtao
Signal Processing, Vol. 104
DOI: 10.1016/j.sigpro.2014.04.026

An introduction to wavelets
journal, July 1995

Graps, A.
IEEE Computational Science and Engineering, Vol. 2, Issue 2
DOI: 10.1109/99.388960

Speech denoising using discrete wavelet packet decomposition technique
conference, May 2016

Oktar, Mehmet Alper; Nibouche, Mokhtar; Baltaci, Yusuf
2016 24th Signal Processing and Communication Application Conference (SIU)
DOI: 10.1109/SIU.2016.7495865

Warm dark matter as a solution to the small scale crisis: New constraints from high redshift Lyman- $α$ forest data
journal, August 2013

Viel, Matteo; Becker, George D.; Bolton, James S.
Physical Review D, Vol. 88, Issue 4
DOI: 10.1103/PhysRevD.88.043502

Wavelet Threshold Estimators for Data with Correlated Noise
journal, May 1997

Johnstone, Iain M.; Silverman, Bernard W.
Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 59, Issue 2
DOI: 10.1111/1467-9868.00071

The what, how, and why of wavelet shrinkage denoising
journal, January 2000

Taswell, C.
Computing in Science & Engineering, Vol. 2, Issue 3
DOI: 10.1109/5992.841791

A New Wavelet Denoising Method for Selecting Decomposition Levels and Noise Thresholds
journal, January 2016

Srivastava, Madhur; Anderson, C. Lindsay; Freed, Jack H.
IEEE Access, Vol. 4
DOI: 10.1109/ACCESS.2016.2587581

Adapting to Unknown Smoothness via Wavelet Shrinkage
journal, December 1995

Donoho, David L.; Johnstone, Iain M.
Journal of the American Statistical Association, Vol. 90, Issue 432
DOI: 10.1080/01621459.1995.10476626

A novel technique for selecting mother wavelet function using an intelli gent fault diagnosis system
journal, April 2009

Rafiee, J.; Tse, P. W.; Harifi, A.
Expert Systems with Applications, Vol. 36, Issue 3
DOI: 10.1016/j.eswa.2008.05.052

A wavelet-based data pre-processing analysis approach in mass spectrometry
journal, April 2007

Li, Xiaoli; Li, Jin; Yao, Xin
Computers in Biology and Medicine, Vol. 37, Issue 4
DOI: 10.1016/j.compbiomed.2006.08.009

Adaptive wavelet packet thresholding with iterative Kalman filter for speech enhancement
conference, November 2017

Zhao, Mengjiao; Zhu, Wei-Ping
2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP)
DOI: 10.1109/GlobalSIP.2017.8308606

Image Quality Assessment: From Error Visibility to Structural Similarity
journal, April 2004

Wang, Z.; Bovik, A. C.; Sheikh, H. R.
IEEE Transactions on Image Processing, Vol. 13, Issue 4
DOI: 10.1109/TIP.2003.819861

A New Approach to Linear Filtering and Prediction Problems
journal, March 1960

Kalman, R. E.
Journal of Basic Engineering, Vol. 82, Issue 1
DOI: 10.1115/1.3662552

Improved Image Denoising Technique Using Neighboring Wavelet Coefficients of Optimal Wavelet with Adaptive Thresholding
journal, January 2012

Kumar, Rakesh; Saini, B. S.
International Journal of Computer Theory and Engineering
DOI: 10.7763/IJCTE.2012.V4.491

Wavelet-Based Combined Signal Filtering and Prediction
journal, December 2005

Renaud, O.; Starck, J. -L.; Murtagh, F.
IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics), Vol. 35, Issue 6
DOI: 10.1109/TSMCB.2005.850182

Adaptive wavelet thresholding for image denoising and compression
journal, January 2000

Chang, S. G.; Vetterli, M.
IEEE Transactions on Image Processing, Vol. 9, Issue 9
DOI: 10.1109/83.862633

Covariance matching based adaptive unscented Kalman filter for direct filtering in INS/GNSS integration
journal, March 2016

Meng, Yang; Gao, Shesheng; Zhong, Yongmin
Acta Astronautica, Vol. 120
DOI: 10.1016/j.actaastro.2015.12.014

Ideal spatial adaptation by wavelet shrinkage
journal, September 1994

Donoho, David L.; Johnstone, Iain M.
Biometrika, Vol. 81, Issue 3
DOI: 10.1093/biomet/81.3.425

Auto Regressive Moving Average (ARMA) Modeling Method for Gyro Random Noise Using a Robust Kalman Filter
journal, September 2015

Huang, Lei
Sensors, Vol. 15, Issue 10
DOI: 10.3390/s151025277

On-line Support Vector Regression of the transition model for the Kalman filter
journal, June 2013

Salti, Samuele; Di Stefano, Luigi
Image and Vision Computing, Vol. 31, Issue 6-7
DOI: 10.1016/j.imavis.2012.09.008

Analysis of time-varying signals using continuous wavelet and synchrosqueezed transforms
journal, July 2018

Tary, Jean Baptiste; Herrera, Roberto Henry; van der Baan, Mirko
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 376, Issue 2126
DOI: 10.1098/rsta.2017.0254

Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures
journal, January 2009

Zhou Wang, ; Bovik, A. C.
IEEE Signal Processing Magazine, Vol. 26, Issue 1
DOI: 10.1109/MSP.2008.930649

Learning to swim in a sea of wavelets
journal, January 1995

Bultheel, Adhemar
Bulletin of the Belgian Mathematical Society - Simon Stevin, Vol. 2, Issue 1
DOI: 10.36045/bbms/1103408773

New constraints on the free-streaming of warm dark matter from intermediate and small scale Lyman- $α$ forest data
journal, July 2017

Iršič, Vid; Viel, Matteo; Haehnelt, Martin G.
Physical Review D, Vol. 96, Issue 2
DOI: 10.1103/PhysRevD.96.023522

Adaptive unscented Kalman filter for parameter and state estimation of nonlinear high-speed objects
journal, August 2013

Deng, Fang; Chen, Jie; Chen, Chen
Journal of Systems Engineering and Electronics, Vol. 24, Issue 4
DOI: 10.1109/JSEE.2013.00076

A new extensive library of PHOENIX stellar atmospheres and synthetic spectra
journal, April 2013

Husser, T. -O.; Wende-von Berg, S.; Dreizler, S.
Astronomy & Astrophysics, Vol. 553
DOI: 10.1051/0004-6361/201219058

Similar Records in DOE PAGES and OSTI.GOV collections:

Empirical Mode Decomposition Technique with Conditional Mutual Information for Denoising Operational Sensor Data

Journal Article Omitaomu, Olufemi A. ; Protopopescu, Vladimir A. ; Ganguly, Auroop R. - IEEE Sensors Journal

A new approach is developed for denoising signals using the Empirical Mode Decomposition (EMD) technique and the Information-theoretic method. The EMD technique is applied to decompose a noisy sensor signal into the so-called intrinsic mode functions (IMFs). These functions are of the same length and in the same time domain as the original signal. Therefore, the EMD technique preserves varying frequency in time. Assuming the given signal is corrupted by high-frequency Gaussian noise implies that most of the noise should be captured by the first few modes. Therefore, our proposition is to separate the modes into high-frequency and low-frequency groups.more »« less
https://doi.org/10.1109/JSEN.2011.2142302
A Step-Down Test Procedure for Wavelet Shrinkage Using Bootstrapping

Journal Article Lim, Munwon ; Omitaomu, Olufemi A. ; Bae, Suk Joo - IEEE Access

Wavelet thresholding (or shrinkage) attempts to remove the noises existing in the signals while preserving inherent pattern characteristics in the reconstruction of true signals. For data-denoising purpose, we present a new wavelet thresholding procedure which employs the step-down testing idea of identifying active contrasts in unreplicated fractional factorial experiments. The proposed method employs bootstrapping methods to a step-down test for thresholding wavelet coefficients. By introducing the concept of a false discovery error rate in testing wavelet coefficients, we shrink the wavelet coefficients with p -values higher than the error rate. The error rate controls the expected proportion of wrongly acceptedmore »« less
https://doi.org/10.1109/access.2020.3025103

Full Text Available
Denoising solar radiation data using coiflet wavelets

Journal Article Karim, Samsul Ariffin Abdul, E-mail: samsul-ariffin@petronas.com.my ; Janier, Josefina B., E-mail: josefinajanier@petronas.com.my ; Muthuvalu, Mohana Sundaram, E-mail: mohana.muthuvalu@petronas.com.my ; ... - AIP Conference Proceedings

Signal denoising and smoothing plays an important role in processing the given signal either from experiment or data collection through observations. Data collection usually was mixed between true data and some error or noise. This noise might be coming from the apparatus to measure or collect the data or human error in handling the data. Normally before the data is use for further processing purposes, the unwanted noise need to be filtered out. One of the efficient methods that can be used to filter the data is wavelet transform. Due to the fact that the received solar radiation data fluctuatesmore »« less
https://doi.org/10.1063/1.4898497
Non parametric denoising methods based on wavelets: Application to electron microscopy images in low exposure time

Journal Article Soumia, Sid Ahmed, E-mail: samasoumia@hotmail.fr ; Messali, Zoubeida ; Ouahabi, Abdeldjalil ; ... - AIP Conference Proceedings

The 3D reconstruction of the Cryo-Transmission Electron Microscopy (Cryo-TEM) and Energy Filtering TEM images (EFTEM) hampered by the noisy nature of these images, so that their alignment becomes so difficult. This noise refers to the collision between the frozen hydrated biological samples and the electrons beam, where the specimen is exposed to the radiation with a high exposure time. This sensitivity to the electrons beam led specialists to obtain the specimen projection images at very low exposure time, which resulting the emergence of a new problem, an extremely low signal-to-noise ratio (SNR). This paper investigates the problem of TEM imagesmore »« less
https://doi.org/10.1063/1.4906004
The application of wavelet shrinkage denoising to magnetic Barkhausen noise measurements

Journal Article Thomas, James - AIP Conference Proceedings

The application of Magnetic Barkhausen Noise (MBN) as a non-destructive method of defect detection has proliferated throughout the manufacturing community. Instrument technology and measurement methodology have matured commensurately as applications have moved from the R and D labs to the fully automated manufacturing environment. These new applications present a new set of challenges including a bevy of error sources. A significant obstacle in many industrial applications is a decrease in signal to noise ratio due to (i) environmental EMI and (II) compromises in sensor design for the purposes of automation. The stochastic nature of MBN presents a challenge to anymore »« less
https://doi.org/10.1063/1.4864969

Similar Records

Title: Automatic Kalman-filter-based wavelet shrinkage denoising of 1D stellar spectra

Abstract

Citation Formats

A New Signal Denoising Algorithm from Wavelet Modulus Maxima conference, August 2007

The Effect of AGN Heating on the Low-redshift Ly α Forest journal, January 2017

Bayesian unscented Kalman filter for state estimation of nonlinear and non-Gaussian systems conference, August 2016

Multiwavelets denoising using neighboring coefficients journal, July 2003

Parameter estimation for leaky aquifers using the extended Kalman filter, and considering model and data measurement uncertainties journal, February 2005

A theory for multiresolution signal decomposition: the wavelet representation journal, July 1989

A method of eliminating the signal-dependent random noise from the raw CMOS image sensor data based on Kalman filter journal, November 2014

An introduction to wavelets journal, July 1995

Speech denoising using discrete wavelet packet decomposition technique conference, May 2016

Warm dark matter as a solution to the small scale crisis: New constraints from high redshift Lyman- α forest data journal, August 2013

Wavelet Threshold Estimators for Data with Correlated Noise journal, May 1997

The what, how, and why of wavelet shrinkage denoising journal, January 2000

A New Wavelet Denoising Method for Selecting Decomposition Levels and Noise Thresholds journal, January 2016

Adapting to Unknown Smoothness via Wavelet Shrinkage journal, December 1995

A novel technique for selecting mother wavelet function using an intelli gent fault diagnosis system journal, April 2009

A wavelet-based data pre-processing analysis approach in mass spectrometry journal, April 2007

Adaptive wavelet packet thresholding with iterative Kalman filter for speech enhancement conference, November 2017

Image Quality Assessment: From Error Visibility to Structural Similarity journal, April 2004

A New Approach to Linear Filtering and Prediction Problems journal, March 1960

Improved Image Denoising Technique Using Neighboring Wavelet Coefficients of Optimal Wavelet with Adaptive Thresholding journal, January 2012

Wavelet-Based Combined Signal Filtering and Prediction journal, December 2005

Adaptive wavelet thresholding for image denoising and compression journal, January 2000

Covariance matching based adaptive unscented Kalman filter for direct filtering in INS/GNSS integration journal, March 2016

Ideal spatial adaptation by wavelet shrinkage journal, September 1994

Auto Regressive Moving Average (ARMA) Modeling Method for Gyro Random Noise Using a Robust Kalman Filter journal, September 2015

On-line Support Vector Regression of the transition model for the Kalman filter journal, June 2013

Analysis of time-varying signals using continuous wavelet and synchrosqueezed transforms journal, July 2018

Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures journal, January 2009

Learning to swim in a sea of wavelets journal, January 1995

New constraints on the free-streaming of warm dark matter from intermediate and small scale Lyman- α forest data journal, July 2017

Adaptive unscented Kalman filter for parameter and state estimation of nonlinear high-speed objects journal, August 2013

A new extensive library of PHOENIX stellar atmospheres and synthetic spectra journal, April 2013

A New Signal Denoising Algorithm from Wavelet Modulus Maxima
conference, August 2007

The Effect of AGN Heating on the Low-redshift Ly α Forest
journal, January 2017

Bayesian unscented Kalman filter for state estimation of nonlinear and non-Gaussian systems
conference, August 2016

Multiwavelets denoising using neighboring coefficients
journal, July 2003

Parameter estimation for leaky aquifers using the extended Kalman filter, and considering model and data measurement uncertainties
journal, February 2005

A theory for multiresolution signal decomposition: the wavelet representation
journal, July 1989

A method of eliminating the signal-dependent random noise from the raw CMOS image sensor data based on Kalman filter
journal, November 2014

An introduction to wavelets
journal, July 1995

Speech denoising using discrete wavelet packet decomposition technique
conference, May 2016

Warm dark matter as a solution to the small scale crisis: New constraints from high redshift Lyman- $α$ forest data
journal, August 2013

Wavelet Threshold Estimators for Data with Correlated Noise
journal, May 1997

The what, how, and why of wavelet shrinkage denoising
journal, January 2000

A New Wavelet Denoising Method for Selecting Decomposition Levels and Noise Thresholds
journal, January 2016

Adapting to Unknown Smoothness via Wavelet Shrinkage
journal, December 1995

A novel technique for selecting mother wavelet function using an intelli gent fault diagnosis system
journal, April 2009

A wavelet-based data pre-processing analysis approach in mass spectrometry
journal, April 2007

Adaptive wavelet packet thresholding with iterative Kalman filter for speech enhancement
conference, November 2017

Image Quality Assessment: From Error Visibility to Structural Similarity
journal, April 2004

A New Approach to Linear Filtering and Prediction Problems
journal, March 1960

Improved Image Denoising Technique Using Neighboring Wavelet Coefficients of Optimal Wavelet with Adaptive Thresholding
journal, January 2012

Wavelet-Based Combined Signal Filtering and Prediction
journal, December 2005

Adaptive wavelet thresholding for image denoising and compression
journal, January 2000

Covariance matching based adaptive unscented Kalman filter for direct filtering in INS/GNSS integration
journal, March 2016

Ideal spatial adaptation by wavelet shrinkage
journal, September 1994

Auto Regressive Moving Average (ARMA) Modeling Method for Gyro Random Noise Using a Robust Kalman Filter
journal, September 2015

On-line Support Vector Regression of the transition model for the Kalman filter
journal, June 2013

Analysis of time-varying signals using continuous wavelet and synchrosqueezed transforms
journal, July 2018

Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures
journal, January 2009

Learning to swim in a sea of wavelets
journal, January 1995

New constraints on the free-streaming of warm dark matter from intermediate and small scale Lyman- $α$ forest data
journal, July 2017

Adaptive unscented Kalman filter for parameter and state estimation of nonlinear high-speed objects
journal, August 2013

A new extensive library of PHOENIX stellar atmospheres and synthetic spectra
journal, April 2013