Bayesian K-SVD Using Fast Variational Inference

Serra, Juan G.; Testa, Matteo; Molina, Rafael; Katsaggelos, Aggelos K.

doi:10.1109/TIP.2017.2681436

Bayesian K-SVD Using Fast Variational Inference

Journal Article · Fri Mar 10 04:00:00 EST 2017 · IEEE Transactions on Image Processing

DOI:https://doi.org/10.1109/TIP.2017.2681436· OSTI ID:1487464

^[1]; ^[2]; Molina, Rafael ^[3]; ^[4]

Univ. de Granada, Granada (Spain); Northwestern University
Politecnico di Torino, Turin (Italy)
Univ. de Granada, Granada (Spain)
Northwestern Univ., Evanston, IL (United States)

Recent work in signal processing in general and image processing in particular deals with sparse representation related problems. Two such problems are of paramount importance: an overriding need for designing a well-suited overcomplete dictionary containing a redundant set of atoms— i.e., basis signals—and how to find a sparse representation of a given signal with respect to the chosen dictionary. Dictionary learning techniques, among which we find the popular K-singular value decomposition algorithm, tackle these problems by adapting a dictionary to a set of training data. A common drawback of such techniques is the need for parameter-tuning. In order to overcome this limitation, we propose a fullyautomated Bayesian method that considers the uncertainty of the estimates and produces a sparse representation of the data without prior information on the number of non-zeros in each representation vector. We follow a Bayesian approach that uses a three-tiered hierarchical prior to enforce sparsity on the representations and develop an efficient variational inference framework that reduces computational complexity. Furthermore, we describe a greedy approach that speeds up the whole process. Lastly, we present experimental results that show superior performance on two different applications with real images: denoising and inpainting.

Research Organization:: Northwestern Univ., Evanston, IL (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA), Office of Nonproliferation and Verification Research and Development (NA-22)

DOE Contract Number:: NA0002520

OSTI ID:: 1487464

Journal Information:: IEEE Transactions on Image Processing, Journal Name: IEEE Transactions on Image Processing Journal Issue: 7 Vol. 26; ISSN 1057-7149

Publisher:: IEEE

Country of Publication:: United States

Language:: English

References (43)

Dictionaries for Sparse Representation Modeling Rubinstein, Ron; Bruckstein, Alfred M.; Elad, Michael Proceedings of the IEEE, Vol. 98, Issue 6 https://doi.org/10.1109/JPROC.2010.2040551	journal	June 2010
Recursive Least Squares Dictionary Learning Algorithm Skretting, Karl; Engan, Kjersti IEEE Transactions on Signal Processing, Vol. 58, Issue 4 https://doi.org/10.1109/TSP.2010.2040671	journal	April 2010
A non-convex relaxation approach to sparse dictionary learning Shi, Jianping; Ren, Xiang; Dai, Guang CVPR 2011 https://doi.org/10.1109/CVPR.2011.5995592	conference	June 2011
A Variational Method for Learning Sparse and Overcomplete Representations Girolami, Mark Neural Computation, Vol. 13, Issue 11 https://doi.org/10.1162/089976601753196003	journal	November 2001
$rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation Aharon, M.; Elad, M.; Bruckstein, A. IEEE Transactions on Signal Processing, Vol. 54, Issue 11 https://doi.org/10.1109/TSP.2006.881199	journal	November 2006
Emergence of simple-cell receptive field properties by learning a sparse code for natural images Olshausen, Bruno A.; Field, David J. Nature, Vol. 381, Issue 6583 https://doi.org/10.1038/381607a0	journal	June 1996
Discriminative K-SVD for dictionary learning in face recognition Zhang, Qiang; Li, Baoxin 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2010.5539989	conference	June 2010
On the Role of Sparse and Redundant Representations in Image Processing Elad, Michael; Figueiredo, Mário A. T.; Ma, Yi Proceedings of the IEEE, Vol. 98, Issue 6 https://doi.org/10.1109/JPROC.2009.2037655	journal	June 2010
Analysis SimCO Algorithms for Sparse Analysis Model Based Dictionary Learning Dong, Jing; Wang, Wenwu; Dai, Wei IEEE Transactions on Signal Processing, Vol. 64, Issue 2 https://doi.org/10.1109/TSP.2015.2483480	journal	January 2016
Nonparametric factor analysis with beta process priors Paisley, John; Carin, Lawrence ICML '09 Proceedings of the 26th Annual International Conference on Machine Learning, p. 777-784 https://doi.org/10.1145/1553374.1553474	conference	January 2009
Sparse coding with an overcomplete basis set: A strategy employed by V1? Olshausen, Bruno A.; Field, David J. Vision Research, Vol. 37, Issue 23 https://doi.org/10.1016/S0042-6989(97)00169-7	journal	December 1997
Iterative Convex Refinement for Sparse Recovery Mousavi, Hojjat S.; Monga, Vishal; Tran, Trac D. IEEE Signal Processing Letters, Vol. 22, Issue 11 https://doi.org/10.1109/LSP.2015.2438255	journal	November 2015
Augmented Lagrangian-Based Sparse Representation Method with Dictionary Updating for Image Deblurring Liu, Qiegen; Liang, Dong; Song, Ying SIAM Journal on Imaging Sciences, Vol. 6, Issue 3 https://doi.org/10.1137/110857349	journal	January 2013
A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems Beck, Amir; Teboulle, Marc SIAM Journal on Imaging Sciences, Vol. 2, Issue 1 https://doi.org/10.1137/080716542	journal	January 2009
Bayesian Compressive Sensing Using Laplace Priors Babacan, S. D.; Molina, R.; Katsaggelos, A. K. IEEE Transactions on Image Processing, Vol. 19, Issue 1 https://doi.org/10.1109/TIP.2009.2032894	journal	January 2010
A Singular Value Thresholding Algorithm for Matrix Completion Cai, Jian-Feng; Candès, Emmanuel J.; Shen, Zuowei SIAM Journal on Optimization, Vol. 20, Issue 4 https://doi.org/10.1137/080738970	journal	January 2010
Learning Overcomplete Representations Lewicki, Michael S.; Sejnowski, Terrence J. Neural Computation, Vol. 12, Issue 2 https://doi.org/10.1162/089976600300015826	journal	February 2000
Dictionary Learning Algorithms for Sparse Representation Kreutz-Delgado, Kenneth; Murray, Joseph F.; Rao, Bhaskar D. Neural Computation, Vol. 15, Issue 2, p. 349-396 https://doi.org/10.1162/089976603762552951	journal	February 2003
Learning Overcomplete Dictionaries Based on Atom-by-Atom Updating Sadeghi, Mostafa; Babaie-Zadeh, Massoud; Jutten, Christian IEEE Transactions on Signal Processing, Vol. 62, Issue 4 https://doi.org/10.1109/TSP.2013.2295062	journal	February 2014
Random Projections of Smooth Manifolds Baraniuk, Richard G.; Wakin, Michael B. Foundations of Computational Mathematics, Vol. 9, Issue 1 https://doi.org/10.1007/s10208-007-9011-z	journal	December 2007
Sparse signal reconstruction from limited data using FOCUSS: a re-weighted minimum norm algorithm Gorodnitsky, I. F.; Rao, B. D. IEEE Transactions on Signal Processing, Vol. 45, Issue 3 https://doi.org/10.1109/78.558475	journal	March 1997
Learning Incoherent Dictionaries for Sparse Approximation Using Iterative Projections and Rotations Barchiesi, Daniele; Plumbley, Mark D. IEEE Transactions on Signal Processing, Vol. 61, Issue 8 https://doi.org/10.1109/TSP.2013.2245663	journal	April 2013
Family of iterative LS-based dictionary learning algorithms, ILS-DLA, for sparse signal representation Engan, Kjersti; Skretting, Karl; Husøy, John Håkon Digital Signal Processing, Vol. 17, Issue 1 https://doi.org/10.1016/j.dsp.2006.02.002	journal	January 2007
Sparse and Spurious: Dictionary Learning With Noise and Outliers Gribonval, Remi; Jenatton, Rodolphe; Bach, Francis IEEE Transactions on Information Theory, Vol. 61, Issue 11 https://doi.org/10.1109/TIT.2015.2472522	journal	November 2015
Infinite Sparse Factor Analysis and Infinite Independent Components Analysis Knowles, David; Ghahramani, Zoubin Independent Component Analysis and Signal Separation https://doi.org/10.1007/978-3-540-74494-8_48	book	January 2007
Highly Undersampled Magnetic Resonance Image Reconstruction Using Two-Level Bregman Method With Dictionary Updating No authors listed IEEE Transactions on Medical Imaging, Vol. 32, Issue 7 https://doi.org/10.1109/TMI.2013.2256464	journal	July 2013
Histopathological Image Classification Using Discriminative Feature-Oriented Dictionary Learning Vu, Tiep Huu; Mousavi, Hojjat Seyed; Monga, Vishal IEEE Transactions on Medical Imaging, Vol. 35, Issue 3 https://doi.org/10.1109/TMI.2015.2493530	journal	March 2016
Online Bayesian dictionary learning for large datasets Li, Lingbo; Silva, Jorge; Zhou, Mingyuan 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP.2012.6288339	conference	March 2012
Spike and slab variable selection: Frequentist and Bayesian strategies Ishwaran, Hemant; Rao, J. Sunil The Annals of Statistics, Vol. 33, Issue 2 https://doi.org/10.1214/009053604000001147	journal	April 2005
A majorization–minimization approach to variable selection using spike and slab priors Yen, Tso-Jung The Annals of Statistics, Vol. 39, Issue 3 https://doi.org/10.1214/11-AOS884	journal	June 2011
Dictionary Learning for Sparse Coding: Algorithms and Convergence Analysis Bao, Chenglong; Ji, Hui; Quan, Yuhui IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, Issue 7 https://doi.org/10.1109/TPAMI.2015.2487966	journal	July 2016
Automated Recovery of Compressedly Observed Sparse Signals From Smooth Background Chen, Zhaofu; Molina, Rafael; Katsaggelos, Aggelos K. IEEE Signal Processing Letters, Vol. 21, Issue 8, p. 1012-1016 https://doi.org/10.1109/LSP.2014.2321256	journal	April 2014
Turbo inpainting: Iterative K-SVD with a new dictionary Koh, Min-Sung; Rodriguez-Marek, Esteban 2009 IEEE International Workshop on Multimedia Signal Processing https://doi.org/10.1109/MMSP.2009.5293571	conference	October 2009
Image Denoising Via Sparse and Redundant Representations Over Learned Dictionaries Elad, Michael; Aharon, Michal IEEE Transactions on Image Processing, Vol. 15, Issue 12 https://doi.org/10.1109/TIP.2006.881969	journal	January 2006
Sparse and Redundant Representations Elad, Michael Springer New York, NY https://doi.org/10.1007/978-1-4419-7011-4	book	January 2010
Analysis K-SVD: A Dictionary-Learning Algorithm for the Analysis Sparse Model Rubinstein, Ron; Peleg, Tomer; Elad, Michael IEEE Transactions on Signal Processing, Vol. 61, Issue 3 https://doi.org/10.1109/TSP.2012.2226445	journal	February 2013
Sparse Representation for Color Image Restoration Mairal, Julien; Elad, Michael; Sapiro, Guillermo IEEE Transactions on Image Processing, Vol. 17, Issue 1 https://doi.org/10.1109/TIP.2007.911828	journal	January 2008
Bayesian Nonparametric Dictionary Learning for Compressed Sensing MRI Huang, Yue; Paisley, John; Lin, Qin IEEE Transactions on Image Processing, Vol. 23, Issue 12, p. 5007-5019 https://doi.org/10.1109/TIP.2014.2360122	journal	December 2014
Method of optimal directions for frame design Engan, K.; Aase, S. O.; Hakon Husoy, J. 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing https://doi.org/10.1109/ICASSP.1999.760624	conference	January 1999
Shiftable multiscale transforms Simoncelli, E. P.; Freeman, W. T.; Adelson, E. H. IEEE Transactions on Information Theory, Vol. 38, Issue 2 https://doi.org/10.1109/18.119725	journal	March 1992
Classification and clustering via dictionary learning with structured incoherence and shared features Ramirez, Ignacio; Sprechmann, Pablo; Sapiro, Guillermo 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2010.5539964	conference	June 2010
Nonparametric Bayesian Dictionary Learning for Analysis of Noisy and Incomplete Images Zhou, Mingyuan; Chen, Haojun; Paisley, John IEEE Transactions on Image Processing, Vol. 21, Issue 1, p. 130-144 https://doi.org/10.1109/TIP.2011.2160072	journal	January 2012
Online dictionary learning for sparse coding Mairal, Julien; Bach, Francis; Ponce, Jean Proceedings of the 26th Annual International Conference on Machine Learning - ICML '09 https://doi.org/10.1145/1553374.1553463	conference	January 2009

Similar Records

Sparse Overcomplete Denoising: Aggregation Versus Global Optimization

Journal Article · Mon Jul 31 00:00:00 EDT 2017 · IEEE Signal Processing Letters · OSTI ID:1467204

Image fusion using sparse overcomplete feature dictionaries

Patent · Tue Oct 06 00:00:00 EDT 2015 · OSTI ID:1222629

Tensor-Dictionary Learning with Deep Kruskal-Factor Analysis

Conference · Thu Apr 20 00:00:00 EDT 2017 · OSTI ID:1355098

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayesian modeling
denoising
dictionary learning
inpainting
k-svd
sparse representation
variational inference

Bayesian K-SVD Using Fast Variational Inference

Citation Formats

References (43)

Similar Records

Related Subjects