What is the gradient of a scalar function defined on a subspace of square matrices?

Srinivasan, Shriram; Panda, Nishant

doi:10.1007/s13226-024-00594-4

Title: What is the gradient of a scalar function defined on a subspace of square matrices?

Journal Article · 24 April 2024 · Indian Journal of Pure and Applied Mathematics

DOI: https://doi.org/10.1007/s13226-024-00594-4 · OSTI ID:2426769

^[1];

^[1]

Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

We illustrate a technique to calculate the gradient of scalar functions that are defined on any arbitrary matrix subspace. It generalizes our earlier work titled “What is the gradient of a scalar function of a symmetric matrix ?”(Indian Journal of Pure and Applied Mathematics (2022), https://doi.org/10.1007/s13226-022-00313-x), in which we considered the special case of the subspace of symmetric matrices. Extant methods to calculate the gradient in such cases have an inherent flaw which leads to spurious results that populate several publications, as well as respected textbooks and handbooks on matrix calculus. Here, we examine these sources and results in a rigorous and concrete mathematical setting of a finite-dimensional inner-product space and discover the inherent flaw and also a remedy. We demonstrate two ways to calculate the derivative/gradient and second derivative for scalar functions of matrices defined over an arbitrary matrix subspace; the first method is by considering any (differentiable) extension to the space of square matrices and projection of its gradient onto the given subspace. The second method utilizes an ordered basis and computes each component of the gradient through evaluation of the directional derivative. All the ideas presented are illustrated by non-trivial examples, namely, considering the subspace of 3 x 3 circulant and Toeplitz matrices and presenting the results of gradient-descent with both the spurious and correct gradients. Moreover, our bibliography makes it clear that a rigorous approach to matrix calculus is not common in practice, and our presentation of matrix calculus in the language of inner-product spaces will be significant and meaningful for applied mathematicians, engineers and researchers working in inter-disciplinary fields to avoid the conceptual pitfalls that exist.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Organization:: USDOE Laboratory Directed Research and Development (LDRD) Program

Grant/Contract Number:: 89233218CNA000001

OSTI ID:: 2426769

Report Number(s):: LA-UR--23-23471

Journal Information:: Indian Journal of Pure and Applied Mathematics, Journal Name: Indian Journal of Pure and Applied Mathematics; ISSN 0975-7465; ISSN 0019-5588

Country of Publication:: United States

Language:: English

References (29)

Maximum likelihood solution to factor analysis when some factors are completely specified Gebhardt, Friedrich Psychometrika, Vol. 36, Issue 2 https://doi.org/10.1007/BF02291395	journal	June 1971
Structured Matrices and Their Application in Neural Networks: A Survey Kissel, Matthias; Diepold, Klaus New Generation Computing, Vol. 41, Issue 3 https://doi.org/10.1007/s00354-023-00226-1	journal	July 2023
What is the gradient of a scalar function of a symmetric matrix? Srinivasan, Shriram; Panda, Nishant Indian Journal of Pure and Applied Mathematics, Vol. 54, Issue 3 https://doi.org/10.1007/s13226-022-00313-x	journal	August 2022
Tensor products and matrix differential calculus Pollock, D. S. G. Linear Algebra and its Applications, Vol. 67 https://doi.org/10.1016/0024-3795(85)90194-6	journal	June 1985
On some pattern-reduction matrices which appear in statistics Wiens, Douglas P. Linear Algebra and its Applications, Vol. 67 https://doi.org/10.1016/0024-3795(85)90199-5	journal	June 1985
About the concept of the matrix derivative Parring, A. -M. Linear Algebra and its Applications, Vol. 176 https://doi.org/10.1016/0024-3795(92)90220-5	journal	November 1992
The matrix minimum principle Athans, Michael Information and Control, Vol. 11, Issue 5-6 https://doi.org/10.1016/S0019-9958(67)90803-0	journal	November 1967
On the concept of matrix derivative Magnus, Jan R. Journal of Multivariate Analysis, Vol. 101, Issue 9 https://doi.org/10.1016/j.jmva.2010.05.005	journal	October 2010
On the use of coordinate-free matrix calculus Brinkhuis, Jan Journal of Multivariate Analysis, Vol. 133 https://doi.org/10.1016/j.jmva.2014.09.019	journal	January 2015
Matrix differential calculus with applications in the multivariate linear model and its diagnostics Liu, Shuangzhe; Leiva, Víctor; Zhuang, Dan Journal of Multivariate Analysis, Vol. 188 https://doi.org/10.1016/j.jmva.2021.104849	journal	March 2022
On Kronecker products, tensor products and matrix differential calculus Pollock, D. S. G. International Journal of Computer Mathematics, Vol. 90, Issue 11 https://doi.org/10.1080/00207160.2013.783696	journal	November 2013
Optimization of feedback systems with constrained information flow Yanchevsky, A. E.; Hirvonen, V. J. International Journal of Systems Science, Vol. 12, Issue 12 https://doi.org/10.1080/00207728108963830	journal	January 1981
Some Applications of Matrix Derivatives in Multivariate Analysis Dwyer, Paul S. Journal of the American Statistical Association, Vol. 62, Issue 318 https://doi.org/10.1080/01621459.1967.10482934	journal	June 1967
On Circulant Matrices Kra, Irwin; Simanca, Santiago R. Notices of the American Mathematical Society, Vol. 59, Issue 03 https://doi.org/10.1090/noti804	journal	March 2012
Finding patterned complex-valued matrix derivatives by using manifolds Hjorungnes, Are; Palomar, Daniel P. 2008 First International Symposium on Applied Sciences on Biomedical and Communication Technologies https://doi.org/10.1109/ISABEL.2008.4712619	conference	October 2008
Estimation of structured covariance matrices Burg, J. P.; Luenberger, D. G.; Wenger, D. L. Proceedings of the IEEE, Vol. 70, Issue 9 https://doi.org/10.1109/PROC.1982.12427	journal	January 1982
Patterned complex-valued matrix derivatives Hjorungnes, Are; Palomar, Daniel P. 2008 5th IEEE Sensor Array and Multichannel Signal Processing Workshop https://doi.org/10.1109/SAM.2008.4606875	conference	July 2008
Derivative operations on matrices Vetter, W. IEEE Transactions on Automatic Control, Vol. 15, Issue 2 https://doi.org/10.1109/TAC.1970.1099409	journal	April 1970
On calculating gradient matrices Geering, H. IEEE Transactions on Automatic Control, Vol. 21, Issue 4 https://doi.org/10.1109/TAC.1976.1101267	journal	August 1976
The gradient with respect to a symmetric matrix Brewer, J. IEEE Transactions on Automatic Control, Vol. 22, Issue 2 https://doi.org/10.1109/TAC.1977.1101459	journal	April 1977
On symmetric matrices and the matrix minimum principle Walsh, P. IEEE Transactions on Automatic Control, Vol. 22, Issue 6 https://doi.org/10.1109/TAC.1977.1101664	journal	December 1977
Downlink MIMO-RSMA With Successive Null-Space Precoding Krishnamoorthy, Aravindh; Schober, Robert IEEE Transactions on Wireless Communications, Vol. 21, Issue 11 https://doi.org/10.1109/TWC.2022.3173463	journal	November 2022
Matrix Calculus Operations and Taylor Expansions Vetter, William J. SIAM Review, Vol. 15, Issue 2 https://doi.org/10.1137/1015034	journal	April 1973
Symbolic Matrix Derivatives Dwyer, Paul S.; Macphail, M. S. The Annals of Mathematical Statistics, Vol. 19, Issue 4 https://doi.org/10.1214/aoms/1177730148	journal	December 1948
Toeplitz and Circulant Matrices: A Review Gray, Robert M. Foundations and Trends® in Communications and Information Theory, Vol. 2, Issue 3 https://doi.org/10.1561/0100000006	journal	January 2005
Multivariate Maxima and Minima with Matrix Derivatives Tracy, Derrick S.; Dwyer, Paul S. Journal of the American Statistical Association, Vol. 64, Issue 328 https://doi.org/10.2307/2286090	journal	December 1969
Patterned matrix derivatives Tracy, D. S.; Jinadasa, K. G. Canadian Journal of Statistics, Vol. 16, Issue 4 https://doi.org/10.2307/3314938	journal	December 1988
Vec and vech operators for matrices, with some uses in jacobians and multivariate statistics Henderson, Harold V.; Searle, S. R. Canadian Journal of Statistics, Vol. 7, Issue 1 https://doi.org/10.2307/3315017	journal	January 1979
The Influence Function of Graphical Lasso Estimators Louvet, Gaëtan; Raymaekers, Jakob; Van Bever, Germain arXiv https://doi.org/10.48550/arXiv.2209.07374	preprint	January 2022