Discovering causal structure with reproducing-kernel Hilbert space ε-machines

Brodu, Nicolas; Crutchfield, James P.

doi:10.1063/5.0062829

Title: Discovering causal structure with reproducing-kernel Hilbert space ε-machines

Journal Article · 31 January 2022 · Chaos: An Interdisciplinary Journal of Nonlinear Science

DOI: https://doi.org/10.1063/5.0062829 · OSTI ID:1978952

^[1];

^[2]

Centre Inria de l'université de Bordeaux (France); INDIRA, Talence Cedex (France)
University of California, Davis, CA (United States)

We merge computational mechanics’ definition of causal states (predictively equivalent histories) with reproducing-kernel Hilbert space (RKHS) representation inference. The result is a widely applicable method that infers causal structure directly from observations of a system’s behaviors whether they are over discrete or continuous events or time. A structural representation—a finite- or infinite-state kernel ϵ-machine—is extracted by a reduced-dimension transform that gives an efficient representation of causal states and their topology. In this way, the system dynamics are represented by a stochastic (ordinary or partial) differential equation that acts on causal states. We introduce an algorithm to estimate the associated evolution operator. Paralleling the Fokker–Planck equation, it efficiently evolves causal-state distributions and makes predictions in the original data space via an RKHS functional mapping. We demonstrate these techniques, together with their predictive abilities, on discrete-time, discrete-value infinite Markov-order processes generated by finite-state hidden Markov models with (i) finite or (ii) uncountably infinite causal states and (iii) continuous-time, continuous-value processes generated by thermally driven chaotic flows. The method robustly estimates causal structure in the presence of varying external and measurement noise levels and for very high-dimensional data.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: University of California, Davis, CA (United States)

Sponsoring Organization:: Army Research Laboratory; Army Research Office (ARO); Foundational Questions Institute; USDOE; USDOE Office of Science (SC)

Grant/Contract Number:: SC0017324

OSTI ID:: 1978952

Journal Information:: Chaos: An Interdisciplinary Journal of Nonlinear Science, Journal Name: Chaos: An Interdisciplinary Journal of Nonlinear Science Journal Issue: 2 Vol. 32; ISSN 1054-1500

Publisher:: American Institute of Physics (AIP)Copyright Statement

Country of Publication:: United States

Language:: English

References (54)

Approximating a Gram Matrix for Improved Kernel-Based Learning Drineas, Petros; Mahoney, Michael W. Learning Theory https://doi.org/10.1007/11503415_22	book	January 2005
Diffusion maps Coifman, Ronald R.; Lafon, Stéphane Applied and Computational Harmonic Analysis, Vol. 21, Issue 1 https://doi.org/10.1016/j.acha.2006.04.006	journal	July 2006
Automated pattern detection—An algorithm for constructing optimally synchronizing multi-regular language filters McTague, Carl S.; Crutchfield, James P. Theoretical Computer Science, Vol. 359, Issue 1-3 https://doi.org/10.1016/j.tcs.2006.05.002	journal	August 2006
Spectral simplicity of apparent complexity. II. Exact complexities and complexity spectra Riechers, Paul M.; Crutchfield, James P. Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 28, Issue 3 https://doi.org/10.1063/1.4986248	journal	March 2018
Quantifying Self-Organization with Optimal Predictors Shalizi, Cosma Rohilla; Shalizi, Kristina Lisa; Haslinger, Robert Physical Review Letters, Vol. 93, Issue 11 https://doi.org/10.1103/physrevlett.93.118701	journal	September 2004
Reconstruction of Epsilon-Machines in Predictive Frameworks and Decisional States Brodu, Nicolas arXiv https://doi.org/10.48550/arxiv.0902.0600	preprint	January 2009
Time's Barbed Arrow: Irreversibility, Crypticity, and Stored Information Crutchfield, James P.; Ellison, Christopher J.; Mahoney, John R. arXiv https://doi.org/10.48550/arxiv.0902.1209	text	January 2009
Prediction, Retrodiction, and The Amount of Information Stored in the Present Ellison, Christopher J.; Mahoney, John R.; Crutchfield, James P. arXiv https://doi.org/10.48550/arxiv.0905.3587	text	January 2009
Circumventing the Curse of Dimensionality in Prediction: Causal Rate-Distortion for Infinite-Order Markov Processes Marzen, Sarah; Crutchfield, James P. arXiv https://doi.org/10.48550/arxiv.1412.2859	preprint	January 2014
Local Causal States and Discrete Coherent Structures Rupe, Adam; Crutchfield, James P. arXiv https://doi.org/10.48550/arxiv.1801.00515	text	January 2018
Kernel Conditional Density Operators Schuster, Ingmar; Mollenhauer, Mattes; Klus, Stefan arXiv https://doi.org/10.48550/arxiv.1905.11255	preprint	January 2019
Spacetime Autoencoders Using Local Causal States Rupe, Adam; Crutchfield, James P. arXiv https://doi.org/10.48550/arxiv.2010.05451	preprint	January 2020
Blind Construction of Optimal Nonlinear Recursive Predictors for Discrete Sequences Shalizi, Cosma Rohilla; Shalizi, Kristina Lisa arXiv https://doi.org/10.48550/arxiv.cs/0406011	text	January 2004
A Generalized Representer Theorem Schölkopf, Bernhard; Herbrich, Ralf; Smola, Alex J. Lecture Notes in Computer Science https://doi.org/10.1007/3-540-44581-1_27	book	January 2001
Eigendecompositions of Transfer Operators in Reproducing Kernel Hilbert Spaces Klus, Stefan; Schuster, Ingmar; Muandet, Krikamol Journal of Nonlinear Science, Vol. 30, Issue 1 https://doi.org/10.1007/s00332-019-09574-z	journal	August 2019
Prediction, Retrodiction, and the Amount of Information Stored in the Present Ellison, Christopher J.; Mahoney, John R.; Crutchfield, James P. Journal of Statistical Physics, Vol. 136, Issue 6 https://doi.org/10.1007/s10955-009-9808-z	journal	September 2009
Predictive Rate-Distortion for Infinite-Order Markov Processes Marzen, Sarah E.; Crutchfield, James P. Journal of Statistical Physics, Vol. 163, Issue 6 https://doi.org/10.1007/s10955-016-1520-1	journal	May 2016
Informational and Causal Architecture of Continuous-time Renewal Processes Marzen, Sarah; Crutchfield, James P. Journal of Statistical Physics, Vol. 168, Issue 1 https://doi.org/10.1007/s10955-017-1793-z	journal	April 2017
Structure and Randomness of Continuous-Time, Discrete-Event Processes Marzen, Sarah E.; Crutchfield, James P. Journal of Statistical Physics, Vol. 169, Issue 2 https://doi.org/10.1007/s10955-017-1859-y	journal	August 2017
Shannon Entropy Rate of Hidden Markov Processes Jurgens, Alexandra M.; Crutchfield, James P. Journal of Statistical Physics, Vol. 183, Issue 2 https://doi.org/10.1007/s10955-021-02769-3	journal	May 2021
Computational mechanics of cellular automata: An example Hanson, James E.; Crutchfield, James P. Physica D: Nonlinear Phenomena, Vol. 103, Issue 1-4 https://doi.org/10.1016/S0167-2789(96)00259-X	journal	April 1997
Diffusion maps Coifman, Ronald R.; Lafon, Stéphane Applied and Computational Harmonic Analysis, Vol. 21, Issue 1 https://doi.org/10.1016/j.acha.2006.04.006	journal	July 2006
Variable bandwidth diffusion kernels Berry, Tyrus; Harlim, John Applied and Computational Harmonic Analysis, Vol. 40, Issue 1 https://doi.org/10.1016/j.acha.2015.01.001	journal	January 2016
Local kernels and the geometric structure of data Berry, Tyrus; Sauer, Timothy Applied and Computational Harmonic Analysis, Vol. 40, Issue 3 https://doi.org/10.1016/j.acha.2015.03.002	journal	May 2016
Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques Alexander, Romeo; Giannakis, Dimitrios Physica D: Nonlinear Phenomena, Vol. 409 https://doi.org/10.1016/j.physd.2020.132520	journal	August 2020
Approaching complexity by stochastic methods: From biological systems to turbulence Friedrich, Rudolf; Peinke, Joachim; Sahimi, Muhammad Physics Reports, Vol. 506, Issue 5 https://doi.org/10.1016/j.physrep.2011.05.003	journal	September 2011
Automated pattern detection—An algorithm for constructing optimally synchronizing multi-regular language filters McTague, Carl S.; Crutchfield, James P. Theoretical Computer Science, Vol. 359, Issue 1-3 https://doi.org/10.1016/j.tcs.2006.05.002	journal	August 2006
Stochastic Processes for Physicists Jacobs, Kurt https://doi.org/10.1017/CBO9780511815980	book	June 2012
Computational Mechanics: Pattern and Prediction, Structure and Simplicity Shalizi, Cosma Rohilla; Crutchfield, James P. Journal of Statistical Physics, Vol. 104, Issue 3/4, p. 817-879 https://doi.org/10.1023/A:1010388907793	journal	August 2001
Between order and chaos Crutchfield, James P. Nature Physics, Vol. 8, Issue 1 https://doi.org/10.1038/nphys2190	journal	December 2011
Spectral simplicity of apparent complexity. II. Exact complexities and complexity spectra Riechers, Paul M.; Crutchfield, James P. Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 28, Issue 3 https://doi.org/10.1063/1.4986248	journal	March 2018
Local causal states and discrete coherent structures Rupe, Adam; Crutchfield, James P. Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 28, Issue 7 https://doi.org/10.1063/1.5021130	journal	July 2018
Divergent predictive states: The statistical complexity dimension of stationary, ergodic hidden Markov processes Jurgens, Alexandra M.; Crutchfield, James P. Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 31, Issue 8 https://doi.org/10.1063/5.0050460	journal	August 2021
Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps Coifman, R. R.; Lafon, S.; Lee, A. B. Proceedings of the National Academy of Sciences, Vol. 102, Issue 21 https://doi.org/10.1073/pnas.0500334102	journal	May 2005
Geometric diffusions as a tool for harmonic analysis and structure definition of data: Multiscale methods Coifman, R. R.; Lafon, S.; Lee, A. B. Proceedings of the National Academy of Sciences, Vol. 102, Issue 21, p. 7432-7437 https://doi.org/10.1073/pnas.0500896102	journal	May 2005
Theory of reproducing kernels Aronszajn, N. Transactions of the American Mathematical Society, Vol. 68, Issue 3 https://doi.org/10.1090/S0002-9947-1950-0051437-7	journal	March 1950
Automatic filters for the detection of coherent structure in spatiotemporal systems Shalizi, Cosma Rohilla; Haslinger, Robert; Rouquier, Jean-Baptiste Physical Review E, Vol. 73, Issue 3 https://doi.org/10.1103/PhysRevE.73.036104	journal	March 2006
Bayesian structural inference for hidden processes Strelioff, Christopher C.; Crutchfield, James P. Physical Review E, Vol. 89, Issue 4 https://doi.org/10.1103/PhysRevE.89.042119	journal	April 2014
Nonparametric forecasting of low-dimensional dynamical systems Berry, Tyrus; Giannakis, Dimitrios; Harlim, John Physical Review E, Vol. 91, Issue 3 https://doi.org/10.1103/PhysRevE.91.032915	journal	March 2015
Nearly maximally predictive features and their dimensions Marzen, Sarah E.; Crutchfield, James P. Physical Review E, Vol. 95, Issue 5 https://doi.org/10.1103/PhysRevE.95.051301	journal	May 2017
Time’s Barbed Arrow: Irreversibility, Crypticity, and Stored Information Crutchfield, James P.; Ellison, Christopher J.; Mahoney, John R. Physical Review Letters, Vol. 103, Issue 9 https://doi.org/10.1103/PhysRevLett.103.094101	journal	August 2009
Inferring statistical complexity Crutchfield, James P.; Young, Karl Physical Review Letters, Vol. 63, Issue 2 https://doi.org/10.1103/PhysRevLett.63.105	journal	July 1989
Quantifying Self-Organization with Optimal Predictors Shalizi, Cosma Rohilla; Shalizi, Kristina Lisa; Haslinger, Robert Physical Review Letters, Vol. 93, Issue 11 https://doi.org/10.1103/PhysRevLett.93.118701	journal	September 2004
Fraudulent white noise: Flat power spectra belie arbitrarily complex processes Riechers, Paul M.; Crutchfield, James P. Physical Review Research, Vol. 3, Issue 1 https://doi.org/10.1103/PhysRevResearch.3.013170	journal	February 2021
Quantifying the Effect of Learning on Recurrent Spikin Neurons Brodu, Nicolas 2007 International Joint Conference on Neural Networks https://doi.org/10.1109/IJCNN.2007.4371009	conference	August 2007
DisCo: Physics-Based Unsupervised Discovery of Coherent Structures in Spatiotemporal Systems Rupe, Adam; Prabhat, Mr; Crutchfield, James P. 2019 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC) https://doi.org/10.1109/MLHPC49564.2019.00013	conference	November 2019
Solving the pre-image problem in kernel machines: A direct method Honeine, Paul; Richard, Cedric 2009 IEEE International Workshop on Machine Learning for Signal Processing (MLSP) https://doi.org/10.1109/MLSP.2009.5306204	conference	September 2009
Kernel Embeddings of Conditional Distributions: A Unified Kernel Framework for Nonparametric Inference in Graphical Models Song, Le; Fukumizu, Kenji; Gretton, Arthur IEEE Signal Processing Magazine, Vol. 30, Issue 4 https://doi.org/10.1109/MSP.2013.2252713	journal	July 2013
Graph Laplacian Tomography From Unknown Random Projections Coifman, R. R.; Shkolnisky, Y.; Sigworth, F. J. IEEE Transactions on Image Processing, Vol. 17, Issue 10 https://doi.org/10.1109/TIP.2008.2002305	journal	October 2008
Reconstruction of Epsilon-Machines in Predictive Frameworks and Decisional States Brodu, Nicolas Advances in Complex Systems, Vol. 14, Issue 05 https://doi.org/10.1142/S0219525911003347	journal	October 2011
A training algorithm for optimal margin classifiers Boser, Bernhard E.; Guyon, Isabelle M.; Vapnik, Vladimir N. Proceedings of the fifth annual workshop on Computational learning theory - COLT '92 https://doi.org/10.1145/130385.130401	conference	January 1992
Tailoring density estimation via reproducing kernel moment matching Song, Le; Zhang, Xinhua; Smola, Alex Proceedings of the 25th international conference on Machine learning - ICML '08 https://doi.org/10.1145/1390156.1390281	conference	January 2008
Hilbert space embeddings of conditional distributions with applications to dynamical systems Song, Le; Huang, Jonathan; Smola, Alex Proceedings of the 26th Annual International Conference on Machine Learning - ICML '09 https://doi.org/10.1145/1553374.1553497	conference	January 2009
Time resolution dependence of information measures for spiking neurons: scaling and universality Marzen, Sarah E.; DeWeese, Michael R.; Crutchfield, James P. Frontiers in Computational Neuroscience, Vol. 9 https://doi.org/10.3389/fncom.2015.00105	journal	August 2015

Cited By (1)

Assessing planetary complexity and potential agnostic biosignatures using epsilon machines Bartlett, Stuart; Li, Jiazheng; Gu, Lixiang Nature Astronomy, Vol. 6, Issue 3 https://doi.org/10.1038/s41550-021-01559-x	journal	February 2022

Similar Records

Reproducing kernel Hilbert C*-modules and kernels associated with cocycles

Journal Article · 2008 · Journal of Mathematical Physics · OSTI ID:21175675

Heo, Jaeseong

AN ASYMPTOTIC CAUSALITY REQUIREMENT FOR SYSTEMS WITH CONSTRAINED INPUTS

Journal Article · 1959 · Nuclear Phys. · OSTI ID:4199082

Sucher, J

Optimizing Kernel Machines Using Deep Learning

Journal Article · 2018 · IEEE Transactions on Neural Networks and Learning Systems · OSTI ID:1463836

Song, Huan; Thiagarajan, Jayaraman J.; Sattigeri, Prasanna; +1 more

Related Subjects

71 CLASSICAL AND QUANTUM MECHANICS
GENERAL PHYSICS
97 MATHEMATICS AND COMPUTING
Hilbert space
Markov processes
chaotic dynamics
complex systems theory
machine learning
probability theory
stochastic processes

Title: Discovering causal structure with reproducing-kernel Hilbert space ε-machines

Citation Formats

References (54)

Cited By (1)

Similar Records

Related Subjects