Factorized visual representations in the primate visual system and deep neural networks

Lindsey, Jack W.; Issa, Elias B.

doi:10.7554/elife.91685

Factorized visual representations in the primate visual system and deep neural networks

Journal Article · Fri Jul 05 04:00:00 EDT 2024 · eLife

DOI:https://doi.org/10.7554/elife.91685· OSTI ID:2471917

^[1]; ^[2]

Columbia University, New York, NY (United States); OSTI
Columbia University, New York, NY (United States)

Object classification has been proposed as a principal objective of the primate ventral visual stream and has been used as an optimization target for deep neural network models (DNNs) of the visual system. However, visual brain areas represent many different types of information, and optimizing for classification of object identity alone does not constrain how other information may be encoded in visual representations. Information about different scene parameters may be discarded altogether (‘invariance’), represented in non-interfering subspaces of population activity (‘factorization’) or encoded in an entangled fashion. In this work, we provide evidence that factorization is a normative principle of biological visual representations. In the monkey ventral visual hierarchy, we found that factorization of object pose and background information from object identity increased in higher-level regions and strongly contributed to improving object identity decoding performance. We then conducted a large-scale analysis of factorization of individual scene parameters – lighting, background, camera viewpoint, and object pose – in a diverse library of DNN models of the visual system. Models which best matched neural, fMRI, and behavioral data from both monkeys and humans across 12 datasets tended to be those which factorized scene parameters most strongly. Notably, invariance to these parameters was not as consistently associated with matches to neural and behavioral data, suggesting that maintaining non-class information in factorized activity subspaces is often preferred to dropping it altogether. Thus, we propose that factorization of visual scene information is a widely used strategy in brains and DNN models thereof.

View Accepted Manuscript (DOE)

Research Organization:: Krell Institute, Ames, IA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: SC0020347

OSTI ID:: 2471917

Journal Information:: eLife, Journal Name: eLife Vol. 13; ISSN 2050-084X

Publisher:: eLife Sciences Publications, Ltd.Copyright Statement

Country of Publication:: United States

Language:: English

References (38)

The Code for Facial Identity in the Primate Brain Chang, Le; Tsao, Doris Y. Cell, Vol. 169, Issue 6 https://doi.org/10.1016/j.cell.2017.05.011	journal	June 2017
The Geometry of Abstraction in the Hippocampus and Prefrontal Cortex Bernardi, Silvia; Benna, Marcus K.; Rigotti, Mattia Cell, Vol. 183, Issue 4 https://doi.org/10.1016/j.cell.2020.09.031	journal	November 2020
Brain hierarchy score: Which deep neural networks are hierarchically brain-like? Nonaka, Soma; Majima, Kei; Aoki, Shuntaro C. iScience, Vol. 24, Issue 9 https://doi.org/10.1016/j.isci.2021.103013	journal	September 2021
A Channel for 3D Environmental Shape in Anterior Inferotemporal Cortex Vaziri, Siavash; Carlson, Eric T.; Wang, Zhihong Neuron, Vol. 84, Issue 1 https://doi.org/10.1016/j.neuron.2014.08.043	journal	October 2014
Optimal Degrees of Synaptic Connectivity Litwin-Kumar, Ashok; Harris, Kameron Decker; Axel, Richard Neuron, Vol. 93, Issue 5 https://doi.org/10.1016/j.neuron.2017.01.030	journal	March 2017
Tuned geometries of hippocampal representations meet the computational demands of social memory Boyle, Lara M.; Posani, Lorenzo; Irfan, Sarah Neuron, Vol. 112, Issue 8 https://doi.org/10.1016/j.neuron.2024.01.021	journal	April 2024
Untangling invariant object recognition DiCarlo, James J.; Cox, David D. Trends in Cognitive Sciences, Vol. 11, Issue 8 https://doi.org/10.1016/j.tics.2007.06.010	journal	August 2007
The ventral visual pathway: an expanded neural framework for the processing of object quality Kravitz, Dwight J.; Saleem, Kadharbatcha S.; Baker, Chris I. Trends in Cognitive Sciences, Vol. 17, Issue 1 https://doi.org/10.1016/j.tics.2012.10.011	journal	January 2013
Representational geometry: integrating cognition, computation, and the brain Kriegeskorte, Nikolaus; Kievit, Rogier A. Trends in Cognitive Sciences, Vol. 17, Issue 8 https://doi.org/10.1016/j.tics.2013.06.007	journal	August 2013
Norm-based face encoding by single neurons in the monkey inferotemporal cortex Leopold, David A.; Bondar, Igor V.; Giese, Martin A. Nature, Vol. 442, Issue 7102 https://doi.org/10.1038/nature04951	journal	July 2006
Identifying natural images from human brain activity Kay, Kendrick N.; Naselaris, Thomas; Prenger, Ryan J. Nature, Vol. 452, Issue 7185 https://doi.org/10.1038/nature06713	journal	March 2008
The importance of mixed selectivity in complex cognitive tasks Rigotti, Mattia; Barak, Omri; Warden, Melissa R. Nature, Vol. 497, Issue 7451 https://doi.org/10.1038/nature12160	journal	May 2013
A face feature space in the macaque temporal lobe Freiwald, Winrich A.; Tsao, Doris Y.; Livingstone, Margaret S. Nature Neuroscience, Vol. 12, Issue 9 https://doi.org/10.1038/nn.2363	journal	August 2009
Parallel, multi-stage processing of colors, faces and shapes in macaque inferior temporal cortex Lafer-Sousa, Rosa; Conway, Bevil R. Nature Neuroscience, Vol. 16, Issue 12 https://doi.org/10.1038/nn.3555	journal	October 2013
Explicit information for category-orthogonal object properties increases along the ventral stream Hong, Ha; Yamins, Daniel L. K.; Majaj, Najib J. Nature Neuroscience, Vol. 19, Issue 4 https://doi.org/10.1038/nn.4247	journal	February 2016
A new neural framework for visuospatial processing Kravitz, Dwight J.; Saleem, Kadharbatcha S.; Baker, Chris I. Nature Reviews Neuroscience, Vol. 12, Issue 4 https://doi.org/10.1038/nrn3008	journal	March 2011
Primary visual cortex straightens natural video trajectories Hénaff, Olivier J.; Bai, Yoon; Charlton, Julie A. Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-021-25939-z	journal	October 2021
Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons Higgins, Irina; Chang, Le; Langston, Victoria Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-021-26751-5	journal	November 2021
Abstract representations emerge naturally in neural networks trained to perform multiple tasks Johnston, W. Jeffrey; Fusi, Stefano Nature Communications, Vol. 14, Issue 1 https://doi.org/10.1038/s41467-023-36583-0	journal	February 2023
Capturing the objects of vision with neural networks Peters, Benjamin; Kriegeskorte, Nikolaus Nature Human Behaviour, Vol. 5, Issue 9 https://doi.org/10.1038/s41562-021-01194-6	journal	September 2021
Perceptual straightening of natural videos Hénaff, Olivier J.; Goris, Robbe L. T.; Simoncelli, Eero P. Nature Neuroscience, Vol. 22, Issue 6 https://doi.org/10.1038/s41593-019-0377-4	journal	April 2019
Performance-optimized hierarchical models predict neural responses in higher visual cortex Yamins, D. L. K.; Hong, H.; Cadieu, C. F. Proceedings of the National Academy of Sciences, Vol. 111, Issue 23 https://doi.org/10.1073/pnas.1403112111	journal	May 2014
Neural representational geometry underlies few-shot concept learning Sorscher, Ben; Ganguli, Surya; Sompolinsky, Haim Proceedings of the National Academy of Sciences, Vol. 119, Issue 43 https://doi.org/10.1073/pnas.2200800119	journal	October 2022
Classification and Geometry of General Perceptual Manifolds Chung, SueYeon; Lee, Daniel D.; Sompolinsky, Haim Physical Review X, Vol. 8, Issue 3 https://doi.org/10.1103/PhysRevX.8.031003	journal	July 2018
Discovering important people and objects for egocentric video summarization No authors listed 2012 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2012.6247820	conference	June 2012
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Momentum Contrast for Unsupervised Visual Representation Learning He, Kaiming; Fan, Haoqi; Wu, Yuxin 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR42600.2020.00975	conference	June 2020
Unsupervised Visual Representation Learning by Context Prediction Doersch, Carl; Gupta, Abhinav; Efros, Alexei A. 2015 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2015.167	conference	December 2015
Mask R-CNN He, Kaiming; Gkioxari, Georgia; Dollar, Piotr 2017 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2017.322	conference	October 2017
A Cortical Region Consisting Entirely of Face-Selective Cells Tsao, D. Y. Science, Vol. 311, Issue 5761 https://doi.org/10.1126/science.1119983	journal	February 2006
Functional Compartmentalization and Viewpoint Generalization Within the Macaque Face-Processing System Freiwald, W. A.; Tsao, D. Y. Science, Vol. 330, Issue 6005 https://doi.org/10.1126/science.1194908	journal	November 2010
What Is the Goal of Sensory Coding? Field, David J. Neural Computation, Vol. 6, Issue 4 https://doi.org/10.1162/neco.1994.6.4.559	journal	July 1994
Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition Cadieu, Charles F.; Hong, Ha; Yamins, Daniel L. K. PLoS Computational Biology, Vol. 10, Issue 12 https://doi.org/10.1371/journal.pcbi.1003963	journal	December 2014
Deep image reconstruction from human brain activity Shen, Guohua; Horikawa, Tomoyasu; Majima, Kei PLOS Computational Biology, Vol. 15, Issue 1 https://doi.org/10.1371/journal.pcbi.1006633	journal	January 2019
Selectivity and Tolerance ("Invariance") Both Increase as Visual Information Propagates from Cortical Area V4 to IT Rust, N. C.; DiCarlo, J. J. Journal of Neuroscience, Vol. 30, Issue 39 https://doi.org/10.1523/JNEUROSCI.0179-10.2010	journal	September 2010
Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks Rajalingham, Rishi; Issa, Elias B.; Bashivan, Pouya The Journal of Neuroscience, Vol. 38, Issue 33 https://doi.org/10.1523/JNEUROSCI.0388-18.2018	journal	July 2018
Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance Majaj, N. J.; Hong, H.; Solomon, E. A. Journal of Neuroscience, Vol. 35, Issue 39 https://doi.org/10.1523/JNEUROSCI.5181-14.2015	journal	September 2015
Balanced Increases in Selectivity and Tolerance Produce Constant Sparseness along the Ventral Visual Stream Rust, N. C.; DiCarlo, J. J. Journal of Neuroscience, Vol. 32, Issue 30 https://doi.org/10.1523/JNEUROSCI.6125-11.2012	journal	July 2012

Similar Records

Uncertainty-Informed Volume Visualization using Implicit Neural Representation

Conference · Tue Oct 01 00:00:00 EDT 2024 · OSTI ID:2538110

Visualization for Classification in Deep Neural Networks

Conference · Sun Oct 01 00:00:00 EDT 2017 · OSTI ID:1407764

Chromatic information and feature detection in fast visual analysis

Journal Article · Mon Aug 01 00:00:00 EDT 2016 · PLoS ONE · OSTI ID:1333683

Related Subjects

59 BASIC BIOLOGICAL SCIENCES
Life Sciences & Biomedicine - Other Topics

Factorized visual representations in the primate visual system and deep neural networks

Citation Formats

References (38)

Similar Records

Related Subjects