- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the extent to which latent representations from vision models can be decoded from neural activity in the human medial temporal lobe (MTL) during naturalistic movie viewing. Using the existing SUMMER dataset, the authors train decoding models to predict frame-level features from pretrained vision models using neural activity. The authors also propose reducing dimensions to 128 as an appropriate preprocessing. The principal findings are that learned visual representations can be decoded from MTL activity, decoding performance continues to improve with the number of recorded units, and the parahippocampal cortex (PHC) outperforms control populations. The paper also reports higher decodability for large-scale pretrained models.
Critical strengths
- The authors leverage a recently published large human MTL spiking dataset to predict continuous features from vision models. Establishing the extent to which these representations can be decoded from human MTL spiking activity under naturalistic stimulation has empirical value.
- The authors compare the PHC population against equally sized controls. The PHC advantage persists across comparisons and is consistent with the unit-level encoding analysis. The paper additionally reports performance as the population size increases, showing no saturation within the observed range.
Critical weaknesses
- It remains unclear what has been newly learned about the MTL by decoding representations from multiple DNNs and comparing their performance:
Encoding and decoding with DNN representations, as well as comparisons across model architectures and training conditions, are already well-established across several domains. In vision, examples include hierarchical visual-feature decoding by Horikawa and Kamitani (2017, Nature Communications) and large-scale model comparisons by Conwell et al. (2024, Nature Communications). In audition, Kell et al. (2018, Neuron) and Tuckute et al. (2023, PLOS Biology) examined neural-response prediction using DNNs and its relationship to model hierarchy and training tasks. In language, Schrimpf et al. (2021, PNAS) and Caucheteux and King (2022, Communications Biology) used internal model representations to predict brain responses and compared models or training conditions. These studies do not establish the present results, but they demonstrate that the general framework of using model representations to decode or explain neural activity is not itself novel.
More directly, Choksi et al. (2022, Neural Networks) used representational similarity analysis to compare hippocampal fMRI patterns with representations from unimodal and multimodal models, including CLIP. Gerken et al. (2025) decoded characters, indoor/outdoor settings, and visual transitions from the same movie-viewing MTL recordings used here. It is therefore important to explain what additional insight is gained by decoding continuous model features.
In particular, the paper motivates continuous representations by arguing that predefined labels capture only selected aspects of visual information. However, using a continuous feature vector is not equivalent to demonstrating neural information beyond what existing character, location, or scene labels capture. It remains unclear whether the decoded features capture additional information or simply express familiar category-level information in another format.
The PHC advantage is a useful quantitative result, but the authors themselves interpret it as consistent with PHC’s established involvement in visual and scene processing.
I recognize the empirical value of this descriptive comparison, but do not find the additional neuroscientific insight sufficiently established to support a strong scientific contribution.
- The paper aims to characterize MTL representations while presenting control of feature-space geometry through PCA whitening as a central methodological point. However, the relationship between these objectives is insufficiently established:
Three questions need to be distinguished: which representation supports better frame retrieval from neural activity; how accurately individual model features can be predicted from neural activity; and which model better explains MTL representations. These questions are related, but answering one does not automatically answer the others.
I agree with the authors that identification performance depends both on feature-prediction accuracy and on how the feature space arranges candidate frames. However, this raises a more basic question about using identification accuracy as the primary measure of feature recoverability. If frame retrieval is the objective, identification is relevant, and feature-space geometry is part of the retrieval system’s performance. If the objective is to determine how accurately model features can be recovered from MTL activity, direct held-out feature-prediction measures would address that question more directly. Whitening may reduce certain differences in retrieval difficulty, but this does not by itself establish that the resulting scores quantify feature recoverability or provide a more appropriate ranking of correspondence with the MTL.
My concern is therefore not that whitening is intrinsically inappropriate, nor simply that PCA discards information. Rather, the authors need to clarify what scientific property the comparison is intended to measure, why identification accuracy is an appropriate indicator of that property, and how whitening improves its measurement. Without this connection, it is difficult to assess whitening as a methodological contribution to understanding MTL representations, rather than a normalization choice for a frame-retrieval benchmark.
Questions for authors
-
What specific finding advances our understanding beyond previously demonstrated character, setting, and transition decoding?
-
What scientific question is the whitened identification score intended to answer?
7 comments
Rebuttal: Reviewer MQvZ (2/2)
Takeaway: Continuous feature decoding shows that human MTL spiking carries within-scene visual information beyond predefined character, setting, and transition labels or scene identity alone, while uncovering a between- versus within-scene dissociation in PHC that binary labels could not resolve.
Weakness 2 & Question 2: Scientific question answered by whitened identification, and relation to direct feature prediction and neural encoding
We appreciate the reviewer's clear separation of the three questions: (i) frame-retrieval system performance, (ii) held-out feature-prediction accuracy, and (iii) correspondence with MTL representations. We fully agree that answering one does not automatically answer the others, and that if native frame retrieval were the engineering goal, native target geometry would simply be part of the retrieval system.
Here, our goal is neither to benchmark native retrieval pipelines nor to claim that a vision model implements the MTL's internal computations (as noted in Discussion §5, linear decodability cannot establish shared mechanisms). Rather, as we now state explicitly in Methods §3.6 and Discussion §5, whitened identification measures how much frame-discriminative scene information encoded in each model's principal representational subspace is linearly readable from human MTL spiking.
We pair PCA whitening with identification rather than relying on unwhitened metrics for two reasons: First, unwhitened direct regression metrics (, cosine similarity) are poorly suited for cross-model comparison here, because heavy ridge shrinkage on noisy single-trial spikes drives unscaled near zero while unwhitened cosine similarity and rescaled weight dimensions by native variance—favoring anisotropic spaces dominated by a few slow components (Discussion §5, Appendix Table 2, Appendix A.2). Second, unwhitened identification is scale-invariant, but native crowding causes comparable prediction errors to yield inconsistent rankings across spaces (e.g., DINOv2 exceeding Places365 on raw identification while reversing on calibrated signal fraction ; Spearman ρ = 0.92; Results §4.2). Equalizing training variance across k=128 dimensions places all models on a common isotropic subspace where identification becomes a consistent function of directional recovery and aligns closely with (Fig. 1d–e), while dedicated circular-shift nulls account for remaining temporal smoothness (Appendix Table 4).
Crucially, to test directly whether our model hierarchy depends on using a retrieval score rather than direct feature prediction or forward neural encoding, we compared all metrics across the 13 models at k=128 (Results §4.2, Discussion §5, Appendix A.2, A.4):
- Agreement with direct held-out feature prediction (Results §4.2, Discussion §5, Appendix A.2, Appendix Table 4): Across all 13 spaces, nulled k=128 identification produces virtually the same model ordering as direct held-out mean cosine similarity and scaled R^2 (both reported per model in Appendix Table 4).
- Agreement with retrieval-free forward neural encoding (Results §4.2, Discussion §5, Appendix A.2, Appendix A.4, Fig. 9c) Reversing the mapping—predicting each individual neuron's spike counts from the k=128 features, which involves no candidate pool or frame retrieval at all—likewise yields the same hierarchy across models (ρ = 0.97) , and rankings remain stable across whitening dimensionalities k64,128,256,512 (Appendix Fig. 6).
Takeaway: Whitening is therefore not an arbitrary retrieval normalization, but a control that isolates linear mutual predictability between MTL spiking and each model's principal representational subspace—yielding a model hierarchy that converges across frame identification, direct feature recovery, and per-neuron spike encoding (ρ = 0.97).
Summary. We thank the reviewer again for their thoughtful and constructive critique, which prompted the new annotation- and scene-conditioned analyses as well as a clearer formulation and triangulation of our geometry-controlled evaluation. Together, we believe these additions substantially strengthen both the neuroscientific contribution and the clarity of the manuscript. We hope that our new results and revisions address the reviewer's concerns and invite them to reconsider their assessment; we would be glad to provide any further clarification during the discussion period.
Thank you for your detailed responses, as well as the substantial additional analyses. I appreciate the authors' efforts to address my concerns, and I agree that the new controls and metric comparisons strengthen the empirical results.
1. Neuroscientific contribution
The new analyses provide useful evidence that continuous feature decoding captures information beyond previously studied categorical labels and scene identity. I appreciate this clarification.
However, I remain somewhat cautious about the magnitude of the scientific advance. While continuous representations allow finer-grained information to be detected than predefined labels, demonstrating such additional decodable information does not necessarily establish a qualitatively new understanding of MTL representations. In particular, the findings regarding PHC seem broadly consistent with its previously established involvement in visual and scene processing. The specific nature of the additional information also remains unclear. I therefore see this as a valuable empirical extension of prior findings, but one whose incremental neuroscientific contribution remains relatively modest, particularly given the use of existing data and established decoding approaches.
2. Scientific motivation for whitening
I appreciate the clarification that whitening is intended to equalize feature-space geometry and the additional comparisons with direct feature prediction and forward neural encoding. The consistency across these metrics is reassuring.
However, I am still not fully convinced that this establishes the scientific motivation for whitening as a methodological contribution. In particular, while equalizing feature variance provides a reasonable normalization for cross-model comparisons, it is not yet clear why this particular normalization should provide a more scientifically meaningful measure of correspondence with MTL representations.
The agreement across different metrics within the whitened feature spaces supports the robustness of the resulting rankings, but does not necessarily establish why the whitened representation should be preferred over the native representation for the underlying neuroscientific question.
Could the authors further clarify this distinction, particularly what scientific insight the whitened identification measure provides that cannot be obtained from more direct feature-prediction or neural-encoding analyses?
Rebuttal: Reviewer MQvZ (1/2)
We thank the reviewer for their thoughtful and rigorous assessment of both our neuroscientific contribution and the conceptual role of our evaluation metric. Their comments directly motivated two major additions to the revised manuscript that resolve both concerns empirically: (1) adding targeted annotation- and scene-controlled analyses across the full MTL and PHC—demonstrating that continuous scene decoding and the 13-model hierarchy persist beyond all seven previously decoded binary labels and within exact movie scenes, while uncovering a regional dissociation in PHC between coarse scene context and finer within-scene visual discrimination that binary labels could not resolve; and (2) explicitly defining whitened identification as a measure of linearly readable scene-discriminative information in a variance-equalized subspace, and showing that it converges on virtually the same cross-model hierarchy as direct held-out feature prediction and retrieval-free forward per-neuron encoding (Spearman ρ=0.97 for all comparisons). Below, we address both points in detail and identify the corresponding manuscript revisions.
We agree with the reviewer that comparing DNN representations to brain activity is well established across vision, audition, and language (including Horikawa & Kamitani, 2017; Kell et al., 2018; Schrimpf et al., 2021; Caucheteux & King, 2022; Choksi et al., 2022; Tuckute et al., 2023; Conwell et al., 2024; added in Related Work §2). However, in humans these studies rely almost entirely on non-invasive recordings (or ECoG) that average activity across tens of thousands of neurons. In contrast, single-neuron recordings in the human MTL probe a distinct coding regime: classic single-unit studies characterized sparse, invariant "concept cells" using isolated static images (Quiroga et al., 2005), whereas during continuous naturalistic movie viewing Gerken et al. (2025) decoded only discrete, predefined binary labels (characters, indoor/outdoor setting, and visual transitions).
Motivated by the reviewer's feedback, we added new control analyses in this revision that hold those binary labels and exact movie-scene identity fixed (Methods §3.7, new Results §4.3 and §4.5, Figs. 2–4, Appendix A.3–A.4). Pairing continuous vision-model embeddings with these new controls lets us test directly whether human MTL spiking can differentiate within-scene visual structure, how between-scene context and finer within-scene content separate anatomically in PHC, and which model training factors govern single-unit alignment.