acceptodds
Under review as a conference paper at ICLR 2027

Visual Representations in the Human Medial Temporal Lobe During Naturalistic Movie Viewing

Abstract

Neurons in the human medial temporal lobe (MTL) respond to specific concepts, such as a particular person, across very different images. So far, this selectivity has mostly been measured with isolated images and predefined labels. Representations from vision models provide continuous descriptions of complete visual scenes, enabling MTL activity to be studied without such labels. Using the SUMMER dataset, comprising 2,286 units recorded from 29 intracranially implanted participants while they viewed a full-length commercial movie, we decode frame-wise representations of a wide range of vision models from MTL population activity. We show that MTL activity identifies held-out frames above a temporally matched baseline and locates them coarsely in the film, with performance still increasing at the full population size, whereas low-level image statistics cannot be reliably decoded. Importantly, size and diversity of training data affect decodability. For example, for the same ResNet-50 architecture, decoding performance markedly improves when trained with language supervision on larger datasets. Across all visual feature spaces, a parahippocampal cortex population outperforms size- and firing-rate-matched control populations from the remaining MTL, as well as control populations with twice as many neurons. Our results show that human MTL populations, in particular in parahippocampal cortex, carry information about the visual content of movies that can be used to compare representations from vision models.

Primary area:
applications to neuroscience & cognitive science
Keywords:
human single-neuron recordings, medial temporal lobe, naturalistic movie viewing, neural decoding, vision foundation models, visual movie representations, representational alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MQvZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the extent to which latent representations from vision models can be decoded from neural activity in the human medial temporal lobe (MTL) during naturalistic movie viewing. Using the existing SUMMER dataset, the authors train decoding models to predict frame-level features from pretrained vision models using neural activity. The authors also propose reducing dimensions to 128 as an appropriate preprocessing. The principal findings are that learned visual representations can be decoded from MTL activity, decoding performance continues to improve with the number of recorded units, and the parahippocampal cortex (PHC) outperforms control populations. The paper also reports higher decodability for large-scale pretrained models.

Critical strengths
  • The authors leverage a recently published large human MTL spiking dataset to predict continuous features from vision models. Establishing the extent to which these representations can be decoded from human MTL spiking activity under naturalistic stimulation has empirical value.
  • The authors compare the PHC population against equally sized controls. The PHC advantage persists across comparisons and is consistent with the unit-level encoding analysis. The paper additionally reports performance as the population size increases, showing no saturation within the observed range.
Critical weaknesses
  1. It remains unclear what has been newly learned about the MTL by decoding representations from multiple DNNs and comparing their performance:

Encoding and decoding with DNN representations, as well as comparisons across model architectures and training conditions, are already well-established across several domains. In vision, examples include hierarchical visual-feature decoding by Horikawa and Kamitani (2017, Nature Communications) and large-scale model comparisons by Conwell et al. (2024, Nature Communications). In audition, Kell et al. (2018, Neuron) and Tuckute et al. (2023, PLOS Biology) examined neural-response prediction using DNNs and its relationship to model hierarchy and training tasks. In language, Schrimpf et al. (2021, PNAS) and Caucheteux and King (2022, Communications Biology) used internal model representations to predict brain responses and compared models or training conditions. These studies do not establish the present results, but they demonstrate that the general framework of using model representations to decode or explain neural activity is not itself novel.

More directly, Choksi et al. (2022, Neural Networks) used representational similarity analysis to compare hippocampal fMRI patterns with representations from unimodal and multimodal models, including CLIP. Gerken et al. (2025) decoded characters, indoor/outdoor settings, and visual transitions from the same movie-viewing MTL recordings used here. It is therefore important to explain what additional insight is gained by decoding continuous model features.

In particular, the paper motivates continuous representations by arguing that predefined labels capture only selected aspects of visual information. However, using a continuous feature vector is not equivalent to demonstrating neural information beyond what existing character, location, or scene labels capture. It remains unclear whether the decoded features capture additional information or simply express familiar category-level information in another format.

The PHC advantage is a useful quantitative result, but the authors themselves interpret it as consistent with PHC’s established involvement in visual and scene processing.

I recognize the empirical value of this descriptive comparison, but do not find the additional neuroscientific insight sufficiently established to support a strong scientific contribution.

  1. The paper aims to characterize MTL representations while presenting control of feature-space geometry through PCA whitening as a central methodological point. However, the relationship between these objectives is insufficiently established:

Three questions need to be distinguished: which representation supports better frame retrieval from neural activity; how accurately individual model features can be predicted from neural activity; and which model better explains MTL representations. These questions are related, but answering one does not automatically answer the others.

I agree with the authors that identification performance depends both on feature-prediction accuracy and on how the feature space arranges candidate frames. However, this raises a more basic question about using identification accuracy as the primary measure of feature recoverability. If frame retrieval is the objective, identification is relevant, and feature-space geometry is part of the retrieval system’s performance. If the objective is to determine how accurately model features can be recovered from MTL activity, direct held-out feature-prediction measures would address that question more directly. Whitening may reduce certain differences in retrieval difficulty, but this does not by itself establish that the resulting scores quantify feature recoverability or provide a more appropriate ranking of correspondence with the MTL.

My concern is therefore not that whitening is intrinsically inappropriate, nor simply that PCA discards information. Rather, the authors need to clarify what scientific property the comparison is intended to measure, why identification accuracy is an appropriate indicator of that property, and how whitening improves its measurement. Without this connection, it is difficult to assess whitening as a methodological contribution to understanding MTL representations, rather than a normalization choice for a frame-retrieval benchmark.

Questions for authors
  • What specific finding advances our understanding beyond previously demonstrated character, setting, and transition decoding?

  • What scientific question is the whitened identification score intended to answer?

7 comments
AuthorsOpenReview ↗

Rebuttal: Reviewer MQvZ (1/2)

We thank the reviewer for their thoughtful and rigorous assessment of both our neuroscientific contribution and the conceptual role of our evaluation metric. Their comments directly motivated two major additions to the revised manuscript that resolve both concerns empirically: (1) adding targeted annotation- and scene-controlled analyses across the full MTL and PHC—demonstrating that continuous scene decoding and the 13-model hierarchy persist beyond all seven previously decoded binary labels and within exact movie scenes, while uncovering a regional dissociation in PHC between coarse scene context and finer within-scene visual discrimination that binary labels could not resolve; and (2) explicitly defining whitened identification as a measure of linearly readable scene-discriminative information in a variance-equalized subspace, and showing that it converges on virtually the same cross-model hierarchy as direct held-out feature prediction and retrieval-free forward per-neuron encoding (Spearman ρ=0.97\rho = 0.97 for all comparisons). Below, we address both points in detail and identify the corresponding manuscript revisions.

  • Weakness 1 & Question 1: Neuroscientific advance beyond prior DNN-brain studies and predefined label decoding (Gerken et al., 2025)

We agree with the reviewer that comparing DNN representations to brain activity is well established across vision, audition, and language (including Horikawa & Kamitani, 2017; Kell et al., 2018; Schrimpf et al., 2021; Caucheteux & King, 2022; Choksi et al., 2022; Tuckute et al., 2023; Conwell et al., 2024; added in Related Work §2). However, in humans these studies rely almost entirely on non-invasive recordings (or ECoG) that average activity across tens of thousands of neurons. In contrast, single-neuron recordings in the human MTL probe a distinct coding regime: classic single-unit studies characterized sparse, invariant "concept cells" using isolated static images (Quiroga et al., 2005), whereas during continuous naturalistic movie viewing Gerken et al. (2025) decoded only discrete, predefined binary labels (characters, indoor/outdoor setting, and visual transitions).

Motivated by the reviewer's feedback, we added new control analyses in this revision that hold those binary labels and exact movie-scene identity fixed (Methods §3.7, new Results §4.3 and §4.5, Figs. 2–4, Appendix A.3–A.4). Pairing continuous vision-model embeddings with these new controls lets us test directly whether human MTL spiking can differentiate within-scene visual structure, how between-scene context and finer within-scene content separate anatomically in PHC, and which model training factors govern single-unit alignment.

  • Visual scene information beyond all seven prior labels and scene identity (Results §4.3, Fig. 2, Appendix A.3): Restricting candidate frames (without refitting) to match the exact joint state across all seven binary labels from Gerken et al. (2025)—Summer, Tom, McKenzie, Persons, Indoor/Outdoor, Scene Cuts, and Camera Cuts—reduced CLIP decoding only modestly relative to a label-preserving shift null (Fig. 2b), as did linearly removing those seven labels on training data and refitting whitening and the decoder on the residuals (Fig. 2c). Restricting candidates to the query's exact movie-scene segment likewise remained clearly above a scene-preserving null across all five buffered folds (Fig. 2d, Appendix Fig. 7).
  • A regional dissociation in PHC between scene context and within-scene discrimination (Results §4.5, Fig. 4c, Appendix A.4, Table 5, Figs. 10–11): Rather than merely confirming that PHC is scene-responsive, continuous features separate coarse scene context from finer within-scene coding. PHC's advantage over size- and firing-rate-matched A+H+EC populations remained intact when controlling for the seven binary annotations (Appendix Fig. 11b) and across all single- and multi-patient omissions (Appendix Fig. 11c,d). Restricting evaluation to the exact same movie scene substantially reduced—yet consistently preserved across all five folds—this PHC margin (Fig. 4c). Together with equal PHC performance for scene- versus object-supervised ResNet-50, this shows that PHC combines between-scene context with finer within-scene visual discrimination rather than acting solely as a discrete scene-category or boundary detector.
  • Factors governing single-unit MTL alignment across models (Results §4.4, Fig. 3, Appendix Fig. 8): At single-neuron resolution under strict temporal and geometric controls, decodability is shaped by training regime and dataset composition rather than convolutional versus transformer architecture, and the broad 13-model hierarchy persists both when the seven annotations are fixed and within exact movie scenes (Appendix A.3, Fig. 8).
AuthorsOpenReview ↗

Rebuttal: Reviewer MQvZ (2/2)

Takeaway: Continuous feature decoding shows that human MTL spiking carries within-scene visual information beyond predefined character, setting, and transition labels or scene identity alone, while uncovering a between- versus within-scene dissociation in PHC that binary labels could not resolve.

Weakness 2 & Question 2: Scientific question answered by whitened identification, and relation to direct feature prediction and neural encoding

We appreciate the reviewer's clear separation of the three questions: (i) frame-retrieval system performance, (ii) held-out feature-prediction accuracy, and (iii) correspondence with MTL representations. We fully agree that answering one does not automatically answer the others, and that if native frame retrieval were the engineering goal, native target geometry would simply be part of the retrieval system.

Here, our goal is neither to benchmark native retrieval pipelines nor to claim that a vision model implements the MTL's internal computations (as noted in Discussion §5, linear decodability cannot establish shared mechanisms). Rather, as we now state explicitly in Methods §3.6 and Discussion §5, whitened identification measures how much frame-discriminative scene information encoded in each model's principal representational subspace is linearly readable from human MTL spiking.

We pair PCA whitening with identification rather than relying on unwhitened metrics for two reasons: First, unwhitened direct regression metrics (R2R^2, cosine similarity) are poorly suited for cross-model comparison here, because heavy ridge shrinkage on noisy single-trial spikes drives unscaled R2R^2 near zero while unwhitened cosine similarity and rescaled R2R^2 weight dimensions by native variance—favoring anisotropic spaces dominated by a few slow components (Discussion §5, Appendix Table 2, Appendix A.2). Second, unwhitened identification is scale-invariant, but native crowding causes comparable prediction errors to yield inconsistent rankings across spaces (e.g., DINOv2 exceeding Places365 on raw identification while reversing on calibrated signal fraction ; Spearman ρ = 0.92; Results §4.2). Equalizing training variance across k=128 dimensions places all models on a common isotropic subspace where identification becomes a consistent function of directional recovery and aligns closely with (Fig. 1d–e), while dedicated circular-shift nulls account for remaining temporal smoothness (Appendix Table 4).

Crucially, to test directly whether our model hierarchy depends on using a retrieval score rather than direct feature prediction or forward neural encoding, we compared all metrics across the 13 models at k=128 (Results §4.2, Discussion §5, Appendix A.2, A.4):

  • Agreement with direct held-out feature prediction (Results §4.2, Discussion §5, Appendix A.2, Appendix Table 4): Across all 13 spaces, nulled k=128 identification produces virtually the same model ordering as direct held-out mean cosine similarity and scaled R^2 (both reported per model in Appendix Table 4).
  • Agreement with retrieval-free forward neural encoding (Results §4.2, Discussion §5, Appendix A.2, Appendix A.4, Fig. 9c) Reversing the mapping—predicting each individual neuron's spike counts from the k=128 features, which involves no candidate pool or frame retrieval at all—likewise yields the same hierarchy across models (ρ = 0.97) , and rankings remain stable across whitening dimensionalities k64,128,256,512 (Appendix Fig. 6).

Takeaway: Whitening is therefore not an arbitrary retrieval normalization, but a control that isolates linear mutual predictability between MTL spiking and each model's principal representational subspace—yielding a model hierarchy that converges across frame identification, direct feature recovery, and per-neuron spike encoding (ρ = 0.97).

Summary. We thank the reviewer again for their thoughtful and constructive critique, which prompted the new annotation- and scene-conditioned analyses as well as a clearer formulation and triangulation of our geometry-controlled evaluation. Together, we believe these additions substantially strengthen both the neuroscientific contribution and the clarity of the manuscript. We hope that our new results and revisions address the reviewer's concerns and invite them to reconsider their assessment; we would be glad to provide any further clarification during the discussion period.

Thank you for your detailed responses, as well as the substantial additional analyses. I appreciate the authors' efforts to address my concerns, and I agree that the new controls and metric comparisons strengthen the empirical results.

1. Neuroscientific contribution

The new analyses provide useful evidence that continuous feature decoding captures information beyond previously studied categorical labels and scene identity. I appreciate this clarification.

However, I remain somewhat cautious about the magnitude of the scientific advance. While continuous representations allow finer-grained information to be detected than predefined labels, demonstrating such additional decodable information does not necessarily establish a qualitatively new understanding of MTL representations. In particular, the findings regarding PHC seem broadly consistent with its previously established involvement in visual and scene processing. The specific nature of the additional information also remains unclear. I therefore see this as a valuable empirical extension of prior findings, but one whose incremental neuroscientific contribution remains relatively modest, particularly given the use of existing data and established decoding approaches.

2. Scientific motivation for whitening

I appreciate the clarification that whitening is intended to equalize feature-space geometry and the additional comparisons with direct feature prediction and forward neural encoding. The consistency across these metrics is reassuring.

However, I am still not fully convinced that this establishes the scientific motivation for whitening as a methodological contribution. In particular, while equalizing feature variance provides a reasonable normalization for cross-model comparisons, it is not yet clear why this particular normalization should provide a more scientifically meaningful measure of correspondence with MTL representations.

The agreement across different metrics within the whitened feature spaces supports the robustness of the resulting rankings, but does not necessarily establish why the whitened representation should be preferred over the native representation for the underlying neuroscientific question.

Could the authors further clarify this distinction, particularly what scientific insight the whitened identification measure provides that cannot be obtained from more direct feature-prediction or neural-encoding analyses?

Reviewer_BmiiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to use the single-neuron population activity of the human medial temporal lobe (MTL) to decode the various visual model representations (CLIP, DINO, SigLIP, ResNet, etc., 13 types, plus handcrafted low-level features) for each frame when watching a full movie. The authors aim to answer three questions: (1) What visual information is present in the MTL; (2) Where in the MTL is this information located; (3) Which model attributes (architecture, training data, objective function) determine the decodability. After conducting the experiments, the article has reached some conclusions.

Critical strengths
  1. The evaluation design is more rigorous than most model–brain comparisons: buffered folds, circular-shift nulls, a least-similar-frame control, and calibrating identification into a signal fraction λ after whitening. The demonstration that raw identification rankings can invert under geometry control (DINOv2 vs. Places365) is a useful methodological point for the brain-alignment literature.

  2. The PHC advantage holds against size-matched, rate-matched, and 2× larger control populations, in all feature spaces, within firing-rate strata, and on least-similar frames. This is a convincing and novel characterization of visual information in human MTL during naturalistic viewing.

Critical weaknesses
  1. One main concern is that the decoded signal may reflect slowly varying scene or narrative state rather than visual content, and this confounds the model comparison. I'm not sure if my understanding is correct, because in the article, I read that the frame localization is at the scale of minutes. At this level of localization , it's hard to say exactly what kind of information it refers to.

    Model rankings could then reflect how well each embedding clusters frames by scene or character identity, not alignment with MTL visual coding. The paper needs a control that removes this component, e.g., ranking models by within-scene identification only.

  2. The training-data conclusions are overstated and partly contradicted by the paper's own results:

    (1) Sec. 4.3 states that decodability "exhibited a monotonic increase with the size of the training dataset", yet SigLIP2 (~10B) < DINOv3 (1.7B), ViT-21k (14M) ≈ CLIP (400M), and WIT-400M ≈ YFCC-15M. (2) The "structurally equivalent" ResNet-50 comparison is not architecture-matched (Table 1, footnote b: CLIP's RN50 has a different stem, anti-aliasing, and attention pooling), and it also changes the objective. (3) Among the three CLIP-RN50 models, which differ only in data, the gain occurs between CC-12M and YFCC-15M (similar size) and vanishes from 15M to 400M. This points to dataset composition rather than scale.

    The abstract's claim that decoding "markedly improves when trained with language supervision on larger datasets" should be weakened accordingly.

Questions for authors
  1. How do model rankings and the PHC advantage change after removing scene or character information (within-scene identification, or a 10–30 s candidate window as the primary metric)?

  2. All decoding is performed on a single pseudo-population that concatenates units from all 29 patients, with cross-validation over movie time only; no patient is ever held out. This leaves open whether the decoded visual information reflects a representation shared across individuals, or simply the union of idiosyncratic per-patient codes. Was this consideration taken into account when sampling the test set? Is it possible that the test group consists of a fixed number of participants? This might affect the conclusion.

  3. The main text reports CLIP at 0.785, while Table 4 reports 0.7643. Which space does each come from?

4 comments
AuthorsOpenReview ↗

Rebuttal: Reviewer Bmii (1/2)

We thank the reviewer for their careful reading and encouraging assessment of our work, particularly the methodological value of controlling feature-space geometry and the matched regional comparisons supporting the PHC advantage. Their comments directly motivated analyses testing whether model rankings and regional differences persist after controlling for character, setting, and scene state, alongside clearer explanations of our pseudo-population design, our cross-model training conclusions, and the transition between native and whitened CLIP scores. Below, we address each point and identify the corresponding manuscript revisions.

Weakness 1 & Question 1: Controlling for scene/character state in full-MTL decoding, model rankings, and the PHC advantage

We agree that minutes-scale unrestricted localization leaves open whether decoding—and cross-model or regional differences—might reflect slowly varying character, setting, or scene identity. To directly control for character, setting, and scene identity while preserving the 32 s separation buffer, we added three control analyses and included their results as a standalone figure and results section in the revised manuscript (Methods §3.7, new Results §4.3, Fig. 2, Appendix A.3):

  • Full-MTL decoding persists beyond annotations and scene identity (Results §4.3, Fig. 2): Restricting candidate frames (without refitting) to match the exact joint state of all seven previously decoded binary labels—Summer, Tom, McKenzie, Persons, Indoor/Outdoor, Scene Cuts, and Camera Cuts (Gerken et al., 2025)—reduced CLIP identification only modestly relative to a label-preserving shift null (Fig. 2b), as did linearly residualizing those seven labels out on training data and refitting whitening and the decoder (Fig. 2c). Restricting candidates to the query's exact movie-scene segment likewise remained clearly above a scene-preserving null in all five folds (Fig. 2d, Appendix Fig. 7).
  • The 13-model hierarchy remains broadly stable (Results §4.4, Appendix Fig. 8) Re-evaluating all 13 learned models (at k=128, without refitting) under either seven-annotation conditioning or exact within-scene restriction closely preserved the unrestricted above-null model ordering on the same scoreable queries, keeping the same four models at the top (RADIOv4, DINOv3, SigLIP2, CLIP) and the four ImageNet-1k/Places365 CNNs at the bottom, with only minor within-tier shifts inside scenes. Thus, the broad model hierarchy is not an artifact of scene or character clustering (see detailed discussion in Appendix A.3).
  • The PHC advantage dissociates between-scene context from within-scene coding (Results §4.5, Fig. 4c, Appendix Fig. 11b) PHC's advantage over size-matched and size-plus-firing-rate-matched A+H+EC controls stayed unchanged after conditioning on or residualizing out the seven binary annotations (Appendix Fig. 11b). Under exact within-scene evaluation, PHC still outperformed both matched controls in every fold, but its margin decreased substantially (Fig. 4c)—indicating that between-scene context accounts for a meaningful share, though not all, of the regional PHC advantage.

Regarding a 10–30 s primary candidate window: prior work on this dataset showed substantial neural and stimulus autocorrelation within 32 s (Gerken et al., 2025), which is why our splits and candidate pools enforce a 32 s separation buffer throughout. While our narrowest valid local window ([32 s, 60 s]; Fig. 1b) already exceeds chance in all five folds, exact annotation and within-scene conditioning isolate character and scene state much more directly without reintroducing short-lag autocorrelation.

Takeaway: Together, these results provide a substantive insight beyond the binary label decoding in Gerken et al. (2025): while scene-level context clearly contributes—most visibly to the magnitude of the PHC advantage—both the full MTL and PHC also carry finer-grained within-scene visual information captured by continuous model features, a distinction that binary character, setting, and transition labels alone could not resolve.

AuthorsOpenReview ↗

Rebuttal: Reviewer Bmii (2/2)

Weakness 2: Overstated training-data scale and ResNet-50 equivalence claims

We agree with all three points and thank the reviewer for catching these overstatements. We revised the manuscript throughout (Abstract, Introduction §1, Related Work §2, Methods 3.2, Results 4.4, and Discussion §5) accordingly: we weakened the Abstract's claim, removed statements of monotonic scaling with dataset size (explicitly noting counterexamples such as SigLIP2 vs. DINOv3 and ImageNet-21k ViT vs. larger pretraining corpora), and separated the broader supervised-versus-CLIP ResNet-50 comparison—explicitly noting differences in both training objective and stem, anti-aliased downsampling, and attention-pooling details (Appendix Table 1)—from the three strictly architecture-matched CLIP-RN50 checkpoints. Within that matched CLIP-RN50 series, we highlight that almost the entire gain occurs between the two smaller, similarly sized datasets before plateauing on the much larger corpus, pointing to dataset composition and training regime alongside scale.

Question 2: Pseudo-population design and cross-participant consistency

Because clinical electrode placement is governed by medical criteria, single neurons have no neuron-to-neuron correspondence across individuals: each recorded unit is a distinct predictor recorded while all 29 participants watched the same film. Consequently, a linear readout fitted on units from one subset of participants cannot be directly evaluated on a held-out test group of unseen participants without fitting a separate alignment mapping on the movie itself. Partitioning movie time (with 32 s buffers) therefore estimates what visual information is present across the sampled regional population—following standard human single-unit pseudo-population practice (Kamiński et al., 2017; Jamali et al., 2021; Gerken et al., 2025), which we now state explicitly alongside the noise-correlation caveat in Methods §3.1.

At the same time, two analyses confirm that decoding reflects stimulus-locked information shared across participants rather than a few idiosyncratic implants:

  • Patient-disjoint split-half agreement (Appendix A.2): Repeatedly partitioning the 29 participants into two non-overlapping patient groups and fitting separate CLIP decoders to each half yielded consistent held-out frame identification in both halves across all five folds, along with clear frame-by-frame cosine agreement between their predicted embeddings—confirming that the decoded signal reflects stimulus-locked visual information shared across individuals rather than a union of unrelated per-patient codes.
  • Patient-level distribution and omissions in PHC (Results §4.5, Appendix A.4, Table 5, Figs. 10 & 11c,d): PHC recordings span 24 of the 29 participants across 33 electrode bundles (Appendix Table 5, Fig. 10). Omitting each of the 29 patients one at a time (while rematching control unit counts and firing rates), or jointly omitting the top two or three PHC contributors, preserved the PHC advantage in both direction and magnitude across every cross-validation fold.

Question 3: Difference between CLIP identification of 0.785 (main text) and 0.7643 (Table 4)

We thank the reviewer for flagging this ambiguity. Both values score the same CLIP ViT-L/14 features on the same five buffered folds, before versus after the k=128 PCA-whitening step introduced for cross-model geometry control: 0.785 (Section 4.1) comes from the native, unwhitened 768-dimensional CLIP space used for the initial single-space analyses, whereas 0.7643 (Appendix Table 4) comes from the training-PCA-whitened k=128 space used whenever models of differing dimensionality and anisotropy are compared to each other. Because whitening weights lower-variance, faster-varying directions equally rather than letting high-variance slow components dominate cosine similarity, identification scores are slightly lower than in native space. We now explicitly label both spaces and flag this transition in Results 4.1–4.3.

Summary. We appreciate the reviewer's positive assessment of our evaluation framework and their incisive feedback, which helped us make the paper substantially more rigorous. By testing model rankings and PHC decoding within fixed scenes and annotation states, clarifying the pseudo-population setup, and aligning our training-regime conclusions strictly with the empirical evidence, the manuscript now presents a much sharper picture of visual representations in the human MTL. We hope the reviewer finds these revisions convincing and will continue to support the acceptance of our work.

Thank you for the detailed reply from the author. I think this reply has solved most of my concerns. I will keep the score and continue to support the acceptance of the article.

We thank the Reviewer Bmii very much for taking the time to read our revised manuscript and for providing such positive feedback. We are delighted that our revisions successfully addressed their concerns, and we appreciate their continued support for the acceptance of our work.

Reviewer_ebXYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which continuous visual representations can be read out from human medial temporal lobe (MTL) activity and how decodability varies across regions and vision models. It reuses SUMMER recordings from 2,286 single- and multi-units in 29 patients viewing the same 83-minute movie. A movie-aligned pseudo-population supplies 11 neural bins of 80 ms, centered from −400 to +400 ms relative to a target; ridge decoders predict continuous-valued frame embeddings from 13 frozen pretrained image models and a handcrafted low-level control. Decoders are fitted on training segments and selected on validation segments. Five-fold testing uses different segments of the same movie and the same recorded population, with 32-second partition buffers. The main comparison whitens learned features to 128 dimensions and measures the true target's percentile rank among candidates from the same test fold separated from the target by at least 32 seconds, with a median of approximately 6,700 eligible movie times. The reported results support held-out feature decoding, coarse temporal localization, and a parahippocampal cortex (PHC) advantage over matched MTL populations.

Critical strengths
  1. The authors predict several image models' features from MTL activity in movie segments excluded from training and check the effects of similar neighboring frames. Section 3.4 leaves 32-second gaps between training, validation, and test segments. Section 3.6 also shifts the temporal correspondence between neural activity and frames to test whether correct alignment gives better predictions. Section 4.2 illustrates a specific problem in comparing models: DINOv2 scores higher than Places365 on native identification but lower on the authors' calibrated measure. After reducing features to 128 dimensions and PCA-whitening them to equalize variation across directions, the two measures give more consistent model rankings. These results show that assessing which model is easier to predict from neural activity requires checking feature processing and scoring choices.

  2. Section 4.4 compares PHC with other regions while matching both unit counts and firing rates. The 293-unit PHC population outperforms equally sized, firing-rate-matched controls from other MTL regions and remains better when the control population is increased to 586 units. This makes differences in unit counts or firing rates alone insufficient to explain the PHC advantage. Gerken et al. previously reported movie-label decoding and PHC advantages in some tasks. The present paper tests this advantage when predicting features from multiple image models and adds count- and firing-rate-matched comparisons.

Critical weaknesses
  1. The authors have not specified what the predicted model features reveal about movie content beyond previously decoded character and location labels. For example, a predictive mapping could arise if neural activity distinguishes characters and locations and the image-model features contain the same differences. Successful prediction alone does not establish whether the model features help identify other visual differences when characters and locations are unchanged. The current results do not sufficiently answer this question, leaving the additional knowledge relative to prior work unclear.

    The low-level comparison in Section 4.1 does not directly resolve this issue. Poor prediction of luminance, color, and edge features could reflect weak information in the recordings or limited ability of the measurements and prediction method to detect it. The discussion acknowledges differences in feature dimensionality and temporal variation. The result therefore provides limited support for interpreting MTL activity as favoring more abstract content. In Section 4.3, larger training datasets also differ in content and source, while supervised-to-CLIP comparisons differ in objective and some architectural details. Performance differences therefore cannot be attributed to data scale or language supervision alone.

  2. The paper does not establish whether multiple patients support the PHC advantage. Recordings are pooled by movie time, but Section 3.7 and Appendix A.3 do not report how many patients or which electrode bundles contribute to each region, or provide patient-level checks. For example, if strongly predictable activity comes mainly from a few patients, PHC could remain better after matching unit counts and firing rates. The current evidence neither establishes nor rules out this possibility. It supports a PHC advantage in these pooled recordings, while leaving its consistency across other patients unclear.

The specific grounds for this recommendation are model-ranking sensitivity to target-space processing and the PHC comparisons with equally sized, firing-rate-matched and twice-sized populations. Limited discrimination among content interpretations, the low-level result's detection conditions, and unresolved patient-level robustness constrain the contribution. The recommendation should be reconsidered if the evidence cannot establish what these comparisons add to prior work, or if the regional advantage substantially depends on a few recording sources; narrower wording alone would not add a substantive contribution.

Questions for authors
  1. What specific additional knowledge about MTL processing of movie content follows from predicting these model features? For example, do the existing results show which visual differences can still be predicted when characters and locations are unchanged? Other existing results that establish an additional insight would also be relevant. If poor low-level prediction supports an interpretation in terms of more abstract content, what validation shows that the current method can detect low-level signals when they are present? Please connect the answer to specific results so that the advance over prior label decoding can be assessed.

  2. How many patients and which electrode bundles contribute recordings to PHC and the other regions? If one patient's recordings are omitted at a time and unit counts and firing rates are rematched, does the PHC advantage remain? Please compare its magnitude and direction rather than only whether each check reaches statistical significance. An advantage mainly dependent on a few patients would weaken the interpretation as a general regional difference; support from multiple patients would strengthen it. If the sampling prevents such a check, please explain why.

4 comments
AuthorsOpenReview ↗

Rebuttal: Reviewer ebXY (1/2)

We thank the reviewer for their constructive assessment and for recognizing the value of our evaluation controls and matched PHC comparisons. Their questions directly motivated new analyses testing whether decoding extends beyond the seven binary annotations previously decoded by Gerken et al. (2025) and whether the pooled PHC advantage depends on a small number of participants, alongside a complementary within-scene evaluation controlling for movie-scene identity. Across all tests, our core findings hold and reveal additional regional structure: MTL frame decoding and the 13-model hierarchy persist when holding all seven annotations or exact movie-scene identity fixed (as well as after residualizing out the annotations), and the PHC advantage remains positive in every fold across all 29 single-patient and joint top-contributor omissions while combining coarse between-scene context with finer within-scene discrimination. Finally, we narrowed our interpretation of the low-level comparison and clarified the roles of training regime and dataset composition versus raw scale.

Below, we address each point in detail and identify the corresponding manuscript revisions.

Weakness 1a & Question 1a: What predicted model features reveal beyond previous decoded character/location/setting labels (Gerken et al., 2025)

We thank the reviewer for identifying this important gap. We agree that decoding continuous model features does not, by itself, establish that the decoded information extends beyond previously studied character and setting labels. Using the k=128 CLIP space and the five buffered folds (32 s separation), we added three controls (Methods §3.7, new Results §4.3, Fig. 2, Appendix A.3):

  • Seven-annotation state fixed (Fig. 2b): Restricting candidates (without refitting) to frames sharing the exact joint state across Summer, Tom, McKenzie, Persons, Indoor/Outdoor, Scene Cuts, and Camera Cuts (the 7 annotations decoded in Gerken et al., 2025) lowered identification only modestly from 0.764 to 0.721 vs. a label-preserving 100-shift null of 0.581.
  • Train-only residualization (Fig. 2c): Removing the training-fitted linear CLIP component predicted by those seven labels and refitting whitening and the decoder on residuals yielded 0.733 identification vs. a 0.576 null.
  • Movie-scene identity fixed (Fig. 2d): Restricting candidates to the query's exact movie-scene segment still yielded 0.627 vs. a 0.501 scene-preserving null (gap +0.127, p=0.0099, 5/5 folds).
  • 13-model hierarchy & regional PHC dissociation (Results §4.4–4.5, Fig. 4c, Appendix Figs. 8 & 11b): Re-evaluating all 13 spaces preserved the broad hierarchy. Regionally, PHC's advantage over size- and rate-matched A+H+EC controls stayed intact under annotation conditioning (0.076/0.073) and residualization (0.078/0.074, vs. 0.081/0.076 unrestricted), whereas within exact scenes it dropped to 0.048/0.045 while remaining positive in all 5/5 folds (Fig. 4c)—revealing that PHC combines between-scene context with finer within-scene visual discrimination that binary labels could not resolve (while leaving exact within-scene visual attributes and multimodal covariates for future work).

Takeaway: Together, these results show that human MTL spiking carries visual-scene information extending beyond the seven prior binary labels and scene identity alone. In addition, it uncovers a regional dissociation in PHC—combining coarse between-scene context with finer within-scene visual discrimination—that binary labels could not resolve. Defining the exact visual attributes driving this remaining within-scene signal, and disentangling them from correlated narrative or auditory factors, provides a clear direction for future work.

AuthorsOpenReview ↗

Rebuttal: Reviewer ebXY (2/2)

Weakness 1b & Question 1b: Low-level feature detection and abstract coding interpretation

We agree that weaker low-level decoding does not by itself establish preferential representation of abstract content. Nevertheless, the handcrafted low-level features were decoded above their dedicated circular-shift null, demonstrating that the pipeline detected low-level information, albeit more weakly and less consistently than learned representations. This difference persisted after matching target dimensionality and training covariance, indicating that these factors do not fully explain the result. We have therefore refined the corresponding manuscript statements (Methods 3.3, Results 4.1, Discussion §5): the comparison shows lower and less consistent decodability of the tested handcrafted features, but does not establish an absence of low-level information or a general MTL preference for abstract content.

The advance over prior label decoding is instead supported by the new conditioning analyses. CLIP decoding persisted after controlling for the seven previously decoded annotations and movie-scene identity, and the broad hierarchy across visual spaces remained stable. These results establish visual information beyond the prior labels, although its specific attributes remain to be identified.

Weakness 1c: Dataset content/source and supervised-to-CLIP architecture/objective differences in Section 4.3 (now §4.4)

We agree with the reviewer that larger training data also differ in data content and source, and that comparing label-supervised ResNet-50 to CLIP-RN50 changes both the training objective and stem/downsampling details—meaning performance gains cannot be attributed to dataset scale or language supervision alone. We have revised the manuscript throughout (Abstract, §1, §2, Methods 3.2, Results 4.4, and Discussion §5) to reflect this distinction. Specifically, we removed claims of monotonic scaling with dataset size and explicitly separated the broader supervised-versus-CLIP ResNet-50 family comparison—noting their objective and stem/downsampling differences in the main text (§4.4) as well as Appendix Table 1—from the three strictly matched CLIP-RN50 checkpoints (CC-12M, YFCC-15M, and WIT-400M), which share an identical architecture, readout, resolution, and contrastive objective. Within that strictly matched series, we highlight that most of the decoding gain already occurs between the similarly sized CC-12M and YFCC-15M datasets before plateauing on WIT-400M, indicating that dataset composition and training regime play a central role alongside raw data scale.

Weakness 2 & Question 2: Patient/bundle contributions across regions and leave-one-patient-out checks

We thank the reviewer for raising this important concern. We agree that matching unit counts and firing rates does not by itself exclude disproportionate dependence on a few participants or recording sites. We have therefore added a patient contribution plot (Appendix Fig. 10) and a complete region-wise source table for the contributions of neurons/bundles (Methods §3.8, Appendix Table 5).

Following the reviewer’s suggestion, we assessed the robustness of the PHC advantage to individual and joint patient omissions. We omitted each of the 29 patients one at a time from both PHC and A+H+EC, refitted the remaining PHC population, and redrew size-matched and size-and-firing-rate-matched controls, and additionally omitted the top two and top three PHC contributors jointly (Results §4.5, Appendix A.4, Fig. 11c,d).

  • Single-patient omissions (all 29 patients): Holding out any individual patient preserved the PHC advantage in both direction and magnitude against size-matched and size-plus-rate-matched controls, remaining positive in every cross-validation fold across all 29 omissions (on both identification accuracy and signal fraction ), with even the single most influential omission retaining the vast majority of the full-cohort margin.

  • Joint top-contributor omissions: Simultaneously removing the two or three patients contributing the most PHC units likewise left the PHC advantage unchanged across all folds.

Together, these checks confirm that the pooled PHC advantage is supported across participants rather than driven by a few recording sources alone.

Summary. We thank the reviewer once more for their thorough and constructive feedback. Their questions directly shaped the core empirical additions of this revision—establishing decodable visual information beyond prior binary labels and scene identity, clarifying the regional dissociation within PHC, and verifying that the PHC advantage is supported across participants rather than driven by a few recording sources. We hope the reviewer finds these new analyses valuable in solidifying the manuscript's scientific contribution and will continue to support the acceptance of our work. We remain happy to address any follow-up questions.

I appreciate the author's response, which addressed most of my concerns. I maintain my positive recommendation.

We thank Reviewer ebXY for the positive feedback and are delighted that our revisions successfully addressed their concerns. We greatly appreciate the time they took to provide the constructive comments and for maintaining their positive recommendation.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Rebuttal (1/2)

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful and constructive feedback! Methodologically, we are pleased that the reviewers found our evaluation "more rigorous than most model–brain comparisons" (R Bmii), showing why model comparison "requires checking feature processing and scoring choices" (R ebXY) and offering "a useful methodological point for the brain-alignment literature" through geometry control (R Bmii). Empirically, they noted that decoding continuous vision-model features and population scaling from full-movie human single-unit activity "has empirical value" (R MQvZ), while our matched parahippocampal cortex (PHC) analysis provides "a convincing and novel characterization of visual information in human MTL during naturalistic viewing" (R ebXY, R Bmii).

Across the reviews, the constructive critiques converged on a clear set of questions: (i) establishing what continuous feature decoding reveals about human MTL activity beyond the seven binary character, setting, and transition labels decoded previously by Gerken et al. (2025) and beyond coarse movie-scene identity; (ii) testing whether the 13-model hierarchy and the regional PHC advantage survive those controls; (iii) verifying that the pooled PHC advantage is supported across multiple participants rather than driven by a few recording sources; and (iv) clarifying the exact estimand of whitened identification, low-level feature comparisons, and training-regime claims.

Motivated by your suggestions, we conducted an extensive set of new control and sensitivity experiments and revised the manuscript accordingly. We are happy to report that these new analyses not only corroborate our core findings, but also yield interesting new insights—most notably a regional dissociation in PHC separating coarse scene-level context from finer within-scene discrimination. Below, we summarize all updates:

New Experimental Results

  • E1: Decoding beyond prior binary annotations and movie-scene identity (Methods 3.7, new Results 4.3, Fig. 2, Appendix A.3 & Fig. 7) Addressing R ebXY, R Bmii, and R MQvZ, we tested whether CLIP decoding was explained by the seven character, setting, and transition labels decoded in Gerken et al. (2025) or by movie-scene identity. Decoding remained consistently above dedicated circular-shift nulls when holding the complete seven-annotation state fixed, when predicting residuals after removing annotation-explainable variance, and when restricting candidates to the exact same movie scene.
  • E2: Stability of the 13-model hierarchy under annotation and scene controls (Results 4.4, Appendix A.3 & Fig. 8) Addressing R Bmii (and R ebXY, R MQvZ), we re-evaluated all 13 visual feature spaces under both seven-annotation conditioning and exact within-scene identification, confirming that the broad model hierarchy persists (same-query Spearman ρ = 0.98 and ρ = 0.93, resp.) and is not driven by character or scene clustering (Appendix A.3).
  • E3: Regional dissociation within PHC (Results 4.5, Fig. 4, Appendix A.4, Table 7 & Fig. 11) Addressing R ebXY, R Bmii, and R MQvZ, we completed the 13-space PHC comparison table (Appendix Table 7) and applied the annotation and scene controls regionally. While annotation controls left the PHC advantage unchanged (margins 0.076–0.078 size-matched, 0.073–0.074 rate-matched, vs. 0.081/0.076 unrestricted), within-scene evaluation reduced—but did not eliminate—the margin over matched MTL controls (0.048/0.045, remaining positive in 5/5 folds). Together with full-population robustness under these controls, this reveals an interesting functional dissociation within PHC between coarse between-scene context and finer-grained within-scene visual discrimination.
  • E4: Multi-patient composition and omission sensitivity for PHC (Methods 3.8, Results 4.5, Appendix A.4, Table 5 & Figs. 10–11) Addressing R ebXY and R Bmii, we added a patient-disjoint split-half decoding check (Appendix A.2), a complete breakdown of patients, depth-electrode bundles, and microwires across regions, and both leave-one-patient-out and joint top-contributor omissions, confirming that our results reflect stimulus-locked visual information shared across participants rather than a few recording sources.
  • E5: Metric triangulation and matched low-level diagnostics (Methods 3.3 & 3.6, Results 4.1–4.2, Discussion §5, Appendix A.2 & A.4, Table 4, Fig. 9c) Addressing R MQvZ and R ebXY, we verified that k=128 whitened identification rankings closely match direct held-out feature-prediction metrics (cosine similarity and rescaled R²) and retrieval-free forward per-neuron spike encoding (Spearman ρ = 0.97 for all comparisons), and added a dimensionality- and covariance-matched (k = 64) low-level comparisons.
AuthorsOpenReview ↗

General Rebuttal (2/2)

Manuscript Updates & Clarifications

  • M1: Refined training-regime and ResNet-50 conclusions (Abstract, §1, §2, §3.2, §4.4, §5, Appendix Table 1) Following R ebXY and R Bmii, we removed claims of monotonic scaling with dataset size, separated ResNet-50 backbone-family comparisons (noting objective and stem/downsampling differences) from strictly matched CLIP-RN50 checkpoints, and emphasized the joint role of training regime and dataset composition.
  • M2: Narrowed low-level feature claims (Abstract, §1, §3.3, §4.1, §5, Appendix A.2) Following R ebXY, we refined our wording to state that the tested handcrafted features are decoded more weakly and less consistently than learned spaces, without claiming an absence of low-level coding.
  • M3: Clarified the pseudo-population estimand, native vs. whitened scores, and prior DNN–brain literature (§2, §3.1, §3.6, §4.1–4.3, §5) Addressing R Bmii and R MQvZ, we explicitly stated the movie-aligned pseudo-population setup, clearly labeled native (0.785) versus k = 128 whitened (0.764) CLIP evaluations, expanded Related Work across visual, auditory, and language DNN–brain studies, and sharpened the scientific framing of whitened identification.

With these new experiments and revisions, we believe the manuscript now thoroughly addresses every concern raised across the three reviews. Please see our individual responses to each reviewer for point-by-point details, and let us know if you have any further questions!

AuthorsOpenReview ↗

Follow-Up: General Rebuttal

Dear Area Chair, dear Reviewers,

We would like to thank all reviewers once again for their detailed feedback on our work. Additionally, we sincerely thank Reviewer MQvZ for their follow-up comments and for acknowledging that the “new controls and metric comparisons strengthen our empirical results” and for Reviewer Bmii to continue to support the acceptance of our work after the revision.

Extended annotation-conditioned decoding controls

Prompted by Reviewer MQvZ’s feedback, we have extended our controls from the seven labels decoded by Gerken et al. (2025) to the full set of annotations released in the SUMMER dataset (Darcher et al., 2026), covering a larger set of annotations for the movie stimulus (including characters, faces, person, visibility, speech, locations, setting, visual-transitions and film-specific events). Our updated results demonstrate that the decoding performance on visual features cannot be explained by any of the 53 released annotations, nor by the narrative structure of the movie. This holds true for the advantage of the PHC populations over the remaining MTL areas, as well.

These results show that the comprehensive content annotations are insufficient to explain the decoded MTL information, and not that these attributes are absent from the representation. Identifying the remaining information and its behavioral relevance is an important direction for future work.

We have revised our manuscript and incorporated the extended results into Figure 2, Appendices A.3 and A.4, and Appendix Figure 12. We are happy to address any further questions and feedback until the end of the rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.