acceptodds
Under review as a conference paper at ICLR 2027

Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex

Abstract

Understanding how the brain parses actions and events from time-varying natural inputs is a central challenge in neuroscience. Recent work has used deep neural network (DNN) models to build stimulus-computable fMRI encoding models that predict single-voxel responses to complex natural videos. However, the majority of video-computable encoding models predict responses using simple linear mappings from model tokens, overlooking the spatiotemporal structure shared by video representations and neural responses. Recent cross-attention encoding models address this limitation for static images, enabling flexible stimulus-dependent weighting of image content across space. Here, we extend this framework to naturalistic video, using per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, fitting this model to fMRI responses to short video clips. We compare joint spatiotemporal attention with factorized and selectively constrained alternatives, and find that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas associated with dynamic motion perception. Moreover, our method provides interpretable, stimulus-specific attention maps that dynamically follow moving objects, revealing which locations and temporal moments contribute to each neural response. We further show that attention maps from parcels in different category-selective networks (face-, body-, scene-selective) differentially weight content in accordance with expected semantic selectivity. Together, this work provides a new computational framework for understanding how visual information is adaptively weighted by cortical populations during dynamic visual perception.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Encoding model, fMRI, neuroscience, visual cortex, video, dynamic, spatiotemporal

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rUMJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper extends cross-attention encoding models from static images to naturalistic videos. The proposed approach learns one stimulus-independent query per parcel, while keys and values are shared across parcels. Cross-attention is followed by a linear projection and a residual 2-layer feed-forward neural network (FFN) with GELU nonlinearity, both shared across parcels, whose output is then passed to voxel-specific linear readouts. The authors compare this joint attention against several alternatives: (1) mean pooling, which weights all frames and locations equally; (2) temporal attention, which selects moments while pooling uniformly over spatial locations; (3) spatial attention, which selects locations while pooling uniformly over time; and (4) factorized attention, which selects both moments and locations but uses the same spatial attention pattern at every moment. Additionally, the authors include voxelwise ridge regression on spatially and temporally pooled features. Experiments use V-JEPA 2 ViT-g, VideoMAE v2 ViT-g, DINO v2 ViT-B/14, and CLIP RN50. For each model, the main analysis uses the layer selected by voxelwise ridge regression on pooled features, with an earlier-layer comparison reported in the appendix. Held-out signed r2r^2 on 102 test clips shows that spatial routing improves over mean pooling and ridge regression. Joint routing gives small additional gains over spatial and factorized attention, mainly in lateral/face/body and parietal regions with V-JEPA 2 late-layer features, but not for all backbones or layers. Statistical comparisons use FDR-corrected paired t-tests across subjects. The authors also present qualitative attention maps that appear to track moving objects and compare attended tokens with SAM 3 semantic masks. Although the mask analysis differentiates face-, body-, and scene-selective parcels, a mismatched-mask control finds no significant advantage for the true masks over mismatched masks. This limits the evidence that the attention maps specifically capture category-selective neural representations.

Critical strengths

The paper is overall well-written and well organized. The application of spatiotemporal cross-attention to brain encoding models of naturalistic videos is novel and interesting for the research community.

Critical weaknesses
  • The use of fMRI with TR=1.75s with 3s clips does not provide sufficient temporal resolution, as this results in one scalar per voxel per clip. With the current setup, claims about which moments contribute to each neural response are therefore unconstrained by the data, and the temporal structure of attention maps depends on what the readout can exploit, not by the neural timing. The authors mention this as a limitation in the discussion section and indicate the use of EEG/MEG data as future work, but this should be stated much earlier, as claims about temporal routing giving zero improvement might reflect missing temporal information.
  • The authors claim to “demonstrate how joint spatiotemporal attention leads to more interpretable and accurate attention maps compared to factorized attention, tracking the movement of relevant objects across a video clip”. But in Appendix 9.12, they also specify that they “chose the clips by eye as examples of moving people and animals”. They never provide a quantitative test, for instance by using object-track masks from SAM3 and comparing joint and factorized attention maps across all test clips. The authors should consider adding such quantitative test, or refraining from making claims about interpretability and accuracy of attention maps. Additionally, attention weights are not explanations. ViT-g tokens at layer 32 already integrate information across the whole clip through self-attention. So, a token at a given position does not correspond to pixel content at that position, as it carries global context. Additionally, the attention output is passed through a projection layer and FFN. Therefore, the fact that an attention map highlights the panda’s head does not mean that the model is necessarily looking just at the panda’s head. This is a well-known problem with using attention as explanation [e.g., 1,2]. The authors should consider running causal ablations by masking the attended regions and measuring the prediction drop for that parcel. Factorized attention uses the same spatial map at every frame by design, so showing the factorized model fails to follow moving objects is expected by construction.
  • The main text states that “attention maps reliably differentiate the selectivity of distinct functionally-defined cortical networks” and that localizer overlaps are above chance for all three categories. However, Appendix 9.14 shows that for scenes the mismatched-mask control gives almost identical overlap (0.26 vs 0.28, n.s.), and is itself above chance. Similarly, for bodies the correct masks are not significantly better than the mismatched ones. Only for faces the correct masks outperform the control, but even there the correct-mask overlap is 0.19 with only 5/10 subjects being individually significant. Therefore, claims in the abstract and the main text do not reflect the actual presented evidence, and should be revised. The authors should also consider moving the control analysis to the main text. [1] Jain, S., & Wallace, B. C. (2019, June). Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 3543-3556). [2] Serrano, S., & Smith, N. A. (2019, July). Is attention interpretable?. In Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 2931-2951).
Questions for authors

Clarifications

  1. Was layer choice in Appendix 9.5 made on validation data or on the test clips? Please redo it on held-out validation if the latter.
  2. Why does joint routing underperform spatial routing for some backbones and layers? Can the authors establish whether this reflects optimization, model capacity, or properties of the learned features?
  3. Can you show a quantitative comparison (e.g., attention overlap with SAM 3 moving-object masks or optical flow, across the 102 test clips and many parcels) of joint vs. factorized attention, rather than hand-picked clips?
  4. Could you perform causal ablations by masking the attended regions and measuring the prediction drop for that parcel, to verify that the attended tokens is actually what the model is looking at to predict voxel response?

Future work

  1. CLIP is the only model where ridge regression is best overall. Could this be due to the fact that it is the only model with a non-transformer-based backbone (ResNet50)? Might this be related to the fact that ViT tokens at later layers already integrate global context from the whole clip through self-attention? This might be interesting questions for future work.
4 comments
AuthorsOpenReview ↗

Response to rUMJ (1/2)

We thank the reviewer for a thorough and constructive review. The questions about temporal resolution, attention as explanation and the localization claims highlighted points that needed to be explained more clearly in the text, and we appreciate the opportunity to strengthen the paper by addressing them.

Temporal resolution: the single response amplitude reflects nonlinear temporal integration (Weakness 1)

The reviewer pointed out that with a TR of 1.75 s and 3 s clips there is one value per voxel and clip, and asked that this be stated much earlier. We agree, and the revised Introduction now states it directly (lines 106-107). The Discussion keeps the slow temporal resolution of fMRI as a limitation (line 539).

This single value still carries temporal information, because the BOLD response integrates the clip nonlinearly (General response 3). The attention maps therefore show which moments and locations of the clip the model reads to predict this integrated response. The reviewer also asked whether the lack of improvement from temporal routing alone (lines 315-316) might reflect missing temporal information. It does not: the joint model, which selects moments together with the locations that matter at each moment, is more accurate than the temporal, spatial and factorized models in all 10 subjects (lines 303-307). Selecting moments helps only when it is combined with selecting the locations that matter at each moment (lines 482-485).

Quantitative tests over all test clips, and whether attention reflects what the model uses (Weakness 2; Questions 3 and 4)

The reviewer noted that the tracking examples were chosen by eye and that attention weights are not explanations [Jain & Wallace; Serrano & Smith]. General response 1 addresses both with tests over all 102 test clips. Here we connect each test to the reviewer's request.

  1. A quantitative comparison over all test clips (Question 3). The joint model's attention aligns with the object motion in each frame (Table 10). Because the factorized model cannot follow motion by construction, we compared joint and factorized routing through their predictions instead of their maps: joint routing helps most on clips whose motion changes position in the frame, supporting the interpretation that joint routing's advantage is flexible modeling of motion (App. 9.17, Table 11).
  2. Causal ablation of the attended tokens (Question 4). We removed each parcel's own most-attended tokens from its attention, with all other weights fixed. Because each voxel is predicted from its own parcel's attention, the change in its accuracy measures the effect on that parcel. Accuracy drops sharply, whereas removing the same number of random tokens has no effect (Fig. 3).
  3. The projection and the FFN. The projection is linear, so it cannot change how much each token contributes, and the FFN's MLP can be held at its training mean without changing the maps or the accuracy (App. 9.9). This modified model uses the same attention calculation and reaches the same accuracy, and in it attention is the only stimulus-dependent nonlinearity.
  4. Mixing in the backbone. We agree that a layer-32 token does not correspond only to the pixels at its position, and the revised Discussion states that the maps should not be viewed as a true stimulus-level attribution (lines 533-535). Despite this mixing, the maps still align with the local motion in each frame (lines 414-417).
AuthorsOpenReview ↗

Response to rUMJ (2/2)

Attention alone identifies category-selective parcels from 102 natural clips (Weakness 3)

The reviewer noted that only face localization exceeds the mismatched-mask control, and that even there the overlap is 0.19, with only 5 of 10 subjects individually significant. General response 4 addresses this in detail; in short, we make two points.

  • The analysis reliably distinguishes different functional networks. Our results show that from attention alone, using a small set of 102 natural clips in which faces, bodies and scenery appear together, we can recover face-, body- and scene-selective parcels at about 2, 6 and 3 times chance (Table 12). An overlap of 0.19, with 5 of 10 subjects individually significant, is a strong result in this setting. Note that this analysis is not intended as a competitor to standard functional localizer experiments.
  • The scene result is what the statistics of scenes predict. Analysis of category masks indicates that scenery occupies stereotyped positions in the frame, far more than faces or bodies: the scene masks of two different clips overlap with a mean IoU of 0.224, against 0.019 for faces and 0.098 for bodies. These overlapping positions can explain the correct localization of scene regions when using mismatched masks, and suggest scene regions are aligned with the actual spatial priors of scene-relevant content (App. 9.19, Fig. 26).

Layer choice uses only the training clips (Question 1)

The reviewer asked whether the layer choice in Appendix 9.5 was made on the test clips, and to redo it on held-out validation data if so. It was, and we redid the backbone and layer selection with 10-fold cross-validation within the 1,000 training clips. As General response 2 describes, it selects the same layer for every backbone and the same main model, so no result changes (App. 9.5, Table 2).

When joint routing outperforms spatial routing (Question 2)

The reviewer asked why joint routing underperforms spatial routing for some backbones and layers. In all ViT backbones, joint routing is less accurate than spatial routing at an early layer and more accurate at a late layer (lateral cortex; Table 8). This raises two questions: what makes some layers better for joint routing, and what makes some backbones better. We do not yet know the answer to either, as the revised Discussion states (lines 535-537).

Layers. At an early layer of each backbone, we trained a model that can use both the joint and the spatial attention scores. It does as well as the spatial model, as if it relies on the spatial scores, but we do not know why early-layer features favor them.

Properties of the features. We measured a variety of properties of the tokens to see whether they follow the pattern across layers and backbones: how well single tokens separate objects from the background, how the tokens change over time, and how they represent motion. We also checked model size, which does not follow the pattern either (CLIP RN50, with about 38M parameters, gains as much as V-JEPA 2, with about 1B). None of these has given insight yet.

Ridge regression and CLIP: the advantage is specific to CLIP's features (Future work)

The reviewer asked whether ridge regression is best for CLIP because CLIP RN50 is the only non-transformer backbone, whose tokens do not integrate context from the whole clip. To test this, we fit ridge regression and the routing models to two further convolutional networks: ResNet-50 trained on ImageNet, which has CLIP's image architecture with a different objective, and the video network R(2+1)D-18 (App. 9.14, Table 9). For both, spatial routing is more accurate than ridge regression in every region group, and the joint model is more accurate than ridge regression across all voxels and in lateral cortex. The advantage of ridge regression for CLIP therefore reflects CLIP's features rather than convolutional networks in general. However, for both CNNs, as at the early layers of the transformers, the joint model is less accurate than the spatial model, so whether context integrated across the clip matters for joint routing remains part of the open question above.

Reviewer_rUMJOpenReview ↗

Thanks for the clarifications

Thank you for your detailed response and for the effort you put in the rebuttal. The quantitative motion alignment analysis, the causal ablation and the FFN control address my main concerns about attention maps. Removing the reliance of layer selection on test data also addresses another of my major concerns. For these reasons, I increase my score to a weak accept.

Concerning temporal resolution, I appreciate the addition of a clarification in the introduction. While the argument that BOLD integrates the clip nonlinearly is plausible, the addition of EEG/MEG data would allow to directly test the advantage of temporal routing, by checking whether the moments the model attends to match the ones driving the neural response.

AuthorsOpenReview ↗

Thank you, and EEG/MEG as future work

Thank you for raising your score, and for your careful and constructive comments throughout the review, which improved the paper. We agree that EEG/MEG data would allow a more direct test of temporal routing, and we view it as an important future step. (edited)

Reviewer_CH5COpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a cross-attention encoding framework for predicting human fMRI responses to natural videos, which is an extension on previous work from Adeli, etc. (2023;2025). The authors use pretrained V-JEPA 2 features and introduce parcel-specific queries that dynamically select spatial and temporal information from video tokens. They compare joint spatiotemporal attention with spatial-only, temporal-only, factorized, and mean-pooling alternatives. Experiments on the BOLD Moments Dataset show that joint attention improves prediction accuracy, particularly in lateral and parietal visual regions. The authors also visualize the learned attention maps and investigate whether they capture category-selective responses associated with faces, bodies, and scenes.

Critical strengths

The proposed framework is relatively straightforward, and the comparison between different attention configurations provides a useful way to examine this question. I also appreciate the extensive experiments across multiple model backbones, layers, and cortical regions. The improvement in lateral visual cortex appears relatively consistent across subjects, especially in MT and EBA. The authors also include additional controls for readout architecture and spatial priors, which help clarify where the improvements come from.

Critical weaknesses
  • One of my biggest methodological concern is the use of the Schaefer-1000 atlas to define the attention queries. Although Appendix 9.2 explains why only 423–433 parcel queries remain after intersecting the atlas with the selected voxel masks, I do not think the authors provide convincing evidence that this is an appropriate spatial granularity. I would like to see a sensitivity analysis across different atlas resolutions or alternative parcellation strategies (BMD under MOSAIC framework uses HCP parcellations, or even Schaefer-100 with more coarse spatial resolution). Without this, it is difficult to determine how much the results depend on an arbitrary architectural choice.

  • The authors interpret learned attention maps as evidence that cortical regions dynamically track relevant objects. However, I think the distinction between predictive attention weights and neural response attribution needs to be much clearer. The model is trained to predict a single fMRI response amplitude per voxel for each three-second clip. It does not receive temporally resolved neural supervision within the clip. Therefore, the frame-by-frame attention maps are inferred from video representations under clip-level supervision, rather than directly validated against frame-specific neural responses. This should be discussed more clearly.

    Moreover, visually plausible attention maps do not necessarily indicate that the attended features are responsible for the predictions. I would expect a perturbation-based analysis showing that removing highly attended tokens produces a larger reduction in prediction accuracy than removing less-attended tokens.

  • I found the spatial-prior control in Appendix 9.14 particularly important. The authors compare their attention maps with category masks from mismatched video clips to assess whether localization depends on stimulus content. However, Table 8 shows that the scene-localization overlap is nearly identical for correct and mismatched masks (0.28 versus 0.26), with no significant difference. The body-localization results also fail to show a significant advantage over mismatched masks. Only face localization clearly benefits from the correct stimulus correspondence. To me, this substantially weakens the claim that attention maps reveal category-specific dynamic information. At least for scene-selective regions, the reported localization appears largely explained by spatial priors rather than stimulus-specific attention.

  • The authors argue that comparing joint and factorized attention isolates the benefit of coupled spatiotemporal routing. However, the factorized model computes its spatial and temporal attention using tokens averaged over the other dimension, with separate key projections.

    Consequently, the models differ not only in whether attention weights are factorized, but also in how the attention scores are constructed. I am not convinced that the observed performance difference can be attributed exclusively to the coupling of spatial and temporal information.

  • I feel that the methodological novelty is relatively limited. The overall architecture closely follows previous cross-attention fMRI encoding models, particularly Adeli et al. (NeurIPS 2025), Hwang et al (NeurIPS 2025). Universal Brain Encoder (CCN 2026) also adopted a similar framework. Importantly, from a recent work published at CCN 2026, Feng et al. already extended parcel-query cross-attention encoding to naturalistic video stimuli based on cneuromod dataset.

Questions for authors
  • What motivates using the Schaefer-1000 atlas rather than a coarser atlas or functionally defined visual parcels? Could the authors report the distribution of voxel counts per query and evaluate how prediction accuracy and attention-map stability change with parcellation granularity?
  • How can the authors establish that the learned attention weights reflect features causally important for encoding predictions? Could they perform token-level perturbation experiments or compare attention maps with independent feature-attribution methods?
  • Given that the spatial-prior control in Table 8 explains much of the scene-localization result, and that body localization does not significantly exceed the mismatched-mask control, what evidence supports interpreting these maps as stimulus-specific category selectivity rather than learned spatial preferences?
  • Can the authors provide a more strictly controlled comparison between joint and factorized routing, using matched attention-score computations, to isolate the benefit of nonseparable spatiotemporal attention?
  • Were the backbone and layer selected using the 102 held-out test clips? If so, how would the results change if all model-selection decisions were made exclusively using training and validation data?
8 comments
AuthorsOpenReview ↗

Response to CH5C (1/2)

We thank the reviewer for a detailed and expert review. The questions about parcel granularity, the spatial-prior control and the matched comparison pointed out key methodological and conceptual ambiguities, and we appreciate the opportunity they gave us to strengthen the paper.

Parcel granularity: the results hold with 100, 400 and 1000 parcels (Weakness 1; Question 1)

The reviewer asked what motivates the Schaefer-1000 atlas and how the results change with granularity.

Motivation (App. 9.1). The number of queries sets a trade-off. With one query per voxel, each attention map would be estimated from a single noisy voxel. With one query per large region, one map would be imposed on a heterogeneous population, and the regions would have to be defined in advance. Queries defined by functional ROIs would also put about half of the measured cortex into a single catch-all query: 52% of the voxels we model lie outside all 23 functional ROIs released with the dataset. Atlas parcels sit between these extremes. Because the atlas is defined independently of our data and of the functional localizers, any category selectivity that appears in the maps is not built into the model. Recent whole-brain cross-attention encoders also use one query per parcel of the same Schaefer 1000-parcel atlas (Hwang et al.; Feng et al.).

Sensitivity analysis (App. 9.12). We retrained the joint, factorized, spatial and mean-pool models with the Schaefer-100 and -400 atlases, with everything else unchanged.

  • Voxels per query (Fig. 10): medians of 155, 66 and 32 voxels with 100, 400 and 1000 parcels.
  • Accuracy is flat: the joint model reaches 0.189, 0.191 and 0.191 across all voxels, with no significant difference between granularities.
  • The joint advantage holds: in lateral cortex, the joint model is more accurate than the factorized and spatial models at both coarser granularities (Table 7, Figure 11, Figure 12).
  • The maps are stable: maps from different parcellations are as similar as maps from two data splits of one parcellation (cosine similarity 0.885-0.897), and far more similar than maps of different clips (0.73-0.74; App. 9.12).

Clip-level supervision (Weakness 2)

The reviewer asked that the lack of temporally resolved supervision, and the distinction between predictive attention weights and neural response attribution, be discussed more clearly. We agree, and the revision now discusses both. The revised Introduction states the target directly (lines 106-107). The Discussion states that the maps should not be viewed as a true stimulus-level attribution (lines 533-535) and names the slow temporal resolution of fMRI as a limitation (line 539).

Clip-level supervision still provides insight into how cortex temporally processes the clip. As General response 3 explains, the response amplitude integrates the clip nonlinearly, so predicting it requires dynamically selecting the information that the visual system integrates. Joint routing, which can attend to different locations in different frames, does this better than factorized routing in all 10 subjects, most in lateral cortex (lines 303-307 and 370-373). This is less direct than validation against frame-specific responses, but it supports the interpretation that cortical regions dynamically select relevant content, which the maps make visible (lines 482-485).

Perturbation: removing the most-attended tokens lowers accuracy far more than removing random tokens (Weakness 2; Question 2)

The reviewer asked how we can establish that the attention weights reflect features that are causally important for the predictions, and suggested token-level perturbations. We ran exactly this test (General response 1; Fig. 3): removing each parcel's most-attended tokens sharply lowers accuracy, whereas removing the same number of random tokens has no effect, because attention is spread over many tokens and neighboring tokens carry similar features.

On independent attribution methods: at the level of the tokens, the attribution of our readout can be interpreted as exact. As General response 1 describes, replacing the FFN's MLP by its mean over the training clips gives a modified model with the same attention calculation, the same maps and the same accuracy (0.1907 for both). In this modified model, each prediction is the sum over tokens of attention × a fixed linear score of that token, plus a constant, so each token's contribution is exactly its attention times its score (App. 9.9). Because the modified model uses the same attention and reaches the same accuracy, we interpret this exact attribution as also describing how the trained model uses the tokens. This exactness stops at the tokens. Because the backbone mixes information across the clip, attributing a prediction to pixels would require further analysis of the backbone.

AuthorsOpenReview ↗

Response to CH5C (2/2)

Face and body localization depend on clip content; scene localization reflects where scenery usually appears (Weakness 3; Question 3)

The reviewer asked what evidence supports interpreting the maps as stimulus-specific category selectivity rather than learned spatial preferences. General response 4 addresses this in detail; in short, we make two points.

  • Face and body localization depend on clip content. Both overlaps are significantly above chance with the correct masks and no longer significant with the masks of a different clip. That the body overlap does not significantly exceed the mismatched overlap reflects a harder, additional test, rather than evidence against content dependence (Table 12).
  • The scene result is what the statistics of scenes predict. Analysis of category masks indicates that scenery occupies stereotyped positions in the frame, far more than faces or bodies: the scene masks of two different clips overlap with a mean IoU of 0.224, against 0.019 for faces and 0.098 for bodies. These overlapping positions can explain the correct localization of scene regions when using mismatched masks, and suggest scene regions are aligned with the actual spatial priors of scene-relevant content (App. 9.19, Fig. 26).

Matched attention scores: the joint advantage remains (Weakness 4; Question 4)

The reviewer noted that the factorized model computes its scores differently from the joint model, so the two differ in more than factorization. We trained a variant that computes its scores exactly as the joint model does and then makes the resulting attention separable, as the product of its marginals over time and over space. It reaches 0.186 trained end to end (level with the factorized and spatial models; App. 9.10) and 0.155 from the frozen joint model, both below the joint model (0.191). The advantage of the joint model therefore comes from attention that is not separable over space and time, not from how the scores are computed.

Novelty: a controlled test of how routing is structured over space and time (Weakness 5)

The reviewer noted that the architecture closely follows prior cross-attention encoders, and that Feng et al. already extended parcel-query cross-attention encoding to naturalistic video. We agree that our architecture builds on these models, and we keep it close to them deliberately. Our contribution is not a new architecture but a controlled test of how routing is structured over space and time to best predict temporally integrated BOLD signals, which none of these works performs.

  • Adeli et al. and Hwang et al. route over space in static images.
  • The Universal Brain Encoder (Beliy et al.) learns per-voxel queries shared across subjects.
  • Feng et al. (CCN 2026; full version MM-BEAR, NeurIPS 2026) are closest to our work. Their parcel queries attend jointly over visual tokens from many frames, together with audio and text tokens, to predict the fMRI time series of movies. They compare this model with other ways of combining the modalities (a transformer encoder and a concatenation baseline), but not with attention restricted to space, to time, or to a factorized form. Whether routing must change over time to predict cortical responses is therefore not tested there. They also note that attention weights do not directly quantify predictive contributions, and test reliance by removing whole modalities rather than the attended tokens.

All of these works are now cited (lines 93-94 and 161-164). Our contributions (lines 108-123):

  1. A framework that isolates each form of spatiotemporal routing, with backbone, queries, readout and training identical, to test whether cortex selects moments, locations, or locations that change over time.
  2. Evidence that higher visual cortex depends on routing that changes over time. Joint routing improves accuracy in every subject, most in lateral cortex, at all three parcellation granularities and in the late layers of all three ViT backbones.
  3. The clearest evidence to date that the attention of a cross-attention encoder is interpretable: the predictions depend on the most-attended tokens, and the FFN common to these models does not confound the maps (lines 117-120).

Model selection without the test clips (Question 5)

The reviewer asked whether the backbone and layer were selected with the test clips. They were, and we redid the backbone and layer selection with 10-fold cross-validation within the 1,000 training clips. As General response 2 describes, it selects the same layer for every backbone and the same main model, so no result changes (App. 9.5, Table 2).

Reviewer_CH5COpenReview ↗

Reply to authors in terms of parcel granularity

Thank you for the additional parcellation experiments. I think the results across Schaefer-100, -400, and -1000 largely address my original concern about the sensitivity to atlas granularity. I still have some questions about the exact voxel-selection and parcel-assignment procedure. The original paper reports 423–433 queries per subject using Schaefer-1000, but I could not easily reproduce this procedure when working with the released BMD data myself.

Could the authors clarify the following?

  • Could you please explain more about voxel selection and voxel assignment? And with your aapraoch, is final voxel coverage in each parcel balanced?
  • How many queries are retained for Schaefer-100 and Schaefer-400, respectively? It would be helpful to report the number of retained parcels and voxels per subject for all three atlas resolutions.
  • Is the same voxel-selection mask used for all three parcellation resolutions, with only the parcel assignments changing? Also, how are partially overlapping parcels and voxels outside the atlas handled?
Reviewer_nqnBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends the parcel-query cross-attention encoding framework, previously applied to static images, to natural video. Frozen features from a self-supervised video model (V-JEPA-2) are aggregated with joint spatiotemporal attention, and the attended representation is mapped to repeat-averaged voxel responses in the BOLD Moments Dataset. The authors compare spatial, temporal, factorized, and joint routing under one fitting procedure, with the backbone, readout, and training held fixed, plus separate controls for the FFN, shared projections, and optimization. Joint routing performs best in parts of higher visual cortex, with the clearest advantage in lateral regions. They also analyze the resulting attention maps and check them against independently defined category-selective functional regions.

Critical strengths

The comparison design is the paper's real strength. Spatial, temporal, factorized, and joint routing are treated as constrained versions of the same operation, with the backbone, readout, and training procedure held fixed, and separate controls for the FFN, shared projections, and optimization. That discipline matters for the specific claim being made: if the routing variants had been allowed to differ in capacity or in how they were optimized, the gains could not be attributed to joint spatiotemporal routing.

The improvements also hold up across subjects, most clearly in lateral visual regions, and they survive subject-level paired tests with multiple-comparison correction. The extra backbone experiments, the independently defined functional localizers, and the spatial-prior control go beyond the qualitative attention examples. That is what convinces me on the predictive side.

Critical weaknesses

The model-selection protocol worries me. Appendix 9.3 states that the 102 test clips are used only for final evaluation, whereas Appendix 9.5 describes backbone and layer selection through a sweep and reports the best layers using test-set scores. I could not tell whether selection relied on independent validation data. If test scores informed selection, the generalization results need to be re-derived with a validation-only selection procedure.

The mechanistic claims also outrun the evidence. The prediction target is one response amplitude per voxel per clip, not a within-clip neural time course, so nothing in the fitting constrains the model to recover the temporal moment that drove the response. And because deep backbone tokens are contextualized, attention over token positions is not automatically a faithful attribution to local stimulus content. The qualitative tracking examples do not show that the maps are stable across fitted models, and there is no quantitative temporal correspondence over the full test set.

Questions for authors
  1. What exact validation criterion selected the main backbone and layer, independently of the test clips?

  2. How stable are the attention maps across ensemble members and data splits, and which maps are used in the interpretability analyses?

  3. Do quantitative temporal-correspondence and model-level perturbation tests support the attention interpretation beyond the selected examples?

3 comments
AuthorsOpenReview ↗

Response to nqnB (1/2)

We thank the reviewer for a careful review. The concerns about model selection and about what the attention maps can and cannot show were valuable critiques, and we appreciate the opportunity to strengthen the paper by addressing them.

Model selection: the backbone and layer are now selected without the test clips (Weakness 1; Question 1)

The reviewer asked what validation criterion selected the main backbone and layer, independently of the test clips. In the submission, the layer sweep of Appendix 9.5 used test-set scores, so the selection did not follow model selection protocol. We redid the backbone and layer selection with 10-fold cross-validation of ridge regression within the 1,000 training clips, averaging the cross-validated R2R^2 over voxels and subjects, without the test clips at any step. As General response 2 describes, this selects the same layer for every backbone and the same main model, so all results stand (App. 9.5, Table 2).

The target is one response amplitude per clip (Weakness 2)

The reviewer noted that with one response amplitude per clip, nothing in the fitting constrains the model to recover the moment that drove the response. We agree, and the revised Introduction now states the target directly (lines 106-107). The attention maps therefore show which moments and locations of the clip the model reads to predict this one number. The Discussion also keeps the slow temporal resolution of fMRI as a limitation (line 539).

As General response 3 explains, this amplitude still carries temporal information, because the BOLD response integrates the clip nonlinearly. Our model comparison asks which form of routing best captures this integration: the joint model, which can attend to different locations in different frames, is more accurate than spatial and factorized routing in all 10 subjects (lines 303-307), which the revised Discussion interprets as evidence that temporal and spatial information must interact dynamically to predict the response (lines 482-485).

Contextualized tokens: the maps describe which tokens the model reads (Weakness 2)

The reviewer pointed out that deep backbone tokens are contextualized, so attention over token positions is not automatically a faithful attribution to local stimulus content. We agree. Before our readout, the backbone mixes information across the clip through self-attention, so a token at one position of layer 32 also carries information from other locations and frames. The attention map therefore tells us which tokens the model reads, but not by itself which pixels of the video drive the prediction. The revised Discussion says this directly (lines 533-535), and the Results describe the attended tokens as carrying mixed information over space and time (lines 414-417).

However, as described below under "Quantitative tests over the full test set", we can still support the interpretability of the attention maps in relation to local stimulus content: across all 102 test clips, and despite the mixing, they align with the motion within each frame.

AuthorsOpenReview ↗

Response to nqnB (2/2)

Stability across fitted models, and which maps are used (Weakness 2; Question 2)

The reviewer asked how stable the attention maps are across ensemble members and data splits, and which maps are used in the analyses.

  • Which maps. The example figures and the category localization use the maps of the first ensemble member, as the revised Methods and App. 9.18 now state (lines 284-285); the token-removal test and the motion alignment use the full 10-member ensemble.
  • Stability across data splits. Maps of two ensemble members, trained on different random splits of the training clips, are far more similar for the same test clip than for different clips (cosine similarity 0.887 vs 0.724), in every region group (App. 9.16).
  • The examples are typical. New Fig. 25 shows the first member's maps next to the average over all 10 members for three clips of the main text. The average attends to the same content, with less scattered attention on single tokens.
  • Stability across parcellations. Maps from models trained with 100, 400 and 1000 parcels are as similar to each other as maps from two data splits of one parcellation (App. 9.12).

Quantitative tests over the full test set (Weakness 2; Question 3)

The reviewer asked whether quantitative temporal-correspondence and model-level perturbation tests support the interpretation beyond the selected examples. We added both, over all 102 test clips and all parcels; General response 1 gives the results, together with the test showing that the FFN does not confound the maps (App. 9.16). In the reviewer's terms:

  1. Model-level perturbation. Removing each parcel's most-attended tokens from its attention, with all other weights fixed, sharply lowers accuracy, whereas removing the same number of random tokens has no effect (Fig. 3). Random removal has no effect because attention is spread over many tokens and neighboring tokens carry similar features, so it removes important and unimportant tokens in proportion.
  2. Temporal correspondence. The alignment with object motion is computed separately in every frame, so it tests whether attention follows moving content over time rather than where motion usually is. Attention aligns with the motion of the same clip far more than with the motion of other clips (Table 10).
  3. Which clips benefit from joint routing. Joint routing helps most on clips whose moving content changes position in the frame, supporting the interpretation that joint routing's advantage is flexible modeling of motion (App. 9.17).
AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your careful review. We wanted to check whether our revisions address your concerns before the discussion period closes. In particular, we have added quantitative evaluations of the attention maps, examining their stability across data splits, the effect of removing the top-attended tokens on prediction performance, and their alignment with motion. We have also changed our backbone and layer selection procedure to use cross-validation on the training clips, which resulted in same selection.

If these address your concerns, we would be grateful if you would consider updating your assessment. We are also happy to answer any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2)

We thank the reviewers for their careful reviews. All three highlighted the controlled design, in which joint, factorized, spatial and temporal routing are constrained versions of one attention operation with the backbone, readout and training held fixed. We first address the points raised by several reviewers, then list every change in the revision. The new text is in blue in the revised PDF, to which all line numbers refer.

1. The attention maps are faithful to the model's computation and track motion across all test clips (nqnB, CH5C, rUMJ)

We agree that plausible-looking attention maps are not by themselves explanations: attention can be a poor guide to what drives a model's predictions [Jain & Wallace, 2019; Serrano & Smith, 2019], and our attention output passes through a projection and an FFN. The revision tests faithfulness directly and replaces the qualitative tracking examples with quantitative measures over all 102 test clips (Results, lines 404-421).

  • Causal token removal. For each parcel and test clip, we removed 5-50% of the tokens from the attention (scores set to −∞-\infty before the softmax, all other weights fixed): either the most-attended tokens or the same number of random tokens. Removing the top 20% lowers the signed r2r^2 from 0.191 to 0.095, whereas random removal has no effect at any fraction (Fig. 3; App. 9.16).
  • The FFN does not confound the maps. Holding the FFN's MLP at its training mean leaves the attention maps and the accuracy unchanged (0.1907 vs 0.1907). An affine map fits the whole FFN block with R2R^2 = 0.947 but the MLP alone only with 0.30: the residual path dominates. In this modified model, which uses the same attention calculation, each prediction is exactly an attention-weighted sum of fixed linear scores of the tokens, so attention is its only stimulus-dependent nonlinearity. Both cited studies attribute unfaithful attention mainly to encoders that mix inputs before the attention; with respect to the backbone tokens, our readout has no such mixing (App. 9.9). Because the backbone itself mixes information, the revised Discussion states that the maps should not be viewed as a true stimulus-level attribution (lines 533-535).
  • Attention aligns with motion. Within each frame, the ensemble-averaged attention correlates with camera-compensated optical flow (Spearman ρ\rho = 0.133, vs 0.011 with the flow of other clips), most in MT and OFA and least in the scene-selective PPA and RSC. For scale, two motion measures of the same clip agree only 0.32-0.49 under this measure. The alignment also holds over the whole video (App. 9.16, Table 10).
  • Joint routing helps most on clips whose motion layout changes. Among 77 clip properties covering content, content layout, image statistics and motion, the per-clip advantage of joint over factorized routing is best predicted by measures of motion layout, which a factorized model cannot represent, and motion properties predict it on the held-out test clips as well (App. 9.17).

2. Backbone and layer are now selected on the training clips only (nqnB, CH5C, rUMJ)

We agree that selecting the backbone and layer with test-set scores did not follow model selection protocol. We redid the selection with 10-fold cross-validation of ridge regression within the 1,000 training clips (App. 9.5, Table 2). For every backbone, cross-validation selects the layer used in the paper, so all results stand. V-JEPA 2 ViT-g layer 32 and ViT-L layer 18 are tied, each better in 5 of the 10 subjects, and ahead of every other backbone; we use ViT-g, the larger of the two.

3. Each fMRI response integrates the clip's temporal structure, which joint routing captures (nqnB, CH5C, rUMJ)

We agree that the original text did not make clear what the model predicts or how temporal structure enters. The target is a single response amplitude per voxel and clip. BOLD signals integrate information over time nonlinearly, in a stimulus-dependent and spatially dependent manner [Zhou et al., 2018; Stigliani et al., 2017; Groen et al., 2022], so this amplitude should not be interpreted as an average or linear combination over the clip. Our model shows that it is necessary to include this temporal information and allow it to interact dynamically with spatial information to predict the response accurately: joint routing outperforms spatial and factorized routing in all 10 subjects, most in lateral cortex, also when the factorization is computed from the joint model's own scores (App. 9.10). The revised Introduction now states the target and how it integrates the clip (lines 106-107), and the revised Discussion states what the model shows (lines 482-485).

AuthorsOpenReview ↗

General response (2/2)

4. Attention alone identifies category-selective parcels, and face and body localization follow the content of each clip (CH5C, rUMJ)

We agree that the mismatched-mask control is central to the localization claims. It is now in the main text, and the claims are revised (Results, lines 469-475). We would also like to clarify the purpose of the analysis and how we interpret the control.

What the localization is meant to show. This method is not intended to replace standard functional localizers, which measure responses to controlled stimuli in specified categories. Instead, our analysis uses only the 102 test clips: natural videos in which faces, bodies and scenery appear together. We show that from attention overlap with category masks alone on just these clips, we are able to identify a significant portion of the actual parcels in the face-, body- and scene-selective ROIs (about 2, 6 and 3 times chance; Table 12). This result shows that even using a small stimulus set with mixed content, we can use attention to reliably distinguish functionally-distinct brain networks.

Bodies. With the correct masks, the body overlap is well above chance (0.14 vs 0.02), and with the masks of a different clip it is no longer significantly above chance (0.08), the same pattern as for faces. This is what the control is designed to detect: a localization that reflected only where body-selective parcels usually attend would remain significant with mismatched masks, as the scene localization does. The direct comparison of correct and mismatched masks is an additional, harder test: bodies often appear at similar positions in different clips, which raises the mismatched overlap to 0.08, so the comparison must detect a difference of 0.06, half the margin over chance (0.12).

Scenes. The localization mostly reflects where scene-selective parcels attend in every clip rather than the scenery of each clip. This is expected, because scenery occupies stereotyped positions in the frame, far more than faces or bodies (App. 9.19, Fig. 26).

New results in the revision

  1. Causal token removal (Fig. 3; App. 9.16, Fig. 24; nqnB, CH5C, rUMJ; point 1).
  2. The FFN at test time (App. 9.9; CH5C, rUMJ; point 1).
  3. Map stability (App. 9.16, Fig. 25; nqnB). Maps of ensemble members trained on different data splits are far more similar for the same clip than for different clips.
  4. Alignment of attention with seven motion maps (App. 9.16, Table 10; nqnB, rUMJ; point 1).
  5. Which clips benefit from joint routing (App. 9.17, Table 11; rUMJ, nqnB; point 1).
  6. Cross-validated backbone and layer selection (App. 9.5, Table 2; nqnB, CH5C, rUMJ; point 2).
  7. Schaefer-100 and -400 parcellations (App. 9.12, Table 7, Figs. 10-12; CH5C). Accuracy is flat across granularity, the joint advantage in lateral cortex holds at both, and maps agree across granularity as closely as across data splits.
  8. Separable attention from the joint scores (App. 9.10; CH5C). A model whose attention is the product of the joint softmax's time and space marginals is less accurate than the joint model and level with the factorized model.
  9. Spatial layout of the localization categories (App. 9.19, Fig. 26; CH5C, rUMJ; point 4).
  10. Two further CNN backbones (App. 9.14, Table 9; rUMJ). With ResNet-50 (ImageNet) and R(2+1)D-18, spatial routing beats ridge regression in every region group, so CLIP's ridge advantage is specific to its features.

Revisions to the text

  1. The prediction target (one response amplitude per voxel and clip) and the nonlinear temporal integration of the BOLD response, in the Introduction (lines 47-49, 106-107), Methods (lines 255-256) and Discussion (lines 482-485) (nqnB, CH5C, rUMJ).
  2. A Results paragraph quantifying the attention maps (lines 404-421), a contribution statement on this evidence (lines 117-120), and a Discussion statement that the maps are not a stimulus-level attribution (lines 533-535) (nqnB, CH5C, rUMJ).
  3. Beliy et al. (CCN 2026) and Feng et al. (CCN 2026; NeurIPS 2026) added to the Introduction (lines 93-94) and Related Work (lines 161-164) (CH5C).
  4. The mismatched-mask control moved to the main text, with the localization claims scoped to face and body content and scene spatial preferences (lines 469-475) (CH5C, rUMJ).
  5. The Methods state which ensemble member's maps are shown (lines 284-285) (nqnB).
  6. The Discussion states that the backbone and layer dependence of the advantage for joint routing remains open (lines 535-537) (rUMJ).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.