- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper extends cross-attention encoding models from static images to naturalistic videos. The proposed approach learns one stimulus-independent query per parcel, while keys and values are shared across parcels. Cross-attention is followed by a linear projection and a residual 2-layer feed-forward neural network (FFN) with GELU nonlinearity, both shared across parcels, whose output is then passed to voxel-specific linear readouts. The authors compare this joint attention against several alternatives: (1) mean pooling, which weights all frames and locations equally; (2) temporal attention, which selects moments while pooling uniformly over spatial locations; (3) spatial attention, which selects locations while pooling uniformly over time; and (4) factorized attention, which selects both moments and locations but uses the same spatial attention pattern at every moment. Additionally, the authors include voxelwise ridge regression on spatially and temporally pooled features. Experiments use V-JEPA 2 ViT-g, VideoMAE v2 ViT-g, DINO v2 ViT-B/14, and CLIP RN50. For each model, the main analysis uses the layer selected by voxelwise ridge regression on pooled features, with an earlier-layer comparison reported in the appendix. Held-out signed on 102 test clips shows that spatial routing improves over mean pooling and ridge regression. Joint routing gives small additional gains over spatial and factorized attention, mainly in lateral/face/body and parietal regions with V-JEPA 2 late-layer features, but not for all backbones or layers. Statistical comparisons use FDR-corrected paired t-tests across subjects. The authors also present qualitative attention maps that appear to track moving objects and compare attended tokens with SAM 3 semantic masks. Although the mask analysis differentiates face-, body-, and scene-selective parcels, a mismatched-mask control finds no significant advantage for the true masks over mismatched masks. This limits the evidence that the attention maps specifically capture category-selective neural representations.
Critical strengths
The paper is overall well-written and well organized. The application of spatiotemporal cross-attention to brain encoding models of naturalistic videos is novel and interesting for the research community.
Critical weaknesses
- The use of fMRI with TR=1.75s with 3s clips does not provide sufficient temporal resolution, as this results in one scalar per voxel per clip. With the current setup, claims about which moments contribute to each neural response are therefore unconstrained by the data, and the temporal structure of attention maps depends on what the readout can exploit, not by the neural timing. The authors mention this as a limitation in the discussion section and indicate the use of EEG/MEG data as future work, but this should be stated much earlier, as claims about temporal routing giving zero improvement might reflect missing temporal information.
- The authors claim to “demonstrate how joint spatiotemporal attention leads to more interpretable and accurate attention maps compared to factorized attention, tracking the movement of relevant objects across a video clip”. But in Appendix 9.12, they also specify that they “chose the clips by eye as examples of moving people and animals”. They never provide a quantitative test, for instance by using object-track masks from SAM3 and comparing joint and factorized attention maps across all test clips. The authors should consider adding such quantitative test, or refraining from making claims about interpretability and accuracy of attention maps. Additionally, attention weights are not explanations. ViT-g tokens at layer 32 already integrate information across the whole clip through self-attention. So, a token at a given position does not correspond to pixel content at that position, as it carries global context. Additionally, the attention output is passed through a projection layer and FFN. Therefore, the fact that an attention map highlights the panda’s head does not mean that the model is necessarily looking just at the panda’s head. This is a well-known problem with using attention as explanation [e.g., 1,2]. The authors should consider running causal ablations by masking the attended regions and measuring the prediction drop for that parcel. Factorized attention uses the same spatial map at every frame by design, so showing the factorized model fails to follow moving objects is expected by construction.
- The main text states that “attention maps reliably differentiate the selectivity of distinct functionally-defined cortical networks” and that localizer overlaps are above chance for all three categories. However, Appendix 9.14 shows that for scenes the mismatched-mask control gives almost identical overlap (0.26 vs 0.28, n.s.), and is itself above chance. Similarly, for bodies the correct masks are not significantly better than the mismatched ones. Only for faces the correct masks outperform the control, but even there the correct-mask overlap is 0.19 with only 5/10 subjects being individually significant. Therefore, claims in the abstract and the main text do not reflect the actual presented evidence, and should be revised. The authors should also consider moving the control analysis to the main text. [1] Jain, S., & Wallace, B. C. (2019, June). Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 3543-3556). [2] Serrano, S., & Smith, N. A. (2019, July). Is attention interpretable?. In Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 2931-2951).
Questions for authors
Clarifications
- Was layer choice in Appendix 9.5 made on validation data or on the test clips? Please redo it on held-out validation if the latter.
- Why does joint routing underperform spatial routing for some backbones and layers? Can the authors establish whether this reflects optimization, model capacity, or properties of the learned features?
- Can you show a quantitative comparison (e.g., attention overlap with SAM 3 moving-object masks or optical flow, across the 102 test clips and many parcels) of joint vs. factorized attention, rather than hand-picked clips?
- Could you perform causal ablations by masking the attended regions and measuring the prediction drop for that parcel, to verify that the attended tokens is actually what the model is looking at to predict voxel response?
Future work
- CLIP is the only model where ridge regression is best overall. Could this be due to the fact that it is the only model with a non-transformer-based backbone (ResNet50)? Might this be related to the fact that ViT tokens at later layers already integrate global context from the whole clip through self-attention? This might be interesting questions for future work.
4 comments
Response to rUMJ (2/2)
Attention alone identifies category-selective parcels from 102 natural clips (Weakness 3)
The reviewer noted that only face localization exceeds the mismatched-mask control, and that even there the overlap is 0.19, with only 5 of 10 subjects individually significant. General response 4 addresses this in detail; in short, we make two points.
- The analysis reliably distinguishes different functional networks. Our results show that from attention alone, using a small set of 102 natural clips in which faces, bodies and scenery appear together, we can recover face-, body- and scene-selective parcels at about 2, 6 and 3 times chance (Table 12). An overlap of 0.19, with 5 of 10 subjects individually significant, is a strong result in this setting. Note that this analysis is not intended as a competitor to standard functional localizer experiments.
- The scene result is what the statistics of scenes predict. Analysis of category masks indicates that scenery occupies stereotyped positions in the frame, far more than faces or bodies: the scene masks of two different clips overlap with a mean IoU of 0.224, against 0.019 for faces and 0.098 for bodies. These overlapping positions can explain the correct localization of scene regions when using mismatched masks, and suggest scene regions are aligned with the actual spatial priors of scene-relevant content (App. 9.19, Fig. 26).
Layer choice uses only the training clips (Question 1)
The reviewer asked whether the layer choice in Appendix 9.5 was made on the test clips, and to redo it on held-out validation data if so. It was, and we redid the backbone and layer selection with 10-fold cross-validation within the 1,000 training clips. As General response 2 describes, it selects the same layer for every backbone and the same main model, so no result changes (App. 9.5, Table 2).
When joint routing outperforms spatial routing (Question 2)
The reviewer asked why joint routing underperforms spatial routing for some backbones and layers. In all ViT backbones, joint routing is less accurate than spatial routing at an early layer and more accurate at a late layer (lateral cortex; Table 8). This raises two questions: what makes some layers better for joint routing, and what makes some backbones better. We do not yet know the answer to either, as the revised Discussion states (lines 535-537).
Layers. At an early layer of each backbone, we trained a model that can use both the joint and the spatial attention scores. It does as well as the spatial model, as if it relies on the spatial scores, but we do not know why early-layer features favor them.
Properties of the features. We measured a variety of properties of the tokens to see whether they follow the pattern across layers and backbones: how well single tokens separate objects from the background, how the tokens change over time, and how they represent motion. We also checked model size, which does not follow the pattern either (CLIP RN50, with about 38M parameters, gains as much as V-JEPA 2, with about 1B). None of these has given insight yet.
Ridge regression and CLIP: the advantage is specific to CLIP's features (Future work)
The reviewer asked whether ridge regression is best for CLIP because CLIP RN50 is the only non-transformer backbone, whose tokens do not integrate context from the whole clip. To test this, we fit ridge regression and the routing models to two further convolutional networks: ResNet-50 trained on ImageNet, which has CLIP's image architecture with a different objective, and the video network R(2+1)D-18 (App. 9.14, Table 9). For both, spatial routing is more accurate than ridge regression in every region group, and the joint model is more accurate than ridge regression across all voxels and in lateral cortex. The advantage of ridge regression for CLIP therefore reflects CLIP's features rather than convolutional networks in general. However, for both CNNs, as at the early layers of the transformers, the joint model is less accurate than the spatial model, so whether context integrated across the clip matters for joint routing remains part of the open question above.
Thanks for the clarifications
Thank you for your detailed response and for the effort you put in the rebuttal. The quantitative motion alignment analysis, the causal ablation and the FFN control address my main concerns about attention maps. Removing the reliance of layer selection on test data also addresses another of my major concerns. For these reasons, I increase my score to a weak accept.
Concerning temporal resolution, I appreciate the addition of a clarification in the introduction. While the argument that BOLD integrates the clip nonlinearly is plausible, the addition of EEG/MEG data would allow to directly test the advantage of temporal routing, by checking whether the moments the model attends to match the ones driving the neural response.
Thank you, and EEG/MEG as future work
Thank you for raising your score, and for your careful and constructive comments throughout the review, which improved the paper. We agree that EEG/MEG data would allow a more direct test of temporal routing, and we view it as an important future step. (edited)
Response to rUMJ (1/2)
We thank the reviewer for a thorough and constructive review. The questions about temporal resolution, attention as explanation and the localization claims highlighted points that needed to be explained more clearly in the text, and we appreciate the opportunity to strengthen the paper by addressing them.
Temporal resolution: the single response amplitude reflects nonlinear temporal integration (Weakness 1)
The reviewer pointed out that with a TR of 1.75 s and 3 s clips there is one value per voxel and clip, and asked that this be stated much earlier. We agree, and the revised Introduction now states it directly (lines 106-107). The Discussion keeps the slow temporal resolution of fMRI as a limitation (line 539).
This single value still carries temporal information, because the BOLD response integrates the clip nonlinearly (General response 3). The attention maps therefore show which moments and locations of the clip the model reads to predict this integrated response. The reviewer also asked whether the lack of improvement from temporal routing alone (lines 315-316) might reflect missing temporal information. It does not: the joint model, which selects moments together with the locations that matter at each moment, is more accurate than the temporal, spatial and factorized models in all 10 subjects (lines 303-307). Selecting moments helps only when it is combined with selecting the locations that matter at each moment (lines 482-485).
Quantitative tests over all test clips, and whether attention reflects what the model uses (Weakness 2; Questions 3 and 4)
The reviewer noted that the tracking examples were chosen by eye and that attention weights are not explanations [Jain & Wallace; Serrano & Smith]. General response 1 addresses both with tests over all 102 test clips. Here we connect each test to the reviewer's request.