- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper compares cortical fMRI encoding by observed-video features, autoregressive future-generation features, and non-autoregressive reconstruction features. Across three model pairs, future-generation features align more strongly with higher visual regions. A human preference study further relates layer-wise alignment to the quality of skip-guided video continuations.
Critical strengths
The alignment advantage is replicated across three AR/base model pairs and survives comparison with processed observed tokens, making the central empirical result convincing. Training-only model selection, held-out variance partitioning, and the broad vision-model comparison provide a careful assessment of where the advantage occurs.
Critical weaknesses
- The within-model comparison does not isolate predictive content. Future-token features outperform both input embeddings and processed observed tokens, but these positions differ in computational role and potentially in access to the observed context. The unique contribution in Figure 2 therefore establishes a better readout, while the stronger interpretation in Section 6 would require evidence separating future-specific information from a richer representation of the observed clip.
- The AR/base comparison changes more than the prediction objective. Section 4 supplies clean observed context to the AR model, whereas the base model reconstructs noised observations; fine-tuning and attention structure also differ. These asymmetries leave the higher-order advantage in Figure 3 compatible with improved conditioning or representation learning, limiting how specifically it can be attributed to future generation.
- The Bayesian-coding interpretation remains untested. Section 6 relates the noisiest-step alignment peak to neural representations of a distribution over possible futures. The analysis measures clip-level encoding accuracy, however, without testing how features represent alternative continuations or their uncertainty; the result is compatible with that interpretation but does not distinguish distributional from single-valued coding, leaving a plausible rather than demonstrated computational insight.
- The preference correlation may reflect the shared depth profile. Figure 4 correlates eight layer conditions, with both neural alignment and preference following an inverted U. Without accounting for depth or comparing against the base-model alignment profile, the reported correlation does not establish that future-feature alignment provides additional information about perceptual quality, reducing its force as behavioral validation.
- Skip guidance is not a selective intervention on the measured representation. In Figure 4, STG contrasts intact and skipped-attention denoising predictions, affecting subsequent generation rather than directly scaling the feature used for cortical encoding. Although the paper acknowledges the absence of a causal account, the description of preference gains through representation amplification still attributes a mechanism that this intervention does not isolate.
Questions for authors
- How does access to the observed context differ between observed and future token positions? Can a matched-context readout distinguish the future-feature advantage from improved summarization of the observed clip?
- Does the alignment–preference relationship remain after accounting for layer depth, and does AR alignment predict preference better than the corresponding base-model alignment?
- What evidence supports interpreting the STG manipulation as amplification of the particular representation measured by the encoding model?
7 comments
Response to Reviewer RjKX (2/4)
[W3] Bayesian interpretation revised as a possible relation.
We thank the reviewer for this fair point and agree that our analysis does not distinguish distributional from single-valued coding. The alignment peak at the noisiest step is compatible with the Bayesian interpretation but does not demonstrate it. We have revised Section 6 to present it as a possible relation rather than a finding.
- The Bayesian interpretation is now stated as a possible relation. The sentence in Section 6 has been changed to "This may relate to Bayesian theories of neural coding, in which neural activity represents a range of possible stimuli rather than a single prediction (Ma et al., 2006 and van Bergen and Jehee, 2019), although our analysis does not test whether the representation carries such uncertainty." Testing this interpretation would require relating the variability across alternative continuations to the cortical response, which we consider an important direction for future work.
Response to Reviewer RjKX (3/4)
[W4 & Q2] The preference correlation holds beyond layer depth and is specific to the AR model.
We thank the reviewer for this important point. We computed partial correlations that regress out either the layer index or the alignment of the base model, and showed that the correlation remains after accounting for layer depth and that the alignment of the AR model predicts preference better than that of the base model (Table R3 in our response to Reviewer Yes1, Appendix A.9, Figure 17).
- The correlation is not explained by layer depth or by the base model. With layer index regressed out, the correlation between the alignment of the AR model and preference remained (r = 0.83, p = 0.014). The alignment of the base model follows a similar layer profile. With each profile regressed out of the other, the correlation of the AR model decreased but stayed positive (r = 0.48, p = 0.230), whereas that of the base model disappeared (r = −0.02, p = 0.963).
Response to Reviewer RjKX (1/4)
[W1, W2 & Q1] Past video generation as a matched-context control.
We thank the reviewer for the careful observation. We agree that the within-model comparison involves token positions with different computational roles, and that the AR and base models differ in input noise, fine-tuning, and attention structure. We added past video generation by the same AR model as a matched control (Figure 1, Section 5.3, Figure 4). We showed that future generation aligns better than past generation under the same context, input, and computational role, which links the advantage to the direction of generation rather than to a richer representation of the observed clip.