acceptodds
Under review as a conference paper at ICLR 2027

Future Video Generation Better Aligns with the Human Visual Cortex than Observed Video

13 views

Abstract

Studying the alignment between the internal representations of vision models and the responses of the visual cortex to the same observed visual stimuli has enabled us to better understand human visual processing. However, studies so far have largely overlooked the fact that the human brain not only processes observed visual stimuli, but also predicts upcoming stimuli based on what has been observed. Accordingly, we hypothesize that internal representations for generating future video frames are better aligned with the predictive nature of human visual processing than representations of the observed video itself. To this end, we compare the alignment between human video-watching fMRI responses in the visual cortex and the internal representations from two types of video diffusion models, an autoregressive (AR) model and its non-AR base model. We first conduct a within-model analysis of the AR video diffusion model and show that the representations for future video generation align better with the visual cortex than the representations of the observed video. We then compare the internal representations of the AR model with those of its non-AR base model and again show that the representations for future video generation align better with the visual cortex than the representations for observed video reconstruction by the base model. Specifically, the alignment of observed video reconstruction is concentrated in lower-order visual cortex, whereas that of future video generation is concentrated in higher-order visual cortex. Finally, we show in a human behavioral experiment that humans prefer videos generated by amplifying the contributions of individual layers that align better with the visual cortex.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-model alignment, fMRI, human visual cortex, predictive processing, video diffusion models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RjKXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares cortical fMRI encoding by observed-video features, autoregressive future-generation features, and non-autoregressive reconstruction features. Across three model pairs, future-generation features align more strongly with higher visual regions. A human preference study further relates layer-wise alignment to the quality of skip-guided video continuations.

Critical strengths

The alignment advantage is replicated across three AR/base model pairs and survives comparison with processed observed tokens, making the central empirical result convincing. Training-only model selection, held-out variance partitioning, and the broad vision-model comparison provide a careful assessment of where the advantage occurs.

Critical weaknesses
  1. The within-model comparison does not isolate predictive content. Future-token features outperform both input embeddings and processed observed tokens, but these positions differ in computational role and potentially in access to the observed context. The unique contribution in Figure 2 therefore establishes a better readout, while the stronger interpretation in Section 6 would require evidence separating future-specific information from a richer representation of the observed clip.
  2. The AR/base comparison changes more than the prediction objective. Section 4 supplies clean observed context to the AR model, whereas the base model reconstructs noised observations; fine-tuning and attention structure also differ. These asymmetries leave the higher-order advantage in Figure 3 compatible with improved conditioning or representation learning, limiting how specifically it can be attributed to future generation.
  3. The Bayesian-coding interpretation remains untested. Section 6 relates the noisiest-step alignment peak to neural representations of a distribution over possible futures. The analysis measures clip-level encoding accuracy, however, without testing how features represent alternative continuations or their uncertainty; the result is compatible with that interpretation but does not distinguish distributional from single-valued coding, leaving a plausible rather than demonstrated computational insight.
  4. The preference correlation may reflect the shared depth profile. Figure 4 correlates eight layer conditions, with both neural alignment and preference following an inverted U. Without accounting for depth or comparing against the base-model alignment profile, the reported correlation does not establish that future-feature alignment provides additional information about perceptual quality, reducing its force as behavioral validation.
  5. Skip guidance is not a selective intervention on the measured representation. In Figure 4, STG contrasts intact and skipped-attention denoising predictions, affecting subsequent generation rather than directly scaling the feature used for cortical encoding. Although the paper acknowledges the absence of a causal account, the description of preference gains through representation amplification still attributes a mechanism that this intervention does not isolate.
Questions for authors
  1. How does access to the observed context differ between observed and future token positions? Can a matched-context readout distinguish the future-feature advantage from improved summarization of the observed clip?
  2. Does the alignment–preference relationship remain after accounting for layer depth, and does AR alignment predict preference better than the corresponding base-model alignment?
  3. What evidence supports interpreting the STG manipulation as amplification of the particular representation measured by the encoding model?
7 comments
AuthorsOpenReview ↗

Response to Reviewer RjKX (1/4)

[W1, W2 & Q1] Past video generation as a matched-context control.

We thank the reviewer for the careful observation. We agree that the within-model comparison involves token positions with different computational roles, and that the AR and base models differ in input noise, fine-tuning, and attention structure. We added past video generation by the same AR model as a matched control (Figure 1, Section 5.3, Figure 4). We showed that future generation aligns better than past generation under the same context, input, and computational role, which links the advantage to the direction of generation rather than to a richer representation of the observed clip.

  • Observed and future tokens share access to the observed context but differ in role (Q1). In the AR model, self-attention is not causal, so in every layer the observed tokens and the future tokens attend to each other and both have full access to the observed context. They differ in role. The observed tokens enter clean with a timestep of zero, whereas the future tokens are the noised tokens being denoised. As the reviewer points out, the comparison between these positions therefore does not perfectly isolate predictive content.
  • Past generation provides a matched-context readout (W1, W2). We fed the same 3 s clip to the same AR model with its frames in reverse order, so that the model generates the 1 s preceding the clip. Past and future tokens have the same access to the observed context, the same clean input, the same weights, and the same computational role, and they differ only in the direction of generation. Backward generation was as valid as forward generation within the clip (Table R1 in our response to Reviewer Yes1, Appendix A.3).
  • Future generation is not merely a better summary of the observed clip (W1). If future tokens only reorganized the observed content, past and future generation should align equally with the visual cortex. In the variance partitioning among the three representations, however, future generation had the largest unique contribution in the Visual System (0.016), above observed reconstruction (0.009, p = 0.027) and past generation (0.007, p = 0.023, Figure 4b). Future generation exceeded past generation in every division, significantly in Dorsal Stream, Ventral Stream, and MT+ Complex (p ≤ 0.016).
  • Each representation is oriented to the time it generates (W1). The time-resolved analysis, which uses the response at each second after onset rather than one response per video, separated the past, observed, and future representations in time. Past generation peaked at 4.4 s, observed reconstruction at 5.6 s, and future generation at 7.7 s (Figure 4c). Although both are conditioned on the same observed video, past and future generation are thus oriented to the time each generates.
AuthorsOpenReview ↗

Response to Reviewer RjKX (2/4)

[W3] Bayesian interpretation revised as a possible relation.

We thank the reviewer for this fair point and agree that our analysis does not distinguish distributional from single-valued coding. The alignment peak at the noisiest step is compatible with the Bayesian interpretation but does not demonstrate it. We have revised Section 6 to present it as a possible relation rather than a finding.

  • The Bayesian interpretation is now stated as a possible relation. The sentence in Section 6 has been changed to "This may relate to Bayesian theories of neural coding, in which neural activity represents a range of possible stimuli rather than a single prediction (Ma et al., 2006 and van Bergen and Jehee, 2019), although our analysis does not test whether the representation carries such uncertainty." Testing this interpretation would require relating the variability across alternative continuations to the cortical response, which we consider an important direction for future work.
AuthorsOpenReview ↗

Response to Reviewer RjKX (3/4)

[W4 & Q2] The preference correlation holds beyond layer depth and is specific to the AR model.

We thank the reviewer for this important point. We computed partial correlations that regress out either the layer index or the alignment of the base model, and showed that the correlation remains after accounting for layer depth and that the alignment of the AR model predicts preference better than that of the base model (Table R3 in our response to Reviewer Yes1, Appendix A.9, Figure 17).

  • The correlation is not explained by layer depth or by the base model. With layer index regressed out, the correlation between the alignment of the AR model and preference remained (r = 0.83, p = 0.014). The alignment of the base model follows a similar layer profile. With each profile regressed out of the other, the correlation of the AR model decreased but stayed positive (r = 0.48, p = 0.230), whereas that of the base model disappeared (r = −0.02, p = 0.963).
Reviewer_L4dxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether internal representations used to generate future video frames align better with human visual cortical responses than representations of the observed video. Using the BOLD Moments Dataset, the authors compare observed-video input embeddings, future-token representations from autoregressive (AR) video diffusion models, and observed-video reconstruction representations from their non-AR base models. These representations are evaluated using vertex-wise encoding models and variance partitioning.

The main findings are that future-generation representations provide a larger advantage in higher-order visual cortex and that alignment peaks at the earliest denoising step and early-to-middle layers. The authors also apply layer-wise Spatiotemporal Skip Guidance (STG) and report an association between a layer’s neural alignment and human preference for videos generated under guidance targeting that layer.

Critical strengths
  • The paper distinguishes representations of the observed-video input, representations used to reconstruct that video, and representations used to generate its unobserved continuation.
  • It also asks what the model is processing or generating when the representations are extracted.
  • The regional comparisons and distributions of unique contributions provide information beyond an overall average encoding score or a ranking of models.
  • The evaluation includes multiple video generation models and a broad range of vision models.
  • The authors intervene the generation process and evaluating human preferences for the resulting videos.
Critical weaknesses

W1: The encoding advantage does not distinguish future prediction from improved representations of observed content.

Future-token activations are used to generate future frames, but their higher encoding accuracy could reflect observed objects, motion, and semantics being reorganized into a form more accessible to a linear readout, rather than information about the unobserved future. The analysis uses one fMRI response estimate per video at each vertex, without independently measuring or manipulating participants’ expected continuations. The observed-token comparison in Appendix is useful, but comparing token positions does not distinguish these explanations. The AR/base-model comparison also differs in input-noise conditions and model training, so it does not isolate future generation as the source of the advantage.

W2: The additional neuroscientific insight beyond existing predictive-model accounts remains unclear.

Future-prediction models have already been used to explain sensory cortical properties and perceptual phenomena (Singer et al., 2018; Lotter et al., 2020). More directly related, Nayebi et al. (2023) evaluated latent future-prediction models against neural activity and human physical-prediction judgments. Prior work also examined anticipatory representations of naturalistic actions (de Vries & Wurm, 2023) and cortical alignment across layers and denoising stages of image diffusion models (Takagi & Nishimoto, 2023), both acknowledged in the manuscript.

The extension to AR video diffusion models is meaningful, and I do not suggest that the specific results are already established. However, the analyses primarily identify which representations yield higher encoding accuracy; what they newly reveal about predictive content, uncertainty, or hierarchical computation remains unclear. The preference experiment adds a useful behavioral connection, but, as the authors note, it could reflect richer visual representations producing more natural-looking videos rather than clarify human predictive computation. Even if W1 were resolved, the paper would still need to explain how these findings advance neuroscientific understanding beyond existing accounts.

Questions for authors

Q1. What evidence links the encoding advantage specifically to future-predictive processing rather than improved coding of the observed video?

Q2. What new understanding of predictive visual processing emerges from the layer, denoising-step, and behavioral analyses, beyond identifying more effective encoding features?

8 comments
AuthorsOpenReview ↗

Response to Reviewer L4dx (1/2)

[W1, W2, Q1 & Q2] Past video generation as a control and the spatiotemporal hierarchy of representations.

We sincerely thank the reviewer for this important comment. Because the AR model generates the continuation conditioned on the observed video, distinguishing future information from a better organized representation of the observed video is essential for the meaning of our results and for their neuroscientific interpretation. We also agree that the AR and base models differ in input noise, training objective, and data, which limits their direct comparison. We added past video generation by the same AR model as a control and a time-resolved analysis, and showed that the representation of the AR model is oriented to the time it generates rather than to the observed video, and that past, observed, and future representations follow a temporal and hierarchical order in the visual cortex (Figure 1, Section 5.3, Figure 4).

  • Past video generation keeps everything fixed except the direction of generation. We fed the same 3 s clip to the same AR model with its frames in reverse order, so that the model generates the 1 s preceding the clip. The architecture, weights, inference method, and observed content are identical to future generation, and both run without input noise on the observed video. We verified that backward generation is as valid as forward generation within the clip (Appendix A.3, Table 1).
  • Future video generation is not merely an improved coding of the observed video (W1, Q1). If future tokens only reorganized the observed content, past and future generation should align equally with the visual cortex. However, in the variance partitioning among the three representations, future generation had the largest unique contribution in the Visual System (0.016), above observed reconstruction (0.009, p = 0.027) and past generation (0.007, p = 0.023, Figure 4b).
  • The representation of the AR model is oriented to the time it generates (Q1). The time-resolved analysis, which uses the response at each second after onset rather than one response per video, separated the past, observed, and future representations in time. Past generation peaked at 4.4 s, observed reconstruction at 5.6 s, and future generation at 7.7 s (Figure 4c). Although both are conditioned on the same observed video, past and future generation are thus oriented to the time each generates.
  • Past, observed, and future representations show a spatiotemporal order in the visual cortex (W2, Q2). Past video generation aligned first across lower- and higher-order visual cortex (3 s after video onset), observed video reconstruction then spread from lower- to higher-order visual cortex (5 s), and future video generation spread from higher- to lower-order visual cortex (6 s) (Figure 1b, Figure 4e). This order is consistent with the view that the input rapidly forms a context of the scene, from which predictions are formed in higher-order areas and sent back to lower-order areas [1, 2] (Section 6). Previous studies compared models trained to predict the future with the brain [3, 4, 5], without changing the direction of prediction within a model. Changing only the direction of generation in one model separated these representations in time and along the visual hierarchy. We have added this point to Section 6.

References

[1] Bar, A cortical mechanism for triggering top-down facilitation in visual object recognition, Journal of Cognitive Neuroscience 2003
[2] Bar et al., Top-down facilitation of visual recognition, PNAS 2006
[3] Singer et al., Sensory cortex is optimized for prediction of future input, eLife 2018
[4] Lotter et al., A neural network trained for prediction mimics diverse features of biological neurons and perception, Nature Machine Intelligence 2020
[5] Nayebi et al., Neural foundations of mental simulation: Future prediction of latent representations on dynamic scenes, NeurIPS 2023

AuthorsOpenReview ↗

Response to Reviewer L4dx (2/2)

[W2] An AR-specific link beyond layer depth.

We thank the reviewer for this valuable point. We agree that the preference experiment alone cannot clarify human predictive computation. We tested whether the relation simply reflects a rich visual representation that varies along the layers, and showed with partial correlations that it is specific to the AR model.

  • The correlation does not follow layer depth or the base model. If the relation reflected a rich visual representation along the layers, it would be explained by layer depth or by the base model, which reconstructs the observed video and has a layer profile of alignment similar to that of the AR model. With layer index regressed out, the correlation of the AR model with preference remained (r = 0.83, p = 0.014). With the profile of the base model regressed out, it decreased but stayed positive (r = 0.48, p = 0.230), whereas the correlation of the base model disappeared with the profile of the AR model regressed out (r = −0.02, p = 0.963, Table R3 in our response to Reviewer Yes1, Appendix A.9, Figure 17).

We hope these additions address the reviewer's concerns. All changes are highlighted in blue in the revised manuscript. We welcome any further questions.

Thank you for the thoughtful response and the additional analyses. I appreciate the efforts to address my concerns and would welcome clarification on the following points.

1. Temporal separation versus the timing of observed content.
The BMD dataset paper (Lahner et al., 2024, Fig. 6) showed that features from the first and last seconds of the observed videos preferentially explain earlier and later BOLD responses. Reversing the input could change whether generation features emphasize the beginning or end of the original clip. Thus, the reported temporal separation might reflect different weighting of observed content, without requiring representations of the unobserved past or future. Could you clarify how your analysis distinguishes these explanations, and whether the separation persists after accounting for features from the first and last seconds of the observed clips?

2. Interpretation of the higher-to-lower-order sequence.
The reported spatial sequence is interesting, but a temporal shift in where model features predict BOLD responses does not by itself establish that predictions are generated in higher-order regions and transmitted to lower-order regions. Regional differences in hemodynamic timing or encoding sensitivity could also influence this pattern. Could you clarify how these possibilities were considered, and distinguish the observed spatiotemporal correspondence from the hypothesized feedback mechanism?

3. What is learned about human predictive processing?
My broader concern about the neuroscientific contribution also remains. The new results appear compatible with existing predictive-processing accounts, but it is still unclear what they add to our understanding. I would like to see this framework used for more targeted analyses—for example, to identify what content humans predict, which computations explain the correspondence, or what new constraints the findings place on existing accounts. Could you clarify what specific neuroscientific insight the current results support, beyond identifying representations with better encoding performance and different spatiotemporal profiles?

While I appreciate the additional work, I am currently leaning toward maintaining my original rating of 2, primarily because my concern about the additional neuroscientific insight remains unresolved. I would welcome further clarification on these points.

Reviewer_Yes1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper examines how well video model embeddings predict fMRI responses. It compares the internal features of a diffusion model trained to generate future frames against two baselines: the autoencoder embedding of the observed video, and a diffusion model trained on plain video denoising. The authors claim the results support that future video generation is more similar to brain representations, and that amplifying layers with better brain-prediction accuracy produces videos that humans prefer.

Critical strengths
  • The paper demonstrates a clear advantage for the “predictive model.”
  • It examines a new class of models for brain alignment: autoregressive video diffusion models.
Critical weaknesses
  • The paper lacks a control with the same training objective but without future-frame prediction, for example a model trained to fill in masked intermediate or past frames. Such a control is needed to verify that future-frame prediction is what matters, as the authors claim, rather than a training objective that simply yields better embeddings for this task.
  • Encoding performance varies over a range of values depending on training objective and data, with contrastive learning at the top. The proposed approach falls below the best models with no clear advantage.
  • Following the authors' logic that future video generation encodes better and is therefore closer to what the brain does, one could equally argue that the brain's processing is most similar to V-JEPA, since it performs best.
  • The behavioral experiment is unclear, since "preference" can mean many things. In addition, the results are in no way causal and may be coincidental.
Questions for authors

It is unclear what “human preference” means in the evaluation, since it can mean many different things. A more detailed description of the human evaluation, including the exact instruction given to observers, would be helpful.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Yes1 (1/3)

[W1] Control with the same training objective but without future-frame prediction.

We sincerely thank the reviewer for this insightful suggestion. We agree that such a control allows a more rigorous comparison and a richer neuroscientific interpretation. As the reviewer suggests, the ideal control would be a model trained to fill in masked intermediate or past frames. We respectfully note, however, that the AR model is trained to predict the frames that follow the given frames, and it is therefore not well suited to generating intermediate frames, which would require conditioning on both the preceding and the following frames. We also regret to appeal that retraining the 14B AR model with a past or intermediate frame objective was not feasible within the rebuttal period. Within these constraints, we added past video generation by the same AR model as a control and a time-resolved analysis to the main results (Figure 1, Section 5.3, Figure 4), and showed that the alignment advantage comes from generating the future rather than from the training objective.

  • Past video generation. We kept the architecture, weights, and inference method of the AR model fixed and fed the 3 s clip with its frames in reverse order, so that the model generates the 1 s preceding the clip.
  • Validity of reversed frame generation. We first verified that the model still generates valid content when the frames are reversed. Because each BMD clip lasts 3 s, we used the first 2 s to generate the last 1 s (forward) and the reversed last 2 s to generate the first 1 s (backward). We then compared each generated segment with the ground truth (Table R1). The two directions did not differ significantly on any measure, which indicates that reversal does not degrade the generated content (Appendix A.3, Table 1, Figure 12).

Table R1. Forward versus backward prediction within the clip (102 BMD test clips). Mean ± SEM, p from a paired Wilcoxon signed rank test.

PSNR (dB) ↑SSIM ↑LPIPS ↓VBench imaging quality ↑VBench aesthetic quality ↑VBench subject consistency ↑VBench background consistency ↑
Forward14.70 ± 0.350.460 ± 0.0150.356 ± 0.01450.4 ± 0.94.22 ± 0.040.982 ± 0.0010.984 ± 0.001
Backward14.85 ± 0.320.453 ± 0.0150.350 ± 0.01250.7 ± 0.94.22 ± 0.030.983 ± 0.0010.984 ± 0.001
p0.570.170.760.430.640.390.31
  • Future generation aligns better than past generation and observed reconstruction. We partitioned the variance among the three representations (Section 5.3, Figure 4b). In the Visual System, the unique contribution was largest for future generation (0.016), followed by observed reconstruction (0.009, p = 0.027) and past generation (0.007, p = 0.023). Observed reconstruction contributed most in lower-order visual cortex. Toward higher-order visual cortex the contributions of past and future generation increased, and future generation was the largest. Thus, even with the training objective fixed, the direction of generation matters for brain alignment.
  • Past, observed, and future representations align in temporal order. The time-resolved analysis further separated the three representations (Figure 4c, 4d, and 4e). Past video generation aligned first across lower- and higher-order visual cortex (3 s after video onset), observed video reconstruction then spread from lower- to higher-order visual cortex (5 s), and future video generation spread from higher- to lower-order visual cortex (6 s). This order is consistent with the view that the input rapidly forms a context of the scene, from which predictions are formed in higher-order areas and sent back to lower-order areas [1, 2], as discussed in Section 6.

References

[1] Bar, A cortical mechanism for triggering top-down facilitation in visual object recognition, Journal of Cognitive Neuroscience 2003
[2] Bar et al., Top-down facilitation of visual recognition, PNAS 2006

AuthorsOpenReview ↗

Response to Reviewer Yes1 (2/3)

[W2 & W3] Comparison with the best vision model, V-JEPA2.1.

We sincerely thank the reviewer for this thoughtful comment. As the reviewer notes, V-JEPA2.1-giant aligns best with the visual cortex (0.362 in the Visual System, above 0.334 for future video generation), in line with recent alignment studies [1, 2]. We would like to respectfully clarify that our work studies how the representations of video generative models align with the visual cortex as the generation condition changes, rather than seeking the best model of the visual cortex. In contrast, V-JEPA2.1 predicts latent representations of the video rather than generating the video itself. We have made this scope explicit in the Introduction. By adding a variance partitioning against V-JEPA2.1-giant, we showed that future video generation explains variance in the visual cortex that the best vision model does not, and that this gain comes from the process of generating the future rather than from the generated video.

  • Future generation adds to V-JEPA2.1-giant, most in higher-order visual cortex. We added each representation to the V-JEPA2.1-giant representation of the observed 3 s video and measured the gain in encoding accuracy (Table R2, Appendix A.4, Figures 15 and 16). Future generation added 0.016 in the Visual System, and the gain was largest in Ventral Stream (0.027) and MT+ Complex (0.024). Past generation added 0.012, whereas observed reconstruction by the base model added no significant variance (0.002).
  • The gain comes from the generation process, not from the generated video. The V-JEPA2.1-giant representation of the 1 s future video generated by the AR model did not increase alignment and instead lowered it slightly (by 0.006). Thus the computation of the AR model while generating the future, rather than the content of the generated frames, carries the information that V-JEPA2.1-giant lacks.

Table R2. Gain in encoding accuracy over V-JEPA2.1-giant on the observed 3 s video. Exact sign flip test across ten subjects, ** p < 0.01.

+ V-JEPA2.1, generated future 1 s+ Base model, observed reconstruction+ AR model, past generation+ AR model, future generation
Visual System−0.006**+0.002+0.012**+0.016**

References

[1] Höfling et al., Only brains align with brains: Cross-region alignment patterns expose limits of normative models, ICLR 2026
[2] Tang et al., Many-two-one: Diverse representations across visual pathways emerge from a single objective, bioRxiv 2025

AuthorsOpenReview ↗

Response to Reviewer Yes1 (3/3)

[W4 & Q1] Definition of preference and whether the correlation is coincidental.

We thank the reviewer for pointing out this lack of clarity, and we apologize that our description of the human evaluation was insufficient. In short, the preference in our paper refers to the perceived naturalness of the generated continuation. Its correlation with the alignment of the AR model is not explained by layer depth or by the alignment of the base model. We have added a full description to Appendix A.9 and summarize it below.

  • Observers chose the more natural continuation. Each trial showed two 8 s videos side by side. Each video was the original 3 s stimulus followed by a 5 s continuation that the AR model generated with STG applied to one layer. The two videos showed the same stimulus and looped without sound until the response, and there was no tie option. The exact instruction was "Please choose the more natural video of the two. There is no correct answer, and please choose one even if the two videos look similar." We will use the term perceived naturalness throughout the revised manuscript.
  • The correlation is not explained by layer depth or by the base model. We fully agree that the experiment does not establish a causal relationship between neural alignment and preference. Following this comment and the related comment by Reviewer RjKX, we tested two alternative explanations of the correlation (Table R3, Appendix A.9, Figure 17). First, with layer index regressed out, the correlation between the alignment of the AR model and preference remained (r = 0.83, p = 0.014). Second, the alignment of the base model follows a similar layer profile. With each profile regressed out of the other, the correlation of the AR model decreased but stayed positive (r = 0.48, p = 0.230), whereas that of the base model disappeared (r = −0.02, p = 0.963).

Table R3. Correlation between human preference and encoding accuracy along the eight guided layers. Exact permutation test.

Alignment profileRegressed outrp
AR modelNone0.750.029
AR modelLayer depth0.830.014
AR modelBase model profile0.480.230
Base modelNone0.670.075
Base modelLayer depth0.710.051
Base modelAR model profile−0.020.963

We hope these additions address the reviewer's concerns. All changes are highlighted in blue in the revised manuscript. We welcome any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their careful reading and constructive comments. The reviewers appreciated that the paper examines a new class of models for brain alignment, autoregressive video diffusion models (Yes1), and separates the observed input, its reconstruction, and its unobserved continuation (L4dx). They found the advantage of predictive model clear (Yes1) and convincing, replicated across three model pairs (RjKX). They also valued the regional analysis of unique contributions (L4dx), the broad comparison with vision models (L4dx, RjKX), and the link to human evaluation through intervention on generation (L4dx).

The reviewers raised six shared concerns. We addressed each with new analyses or revisions, highlighted in blue in the manuscript. The main changes are a new time-resolved analysis with past video generation (Section 5.3, Figure 4) and an updated overview (Figure 1).

[Yes1, L4dx, RjKX] Can the AR and base models be compared when they differ in training and input?

We added past video generation by the same AR model as a control, which changes only the direction of generation from the future to the past. Future generation still aligned better than past generation and observed reconstruction (Figure 1, Section 5.3, Appendix A.3).

[L4dx, RjKX] Is the representation for future generation only a better readout of the observed video?

We added a time-resolved analysis that uses the response at each second after video onset. Although conditioned on the same observed video, past and future generation aligned with the visual cortex at the time each generates, with past generation peaking first, 4.4 s after video onset, and future generation later, at 7.7 s (Section 5.3, Figure 4c). The separation remained when the observed video at each second was added to the partitioning (Appendix A.11).

[L4dx] What does the work add to neuroscience?

In the time-resolved analysis, past generation aligned early across both lower- and higher-order visual cortex, and future generation aligned in higher-order before lower-order visual cortex, an order compatible with the top-down facilitation model of visual recognition (Figure 1b, Sections 5.3 and 6). In addition, clips with higher proxy surprisal computed with the AR model evoked larger cortical responses (Appendix A.10).

[Yes1] Does future generation add to the best vision model?

Future generation explained variance that V-JEPA2.1-giant did not, and this gain came from the process of generating the future rather than from the generated video (Appendix A.4).

[Yes1, L4dx, RjKX] What does the behavioral experiment measure, and is the correlation coincidental?

We added the exact instruction for the behavioral experiment and defined the measure as perceived naturalness. Partial correlations showed that the relation holds after regressing out layer depth and is specific to the AR model rather than the base model (Appendix A.9).

[RjKX] Are the Bayesian and STG interpretations supported?

We revised the Bayesian interpretation as a possible relation, and added a formal account of what STG amplifies with a revised description (Section 6, Appendix A.8).


We address each comment in detail in the individual responses below. The new control with past video generation and the time-resolved analysis, which the reviewers' comments motivated, both strengthened the main claim and revealed how past, observed, and future representations unfold in the visual cortex.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.