acceptodds
Under review as a conference paper at ICLR 2027

Scaling Video Generation for Reasoning: At What Cost?

Abstract

We study whether scaling video generation enables models to reason about hidden information from the past frames, and at what computational cost. Our controlled benchmark requires predicting nine prescribed moves of an initially solved Rubik's Cube from a fixed view of three faces. Correct predictions require inferring how actions change hidden states, and the simulator provides exact ground truth for evaluation. Models learn plausible cube geometry early, while correct sticker configurations require substantially more training. Although validation MSE follows approximate power-law scaling, lower MSE loss does not reliably indicate downstream reasoning capabilities. Smaller autoregressive models achieve higher state accuracy with limited compute, while larger models reach higher accuracy after more training. At roughly 0.1 PF-days, the 70M-parameter model correctly predicts the visible sticker configuration in 44.6% of post-action frames, compared with 0.3% for the 1B model, which reaches 83.7% at 3.14 PF-days. Symbolic state supervision raise the 20M model's frame accuracy from 31.1% to 67.3% at the same training-data budget, suggesting that learning representations of state changes can complement scaling.

Primary area:
generative models
Keywords:
Video Models, Scaling Laws, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_24xSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how model size and training compute affect state correctness in action-conditioned video generation. Using a controlled nine-move 2x2x2 Rubik’s Cube task, the authors train autoregressive and bidirectional video models from approximately 20M to 1B parameters. They evaluate action execution and exact sticker configurations alongside validation flow MSE, finding that lower MSE does not reliably imply better state prediction across model sizes.

Discrete-state prediction and full-observation experiments help investigate the difficulty of the task. The authors also introduce symbolic state supervision with predicted-state feedback, improving a 20M model’s average frame accuracy from 31.1% to 67.3% at the same training-data budget.

Critical strengths
  1. A systematic study that measures state correctness rather than relying on generation loss or visual plausibility. The paper compares seven model sizes using a shared training stream and separates action execution, sticker accuracy, exact-frame accuracy, and full-trajectory success. This makes the loss–accuracy mismatch substantially more convincing than a collection of failure examples. The finding is consequential: selecting a video model by validation loss alone can lead to a worse model for accurate state prediction.
  2. The paper provides a constructive intervention, not just a negative result about scaling. Symbolic state supervision and predicted-state feedback produce a large improvement at matched data exposure, without supplying simulator states during inference. The discrete-state and full-observation experiments further support investigating state representations as a way to improve video prediction. Together, these results provide a useful direction beyond simply increasing model size or training duration.
Critical weaknesses
  1. The benchmark does not clearly separate visual-history reasoning from fixed-initial-state action execution. Every episode starts from the same solved cube and provides the complete action sequence upfront, so future states can in principle be computed from the action program without consulting intermediate visual observations. This remains a meaningful state-prediction task, but it does not establish whether models recover and maintain hidden information from visual history—the capability motivating the study. A controlled variant with different initial configurations, fully revealed through a short observation sequence before returning to the three-face view, would make visual information necessary. Combining this with an ablation of visual-history conditioning would better support the hidden-state reasoning interpretation.

  2. Training coverage is not separated from the difficulty of predicting later states. The paper excludes overlap between complete training and evaluation action sequences, but does not report coverage of their shorter prefixes or state–action pairs. Under the sampling scheme, short action prefixes are expected to recur much more frequently than long ones. Consequently, declining accuracy at later actions could reflect less familiar action combinations and states, alongside longer-term state tracking and accumulated errors. This makes it difficult to determine how much of the observed decline specifically concerns hidden-state reasoning. Reporting prefix and state–action coverage, then comparing accuracy across coverage-stratified subsets at fixed action positions, would help distinguish these explanations. Applying this analysis to the existing ground-truth-history diagnostic would further reduce the influence of accumulated generation errors.

  3. The state-guidance improvement lacks ablations that identify what drives the gain. The intervention simultaneously adds state supervision, predicted-state feedback, and an additional state-prediction pass. Front12 and Full24 only change which sticker predictions are fed back; both retain full-state supervision and the additional pass. The comparisons therefore establish the benefit of the combined intervention, but not whether feedback is necessary or auxiliary supervision alone explains most of the improvement. A baseline trained with the same state supervision but without feedback would isolate the feedback mechanism’s contribution. A compute-matched video-only baseline would also clarify whether the gains reflect a more effective learning method rather than simply additional computation.

Questions for authors

see weakness.

3 comments

We thank the reviewer for the detailed and constructive comments.


W1 – Fixed initial state – We design this on purpose. We will define "hidden" and reword the abstract to make this clear.

We reached the same conclusion as the reviewer, and we take the comment as a call to write it more clearly in the paper. The fixed initial state is the stated design of the benchmark. We chose this so that the task has no stochasticity and every generated frame can be scored against exact ground truth. A random initial state observed through the same three-face view would leave the hidden stickers undetermined. Resolving that uncertainty calls for active perception with a decision-making model and requires interaction with the world. This is great future work on video planning or world action models, which are beyond the scope of this paper. We thank the reviewer for pointing this out, and we will make it clear in the introduction, method, and future work.

We totally agree with the reviewer's comment on the claim. In the revised version, we will 1) clearly define "hidden" as information that becomes hidden after the first frame, when the actions move stickers out of view, not information that is hidden before the video starts. The initial state is fixed and known; 2) in the introduction and method, emphasize that no result in the paper requires recovering initially unknown information; 3) to further prevent this reading, replace "reason about hidden information from the past frames" in the abstract with "track how prescribed actions change a hidden state".


W2 – Training coverage – True, coverage decreases. However, we find no evidence that it drives the accuracy decline.

We agree that coverage decreases as the video goes on. We think this is the nature of any long-horizon task: the number of possible action sequences multiplies with every action, so no training set can cover long sequences, and the model has to learn the dynamics rather than recall videos.

Every episode starts from the same solved cube, so an evaluation episode's first j frames were seen in training exactly when some training video begins with the same j actions. With 5M training videos, of our 100 evaluation episodes:

First j actions1–4567–9
Episodes whose prefix appears in training10098320

The question is whether this explains the accuracy decline. Action 6 is the only step where seen and unseen prefixes can be compared at the same position:

Frame accuracy (5M videos)Action 6, seen (32 out of 100 ep.)Action 6, unseen (68 out of 100 ep.)Action 7 (none seen)
20M0%3%0%
30M3%9%1%
70M31%31%8%
120M22%18%6%
270M59%50%28%
550M53%54%31%
1B66%65%47%

Seen and unseen prefixes give the same accuracy: averaged over models the gap is +0.7 points (95% interval −10.5 to +12.2). The larger models also remain well above zero at action 7, where no prefix was seen in training. So coverage does decrease, but we find no evidence that it is what drives the decline over actions.


W3 – What drives the state-guidance gain – The gain comes from feeding the predicted state back, not from the auxiliary loss or the extra computation.

Supervision without feedback. We trained the same 20M model with the same state supervision and the same extra state pass as Front12, but with all predicted states masked before they reach the video pass. Frame accuracy at matched training data:

Training videosVideo onlySupervision only (new)Front12Full24
524k1.9%8.9%8.3%9.4%
1.0M19.0%19.0%21.9%24.1%
1.5M24.9%23.8%34.7%36.4%
2.0M27.2%28.1%52.4%48.9%
2.5M29.6%30.7%60.3%57.4%
3.0M31.1%31.3%67.3%62.9%

Supervision alone stays at the level of video-only training, while both feedback variants pull ahead from 1.5M videos. Since the supervision-only model has the same parameters and the same additional pass, the gain comes from feeding the predicted state back, not from the auxiliary loss or the extra computation.

Compute-matched video-only baseline. The guided model uses about 12% more training compute than video-only at the same data. Giving the video-only 20M model that compute (3.4M videos) yields 31.8%, against 67.3% for Front12. Even with 2.4x the compute (8M videos) it reaches 39.4%.

Thank you for the detailed response and additional experiments. However, my main concern remains unresolved: rewording the claims clarifies the limitation but also narrows the paper’s original contribution, without establishing the claimed visual-history reasoning capability. Given the highly constrained task and the unclear generalizability of the findings to broader video reasoning settings, I will maintain my original score.

Thank you for clarifying your concern! Could you clarify whether your concern about visual-history reasoning is primarily due to our use of a single known initial state?

As we explained in our response, our motivation is to study whether the model can learn to track the effects of actions from data, under the assumption that its knowledge comes from the training data. If the same action history can start from different initial states without the initial state being specified, the resulting state would no longer be uniquely determined by that history. We therefore fix the initial state to isolate the capability we study. We would like to emphasize that the objective of this paper is to study state tracking and reasoning, not decision-making.

More importantly, we believe the simplicity of the setting strengthens our conclusion: even in this highly controlled task, where we can generate essentially unlimited training data, the model still struggles to learn the task. While we agree that broader settings would be valuable to study, this result in a simple setting is itself a strong finding about the limits of current video models.

We appreciate your valuable feedback and hope that our clarification has helped address your concerns. We would be glad to continue the discussion and are happy to provide any further clarification that may be helpful.

Reviewer_Ls85OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how model size and training data affect video models’ ability to predict state changes in a Rubik’s Cube. It finds that lower validation loss does not always mean more accurate state prediction. Larger models improve accuracy with more training, while explicit state supervision and feedback help smaller models achieve better results.

Critical strengths
  1. The experiments are extensive and well organized, covering model size, training data, architecture, and generation strategy.

  2. The simulator provides exact ground truth. The evaluation distinguishes correct actions, correct sticker colors, and fully correct trajectories.

  3. The comparisons with symbolic prediction, full observation, and ground-truth history help explain where prediction errors come from.

Critical weaknesses
  1. Section 3.3 / Figure 7: The curves are extended far beyond the tested compute range to estimate the cost of reaching 98% accuracy. The estimates change dramatically depending on which points are used for fitting. Although the authors acknowledge this, the actual compute needed to reach high accuracy remains unclear.

  2. Section 3.4 / Figure 9: The method adds both state supervision and predicted-state feedback. The experiments show that this combination helps, but do not separate the contributions of these two components.

Questions for authors
  1. For Figure 7, can the authors test their extrapolation method by fitting earlier checkpoints and checking whether it accurately predicts later measured results?

  2. For Figure 9, how much improvement comes from adding state supervision alone, without feeding predicted states back into video generation?

2 comments

We thank the reviewer for the positive and constructive comments.


W1, Q1 – Testing the extrapolation in Fig. 7 – We ran the suggested test. We will state the cost only as a lower bound.

We agree that the fit is not quantitatively reliable, which is why the paper calls it illustrative. During the rebuttal we ran the test the reviewer suggests: fit the earlier checkpoints, then predict the later measured results.

AR accuracy at our largest run (3.14 PF-days)Predicted from earlier fitMeasured
Average frame accuracy73.8%83.7%
Full-trajectory accuracy17%36%

The earlier fit underestimates the later results, but this window dependence is a known property of compute frontiers. For example, In the famous Chinchilla paper[1] Appendix E, it reports that fits to the first, middle and final thirds of their frontier give different projections.

We ran an additional resampling experiment to see what can still be said. We resampled the 100 evaluation episodes 2,000 times and refit each time. The upper end of the estimate is unbounded, but the lower end is stable: in every resample, 98% full-trajectory accuracy needs more than 2x the compute of our largest run, and in 80% of resamples more than 10x. We will report both experiments in the revision and state the cost only as a lower bound, not an exact value.


W2, Q2 – State supervision without feedback – We ran this ablation. The gain comes from feeding the predicted state back, not from the supervision itself.

We ran this ablation. It is the same 20M model with the same state supervision as Front12, but all predicted states are masked before they reach the video pass, so the video generator receives no state feedback. Frame accuracy at matched training data:

Training videosVideo onlySupervision only (new)Front12Full24
524k1.9%8.9%8.3%9.4%
1.0M19.0%19.0%21.9%24.1%
1.5M24.9%23.8%34.7%36.4%
2.0M27.2%28.1%52.4%48.9%
2.5M29.6%30.7%60.3%57.4%
3.0M31.1%31.3%67.3%62.9%

Supervision alone helps the model start earlier (524k), but the gain does not last: from 1.0M videos it stays at the level of video-only training, while both feedback variants pull ahead. So the improvement in Fig. 9 comes from feeding the predicted state back into video generation, not from the state supervision itself.

[1] Hoffmann, Jordan, et al. "An empirical analysis of compute-optimal large language model training." Advances in neural information processing systems 35 (2022): 30016-30030.

Thank you for the detailed response and the additional experiments.

Fig. 7. I remain unsure whether fits from smaller or earlier runs can reliably predict larger-scale performance, since the table shows a notable gap (17% predicted vs. 36% measured). If possible, please add a larger-scale run to validate the extrapolation.

Fig. 9. The supervision-only ablation separates the two components and resolves my question. It is an informative result that the gain comes from predicted-state feedback rather than the supervision itself, so I suggest adding this ablation to the main paper.

My concerns are mostly resolved, and I maintain my positive assessment.

Reviewer_QUjwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how video-generation training compute and validation flow loss relate to exact state correctness. Episodes start from a solved Rubik's Cube, provide nine prescribed turns, and expose only part of the cube. The authors compare autoregressive and bidirectional video DiTs across approximately 20M–1B parameters and score turns, stickers, exact post-action frames, and complete trajectories. They also examine vector-state prediction, visibility, rollout history, and supervised sticker guidance. The main findings are a mismatch between cross-model validation-loss rankings and free-rollout state accuracy, and substantial accuracy and estimated training-compute gains from predicted-state guidance.

Critical strengths
  1. Separating sticker, exact-frame, and full-trajectory correctness makes failures measurable even when videos look plausible. At the same exposure of 1.5M training videos, the 1B autoregressive model has lower validation MSE than the 70M model, yet their average exact-frame accuracies are 3.67% and 41.00%, respectively (Section 3.3; Fig. 8). Validation uses ground-truth history while accuracy measures free rollouts. This is useful evidence that the training objective can rank these models poorly on state consistency under the reported recipe; it does not imply that loss is uninformative within each model.

  2. At 3M videos, 20M Front12 reaches 67.3% exact-frame accuracy versus 31.1% for video-only training. It also attains 67.3% using 0.0445 PF-days, comparable to 67.1% for the 550M video-only model at 1.33 PF-days—about 30 times more estimated generator-training compute (Section 3.4). The estimate includes the state-query pass and feedback adapter, but excludes frozen encoders, optimizer/EMA updates, validation, and sampling (Appendices A.4 and E). Front12 and Full24 both supervise all 24 stickers; their comparison isolates whether predicted hidden-sticker distributions also feed the video pass (Fig. 9). WildWorld’s StateCtrl-AR baseline uses supervised world-state prediction and autoregressive state rollouts to condition game-video generation (Li et al. 2026). The present paper's particular value is its precise state scoring and controlled compute/accuracy measurements.

Critical weaknesses
  1. Every episode starts from the same solved state with a fixed color convention, and the complete action program is supplied before generation (Section 2.1). These inputs determine all future sticker states, including temporarily invisible stickers. Correct videos establish action-conditioned state consistency, but cannot distinguish recovering initially unknown hidden information from propagating a known initial state through the action program. The visibility and ground-truth-history diagnostics are informative, yet do not resolve this ambiguity: the six-face experiment also changes the visual target and scoring, and the prefix intervention supplies true history. Limiting the central claim to fixed-initial-state, nine-action consistency would better match the evidence; broader inference claims need a diagnostic that actually requires recovering unknown state.

  2. The 98% full-trajectory intersections are 248.9 and 1.425×10¹⁴ PF-days for two fitting windows, each using only six frontier points and extending far beyond measured compute (Fig. 7; Table 13). The same 100 episodes support repeated checkpoint/architecture evaluation, with one-percentage-point resolution for full-trajectory accuracy. The paper appropriately calls the intersections illustrative, but they cannot establish a numerical cost of reliable trajectories. In addition, batch size 16 and learning rate 4×10⁻⁴ are transferred from a short 35M pilot to the full size range; Yin et al. (2025) use scale-dependent learning-rate and batch-size choices in their published video-DiT scaling study. Thus the key inversion is established for this recipe, while its broader scaling interpretation remains uncertain. These limitations of central claims lead me narrowly toward rejection despite the useful measurements and guidance result; narrower claims and targeted sensitivity evidence could change that assessment.

References

Li, Zhen, Zian Meng, Shuwei Shi, Wenshuo Peng, Yuwei Wu, Bo Zheng, Chuanhao Li, and Kaipeng Zhang. 2026. “WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG.” arXiv preprint, arXiv:2603.23497. https://arxiv.org/abs/2603.23497.

Yin, Yuanyang, Yaqi Zhao, Mingwu Zheng, Ke Lin, Jiarong Ou, Rui Chen, Victor Shea-Jay Huang, Jiahao Wang, Xin Tao, Pengfei Wan, Di Zhang, Baoqun Yin, Wentao Zhang, and Kun Gai. 2025. “Towards Precise Scaling Laws for Video Diffusion Transformers.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://openaccess.thecvf.com/content/CVPR2025/html/Yin_Towards_Precise_Scaling_Laws_for_Video_Diffusion_Transformers_CVPR_2025_paper.html.

Questions for authors
  1. Will you limit the main conclusion to fixed-initial-state, nine-action state consistency? If you retain the broader hidden-state inference claim, which result requires recovering initially unknown information from visual observations?

  2. How stable are the trajectory fits and 98% intersections to resampling evaluation episodes and changing fit windows? A clear separation of measured results from extrapolation in the abstract and Fig. 1 would also materially improve my assessment of the cost claim.

  3. Does the 70M/1B loss-versus-accuracy inversion at 1.5M videos persist under a small set of scale-appropriate learning-rate and batch-size settings? This would help distinguish the general scaling interpretation from relative optimization under the shared recipe.

1 comment

We thank the reviewer for the positive and constructive comments.


W1, Q1 – Scope of the main claim – We agree. We will define "hidden" and reword the claim in the abstract.

We reached the same conclusion as the reviewer, and we take the comment as a call to write it more clearly in the paper. The fixed initial state is the stated design of the benchmark. We chose this so that the task has no stochasticity and every generated frame can be scored against exact ground truth. A random initial state observed through the same three-face view would leave the hidden stickers undetermined. Resolving that uncertainty calls for active perception with a decision-making model and requires interaction with the world. This is great future work on video planning or world action models, which are beyond the scope of this paper. We thank the reviewer for pointing this out, and we will make it clear in the introduction, method, and future work.

We totally agree with the reviewer's comment on the claim. In the revised version, we will 1) clearly define "hidden" as information that becomes hidden after the first frame, when the actions move stickers out of view, not information that is hidden before the video starts. The initial state is fixed and known; 2) in the introduction and method, emphasize that no result in the paper requires recovering initially unknown information; 3) to further prevent this reading, replace "reason about hidden information from the past frames" in the abstract with "track how prescribed actions change a hidden state".


W2, Q2, Q3 – The cost extrapolation and the shared training recipe – We agree. We will state the cost only as a lower bound, and the paper's recipe is still the best of five for the 70M model.

Extrapolation. We agree that the 98% intersections are not reliable numbers. We resampled the evaluation episodes 2,000 times and refit each time. The upper end of the estimate is unbounded, but the lower end is stable: in every resample, 98% full-trajectory accuracy needs more than 2x the compute of our largest run, and in 80% of resamples more than 10x. Changing the fit window gives the same picture: the smallest estimate over all windows is still 10x our largest run. Our largest run reaches 36%. We will revise the paper to state the cost only as a lower bound, not an exact value.

Our conclusions do not depend on the precision of the 100 evaluation episodes. The standard error is at most 2.1 points for frame accuracy and 4.8 points for full-trajectory accuracy across all checkpoints, while the gaps we report are far larger (e.g. 41.0% vs. 3.7% for 70M vs. 1B at 1.5M videos). Due to the time constraint, we cannot re-evaluate all checkpoints with 1000 episodes, but we will add this in the final version.

Shared recipe. We agree that an extensive hyperparameter search is the most conservative way. Given finite time and compute, whether scaling results depend on per-scale tuning is an open question in the community: DiT ([1], Table 4) uses one learning rate and batch size for all model sizes, and we followed this protocol. Within the rebuttal window, only the 70M runs have trained long enough to compare. We varied its learning rate and batch size on the same data:

70M modelLRBatchFrame acc. at 524k videos
Paper recipe4e-41625.8%
Lower LR2e-41618.1%
Higher LR8e-4161.3%
Larger batch4e-4320.1%
Lower LR, larger batch2e-4320.0%

The recipe used in the paper is the best of the five, so the 70M side of the comparison was not under-tuned. Due to time, we cannot train it to 1.5M videos and the same runs for the 1B model are in progress. We will provide the result in the revised paper.

[1] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.