- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how model size and training compute affect state correctness in action-conditioned video generation. Using a controlled nine-move 2x2x2 Rubik’s Cube task, the authors train autoregressive and bidirectional video models from approximately 20M to 1B parameters. They evaluate action execution and exact sticker configurations alongside validation flow MSE, finding that lower MSE does not reliably imply better state prediction across model sizes.
Discrete-state prediction and full-observation experiments help investigate the difficulty of the task. The authors also introduce symbolic state supervision with predicted-state feedback, improving a 20M model’s average frame accuracy from 31.1% to 67.3% at the same training-data budget.
Critical strengths
- A systematic study that measures state correctness rather than relying on generation loss or visual plausibility. The paper compares seven model sizes using a shared training stream and separates action execution, sticker accuracy, exact-frame accuracy, and full-trajectory success. This makes the loss–accuracy mismatch substantially more convincing than a collection of failure examples. The finding is consequential: selecting a video model by validation loss alone can lead to a worse model for accurate state prediction.
- The paper provides a constructive intervention, not just a negative result about scaling. Symbolic state supervision and predicted-state feedback produce a large improvement at matched data exposure, without supplying simulator states during inference. The discrete-state and full-observation experiments further support investigating state representations as a way to improve video prediction. Together, these results provide a useful direction beyond simply increasing model size or training duration.
Critical weaknesses
-
The benchmark does not clearly separate visual-history reasoning from fixed-initial-state action execution. Every episode starts from the same solved cube and provides the complete action sequence upfront, so future states can in principle be computed from the action program without consulting intermediate visual observations. This remains a meaningful state-prediction task, but it does not establish whether models recover and maintain hidden information from visual history—the capability motivating the study. A controlled variant with different initial configurations, fully revealed through a short observation sequence before returning to the three-face view, would make visual information necessary. Combining this with an ablation of visual-history conditioning would better support the hidden-state reasoning interpretation.
-
Training coverage is not separated from the difficulty of predicting later states. The paper excludes overlap between complete training and evaluation action sequences, but does not report coverage of their shorter prefixes or state–action pairs. Under the sampling scheme, short action prefixes are expected to recur much more frequently than long ones. Consequently, declining accuracy at later actions could reflect less familiar action combinations and states, alongside longer-term state tracking and accumulated errors. This makes it difficult to determine how much of the observed decline specifically concerns hidden-state reasoning. Reporting prefix and state–action coverage, then comparing accuracy across coverage-stratified subsets at fixed action positions, would help distinguish these explanations. Applying this analysis to the existing ground-truth-history diagnostic would further reduce the influence of accumulated generation errors.
-
The state-guidance improvement lacks ablations that identify what drives the gain. The intervention simultaneously adds state supervision, predicted-state feedback, and an additional state-prediction pass. Front12 and Full24 only change which sticker predictions are fed back; both retain full-state supervision and the additional pass. The comparisons therefore establish the benefit of the combined intervention, but not whether feedback is necessary or auxiliary supervision alone explains most of the improvement. A baseline trained with the same state supervision but without feedback would isolate the feedback mechanism’s contribution. A compute-matched video-only baseline would also clarify whether the gains reflect a more effective learning method rather than simply additional computation.
Questions for authors
see weakness.
3 comments
Thank you for the detailed response and additional experiments. However, my main concern remains unresolved: rewording the claims clarifies the limitation but also narrows the paper’s original contribution, without establishing the claimed visual-history reasoning capability. Given the highly constrained task and the unclear generalizability of the findings to broader video reasoning settings, I will maintain my original score.
Thank you for clarifying your concern! Could you clarify whether your concern about visual-history reasoning is primarily due to our use of a single known initial state?
As we explained in our response, our motivation is to study whether the model can learn to track the effects of actions from data, under the assumption that its knowledge comes from the training data. If the same action history can start from different initial states without the initial state being specified, the resulting state would no longer be uniquely determined by that history. We therefore fix the initial state to isolate the capability we study. We would like to emphasize that the objective of this paper is to study state tracking and reasoning, not decision-making.
More importantly, we believe the simplicity of the setting strengthens our conclusion: even in this highly controlled task, where we can generate essentially unlimited training data, the model still struggles to learn the task. While we agree that broader settings would be valuable to study, this result in a simple setting is itself a strong finding about the limits of current video models.
We appreciate your valuable feedback and hope that our clarification has helped address your concerns. We would be glad to continue the discussion and are happy to provide any further clarification that may be helpful.
We thank the reviewer for the detailed and constructive comments.
W1 – Fixed initial state– We design this on purpose. We will define "hidden" and reword the abstract to make this clear.We reached the same conclusion as the reviewer, and we take the comment as a call to write it more clearly in the paper. The fixed initial state is the stated design of the benchmark. We chose this so that the task has no stochasticity and every generated frame can be scored against exact ground truth. A random initial state observed through the same three-face view would leave the hidden stickers undetermined. Resolving that uncertainty calls for active perception with a decision-making model and requires interaction with the world. This is great future work on video planning or world action models, which are beyond the scope of this paper. We thank the reviewer for pointing this out, and we will make it clear in the introduction, method, and future work.
We totally agree with the reviewer's comment on the claim. In the revised version, we will 1) clearly define "hidden" as information that becomes hidden after the first frame, when the actions move stickers out of view, not information that is hidden before the video starts. The initial state is fixed and known; 2) in the introduction and method, emphasize that no result in the paper requires recovering initially unknown information; 3) to further prevent this reading, replace "reason about hidden information from the past frames" in the abstract with "track how prescribed actions change a hidden state".
W2 – Training coverage– True, coverage decreases. However, we find no evidence that it drives the accuracy decline.We agree that coverage decreases as the video goes on. We think this is the nature of any long-horizon task: the number of possible action sequences multiplies with every action, so no training set can cover long sequences, and the model has to learn the dynamics rather than recall videos.
Every episode starts from the same solved cube, so an evaluation episode's first j frames were seen in training exactly when some training video begins with the same j actions. With 5M training videos, of our 100 evaluation episodes:
The question is whether this explains the accuracy decline. Action 6 is the only step where seen and unseen prefixes can be compared at the same position:
Seen and unseen prefixes give the same accuracy: averaged over models the gap is +0.7 points (95% interval −10.5 to +12.2). The larger models also remain well above zero at action 7, where no prefix was seen in training. So coverage does decrease, but we find no evidence that it is what drives the decline over actions.
W3 – What drives the state-guidance gain– The gain comes from feeding the predicted state back, not from the auxiliary loss or the extra computation.Supervision without feedback. We trained the same 20M model with the same state supervision and the same extra state pass as Front12, but with all predicted states masked before they reach the video pass. Frame accuracy at matched training data:
Supervision alone stays at the level of video-only training, while both feedback variants pull ahead from 1.5M videos. Since the supervision-only model has the same parameters and the same additional pass, the gain comes from feeding the predicted state back, not from the auxiliary loss or the extra computation.
Compute-matched video-only baseline. The guided model uses about 12% more training compute than video-only at the same data. Giving the video-only 20M model that compute (3.4M videos) yields 31.8%, against 67.3% for Front12. Even with 2.4x the compute (8M videos) it reaches 39.4%.