acceptodds
Under review as a conference paper at ICLR 2027

STRIKE: Learning Visual State Transitions for Physical World Modeling

Abstract

Physical world modeling requires predicting how interactions change a scene, not merely generating coherent motion. We propose STRIKE, a framework that separates visual state transition learning from dense video generation. We construct event-aligned supervision by extracting observed states from training videos and pairing them with transition descriptions and temporal offsets. An image-based transition model learns to predict the next scene configuration from the current image, a local transition specification, and elapsed time. At inference, a pre-trained vision-language planner predicts time transition specifications, and recursive application of the learned transition model produces a sequence of future visual states. A separately trained dynamic model then generates the complete rollout conditioned on these states and their temporal locations. Experiments on Physics-IQ Verified, PhyGenBench, Pisa-Experiments, and RoboTwin2.0 show improvements over the corresponding video-backbone baselines in benchmark measures of physical consistency and manipulation-video fidelity. These results support learned visual state transitions as an effective intermediate representation for physical world modeling.

Primary area:
generative models
Keywords:
Physical World Model, Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BLCAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed STRIKE to help video generation models handle physical interactions by explicitly predicting how a scene changes step-by-step rather than guessing entire motions at once. E.g., when picking up a cube, the model maps out the timing of events like contact and lifting, then predicts the visual state after each interaction, and finally use such states to guide the diffusion model to synthesize the final sequence. This approach allows STRIKE to significantly improve physical consistency and video fidelity across benchmarks, e.g., Physics-IQ Verified and RoboTwin 2.0.

Critical strengths
  1. Explicitly modeling interaction outcomes before generating the video is a reasonable approach. The separation between state-transition prediction and video dynamics is also a clear structure.

  2. STRIKE improves performance across physical-consistency, object-motion, and robotic-manipulation evaluations, e.g., 76.91 on PhyGenBench (vs. 54.79 for the CogVideoX baseline).

  3. Good ablation studies, e.g., training the transition model improves its state-prediction quality vs. using the pretrained image-editing model directly. I also appreciate the temporal conditioning, first-frame anchoring, and synthetic training data studies.

  4. Also exhaustive results on both real-world and simulated data.

Critical weaknesses
  1. Weak novelty because the idea of planning events, predicting intermediate visual states, and conditioning video generators on those states is already proposed in papers like Key World and CausalMotion. The counterargument could be the introduction of transition model, however, it is not clear on how much gain does is introduce from learning transitions rather than simply providing better intermediate visual conditions to the video generator.

  2. Unfair comparison from model size perspective. STRIKE combines a 20B transition model with a 5B video generator (25B+ in total), whereas several baselines use much smaller models.

  3. The error from each autoregressive state prediction could accumulate. While the paper acknowledges this limitation, a systematic rollout-horizon study or length-dependent evaluation would confirm how severely these errors impact performance over longer sequences.

  4. The role and reliability of the event planner requires further evaluation. Even a strong transition model may generate incorrect videos if the event sequence or timing is wrong, and an evaluation comparing predicted event plans against reference plans would confirm the planner's reliability.

Questions for authors
  1. How much improvement comes from the transition model vs. simply conditioning on intermediate states?

  2. How does performance degrade as the number of predicted transitions increases?

  3. How sensitive is the transition model to the train/test domain gap. E.g., if the training is purely synthetic data, would the result be much worse?

5 comments
AuthorsOpenReview ↗

Response to Reviewer BLCA (1/4)

We thank the reviewer for finding the explicit modeling of interaction outcomes reasonable and the decomposition clear, and for appreciating the ablations and the breadth of real and simulated evaluations. We address each point below.

W1: We do not claim keyframe-guided generation itself as new. Our contribution is a learned transition unit: one shared model, trained on event-aligned state pairs, that maps (current state, local transition specification, elapsed time) → next state. It is composed recursively, and its timed outputs condition a video model that generates the full window jointly.

  • KeyWorld generates keyframes from the initial observation and a single global task description, without per-transition specifications or elapsed-time conditioning. Our new 2×2 ablation (global vs. local descriptions, event-aligned vs. uniform states, matched data) tests this design choice inside our pipeline.
  • CausalMotion guides a pretrained generator with VLM-planned keyframes and trajectories without training. STRIKE outperforms it on all three shared benchmarks: Physics-IQ 41.9 vs. 12.7, PhyGenBench 76.91 vs. 70.21, Pisa L2 / CD / IoU 0.0949 / 0.2495 / 0.1753 vs. 0.1292 / 0.3435 / 0.0966.

How much comes from learning transitions: Table 6 isolates exactly this. With planner, video model, parameters, and FLOPs held fixed, intermediate states from the pretrained editor vs. our learned transition model give:

Physics-IQPhyGenBenchPisa-IoU
CogVideoX-5B-I2V (no states)30.5054.790.1481
+ (a) intermediate states, pretrained editor32.6072.500.1224
+ (b) intermediate states, learned transitions (STRIKE)41.9376.910.1753
+ (c) intermediate states, newer off-the-shelf editor (Qwen-Image-2.1, untrained)31.9676.040.1515
+ (d) intermediate states, Qwen-Image-2.1 trained with our data41.4377.500.1687

Comparing (a) and (b), (c) and (d), which (b) and (d) provide intermediate states using pretrained models, our learned transition model perform better in all three benchmarks.

W2: We thank the reviewer for raising this point. We agree that model size matters for a fair comparison; this is why Table 1 reports both parameter counts and generation cost (EFLOPs) for every method. We believe the comparison does not favor STRIKE, for three reasons.

1. With the same parameters, other systems do much worse. CoECT [1] uses the same composition as STRIKE, a 20B image editor plus a 5B video generator. Yet it scores 19.9 on Physics-IQ, vs. 41.9 for STRIKE, and needs about 3× the generation cost (29.6 vs. 10.02 EFLOPs). The gain therefore comes from what the transition model learns, not from its parameter count.

2. Much larger systems are not clearly better. Cosmos3-Super has 64B parameters, 2.6× STRIKE's 25B, yet scores only 0.8 points higher (42.7 vs. 41.9). MiniMax-H3 has 33B parameters and scores 39.8, 2.1 points below STRIKE. STRIKE is thus on par with systems well above its size.

3. Smaller baselines put their capacity into the video model; STRIKE puts it into an image model, which is cheaper to train and to run. Most baselines below 25B parameters rely on a large video backbone: Wan2.2 (14B), Cosmos3-Nano (16B), LTX-Video (13B), Open-Sora v2 (11B). STRIKE instead places most of its parameters in an image-space transition model and keeps the video model small (5B, or 2B).

Model (Physics-IQ, Table 1)# paramsEFLOPsScore
Cosmos3-Super64B–42.7
MiniMax-H333B–39.8
CoECT20B + 5B29.619.9
Cosmos3-Nano16B11.827.6
Wan2.214B49.134.2
STRIKE (CogVideoX-5B)20B + 5B10.0241.9
STRIKE (Wan2.2-5B)20B + 5B4.1939.7

Within the same model family, STRIKE with Wan2.2-5B outperforms Wan2.2-14B (39.7 vs. 34.2) at 1/12 of the generation cost (4.19 vs. 49.1 EFLOPs). On VideoPhy, STRIKE with CogVideoX-2B obtains the best physical-commonsense score (32.9). This is above its own 2B backbone (27.6) and above all 11–16B video models: Open-Sora v2 20.2, LTX-Video 20.9, WISA 18.0, Cosmos3-Nano 19.5. Its semantic adherence is comparable (70.4 vs. 70.5 for WISA-14B).

In addition, An image model trains on single frames instead of long clips, so sequences are far shorter, memory is lower and batches are larger. A 20B image model is thus much cheaper to train than a video model of similar size, and it runs only once per state at inference.

In short, STRIKE does not win by being larger. Systems of the same size do much worse, much larger systems are not clearly better, and the smaller baselines spend their capacity on the video model, which is the more expensive place to put it.

AuthorsOpenReview ↗

Response to Reviewer BLCA (2/4)

W3: We that two facts bound the issue on our benchmarks:

  • Rollouts are short. On Physics-IQ the planner specifies 1, 2, or 3 future states for 39, 138, and 21 of 198 videos (mean 1.91).
  • The design limits drift. The first-frame sink keeps the input image as a persistent reference (without it, Physics-IQ drops to 40.60 and Pisa-IoU to 0.1528, Table 7), and states are soft conditions.

We added two analyses on our validation dataset:

  1. Step-wise state accuracy on validation dataset (VAL). We find that adding first-frame sink in the transition model during can significantly mitigate the step-wise error accumulation.
States CountPSNR Rec.PSNR TFPSNR Rec. w/o sinkSSIM Rec.SSIM TFSSIM Rec. w/o sinkLPIPS Rec.LPIPS TFLPIPS Rec. w/o sinkΔ\Delta PSNR Rec.Δ\Delta PSNR Rec. w/o sink
122.3322.3322.350.7830.7830.7820.2170.2170.2210.00+0.02
221.2123.2220.850.7520.7810.7440.2670.2200.279−2.01−2.37
319.5722.1318.570.7120.7580.6900.3280.2540.355−2.56−3.56
415.6218.6514.910.6270.6790.5780.4310.3080.473−3.04−3.75
  1. In addition, we ran the following analyses. The first three columns use our validation videos (VAL) with annotated reference plans and states. We report S-IoU, WS-IoU, ST-IoU against the real videos inspired by Physics-IQ verified benchmark. For the benchmark dataset, we extract state frames from original video clips by running our data curation pipeline.
State condition supplied to the video modelS-IoUWS-IoUST-IoUPhysics-IQ
(a) Reference states0.6930.5240.33663.23
(b) Generated, teacher-forced0.5930.3940.22145.59
(c) Generated, recursive w/o sink0.4970.3180.20140.60
(d) Generated, recursive (STRIKE)0.5300.3410.21141.93

Here Reference States means using the ground truth state images as condition for video generation, which is set as an upper bound of the system. Our conclusion is that error accumulation is measurable but accounts for a minority of that headroom.Teacher-forced (b) states improve over our recursive (d) ones by +0.063 S-IoU, about 39% of the oracle gap; the rest is single-step error. For Physics-IQ, the teacher-forced states only advance ours +3.66 on benchmark score, which only accounts for 17% of the difference compared with reference states. In addition, adding first frame sink (comparing (c) and (d)) can effectively mitigate the error accumulation problem.

AuthorsOpenReview ↗

Response to Reviewer BLCA (3/4)

W4: We thank the reviewer for this question. We answer it in two parts: (1) how accurate GPT-5.5's plans are, measured against reference plans and against the real videos; (2) how much its errors cost in the generated videos.

1. How accurate is the planner on our own validation dataset?

(a) Against the reference plan . For every pair of predicted and reference states of our validation dataset, a judging VLM (Claude Opus 5.5) scores whether the two describe the same physical state (same entity, same phase of the process, same outcome), with the matching rule is that pairs scoring ≥ 0.75 with no incompatibility are matched one-to-one. The evaluation results are shown below:

MetricPlanner with GPT 5.5
Event-count (exact-K rate)87.5%
Event precision0.73
Pairwise order agreement on matched events0.99
Mean timestamp error on matched events0.45 s

(b) Against the real video. A judging VLM (Opus 5.5) sees the real clip and one list of states, without times or source. For each state it decides (1) whether the state occurs (yes / partial / no); (2) when it first becomes visible. It also lists clearly visible state changes that the list misses. The results are as follows:

Validation clipsGPT-5.5 planReference annotationΔ
State occurs: yes / partial / no64.5% / 25.4% / 10.1%83.9% / 13.7% / 2.4%"yes" −19.4 pts
Timing error vs. observed onset (states judged "yes")1.11 s0.55 s+0.57 s
Visible state changes not covered, per clip1.831.63+0.20

Our conclusions are that GPT-5.5 gets the order of events right, and its states are realized fully or partly 90% of the time; 10% do not occur. It covers nearly as many visible events as the annotation with its timing is about 0.6 s coarser. However, the planner sees only the first frame, so these are prediction errors, not perception errors.

2. How do planning errors affect generation?

(a) Predicted vs. reference plan, everything else fixed. On the same validation clips we generate each video twice: once from the reference plan and once from GPT-5.5's plan. The transition model, video model and seed are identical. We conduct comparison experiments with the real continuation:

  • Motion-mask IoU, using Physics-IQ's mask algorithm. S-IoU measures where motion occurs; WS-IoU also weights how often it occurs.
  • A VLM judge (Claude Opus 5.5, blind to which plan was used). It compares real and generated frames at the same times and scores event, final-state and timing agreement in [0, 1].

For scale, the table includes a run that uses real frames as states (the best score can be achieved).The results are shown in the following table:

PlanStatesS-IoUWS-IoUVLM judge
GPT-5.5 (predicted)generated0.5110.3160.326
Referencegenerated0.5300.3410.364
Referencereal frames0.6930.525–

Reading from the table, our conclusions are as follows:

  • Planning gap (GPT-5.5 plan − reference plan): S-IoU −0.019, WS-IoU −0.025, VLM judge −0.038.
  • State gap (real frames − generated states): S-IoU +0.163, about 8× the planning gap. The remaining headroom lies mainly in state generation, not in planning.
  • Per clip. Clips with worse plans get slightly worse videos.

(b) Swapping the planner on Physics-IQ.

  • Only the plans change; the transition model, video model, seed and scorer are fixed.
  • Δ is paired vs. GPT-5.5 over the 66 scenarios.
PlannerPhysics-IQΔ vs. GPT-5.5
None (CogVideoX-5B-I2V backbone)30.5–
GPT-5.5, (paper)41.93–
Qwen3-VL-8B39.94−1.99
Qwen2.5-VL-7B35.52−6.41

Both Qwen2.5-VL and Qwen3-VL perform worse than GPT5.5 on physics-IQ benchmark

To summarize:

  • Accuracy. GPT-5.5 orders events correctly (0.99 pairwise agreement), and 90% of its planned states actually occur, fully or partly, in the real video.
  • Effect on generation. These errors cost little. Replacing the reference plan with GPT-5.5's plan lowers spatial IoU by only 0.019, one-eighth of what imperfect state generation costs (0.163). Controlled timing, order and drop errors change Physics-IQ by at most 1.7 points, and none of these changes is significant.
  • Choice of planner. Large losses (6 points) appear in other planner (Qwen2.5-VL) compared with more advanced VLM (GPT 5.5)
Reviewer_oEJEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STRIKE predicts future visual states before generating a video. It is trained on event-aligned frame pairs selected from videos by a VLM, using the current image, a local transition description, and a time interval to predict the next state. At inference, a planner takes the initial image and task description and specifies a sequence of events and their timing. The transition model follows this plan recursively, and its predictions provide soft guidance for a separately trained model that generates the complete video. Experiments on physical video generation and robotic manipulation benchmarks show improvements over the evaluated baselines.

Critical strengths

Strong experimental performance, with consistent gains across backbones and benchmarks. STRIKE substantially improves physical consistency with both CogVideoX and Wan, while also improving trajectory and geometric metrics on Pisa. Ablation studies on transition-model training further show improvements in both state prediction quality and downstream video generation quality.

Critical weaknesses
  1. Error propagation remains underexplored. Both models use ground-truth state conditions during training but generated states at inference. Step-wise error analysis and controlled tests of downstream sensitivity to state errors are missing.
  2. The gains from event-aligned sampling and local transition descriptions are not isolated. Table 6 supports the benefit of fine-tuning, but matched ablations of these two choices are missing.
Questions for authors
  1. Could you test uniform frame sampling and global task descriptions separately, using the same data sources? This would help show how much event selection and local descriptions each contribute.
  2. How does state accuracy change over successive prediction steps? Does using reference states instead of generated states noticeably improve the final video?
  3. How are predicted states matched to reference frames in Table 6, especially if their event times differ? Do both variants use the same video-model checkpoint?
  4. Do the EFLOPs in Table 1 include planning, all state-generation steps, and video generation? How many states are typically generated per video?
3 comments
AuthorsOpenReview ↗

Response to Reviewer oEJE (1/2)

We thank the reviewer for the positive assessment of our experimental results, consistent gains across backbones and benchmarks, and the transition-training ablations. We address each point below.

W1: Two facts bound the issue on our benchmarks:

  • Rollouts are short. On Physics-IQ the planner specifies 1, 2, or 3 future states for 39, 138, and 21 of 198 videos (mean 1.91).
  • The design limits drift. The first-frame sink keeps the input image as a persistent reference (without it, Physics-IQ drops to 40.60 and Pisa-IoU to 0.1528, Table 7), and states are soft conditions.

We added two analyses on our validation dataset:

  1. Step-wise state accuracy on validation dataset (VAL). We find that adding first-frame sink in the transition model during can significantly mitigate the step-wise error accumulation.
States CountPSNR Rec.PSNR TFPSNR Rec. w/o sinkSSIM Rec.SSIM TFSSIM Rec. w/o sinkLPIPS Rec.LPIPS TFLPIPS Rec. w/o sinkΔ\Delta PSNR Rec.Δ\Delta PSNR Rec. w/o sink
122.3322.3322.350.7830.7830.7820.2170.2170.2210.00+0.02
221.2123.2220.850.7520.7810.7440.2670.2200.279−2.01−2.37
319.5722.1318.570.7120.7580.6900.3280.2540.355−2.56−3.56
415.6218.6514.910.6270.6790.5780.4310.3080.473−3.04−3.75
  1. In addition, we ran the following analyses. The first three columns use our validation videos (VAL) with annotated reference plans and states. We report S-IoU, WS-IoU, ST-IoU against the real videos inspired by Physics-IQ verified benchmark. For the benchmark dataset, we extract state frames from original video clips by running our data curation pipeline.
State condition supplied to the video modelS-IoUWS-IoUST-IoUPhysics-IQ
(a) Reference states0.6930.5240.33663.23
(b) Generated, teacher-forced0.5930.3940.22145.59
(c) Generated, recursive w/o sink0.4970.3180.20140.60
(d) Generated, recursive (STRIKE)0.5300.3410.21141.93

Here Reference States means using the ground truth state images as condition for video generation, which is set as an upper bound of the system. Our conclusion is that error accumulation is measurable but accounts for a minority of that headroom.Teacher-forced (b) states improve over our recursive (d) ones by +0.063 S-IoU, about 39% of the oracle gap; the rest is single-step error. For Physics-IQ, the teacher-forced states only advance ours +3.66 on benchmark score, which only accounts for 17% of the difference compared with reference states. In addition, adding first frame sink (comparing (c) and (d)) can effectively mitigate the error accumulation problem.

W2: We agree that Table 6 isolates fine-tuning but not these two choices. We will run an ablation study built from the same source videos, with matched pair counts and training budget, and evaluated with the same video model and event plans:

StatesTask Spec.Physics-IQPhyGenBenchPisa-IoU
Event-alignedLocal41.9376.910.1753
Event-alignedGlobal32.0856.460.1667
Uniform SamplingGlobal30.0457.290.1615

Here, Uniform Sampling means that each video keeps the same number of states as its event-aligned annotation, placed at uniform intervals, and Global means the video caption replaces uku_k in both training and inference, analogous to a KeyWorld-style global-task condition. Including our original specification (top row), we run 2 more experiments: (1) Event-aligned states x Global: this experiments isolate the contribution of transition states, and (2) Uniformly sampled states x Global video caption: this experiments set as a baseline by ablating both event states and transition descriptions.

AuthorsOpenReview ↗

Response to Reviewer oEJE (2/2)

Q3:

  • Predicted States Matching. Each predicted state is compared with the reference frame that the annotation pipeline selected for the same transition: the same index kk in the annotated sequence, at the annotated timestamp. Table 6 drives the transition model with the annotated transition descriptions and timestamps (no planner), recursively from the real first frame. Prediction and reference therefore correspond to the same event and the same time by construction, and no temporal alignment is required. The initial input image is excluded.
  • Checkpoints. Yes. In each benchmark column, the two rows use the same video dynamics checkpoint (CogVideoX-5B state-conditioned), the same GPT-5.5 event plans, the same seed, and the same evaluation protocol. They differ only in the transition-model weights (pretrained Qwen-Image-Edit vs. ours). We will state this in the caption.

Q4:

  • EFLOPs. The reported EFLOPs are full-benchmark totals (198 videos for Physics-IQ). They include all denoising steps of the transition model (40 steps, both CFG passes) for every generated state, video generation including the memory tokens of every state slot, and VAE decoding. The planner is an external API call; we report its cost separately. It is one call per Physics-IQ scenario.

  • States per video. On Physics-IQ the planner specifies 1, 2, or 3 future states for 39, 138, and 21 of the 198 videos (mean 1.91). Terminal hold anchors (111/198) reuse the previous image and are not generated.

BenchmarkVideosGenerated states per video, histogramMean K
Physics-IQ1981: 39, 2: 138, 3: 211.91
PhyGenBench1602: 152, 3: 82.05
Pisa4211: 17, 2: 399, 3: 51.97
RoboTwin2 (val)5002: 161, 3: 175, 4: 85, 5: 31, 7: 39, 8: 5, 9: 43.38
  • Revision. We will add the distribution of KK to the appendix.

We thank the reviewer again and are happy to provide further details.

Thanks for the clarification and detailed additional experiments. I’ll keep my rating.

Reviewer_U7peOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the ability of video generation models to understand and learn state changes induced by physical interactions. The authors propose STRIKE, a framework designed to enable video generation models to go beyond producing visually smooth motions and instead capture the underlying physical interactions between objects. Specifically, STRIKE first employs a VLM-based event planner to predict key physical events and their corresponding timestamps. A visual transition model then recursively predicts the visual states resulting from these events. Finally, a Video Dynamics Model generates the complete video sequence conditioned on the predicted key states and their associated timestamps. Experiments on multiple benchmarks, including Physics-IQ Verified, PhyGenBench, Pisa-Experiments, and RoboTwin2.0, demonstrate that STRIKE outperforms the corresponding video backbone baselines in terms of physical consistency and the fidelity of robotic manipulation videos.

Critical strengths
  1. Figure 1 effectively illustrates an important challenge in current video generation models: understanding state changes caused by physical interactions. Although existing methods can generate visually realistic motions, they often fail to comply with physical laws. The figure provides an intuitive and compelling motivation for addressing this limitation.

  2. STRIKE separates physical state transitions from continuous motion generation, allowing the model to explicitly predict the outcomes of physical events. Its reusable local transition model predicts each subsequent state conditioned on the previous state and the corresponding event.

  3. The authors use a VLM to extract key physical events, their associated timestamps, and visual states before and after each event from raw videos. This enables the automatic construction of state-transition training data, providing direct supervision for learning physical interactions beyond video-level text descriptions.

Critical weaknesses
  1. Limited novelty. Event-guided video generation has been explored in prior work. CoECT models causal relationships between events and generates keyframes through progressive image editing, while RoboEnvision similarly decomposes tasks into instructions and generates corresponding keyframes. The novelty of STRIKE over these approaches remains unclear.

  2. Unclear source of improvements. Although Table 6 demonstrates the benefits of state-transition learning, STRIKE also introduces a powerful state generation model and further trains the video backbone. It remains unclear whether the improvements over the original 5B backbone stem from state-transition learning or simply from the additional model capacity and training.

  3. Lack of robustness analysis. As noted in Appendix A.4, the video model is trained on ground-truth states but conditioned on generated states during inference. The authors do not analyze its robustness to state prediction errors, which may accumulate and degrade generation quality.

Questions for authors
  1. STRIKE relies on GPT-5.5 for event and timestamp prediction, but the paper mainly evaluates final video quality. Could the authors quantify the planner's accuracy and analyze how prediction errors affect downstream generation?

  2. Some existing approaches enforce physical consistency through explicit physical laws or mathematical constraints. Why does STRIKE favor learned state transitions over such physics-based constraints, and what are the advantages of this choice?

  3. STRIKE achieves improvements across four benchmarks but additionally employs an approximately 20B-parameter image state model. Could the authors report the computational overhead and provide comparisons under matched parameter or compute budgets to better assess the significance of these gains?

4 comments
AuthorsOpenReview ↗

Response to Reviewer U7pe (1/3)

We thank the reviewer for the constructive feedback and for appreciating the motivation (Fig. 1), the separation of state transitions from continuous motion generation, and the automatic construction of transition supervision from raw videos. We address each point below.

W1: We agree that event- and keyframe-guided generation has been explored, and we do not claim it as our contribution. The difference lies in what is learned and how it is used:

  1. A learned, reusable transition unit. STRIKE trains a single transition model qϕ(sk∣sk−1,s0,uk,Δτk)q_\phi(s_k \mid s_{k-1}, s_0, u_k, \Delta\tau_k) on event-aligned state pairs mined from raw videos, directly supervising how a local transition specification changes the current scene. CoECT constructs event keyframes through event-chain reasoning and iterative editing with a pretrained editor, without transition-specific training.
  2. Elapsed-time conditioning and recursive composition. The interval Δτk\Delta\tau_k is injected through the temporal RoPE coordinates, and the same model is applied recursively with a first-frame sink that preserves identity and geometry.
  3. Joint conditioned dynamics. RoboEnvision decomposes manipulation tasks into atomic instructions, generates boundary keyframes, and fills each intervening segment. Our video model generates the entire window jointly, with predicted states as timestamped memories, so motion in one interval can depend on later outcomes.
  4. Scope. STRIKE targets open-domain physical consequences (impact, fracture, overflow, deformation, optics) that are not commanded actions and must be predicted, and uses the same interface for robotic manipulation.
  5. Empirical results. With learning transitions, our pipeline improves Physics-IQ from 32.60 to 41.93, PhyGenBench from 72.50 to 76.91, and Pisa-IoU from 0.1224 to 0.1753.

W2: Table 6 already contains a control in which capacity is held fixed. We present it as a decomposition:

ConfigurationPhysics-IQPhyGenBenchPisa-IoU
(a) CogVideoX-5B-I2V30.554.790.1481
(b) + state-conditioned video model, states from pretrained Qwen-Image-Edit32.6072.500.1224
(c) + learned transition model (STRIKE)41.9376.910.1753
(d) CogVideoX-5B fine-tuned on the same video corpus and schedule, without states27.2759.580.1327

Existing video-only controls. On RoboTwin2 the backbone baselines in our table are already fine-tuned on the same 112K training episodes, with the same configuration as our video model; only the state memory is removed. STRIKE still improves EWMScore by +2.47 (CogVideoX) and +3.37 (Wan). Video fine-tuning alone therefore does not account for the gains.

W3: We think that there two facts bound the issue on our benchmarks:

  • Rollouts are short. On Physics-IQ the planner specifies 1, 2, or 3 future states for 39, 138, and 21 of the 198 videos (mean 1.91) and the physical process on the other benchmarks including PhyGenBench and PisaBench also have 3 or less states.
  • The design limits drift. The first-frame sink keeps the input image as a persistent reference (removing it lowers Physics-IQ to 40.60 and Pisa-IoU to 0.1528, Table 7), and states act as soft conditions rather than hard frames.

In addition, we ran the following analyses. The first three columns use our validation videos (VAL) with annotated reference plans and states. We report S-IoU, WS-IoU, ST-IoU against the real videos inspired by Physics-IQ verified benchmark. For the benchmark dataset, we extract state frames from original video clips by running our data curation pipeline.

State condition supplied to the video modelS-IoUWS-IoUST-IoUPhysics-IQ
(a) Reference states0.6930.5240.33663.23
(b) Generated, teacher-forced0.5930.3940.22145.59
(c) Generated, recursive w/o sink0.4970.3180.20140.60
(d) Generated, recursive (STRIKE)0.5300.3410.21141.93

Here Reference States means using the ground truth state images as condition for video generation, which is set as an upper bound of the system. Our conclusion is that error accumulation is measurable but accounts for a minority of that headroom.Teacher-forced (b) states improve over our recursive (d) ones by +0.063 S-IoU, about 39% of the oracle gap; the rest is single-step error. For Physics-IQ, the teacher-forced states only advance ours +3.66 on benchmark score, which only accounts for 17% of the difference compared with reference states. In addition, adding first frame sink (comparing (c) and (d)) can effectively mitigate the error accumulation problem.

AuthorsOpenReview ↗

Response to Reviewer U7pe (2/3)

Q1: We thank the reviewer for this question. We answer it in two parts: (1) how accurate GPT-5.5's plans are, measured against reference plans and against the real videos; (2) how much its errors cost in the generated videos.

1. How accurate is the planner on our own validation dataset?

(a) Against the reference plan . For every pair of predicted and reference states of our validation dataset, a judging VLM (Claude Opus 5.5) scores whether the two describe the same physical state (same entity, same phase of the process, same outcome), with the matching rule is that pairs scoring ≥ 0.75 with no incompatibility are matched one-to-one. The evaluation results are shown below:

MetricPlanner with GPT 5.5
Event-count (exact-K rate)87.5%
Event precision0.73
Pairwise order agreement on matched events0.99
Mean timestamp error on matched events0.45 s

(b) Against the real video. A judging VLM (Opus 5.5) sees the real clip and one list of states, without times or source. For each state it decides (1) whether the state occurs (yes / partial / no); (2) when it first becomes visible. It also lists clearly visible state changes that the list misses. The results are as follows:

Validation clipsGPT-5.5 planReference annotationΔ
State occurs: yes / partial / no64.5% / 25.4% / 10.1%83.9% / 13.7% / 2.4%"yes" −19.4 pts
Timing error vs. observed onset (states judged "yes")1.11 s0.55 s+0.57 s
Visible state changes not covered, per clip1.831.63+0.20

Our conclusions are that GPT-5.5 gets the order of events right, and its states are realized fully or partly 90% of the time; 10% do not occur. It covers nearly as many visible events as the annotation with its timing is about 0.6 s coarser. However, the planner sees only the first frame, so these are prediction errors, not perception errors.

2. How do planning errors affect generation?

(a) Predicted vs. reference plan, everything else fixed. On the same validation clips we generate each video twice: once from the reference plan and once from GPT-5.5's plan. The transition model, video model and seed are identical. We conduct comparison experiments with the real continuation:

  • Motion-mask IoU, using Physics-IQ's mask algorithm. S-IoU measures where motion occurs; WS-IoU also weights how often it occurs.
  • A VLM judge (Claude Opus 5.5, blind to which plan was used). It compares real and generated frames at the same times and scores event, final-state and timing agreement in [0, 1].

For scale, the table includes a run that uses real frames as states (the best score can be achieved).The results are shown in the following table:

PlanStatesS-IoUWS-IoUVLM judge
GPT-5.5 (predicted)generated0.5110.3160.326
Referencegenerated0.5300.3410.364
Referencereal frames0.6930.525–

Reading from the table, our conclusions are as follows:

  • Planning gap (GPT-5.5 plan − reference plan): S-IoU −0.019, WS-IoU −0.025, VLM judge −0.038.
  • State gap (real frames − generated states): S-IoU +0.163, about 8× the planning gap. The remaining headroom lies mainly in state generation, not in planning.
  • Per clip. Clips with worse plans get slightly worse videos.

(b) Swapping the planner on Physics-IQ.

  • Only the plans change; the transition model, video model, seed and scorer are fixed.
  • Δ is paired vs. GPT-5.5 over the 66 scenarios.
PlannerPhysics-IQΔ vs. GPT-5.5
None (CogVideoX-5B-I2V backbone)30.5–
GPT-5.5, (paper)41.93–
Qwen3-VL-8B39.94−1.99
Qwen2.5-VL-7B35.52−6.41

Both Qwen2.5-VL and Qwen3-VL perform worse than GPT5.5 on physics-IQ benchmark

To summarize:

  • Accuracy. GPT-5.5 orders events correctly (0.99 pairwise agreement), and 90% of its planned states actually occur, fully or partly, in the real video.
  • Effect on generation. These errors cost little. Replacing the reference plan with GPT-5.5's plan lowers spatial IoU by only 0.019, one-eighth of what imperfect state generation costs (0.163). Controlled timing, order and drop errors change Physics-IQ by at most 1.7 points, and none of these changes is significant.
  • Choice of planner. Large losses (6 points) appear in other planner (Qwen2.5-VL) compared with more advanced VLM (GPT 5.5)
AuthorsOpenReview ↗

Response to Reviewer U7pe (3/3)

Q2: Our design choices of STRIKE have the following advantages over other methods:

  1. Coverage. Physics-IQ spans solid mechanics, fluids, optics, thermodynamics, and magnetism, and PhyGenBench covers 27 laws. Constraint or simulation-based methods typically require a simulator for a specific phenomenon class (rigid, articulated, elastoplastic) together with geometry, mass, friction, and material parameters estimated from a single RGB frame. Many target events, e.g. a mug fracturing, a napkin soaking, light refracting, a flame extinguishing, do not have a tractable simulator from one image.
  2. Scalable supervision. Transitions are mined from raw videos without physical-parameter labels, so coverage grows with data.
  3. Complementary, not exclusive. The interface (sk−1,uk,Δτk)→sk(s_{k-1}, u_k, \Delta\tau_k) \to s_k does not depend on how a state is produced. When a simulator is available (e.g., rigid-body drops), simulator-derived states or timestamps can be supplied without changing the video model.

Q3: Table 1 reports parameters and full-benchmark EFLOPs (198 videos), and contains several matched or favorable comparisons:

  • Matched parameters and FLOPs: Table 6, (b) vs. (c) above: 32.60 → 41.93.
  • Same 20B+5B scale: CoECT [1] with similar number of parameters scores 19.9 at 29.6 EFLOPs; STRIKE scores 41.9 at 10.02 EFLOPs.
  • Scaling the video model instead: Wan2.2-14B scores 34.2 at 49.1 EFLOPs; STRIKE (Wan2.2-5B) scores 39.7 at 4.19 EFLOPs, i.e., +5.5 points at ~8.5% of the compute.
  • Smaller video backbone: 20B transition + 2B CogVideoX remains competitive with 11–16B video models (Appendix B: PhyGenBench 65.68, VideoPhy PC 32.9).

[1] Wang, Zixuan, et al. "Chain of event-centric causal thought for physically plausible video generation." arXiv preprint arXiv:2603.09094 (2026).

Thank you for the additional ablations and planner analysis. They clarify the contribution of learned state transitions and show how planning errors affect video generation. My concern about long-horizon robustness remains, since the experiments predict at most three future states per video and leave open how errors accumulate over longer sequences. I therefore maintain my weak reject rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful reading, thoughtful questions, and constructive feedback. Your comments have helped us clarify our contribution, strengthen the empirical analysis, and improve the presentation of our evaluation protocols and limitations. We have incorporated these revisions into the updated manuscript, with the main additions highlighted in light blue. The principal updates are:

  • Clarified contribution and positioning (Section 2). We clarify the distinction between our learned, event-conditioned transition model and existing keyframe-guided generation methods, and discuss its relationship to physics-based approaches.
  • Expanded transition-model evaluation (Section 5.3). We report state-image accuracy alongside downstream video quality and clarify the validation setting, annotated event/timestamp conditions, and event-index matching protocol.
  • Analyzed recursive errors and state conditioning (Section 5.4, Tables 7–8). We compare recursive prediction, teacher forcing, and prediction without the first-frame sink. We also evaluate downstream generation using reference and generated states to examine error accumulation, the benefit of anchoring, and the remaining state-quality gap.
  • Evaluated planner reliability and downstream impact (Appendix C.1). We assess event and timing accuracy, compare planned states against actual video continuations, and examine alternative planners. Controlled comparisons between predicted and reference plans further quantify their influence on video generation.
  • Added a synthetic-only training ablation (Section 5.3, Table 5). We compare synthetic-only training with real-only and mixed-data training on Physics-IQ and Pisa to better characterize the contribution of synthetic data and its transfer to real-world evaluation.
  • Clarified computational cost and rollout lengths (Appendix C.2). We specify which generation components are included in the reported EFLOPs and which costs are excluded, and report future-state-count distributions across the evaluated benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.