- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper studies the mismatch between physical predictions made by multimodal world models and the outcomes they generate in videos. The authors distinguish internal consistency from physical correctness and evaluate both using controlled physics experiments. They also investigate whether consistency-based filtering can improve downstream policy learning. The topic is relevant, and the experiments reveal some interesting failure cases. However, I have concerns about the novelty, the generalizability of the findings, and the interpretation of the downstream results
Critical strengths
- The distinction between internal consistency and physical correctness is useful. The finding that 39 out of 40 samples have correct text predictions but incorrect video outcomes is particularly interesting.
- The paper includes several meaningful controls, including incorrect conditioning instructions and action-matched filtering baselines. I also appreciate that the authors report negative results and acknowledge the limitations of their conclusions.
Critical weaknesses
- The paper mainly combines existing ideas in physical consistency evaluation and sample filtering. Related work such as GapEval, WorldModelBench, and WorldEcho/WorldSync has already explored similar problems. The proposed distance formulation and threshold-based filtering are fairly straightforward. I am not convinced that the paper offers a sufficiently distinct contribution beyond a careful empirical study.
- Most experiments use Cosmos3-Super and Cosmos3-Nano, which belong to the same model family. Furthermore, the main discrepancy is largely driven by generated videos that barely move. It is unclear whether these findings reflect a broader limitation of multimodal world models or a specific weakness of these checkpoints. Results from another model family would make the conclusions more convincing.
- Filtering improves push-planning success from 0.338 to 0.635. However, only two of the 60 generated samples are retained, both at the same action value. Once the retained action distribution is matched, the improvement over the control drops to 0.020, with a confidence interval that includes zero. This suggests that much of the gain may come from changing the training data distribution rather than selecting physically consistent outcomes. The reported reductions in label error are encouraging, but the additional benefit for task success remains unclear.
- The video reader passes only three of six magnitude validation checks, and some analytic references differ from simulator outcomes. These issues raise questions about the reliability of the quantitative alignment results, particularly for the T2/T3 experiments.
Questions for authors
- What specific insight does this framework provide that existing physical consistency benchmarks cannot capture? A more direct comparison with related work would help clarify the novelty.
- Can the authors validate the main findings on another world model and provide stronger evidence that consistency-based filtering improves downstream performance when the action distribution is controlled?
Thank you for pointing out additional experiments that helped clarify details in our work. Your two suggestions, a second model family and a filtering test with the action distribution held fixed, produced the strongest new results in the revision (see appendix Q additions with highlighted comments).
Q1. What does the framework capture that existing benchmarks cannot? Physical benchmarks such as WorldModelBench, VideoPhy and Physics-IQ score a video for plausibility or against a reference. GapEval and RealUnify score understanding and generation against reference answers, on images. WorldEcho and WorldSync diagnose action following and align generation for policy improvement, but they do not separate the model’s own forecast from an external reference. None of these measures one physical outcome in both the text and the video of the same model against a simulator, and none tests what happens when agreement between the two is used to choose training data.
Three results show why that matters. Agreement alone cannot tell failures apart: Nano’s 31 disagreeing pairs are 14 video failures, 4 text failures and 13 where both are wrong (Q.11). A motion metric misses the problem: VBench’s dynamic-degree accepts 19 clips that are physically wrong (Q.10). And an agreement filter with a biased forecast admits only wrong examples (Q.9). We agree that the distance and the threshold are simple, and we claim no new algorithm; the contribution is what the study shows.
Q2a. Another family, and “barely moving”. We agree: in this setting, near-static video looks like a Cosmos trait. We ran LTX-2 and LTX-2.5, two models from another family, on the same prompts, frames and seeds. They move the cube in 78/96 and 85/94 clips, against 7/36 for Cosmos3-Super on the new commands. Moving is not the same as being right, though: 71–82% of their videos miss a forecast that is correct (Q.1, Q.14). Stating the distance in the prompt does not help (6/48 pass; Q.3). The two families also fail in opposite directions. Given a wrong “does not swing” prompt, Cosmos stops the pendulum in 11/16 clips, while LTX-2.5 swings it in all 16 (Q.8). And inside LTX-2, the text encoder that conditions the video answers all 96 prompts correctly, while the video misses on 80 (Q.4).
Q2b. Filtering with the action distribution controlled. You noted that the original gate kept only two rows, both at one action. On LTX-2.5 it keeps 26 rows across 15 commands, and 23 of them are correct, so the matched control has real alternatives to choose from. That control still takes its action profile from the gate. Against it, the gate wins by +0.099 [+0.069, +0.130]. When the forecasts are shuffled across commands the gain disappears (mean +0.007; p = .010). An independent reader gives +0.104 (Q.14). The gain holds on new goals and actions (+0.086; Q.13) and with five other learners (post hoc; Q.14), and 100 newly generated clips replicate it (+0.055 and +0.041 with the two readers; Q.16). With the original nearest-neighbour learner and little clean data, gated video also beats clean data alone (+0.335), while unfiltered video makes the planner worse. On Cosmos the same comparison is still unresolved (+0.006 [−0.029, +0.038]; Q.2).
W4. Reader reliability. For Cosmos, the conclusion does not depend on measuring distance well, because the cube does not move. A second reader and a blinded VLM judge, validated on 34 planted clips, both confirm this (Q.6, Q.7). The three-of-six magnitude checks do limit the T2/T3 magnitude comparisons, and those stay qualified as in the paper. Where analytic and simulator references differ we report them separately, and the main push test uses executed endpoints. For LTX, distance does matter. Our reader is within 25% on all 18 planted clips of known travel, tumbles included, and an independent reader re-scored every LTX clip (Q.14). The gate result holds with both (+0.099 and +0.104), and every robustness check passes with each.