- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies probabilistic alignment in video-based world modeling: whether repeated generations under the same initial observation and action reproduce the distribution of possible physical outcomes. It introduces PAWBench, comprising 50 scenarios across eight mechanism groups, divided into PAW-Calibration, where reference outcome probabilities are specified, and PAW-Coverage, where valid outcomes can be enumerated but their probabilities are not reliably known. PAWEval maps generated videos to terminal outcome labels and reports conditional total variation distance or outcome coverage, together with a separate scene pass rate.
The evaluation covers twelve video generators with 50 rollouts per scenario. It finds substantial discrepancies from reference probabilities and incomplete outcome recovery under the evaluated budget. Supporting analyses include matched finite-sample simulations, an alternative outcome evaluator, human judgments, real-image inputs, and paired causal and non-causal interventions. The paper also studies language-based steering, marginal-preserving noise coupling, and LoRA training with controlled outcome proportions. These experiments distinguish target selection, target realization, finite-budget exploration, and changes to the learned output distribution. The main contribution is a broader diagnostic benchmark and an empirical characterization of probabilistic failure modes in current video generators.
Critical strengths
-
The paper makes an important evaluation question concrete and provides a useful extension of existing benchmarks. Whether a model can produce a plausible future, recover alternative outcomes, and assign those outcomes appropriate probabilities are distinct questions with direct relevance to using video generators for prediction and planning. The separation between Calibration and Coverage is well motivated: the benchmark evaluates probability mass where a reference can be justified, while avoiding arbitrary probability assignments elsewhere. Although the repeated-rollout perspective is related to prior work such as CaliBench, the expansion to 50 scenarios across eight mechanisms, together with the accompanying diagnostics, constitutes a meaningful contribution. This gives the work value beyond the performance of any particular model in the reported comparisons.
-
The empirical investigation provides substantive validation and useful distinctions between possible interventions. The matched Monte Carlo analysis is particularly convincing: it preserves each model’s passing scenes and readable sample counts, showing that the observed average TVD of 30.2 is substantially larger than the reference-sampling mean of 8.26. The alternative evaluator and human study provide complementary support for the broad findings. The intervention experiments further distinguish errors in selecting outcomes from errors in realizing them, and distinguish improvements in finite-budget exploration from changes to the learned distribution. In particular, the paper correctly explains that marginal-preserving noise coupling does not change the generator’s underlying single-rollout distribution. These analyses make the study informative beyond a leaderboard of low scores.
Critical weaknesses
-
Human validation does not cover all important readout errors. The reported human comparison only includes videos with a human majority label and successful readouts from both VLMs. It therefore does not measure how often readable videos are incorrectly rejected. Overall label agreement also does not show whether particular outcomes are systematically confused with others. These errors could distort outcome frequencies, and even a few wrong labels could create or remove rare categories in Coverage. For representative model–scene pairs, please human-label all 50 videos, including rejected ones, report each evaluator’s errors by outcome, and compare the resulting TVD, Coverage, and scene pass rates with the automatic scores.
-
Intervention comparisons average over different sets of scenes. TVD and Coverage are averaged over each condition’s own passing scenes. As acknowledged in Appendix F.1, changes in these averages can reflect scenes entering or leaving the calculation, as well as changes in the generated outcome distributions. This makes it difficult to determine how much an intervention actually improves or worsens performance on the same scenes. Please add paired comparisons on scenes that pass in both conditions, while separately reporting newly passing and newly failing scenes. A supplementary Coverage comparison using equal numbers of readable videos would also help distinguish outcome diversity from differences in readout success.
-
Reference probabilities need clearer justification for individual scenarios. The paper explains that references come from symmetry, sector proportions, outcome counts, or visible physical state, but does not provide a complete account of the assumptions for each calibration scenario. For example, translating spinner sector sizes into outcome probabilities requires assumptions about how the spin is randomized. Since TVD directly depends on these references, unclear assumptions make it harder to attribute a mismatch to the video model.
Questions for authors
See weakness.
7 comments
Table R17. Evaluator errors relative to the human majority on the galleries in Table R16 (Gemini / Qwen). These are disagreements with a crowd majority rather than adjudicated errors.
| Scene | Clear outcome rejected | Cannot tell accepted | Category disagreement |
|---|---|---|---|
| Ball into numbered bins | 5/42 / 3/42 | 0/3 / 0/3 | 0/37 / 2/39 |
| Coin flip | 0/35 / 2/35 | 3/3 / 1/3 | 1/35 / 1/33 |
| Ball tossed toward a cup | 0/45 / 0/45 | — | 0/45 / 0/45 |
| Bottle flip | 9/39 / 1/39 | 1/1 / 1/1 | 0/30 / 0/38 |
| Dog pulled by leash near bowl | 4/38 / 0/38 | — | 4/34 / 5/38 |
| Tape peeled from cardboard | 1/8 / 0/8 | — | 7/7 / 4/8 |
| All | 19/207 / 6/207 | 4/7 / 2/7 | 12/188 / 12/201 |
Together, these analyses show that rejections mostly coincide with videos that humans cannot read, that outcome confusions concentrate in a few mechanism groups, and that readout errors neither create the calibration gap nor inflate Coverage in these galleries.
W2: Intervention comparisons on shared scenes.
We thank the reviewer for this suggestion. Table R18 compares each intervention with its Base condition on the scenes that pass in both conditions, for both Calibration and Coverage, using the original rollouts. It also lists the scenes that newly pass or newly fail under each intervention, and the change in Coverage after matching the number of readable videos.
On shared scenes, the main conclusions of Sec. 5 hold. Oracle PE lowers TVD and raises Coverage for every generator, whereas neither PE nor Native PE significantly lowers TVD for any generator. Consistent with Sec. 5.2, C2C raises Coverage for all three generators, significantly for Cosmos 3 Super I2V; its effect on TVD varies.
Interventions mostly add passing scenes rather than remove them. Across the 13 comparisons, scenes newly pass 66 times in Calibration and 18 times in Coverage, while only 3 newly fail in each track. Part of the change in the original averages therefore comes from scenes entering the calculation, which the shared-scene comparison removes.
To separate outcome diversity from readout success, we subsample both conditions in each shared scene to the same number of readable videos (5,000 repetitions) and recompute Coverage. The direction of the change is preserved in 12 of the 13 comparisons; the exception is Wan2.2 with PE, where both estimates are close to zero (+0.66 and −0.39). The gains under Oracle PE and C2C remain after matching, so they reflect a broader set of outcomes rather than more readable videos. We will add Table R18 to the revised appendix.
Table R18. Intervention minus Base on scenes passing in both conditions, with 95% paired bootstrap intervals. Shared: Calibration / Coverage scenes. Equal readable: Coverage change after matching readable counts. Enter / leave: newly passing / failing scenes.
| Model | Intervention | Shared | Δ TVD ↓ [95% CI] | Δ Coverage ↑ [95% CI] | Δ Coverage, equal readable | Enter / leave |
|---|---|---|---|---|---|---|
| Wan2.2 | PE | 16 / 23 | −0.53 [−11.47, 8.63] | +0.66 [−11.95, 11.99] | −0.39 | 7 / 0; 2 / 0 |
| Cosmos 3 Super I2V | PE | 20 / 23 | +12.11 [2.55, 21.53] | +11.49 [0.71, 22.80] | +10.41 | 3 / 0; 2 / 0 |
| MiniMax H3 | PE | 17 / 23 | +13.71 [4.56, 22.83] | +2.24 [−5.87, 10.50] | +1.92 | 7 / 0; 1 / 0 |
| LTX-2.5 Distilled | PE | 15 / 22 | +6.89 [−2.19, 16.25] | −1.02 [−7.95, 6.59] | −1.18 | 9 / 0; 3 / 0 |
| Wan2.2 | Oracle PE | 15 / 22 | −13.47 [−22.75, −5.24] | +16.03 [3.12, 29.18] | +14.46 | 6 / 1; 2 / 1 |
| Cosmos 3 Super I2V | Oracle PE | 20 / 23 | −8.44 [−17.61, −0.07] | +34.00 [21.26, 46.45] | +34.20 | 2 / 0; 2 / 0 |
| MiniMax H3 | Oracle PE | 17 / 23 | −13.19 [−21.25, −5.84] | +33.45 [23.60, 42.36] | +33.51 | 7 / 0; 1 / 0 |
| LTX-2.5 Distilled | Oracle PE | 15 / 22 | −9.93 [−16.79, −2.74] | +16.97 [4.49, 29.74] | +17.28 | 7 / 0; 2 / 0 |
| Cosmos 3 Super I2V | Native PE | 20 / 23 | +13.91 [6.79, 21.65] | −0.77 [−11.55, 10.53] | −1.53 | 3 / 0; 1 / 0 |
| MiniMax H3 | Native PE | 17 / 23 | +15.83 [6.17, 26.12] | −8.51 [−17.18, 0.53] | −8.53 | 8 / 0; 1 / 0 |
| Wan2.2 | C2C | 16 / 22 | +1.13 [−1.92, 4.13] | +6.41 [−0.16, 13.24] | +5.83 | 5 / 0; 0 / 1 |
| LTX-2.3 | C2C | 6 / 17 | −9.65 [−13.59, −4.74] | +2.78 [−6.18, 12.22] | +3.77 | 2 / 0; 1 / 1 |
| Cosmos 3 Super I2V | C2C | 18 / 23 | −2.03 [−6.33, 2.28] | +8.63 [0.87, 16.61] | +8.36 | 0 / 2; 0 / 0 |
We sincerely thank the reviewer for acknowledging the strengths of our work, including the concrete formulation of an important evaluation question, the separation of calibration and coverage, the breadth of the benchmark, and the validation and intervention analyses.
W1: Human validation of rejections and outcome confusions. We thank the reviewer for this suggestion. The comparison in Table 10 tests whether readable labels are correct, but not whether rejections are correct or whether particular outcomes are confused. Our response to Reviewer Yq64 (Q3) compares the evaluators' rejections with the human majority on all annotated videos, including those excluded from Table 10, and shows that rejections mostly coincide with videos that humans also cannot read (Table R6). Confusions between outcome categories depend on the mechanism (Table R15): agreement exceeds 89% for Rotation, Routing, Draw, and Stability, and is lower for Tossing, Collision, Agent, and Material.
As suggested, we select six representative model–scene pairs and collect ten human responses for each of their 50 videos, including those the evaluators reject, taking a majority of at least six as the human label. The pairs were drawn at random under a rule fixed in advance: one Coverage scene from each of the four Coverage mechanism groups and two Calibration scenes, covering four generators. Table R16 compares the scores and Table R17 reports each evaluator's errors. Five of the six pairs pass the gate under human labels. The exception is the tape-peeling scene, where humans reach a majority on only 8 of 50 videos. Both evaluators pass all six.
In Calibration, the human-derived TVD exceeds the 99th percentile expected from finite sampling in both pairs, as in two of the four Wan2.2 scenes in Table R7 (response to Reviewer Yq64). Automatic readout gives a similar or smaller deviation, so it does not inflate the calibration gap. In Coverage, automatic readout never exceeds the human score in the three pairs that pass under human labels. Both evaluators match humans in the ball-and-cup and bottle-flip scenes. In the dog-and-bowl scene, Qwen matches humans, whereas Gemini reads pausing and resisting the leash as continued eating and reaches 16.7% Coverage compared with 66.7% for humans. In these pairs, readout errors remove observed outcomes rather than create them, and therefore lower Coverage rather than raise it.
Table R15. Category agreement with the human majority on videos that both humans and the evaluator can read, by mechanism group.
Table R16. Scores from human and automatic labels on complete 50-video galleries. Score: Calibration TVD (×100, ↓) or Coverage (%, ↑). n: readable outcomes out of 50; —: below the gate (n < 20). Ref. q99 at the human n (Appendix D.5).