acceptodds
Under review as a conference paper at ICLR 2027

PAWBench: How far are we from Probabilistically Aligned World Modeling?

Abstract

Recent video generation models are increasingly framed as world models. Many physical processes can unfold in more than one valid way. Therefore, a world model should reproduce not only a plausible trajectory, but also the distribution of possible behaviors under the same initial observation and action. We call this distribution-level requirement probabilistic alignment. However, existing evaluations largely assess individual-video plausibility and do not test whether repeated generations recover the correct distribution. This raises a central question: how far are current video generators from probabilistically aligned world modeling? To answer it, we formalize probabilistic alignment as a distributional criterion for world models and introduce PAWBench, a benchmark for evaluating video generators as stochastic samplers of world dynamics. We further introduce PAWEval, an outcome-level protocol that converts repeated video rollouts into empirical distributions over possible physical behaviors. Across 50 scenarios and twelve current systems, no model consistently matches the reference probabilities while recovering the range of valid behaviors. Models also respond inconsistently to physical changes while shifting their outcome distributions in response to irrelevant visual or textual cues. Having established this gap, we test whether language prompts, initial noise sampling, or model training can reshape the model's predictive distribution. Prompt engineering is limited by errors in both predicting outcome distributions and realizing requested outcomes, while initial noise sampling broadens exploration at a fixed rollout budget. Fine-tuning shifts generated outcome frequencies toward the training proportions, but does not reliably adapt them to changes in physical state. We believe our work can serve as a foundation for future efforts to move towards probabilistically aligned world modeling.

Primary area:
generative models
Keywords:
Benchmark, Video Generation Models, World Modeling

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Sbs8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies probabilistic alignment in video-based world modeling: whether repeated generations under the same initial observation and action reproduce the distribution of possible physical outcomes. It introduces PAWBench, comprising 50 scenarios across eight mechanism groups, divided into PAW-Calibration, where reference outcome probabilities are specified, and PAW-Coverage, where valid outcomes can be enumerated but their probabilities are not reliably known. PAWEval maps generated videos to terminal outcome labels and reports conditional total variation distance or outcome coverage, together with a separate scene pass rate.

The evaluation covers twelve video generators with 50 rollouts per scenario. It finds substantial discrepancies from reference probabilities and incomplete outcome recovery under the evaluated budget. Supporting analyses include matched finite-sample simulations, an alternative outcome evaluator, human judgments, real-image inputs, and paired causal and non-causal interventions. The paper also studies language-based steering, marginal-preserving noise coupling, and LoRA training with controlled outcome proportions. These experiments distinguish target selection, target realization, finite-budget exploration, and changes to the learned output distribution. The main contribution is a broader diagnostic benchmark and an empirical characterization of probabilistic failure modes in current video generators.

Critical strengths
  1. The paper makes an important evaluation question concrete and provides a useful extension of existing benchmarks. Whether a model can produce a plausible future, recover alternative outcomes, and assign those outcomes appropriate probabilities are distinct questions with direct relevance to using video generators for prediction and planning. The separation between Calibration and Coverage is well motivated: the benchmark evaluates probability mass where a reference can be justified, while avoiding arbitrary probability assignments elsewhere. Although the repeated-rollout perspective is related to prior work such as CaliBench, the expansion to 50 scenarios across eight mechanisms, together with the accompanying diagnostics, constitutes a meaningful contribution. This gives the work value beyond the performance of any particular model in the reported comparisons.

  2. The empirical investigation provides substantive validation and useful distinctions between possible interventions. The matched Monte Carlo analysis is particularly convincing: it preserves each model’s passing scenes and readable sample counts, showing that the observed average TVD of 30.2 is substantially larger than the reference-sampling mean of 8.26. The alternative evaluator and human study provide complementary support for the broad findings. The intervention experiments further distinguish errors in selecting outcomes from errors in realizing them, and distinguish improvements in finite-budget exploration from changes to the learned distribution. In particular, the paper correctly explains that marginal-preserving noise coupling does not change the generator’s underlying single-rollout distribution. These analyses make the study informative beyond a leaderboard of low scores.

Critical weaknesses
  1. Human validation does not cover all important readout errors. The reported human comparison only includes videos with a human majority label and successful readouts from both VLMs. It therefore does not measure how often readable videos are incorrectly rejected. Overall label agreement also does not show whether particular outcomes are systematically confused with others. These errors could distort outcome frequencies, and even a few wrong labels could create or remove rare categories in Coverage. For representative model–scene pairs, please human-label all 50 videos, including rejected ones, report each evaluator’s errors by outcome, and compare the resulting TVD, Coverage, and scene pass rates with the automatic scores.

  2. Intervention comparisons average over different sets of scenes. TVD and Coverage are averaged over each condition’s own passing scenes. As acknowledged in Appendix F.1, changes in these averages can reflect scenes entering or leaving the calculation, as well as changes in the generated outcome distributions. This makes it difficult to determine how much an intervention actually improves or worsens performance on the same scenes. Please add paired comparisons on scenes that pass in both conditions, while separately reporting newly passing and newly failing scenes. A supplementary Coverage comparison using equal numbers of readable videos would also help distinguish outcome diversity from differences in readout success.

  3. Reference probabilities need clearer justification for individual scenarios. The paper explains that references come from symmetry, sector proportions, outcome counts, or visible physical state, but does not provide a complete account of the assumptions for each calibration scenario. For example, translating spinner sector sizes into outcome probabilities requires assumptions about how the spin is randomized. Since TVD directly depends on these references, unclear assumptions make it harder to attribute a mismatch to the video model.

Questions for authors

See weakness.

7 comments

We sincerely thank the reviewer for acknowledging the strengths of our work, including the concrete formulation of an important evaluation question, the separation of calibration and coverage, the breadth of the benchmark, and the validation and intervention analyses.

W1: Human validation of rejections and outcome confusions. We thank the reviewer for this suggestion. The comparison in Table 10 tests whether readable labels are correct, but not whether rejections are correct or whether particular outcomes are confused. Our response to Reviewer Yq64 (Q3) compares the evaluators' rejections with the human majority on all annotated videos, including those excluded from Table 10, and shows that rejections mostly coincide with videos that humans also cannot read (Table R6). Confusions between outcome categories depend on the mechanism (Table R15): agreement exceeds 89% for Rotation, Routing, Draw, and Stability, and is lower for Tossing, Collision, Agent, and Material.

As suggested, we select six representative model–scene pairs and collect ten human responses for each of their 50 videos, including those the evaluators reject, taking a majority of at least six as the human label. The pairs were drawn at random under a rule fixed in advance: one Coverage scene from each of the four Coverage mechanism groups and two Calibration scenes, covering four generators. Table R16 compares the scores and Table R17 reports each evaluator's errors. Five of the six pairs pass the gate under human labels. The exception is the tape-peeling scene, where humans reach a majority on only 8 of 50 videos. Both evaluators pass all six.

In Calibration, the human-derived TVD exceeds the 99th percentile expected from finite sampling in both pairs, as in two of the four Wan2.2 scenes in Table R7 (response to Reviewer Yq64). Automatic readout gives a similar or smaller deviation, so it does not inflate the calibration gap. In Coverage, automatic readout never exceeds the human score in the three pairs that pass under human labels. Both evaluators match humans in the ball-and-cup and bottle-flip scenes. In the dog-and-bowl scene, Qwen matches humans, whereas Gemini reads pausing and resisting the leash as continued eating and reaches 16.7% Coverage compared with 66.7% for humans. In these pairs, readout errors remove observed outcomes rather than create them, and therefore lower Coverage rather than raise it.

Table R15. Category agreement with the human majority on videos that both humans and the evaluator can read, by mechanism group.

GroupGeminiQwen
Tossing175/242 (72.3%)184/263 (70.0%)
Rotation75/83 (90.4%)79/86 (91.9%)
Routing255/258 (98.8%)268/280 (95.7%)
Draw171/173 (98.8%)182/185 (98.4%)
Collision165/213 (77.5%)159/227 (70.0%)
Stability124/132 (93.9%)158/176 (89.8%)
Agent182/284 (64.1%)195/329 (59.3%)
Material83/126 (65.9%)91/127 (71.7%)
All1,230/1,511 (81.4%)1,316/1,673 (78.7%)

Table R16. Scores from human and automatic labels on complete 50-video galleries. Score: Calibration TVD (×100, ↓) or Coverage (%, ↑). n: readable outcomes out of 50; —: below the gate (n < 20). Ref. q99 at the human n (Appendix D.5).

TrackSceneModelHuman nHuman ScoreGemini nGemini ScoreQwen nQwen ScoreRef. q99
CalibrationBall into numbered binsLTX-2.5 Dev4262.53862.54057.523.2
CalibrationCoin flipHappyHorse3550.05036.04545.621.4
CoverageBall tossed toward a cupSeedance 24520.05020.05020.0—
CoverageBottle flipLTX-2.5 Dev3925.03525.04725.0—
CoverageDog pulled by leash near bowlWan2.73866.74616.75066.7—
CoverageTape peeled from cardboardLTX-2.5 Dev8—4580.05080.0—

Table R17. Evaluator errors relative to the human majority on the galleries in Table R16 (Gemini / Qwen). These are disagreements with a crowd majority rather than adjudicated errors.

SceneClear outcome rejectedCannot tell acceptedCategory disagreement
Ball into numbered bins5/42 / 3/420/3 / 0/30/37 / 2/39
Coin flip0/35 / 2/353/3 / 1/31/35 / 1/33
Ball tossed toward a cup0/45 / 0/45—0/45 / 0/45
Bottle flip9/39 / 1/391/1 / 1/10/30 / 0/38
Dog pulled by leash near bowl4/38 / 0/38—4/34 / 5/38
Tape peeled from cardboard1/8 / 0/8—7/7 / 4/8
All19/207 / 6/2074/7 / 2/712/188 / 12/201

Together, these analyses show that rejections mostly coincide with videos that humans cannot read, that outcome confusions concentrate in a few mechanism groups, and that readout errors neither create the calibration gap nor inflate Coverage in these galleries.

W2: Intervention comparisons on shared scenes.

We thank the reviewer for this suggestion. Table R18 compares each intervention with its Base condition on the scenes that pass in both conditions, for both Calibration and Coverage, using the original rollouts. It also lists the scenes that newly pass or newly fail under each intervention, and the change in Coverage after matching the number of readable videos.

On shared scenes, the main conclusions of Sec. 5 hold. Oracle PE lowers TVD and raises Coverage for every generator, whereas neither PE nor Native PE significantly lowers TVD for any generator. Consistent with Sec. 5.2, C2C raises Coverage for all three generators, significantly for Cosmos 3 Super I2V; its effect on TVD varies.

Interventions mostly add passing scenes rather than remove them. Across the 13 comparisons, scenes newly pass 66 times in Calibration and 18 times in Coverage, while only 3 newly fail in each track. Part of the change in the original averages therefore comes from scenes entering the calculation, which the shared-scene comparison removes.

To separate outcome diversity from readout success, we subsample both conditions in each shared scene to the same number of readable videos (5,000 repetitions) and recompute Coverage. The direction of the change is preserved in 12 of the 13 comparisons; the exception is Wan2.2 with PE, where both estimates are close to zero (+0.66 and −0.39). The gains under Oracle PE and C2C remain after matching, so they reflect a broader set of outcomes rather than more readable videos. We will add Table R18 to the revised appendix.

Table R18. Intervention minus Base on scenes passing in both conditions, with 95% paired bootstrap intervals. Shared: Calibration / Coverage scenes. Equal readable: Coverage change after matching readable counts. Enter / leave: newly passing / failing scenes.

ModelInterventionSharedΔ TVD ↓ [95% CI]Δ Coverage ↑ [95% CI]Δ Coverage, equal readableEnter / leave
Wan2.2PE16 / 23−0.53 [−11.47, 8.63]+0.66 [−11.95, 11.99]−0.397 / 0; 2 / 0
Cosmos 3 Super I2VPE20 / 23+12.11 [2.55, 21.53]+11.49 [0.71, 22.80]+10.413 / 0; 2 / 0
MiniMax H3PE17 / 23+13.71 [4.56, 22.83]+2.24 [−5.87, 10.50]+1.927 / 0; 1 / 0
LTX-2.5 DistilledPE15 / 22+6.89 [−2.19, 16.25]−1.02 [−7.95, 6.59]−1.189 / 0; 3 / 0
Wan2.2Oracle PE15 / 22−13.47 [−22.75, −5.24]+16.03 [3.12, 29.18]+14.466 / 1; 2 / 1
Cosmos 3 Super I2VOracle PE20 / 23−8.44 [−17.61, −0.07]+34.00 [21.26, 46.45]+34.202 / 0; 2 / 0
MiniMax H3Oracle PE17 / 23−13.19 [−21.25, −5.84]+33.45 [23.60, 42.36]+33.517 / 0; 1 / 0
LTX-2.5 DistilledOracle PE15 / 22−9.93 [−16.79, −2.74]+16.97 [4.49, 29.74]+17.287 / 0; 2 / 0
Cosmos 3 Super I2VNative PE20 / 23+13.91 [6.79, 21.65]−0.77 [−11.55, 10.53]−1.533 / 0; 1 / 0
MiniMax H3Native PE17 / 23+15.83 [6.17, 26.12]−8.51 [−17.18, 0.53]−8.538 / 0; 1 / 0
Wan2.2C2C16 / 22+1.13 [−1.92, 4.13]+6.41 [−0.16, 13.24]+5.835 / 0; 0 / 1
LTX-2.3C2C6 / 17−9.65 [−13.59, −4.74]+2.78 [−6.18, 12.22]+3.772 / 0; 1 / 1
Cosmos 3 Super I2VC2C18 / 23−2.03 [−6.33, 2.28]+8.63 [0.87, 16.61]+8.360 / 2; 0 / 0
Reviewer_8GQkOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PAWBench, a benchmark for testing whether video generators are probabilistically aligned, i.e., whether repeated rollouts from a fixed image and action prompt induce the right distribution over terminal outcomes. It contains 50 scenarios split into two suites. PAW-Calibration (25 scenarios: tossing, rotation, routing, draw) has analytically derived or symmetry-derived reference distributions and measures the total variation distance between the empirical outcome distribution (50 rollouts) and the reference. PAW-Coverage (25 scenarios: collision, stability, agent interaction, material transition) has enumerable valid outcomes but no justifiable probabilities, and measures the fraction of valid outcomes observed at least once.

Twelve video generators are evaluated (Wan 2.2, LTX 2.3, 2.5, Cosmos 3 Super I2V, LingBot, ...). For each scenario, 50 independent rollouts are sampled from a fixed source image, action prompt, and default inference configuration. The most important metric is the Scene Pass Rate (SPR), which measures the percentage of the 25 scenes per track that pass the evaluation gate. A scene passes the evaluation gate if at least 20 of its 50 rollouts yield readable schema outcomes.

The key findings are:

  1. Current video generators are not probabilistically aligned world models.
  2. Recovering valid futures (i.e., matching the support) is not the same as assigning the right probabilities (i.e., calibration). In particular, recovering the valid support is necessary but not sufficient, because probability mass may be incorrectly allocated among the outcomes within the support.

The main study is followed by causal/non-causal controls and prompt, noise, and fine-tuning interventions. Their key findings are:

  1. Directly specifying the outcome in the prompt significantly improves both calibration and coverage, while providing an estimated outcome (from GPT-5.5) does not reliably improve the metrics. This suggests that the models are capable of synthesizing a plausible outcome, but struggle to predict it on their own. Therefore, the problem does not stem from conditioning; it lies in the generator itself.
  2. Fine-tuning on outcome frequencies controls the generation probabilities (adjusts the support), but in these experiments it does not reliably adapt them to the physical distributions of related scenes (calibration).
Critical strengths
  1. The paper addresses an important and under-studied problem. Rigorously assessing whether (video) generative models capture uncertainty about the future, given the past and an action, can provide useful insights for improving them. It is well known that video generators struggle with future prediction.
  2. Keeping calibration and coverage as separate metrics is a good design decision, since they expose distinct failures.
  3. The robustness checks are thorough: a rollout-budget sweep and human validation of 1,459 rollout videos.
  4. The qualitative examples in Appendix G clearly illustrate the failure modes described in the main paper.
Critical weaknesses
  1. All source images are generated by image models, and the fine-tuning videos are generated by Seedance (one of the benchmarked systems, Table 15). To claim misalignment with the real world, real inputs are needed, but the real-world check covers only 3 videos × 4 generators. This is a major weakness.
  2. The benchmark reduces every rollout to a discrete terminal label, so the study is limited to discrete outcomes. Physics is continuous, and none of the reported metrics consider the quality of the generated motion: a rollout with a physically implausible or discontinuous trajectory that ends in the right state is scored identically to a correct one. The authors acknowledge this limitation, but it remains a major weakness.
  3. The main tables report no confidence intervals, despite several scenes per scenario (a scene passes the evaluation gate if at least 20 of its 50 rollouts yield readable schema outcomes).
  4. The main evaluation uses the closed-source Gemini 3.5 Flash to return a terminal label for each video, which limits the reproducibility of the study.
Questions for authors

Clarifications requests:

  1. How do results in Table 1 change as the number of independent rollouts changes? Only Macro TVD and Macro coverage are presented in the Figure 5.
  2. How would you account for plausibility of entire trajectory (not just the terminal state) in your benchmark?
7 comments

We sincerely thank the reviewer for acknowledging the strengths of our work, including the importance of the problem, the separation of calibration and coverage, the thorough robustness checks, and the clear qualitative examples.

W1: Generated source images and real-world inputs. We thank the reviewer for this comment. PAWBench uses generated source images because they let us specify the initial state exactly: each image shows the apparatus, the objects, and the conditions the action depends on. The reference distribution follows from the physical mechanism shown in the image (Appendix C), not from how the image was produced. The question is therefore whether the same failures appear when the inputs are real photographs. We address it with the existing real-video evaluation and a new experiment.

Fig. 4 evaluates four generators on the first frames of three real videos. The failures seen on generated inputs also appear there. For example, MiniMax H3 produces heads in 32 of its 33 readable coin rollouts, and only faces 4 and 5 in its 47 readable die rollouts.

We add five real photographs, each matched to a mechanism in the benchmark: two Calibration scenes, a four-color spinner and a card draw, and three Coverage scenes, Jenga, a basketball launcher, and ring toss. We generate 50 rollouts per scene with MiniMax H3, Cosmos 3 Super I2V, and Wan2.2 under the main evaluation protocol, with the outcome categories fixed before viewing any output. Table R11 reports the results. The failures observed on generated inputs recur. In the Calibration scenes, the reference assigns probability 1/4 to each color or suit, assuming that the pointer stops at a uniformly random angle and that each card is equally likely to be drawn. In five of the six model–scene pairs, the generated distribution deviates from this reference beyond the 99th percentile expected from finite sampling. For example, MiniMax H3 lands on red in 30 of 40 readable spins, and Cosmos 3 Super I2V draws diamonds in 39 of 47 readable draws. Coverage varies by scene, as it does on generated inputs: all generators realize most outcomes of ring toss, whereas two of the three fail the readability gate in the basketball scene.

To test whether producing an image with an image model affects the results, we edit each real photograph with GPT Image 2.6, replacing the background while keeping the apparatus, objects, and viewpoint, and repeat the evaluation. The edit changes little. In all six Calibration pairs, the most frequent outcome is the same before and after editing, and five of the six reach the same conclusion relative to the sampling baseline. In the Coverage scenes, the number of observed outcomes changes by at most two.

Table R11. Real photographs and their GPT Image 2.6 edits (50 rollouts each). Calibration: percentages over readable rollouts, with counts in parentheses; outcomes are red / blue / green / yellow for the spinner and clubs / diamonds / hearts / spades for the card draw; * beyond the 99th percentile of finite sampling. Coverage: observed / valid outcomes, with readable rollouts in parentheses.

SceneTrackModelRealEdited
Four-color spinnerCalibrationReference25% each25% each
Four-color spinnerCalibrationMiniMax H375% / 3% / 17% / 5% (30 / 1 / 7 / 2) *75% / 5% / 13% / 7% (30 / 2 / 5 / 3) *
Four-color spinnerCalibrationCosmos 3 Super I2V68% / 7% / 16% / 9% (29 / 3 / 7 / 4) *67% / 7% / 12% / 14% (29 / 3 / 5 / 6) *
Four-color spinnerCalibrationWan2.248% / 24% / 8% / 20% (12 / 6 / 2 / 5)48% / 12% / 4% / 36% (12 / 3 / 1 / 9) *
Card drawCalibrationReference25% each25% each
Card drawCalibrationMiniMax H35% / 22% / 63% / 10% (2 / 9 / 26 / 4) *0% / 15% / 73% / 12% (0 / 5 / 24 / 4) *
Card drawCalibrationCosmos 3 Super I2V4% / 83% / 13% / 0% (2 / 39 / 6 / 0) *7% / 85% / 2% / 6% (3 / 40 / 1 / 3) *
Card drawCalibrationWan2.22% / 65% / 28% / 5% (1 / 28 / 12 / 2) *2% / 81% / 15% / 2% (1 / 38 / 7 / 1) *
JengaCoverageMiniMax H31/5 (50)2/5 (50)
JengaCoverageCosmos 3 Super I2V2/5 (40)2/5 (36)
JengaCoverageWan2.22/5 (42)3/5 (36)
Basketball launcherCoverageMiniMax H33/9 (11)5/9 (16)
Basketball launcherCoverageCosmos 3 Super I2V5/9 (13)5/9 (16)
Basketball launcherCoverageWan2.28/9 (24)8/9 (28)
Ring tossCoverageMiniMax H34/5 (50)3/5 (50)
Ring tossCoverageCosmos 3 Super I2V4/5 (50)5/5 (50)
Ring tossCoverageWan2.25/5 (49)5/5 (49)

Together, these results show that the failures reported in the paper also appear with real photographs and do not depend on whether the input image is produced by an image model.

Q2 · W2: Discrete terminal outcomes and trajectory plausibility.

We thank the reviewer for this comment and question. PAWBench evaluates probabilistic alignment, which is complementary to the physical plausibility of individual videos (Sec. 2). The physical processes in PAWBench evolve continuously but resolve into discrete outcomes: a tossed die settles on one face, and a released ball ends in one bin. The probabilities that characterize these processes are probabilities over outcomes, so we map each trajectory to the terminal outcome it reaches. Terminal outcomes form a finite and visually distinguishable set (Sec. 3.2), whose distribution can be estimated from 50 rollouts and compared with a reference. A distribution over full trajectories cannot be treated this way: trajectories are continuous and high-dimensional, and a trajectory-level reference would require ground-truth trajectories or a physical simulation for every scene.

To account for trajectory plausibility, PAWEval runs a separate trustworthiness audit that checks whether the requested action is completed, whether key objects remain identifiable, and whether the transition follows the specified physical mechanism (Appendix D.1, Table 7). For example, a spinner that ends on blue after an implausible rotation receives the label blue, and its trajectory is flagged by the audit. The audit results are reported in Appendix D.2: physical-process errors account for the largest share of flagged failures (Fig. 11). We keep the audit separate from the scores, so that a change in TVD reflects the outcome probabilities rather than a plausibility filter.

Implausible trajectories do not weaken our main finding that current generators do not reproduce the reference outcome distributions, because reaching an outcome through implausible dynamics is itself a failure to model the physical process. In the revision, we will describe the trustworthiness audit in Sec. 3.3 and refer to its results in Appendix D.2.

Reviewer_xcNAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper raises the concept of probabilistic alignment as a weakness of existing video generation methods and proposes PAWBench, a benchmark for evaluating probabilistic alignment in video generation models, and PAWEval, an outcome-level protocol for estimating outcome distributions from repeated video generations. Experiments across 50 scenarios and 12 models reveal substantial misalignment. Additional experiments investigate causal sensitivity, prompt engineering, noise sampling, and fine-tuning as potential means of improving alignment.

Critical strengths
  • Important and underexplored evaluation dimension: The paper highlights the distinction between generating individually plausible futures and correctly modeling the distribution of possible outcomes. This is an important property of world models, particularly for uncertainty-aware planning and decision-making.

  • Extensive empirical evaluation: The benchmark evaluates 12 contemporary video generation models across 50 scenarios and eight physical mechanism groups, providing a broad empirical assessment of their stochastic behavior.

  • Complementary diagnostic experiments: Beyond the main benchmark, the authors investigate causal and non-causal interventions, prompt engineering, initial-noise sampling, and fine-tuning. These experiments provide additional insights into the limitations of current video generators.

  • Evaluation protocol validation: The authors investigate finite-sampling effects, compare automated evaluators, and perform human validation, strengthening confidence that the observed discrepancies are not merely evaluation artifacts.

  • Clear presentation: The paper is generally well written, organized, and easy to follow, with substantial methodological details in the appendices.

Critical weaknesses
  • W1. Limitations of PAW-Coverage:

    • W1.1. Assumed completeness of outcome sets: The metric assumes that the complete set of physically possible outcomes is known, despite their probabilities being unknown (Sec. 3.1, L159–161). It is unclear how the authors establish that the manually enumerated outcomes are exhaustive, particularly for complex physical processes.

    • W1.2. Limited representation of physical outcome diversity: The predefined outcome categories may be overly coarse to capture the stochastic complexity of physical processes. For example, interactions with a cat or breaking a chocolate bar (Fig. 2) can unfold in an enormous variety of ways. A model generating only one stereotypical response or fracture pattern per category could achieve full PAW-Coverage while representing only a narrow subset of physically plausible behaviors. Thus, even with infinitely many samples, full categorical coverage does not imply recovery of the true support of the underlying physical process.

    • W1.3. Finite-sample coverage can favor probabilistically misaligned models: With K=50K=50 rollouts, even a perfectly calibrated model has only a 39.5% chance of observing an outcome with 1% probability, whereas a miscalibrated model that overrepresents rare outcomes may achieve higher coverage. Thus, not only does higher coverage fail to guarantee better probabilistic alignment, but it can actively favor misaligned models over correctly calibrated ones. This calls into question the validity of coverage as a measure of physical world-modeling accuracy.

  • W2. Conditional evaluation complicates model comparisons: The primary metrics exclude unreadable outcomes and are averaged only over scenes that pass the outcome-readout gate. Consequently, models may be compared on different subsets of scenarios, potentially favoring models that fail on more challenging cases. Although the authors report scene pass rates separately, this does not fully resolve the comparability issue.

Questions for authors
  1. PAW-Coverage outcome definitions: How do the authors ensure that the manually defined outcome categories are exhaustive and sufficiently representative of the physical diversity of each process, particularly for complex interactions?

  2. Fairness of model comparisons: Since different models pass the outcome-readout gate on different scenarios (Sec. 4.1, L248–251), their average scores are computed over different subsets of the benchmark. Could the authors also report results on the same subset of scenarios for all models, to enable a fairer comparison?

  3. Code and benchmark availability: Do the authors plan to publicly release the PAWBench benchmark, including the scenarios, reference distributions, evaluation rubrics, and evaluation code? Such a release would be valuable for reproducibility and future research.

4 comments

We sincerely thank the reviewer for acknowledging the strengths of our work, including the research motivation, the extensive experiments and diagnostic analyses, the validation of our evaluation protocol, and the clear presentation.

Q1 · W1.1: Completeness of the outcome sets.

We thank the reviewer for raising this important question. First, knowing which outcomes are possible does not require knowing their probabilities. Each Coverage scene defines its outcomes over the terminal state specified by its rubric (Sec. 3.3). For example, in the seven-pin bowling scene, the terminal state is the number of fallen pins, so the outcomes are zero to seven fallen pins, plus "no pin contact". Every roll ends in exactly one of these categories, even though the probability of each depends on collision dynamics that cannot be determined from the first frame.

Second, the outcome sets are constructed and reviewed to cover the full range of terminal states. Each outcome set spans the terminal states that the action can produce, including cases where the action has little or no effect. For example, the bowling scene includes "zero pins down after contact" and "no pin contact", and the dart scene includes bouncing off the board and missing it entirely. For complex interactions, the categories are ordered by the strength of the response or the extent of the change. The cat categories range from accepting the touch to active defensive contact, and the chocolate categories range from no visible break to breaking into crumbs or small fragments, so that each terminal state can be assigned to one of these categories. In addition, all outcome sets were finalized and reviewed before evaluating any model (Sec. 3.2). The review checks whether the terminal outcomes are distinguishable. When a check fails, we refine the outcome categories, and we discard a scenario if its ambiguity cannot be removed without changing the physical process (Appendix C.2).

Third, regarding physical diversity, the categories are defined at the level of terminal physical states. They separate physically different results, such as whether the chocolate bar breaks and into how many pieces, rather than variations within the same result, such as the exact fracture line. We discuss this choice and the diversity within each category in our response to W1.2.

W1.2: Granularity of the outcome categories.

We appreciate the reviewer's comment. PAW-Coverage is designed to detect whether a model collapses to a subset of the possible outcomes under the same input, in scenes where these outcomes can be listed but their probabilities cannot be specified (Sec. 2).

Reaching every terminal category is a necessary condition for recovering the full range of physical behaviors: if a model never produces a category, that outcome remains missing under any finer description of the same process. Current models do not meet even this necessary condition. Across the twelve models, average Coverage ranges from 41.8% to 71.7% (Table 1), and on the eleven Coverage scenes that pass for all models, no model recovers all valid outcomes (Table R10). The gaps that Coverage reports are therefore true gaps in model behavior, and a finer description would reveal more of them, not fewer.

The current categories separate physically different terminal states. For the chocolate bar, they distinguish no visible break, a crack without separation, a break into two pieces, a break into more than two pieces, and a break into crumbs or small fragments. For the cat, they range from accepting the touch to active defensive contact. Variation within a category, such as the exact fracture line or the cat's final posture, is continuous and has no natural point at which to split further. Any finer split would be one arbitrary choice among many, and the score would reflect that choice rather than the model. With 50 rollouts per scene, finer categories would also make Coverage depend mainly on the sampling budget. We therefore define categories at the level of terminal states that are finite, visually distinct, and can be read consistently across videos.

W1.3: Finite-sample Coverage and probabilistic alignment.

We appreciate the reviewer's comment. Coverage measures whether a model produces all valid outcomes, which is a different ability from assigning them the right probabilities, and this example illustrates why PAWBench evaluates both.

Sec. 2 defines two levels of probabilistic alignment. Support alignment asks whether a model produces all valid outcomes under the same input, while probability-mass alignment asks whether it produces them in the right proportions. Coverage measures only the first, so a high Coverage is not meant to indicate correct probabilities. For this reason, PAWBench reports the two tracks separately rather than combining them into a single score (Sec. 3.1).

The physical world is complex, and for many physical processes the ground-truth outcome probabilities cannot be reliably specified. They depend on factors that a single image and action do not determine, such as the exact force and point of contact, material properties, or the internal state of an animal. This is the case for collision, stability, agent interaction, and material transition (Appendix C.1.2). For these processes, the possible outcomes can still be listed, but no reference distribution can be derived, so Calibration cannot be applied. Coverage makes distribution-level evaluation possible for them, which covers half of PAWBench and four of its eight mechanism groups.

Coverage also exposes a failure that more sampling cannot correct. An outcome to which a model assigns a small but nonzero probability will appear as the number of rollouts grows, whereas an outcome to which it assigns zero probability will never appear at any budget. Coverage identifies such missing outcomes, which show that the model fails to represent part of the valid futures.

Because observed Coverage depends on the number of rollouts, all models are compared at the same budget of K = 50 (Appendix D.3). A model that overrepresents rare outcomes would gain Coverage but score poorly on Calibration, so the two tracks are read together. In our results, the models that recover broad support are not those that best match the reference probabilities (Sec. 4.2).

Q2 · W2: Comparisons across different passing scenes.

We thank the reviewer for this suggestion. A scene enters the average only if at least 20 of its 50 rollouts are readable and in-schema, and the share of passing scenes is reported separately as the Scene Pass Rate (SPR). An alternative is to fold readout failures into the metric itself, for example by counting unreadable videos as an extra outcome. This, however, makes different failures indistinguishable. Consider a fair coin and two models that both produce unreadable videos half of the time but differ in their readable outputs: the first always lands heads, while the second lands heads and tails equally often. Their probabilities over (heads, tails, unreadable) are (0.5,0,0.5)(0.5, 0, 0.5) and (0.25,0.25,0.5)(0.25, 0.25, 0.5). With unreadable videos counted as an extra outcome, both receive a TVD of 0.5, even though only the first is biased. Reporting the conditional TVD (0.5 and 0) and SPR separately keeps these two failures apart.

As suggested, we compare all twelve models on the scenes that pass for every model: three Calibration scenes and eleven Coverage scenes (Table R10). The main findings hold on these shared scenes. In Calibration, every model's TVD exceeds the 99th percentile of its own matched reference baseline (Appendix D.5), with an average TVD of 27.97 compared with an average 99th percentile of 12.37. In Coverage, no model recovers all valid outcomes, with scores ranging from 45.8% to 80.9%, and the model ranking closely follows Table 1 (Spearman ρ = 0.87). Since the three shared Calibration scenes cover only Tossing and Rotation, we do not draw a benchmark-wide Calibration ranking from them.

To cover every Calibration mechanism, we also bound the effect of readout failures on all 25 Calibration scenes (Table R5 in our response to Reviewer Yq64). Even when every rollout without a readable label is assigned to the outcome most favorable to the generators, the average TVD across models (14.19) remains above the 99th percentile of complete 50-rollout reference samples (7.55), assuming the readable labels are correct.

Together, these results show that our main conclusions do not depend on which scenes each model passes. We will add Table R10 to the revised appendix.

Table R10. All twelve models on the scenes that pass for every model: Calibration TVD (×100) on three scenes with each model's Ref. q99, and Coverage (%) on eleven scenes.

ModelCal. TVDRef. 99th pct.Coverage
HappyHorse44.6911.3453.48
Veo3.1 Fast31.9811.5148.33
Kling 3 Std.23.5111.1159.09
Seedance 232.1311.4055.45
Wan2.727.6511.4945.76
Wan2.229.2811.5062.88
LTX-2.3 Dev25.8514.6780.91
LTX-2.5 Dev22.8513.8370.91
Cosmos 3 Super I2V13.7811.8264.24
LingBot-Video-MoE31.3713.8471.67
LingBot-World V228.8412.4963.18
MiniMax H323.6513.4249.39

Q3: Release of the benchmark and code.

We thank the reviewer for this suggestion. We will publicly release PAWBench, including all 50 scenarios with their source images, action prompts, and outcome sets, together with the Calibration reference distributions and their assumptions and derivations. It will also include the scene-specific rubrics and the evaluation code for outcome readout, the scene gate, and metric aggregation. To make the evaluation reproducible without a closed-source model, we will also release the generation and evaluator configurations, including the open-weight Qwen3.8-27B setup (Appendix D.4). The paper already documents scenario construction (Appendix C), the evaluation protocol and metric computation (Appendix D and Fig. 10), and the intervention settings (Appendix F).

Dear Reviewer xcNA,

We appreciate the time you have spent on our paper. Before the discussion period closes, we want to summarize how our response addresses your comments. The outcome sets are built and reviewed to cover the full range of terminal states, and each category corresponds to a physically distinct terminal state; our response explains both choices. Coverage measures support alignment, so it complements Calibration and cannot replace it. Following your suggestion, we compared all twelve models on the scenes that pass for every model. Even there, each model exceeds its own reference 99th percentile, and none recovers all valid outcomes (Table R10). The calibration gap also persists when failed rollouts are assigned in the way most favorable to the generators (Table R5). If anything in the response is unclear or still falls short, we would be glad to clarify it while the discussion is open.

Reviewer_Yq64OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PAWBench to evaluate whether repeated video generations under fixed observations and actions reproduce the distribution of possible terminal outcomes. The benchmark comprises 50 scenarios across eight mechanism groups, divided into probability calibration and outcome coverage tracks. PAWEval extracts terminal labels and measures conditional total variation distance, coverage, and scene pass rate. Experiments evaluate twelve video generators and examine the effects of input changes, prompt engineering, coupled initial noise, and controlled fine-tuning.

Critical strengths
  1. Important evaluation perspective. The paper distinguishes plausible individual videos from correctly distributed outcomes, addressing a meaningful limitation of conventional video evaluation.
  2. Broad experimental coverage. The evaluation spans multiple generators, physical mechanisms, and intervention types, providing a useful diagnostic picture of current systems.
  3. Substantive robustness analysis. Matched Monte Carlo simulations show that finite-sample noise alone cannot explain the observed calibration gap under the assumed references and readout protocol. A second evaluator and human annotations provide additional validation.
  4. Careful interpretation of noise coupling. The paper correctly distinguishes improved finite-budget exploration from changes to the model's learned single-rollout distribution.
Critical weaknesses
  1. Reference probabilities are not fully established. An image and a coarse action instruction do not uniquely determine outcome probabilities when launch conditions, perturbations, or selection procedures remain unspecified. Consequently, some reported misalignment may reflect disagreement with idealized assumptions rather than an incorrect physical distribution.

  2. Conditional evaluation complicates comparisons. Scores exclude unreadable outcomes and average over different passing scenes across models and interventions. Outcome-dependent failures can bias the estimated distributions, while changes in scene composition confound claims that prompting or sampling improves or worsens alignment.

  3. Validation does not cover the full evaluation pipeline. Physically implausible trajectories can still contribute valid endpoint labels. Human validation also excludes videos without valid labels from both evaluators, leaving rejection decisions insufficiently assessed. These limitations weaken the connection between the reported scores and physically faithful world modeling.

  4. Causal conclusions exceed the experimental controls. Some intervention pairs change background or appearance alongside the intended physical factor. Without sufficiently isolated changes and paired uncertainty estimates, observed distribution shifts cannot be attributed confidently to failures of physical causal reasoning.

  5. The contribution beyond the closest benchmark is insufficiently differentiated. The manuscript describes substantial overlap with CaliBench in repeated sampling, terminal outcomes, conditional TVD, and scorability reporting. The additional scenarios and interventions are useful, but their contribution beyond increased scope remains unclear.

Questions for authors
  1. What assumptions about initial conditions and action randomness justify the reference probabilities? Are these assumptions explicitly provided to the generators, or only used when defining the evaluation targets?
  2. Do the conclusions about prompt engineering and coupled noise remain unchanged when comparisons are restricted to the same passing scenes across conditions?
  3. How many videos were excluded from the human study because either evaluator failed to produce a valid label? Does the available human annotation validate these rejection decisions, and do human-derived outcome distributions support the reported calibration gaps? These are clarification requests about the current evaluation, rather than suggestions for future work.
11 comments

We are grateful for the reviewer's careful reading and for recognizing the value of evaluating outcome distributions rather than individual videos. We also appreciate the positive assessment of our experiments, robustness analyses, and interpretation of noise coupling.

Q1 · W1: Assumptions behind the reference probabilities.

We thank the reviewer for this question. We define each Calibration reference before evaluating any model, independently of model outputs (Appendix C.1.1). The derivation combines information visible in the image, such as sector proportions, outcome counts, physical symmetry, or causal state, with idealized assumptions about initial conditions and action randomness. For example, the spinner reference assumes a uniformly distributed final angle relative to the pointer, so each color's probability equals its angular share. Table R1 groups the assumptions for all 25 Calibration scenes into five types.

Table R1. Assumptions and derivations of the Calibration references.

TypeScenesAssumptionReference
Face symmetryCoins and dice (5)Balanced objects; the flick or throw treats all faces symmetrically; multiple objects land independentlyEqual probability for each face (e.g., 1/2 for a coin, 1/6 for a die); combined outcomes follow from independent faces
Reflection symmetrySingle splits, ramps, ridges, and an upright pencil (7)Left–right symmetric geometry and release perturbations, without lateral impulse1/2 left, 1/2 right
Unbiased branchingGalton boards (3)Each peg deflects left or right with probability 1/2 given the path so farEach bin's share of the left–right paths (binomial)
Uniform phase or selectionSpinners, a ring, and a top (4); draws from a visible set (4)Uniform terminal angle relative to the pointer; each selectable item equally likelyEach sector's share of the circle (e.g., 1/4 and 3/4 for a 1:3 spinner); each category's share of the items
Visible causal stateOffset ball and tilted pencil (2)Release from rest without lateral or angular impulseProbability 1 for the outcome set by the visible state

These assumptions are not stated in the prompts. Each prompt specifies the action, while the image shows the apparatus, including the sector proportions of the spinners. Actions such as flipping a coin or spinning a wheel describe familiar random processes. PAWBench evaluates whether repeated generations given the image and action reproduce the reference distribution derived under the assumptions in Table R1 (Sec. 2).

To assess how much the missing assumptions contribute to the calibration gap, we evaluate three generators (MiniMax H3, Cosmos 3 Super I2V, and Wan2.2) on all 25 Calibration scenes with the scene assumptions and full reference distribution appended to each original prompt. All other settings follow the main evaluation protocol (K = 50). Table R2 compares the original and augmented prompts on scenes that pass under both conditions. The TVD change is not statistically significant for MiniMax H3 or Cosmos 3 Super I2V, while Wan2.2 improves by 8.0 points. Under the augmented prompts, as shown in Table R3, the three generators have an average TVD of 22.0 on their respective passing scenes, compared with a finite-sampling reference 99th percentile of 10.5. Each generator also exceeds its own reference 99th percentile. A substantial calibration gap therefore remains even when both the assumptions and target probabilities are provided.

Table R2. Calibration TVD (×100, ↓) with and without the scene assumptions in the prompt, on scenes passing under both. Δ with 95% paired bootstrap intervals. Toss, Rot., Rout.: Tossing, Rotation, Routing; NA: no shared passing scene.

ModelShared scenesAvg.Δ Avg. [95% CI]TossRot.Rout.Draw
MiniMax H31124.226.414.0NA26.5
+Assumptions1124.7+0.5 [−6.6, 8.0]17.710.3NA40.7
Cosmos 3 Super I2V2020.519.43.517.735.5
+Assumptions2023.0+2.5 [−5.8, 9.7]22.47.127.225.7
Wan2.21527.827.015.823.245.9
+Assumptions1519.7−8.0 [−11.9, −4.4]18.715.814.035.3

A related concern is that an idealized reference may differ from the physical outcome distribution, allowing a physically accurate model to score poorly against it. We therefore also evaluate the nine scenes whose reference probabilities follow from the assumptions of balanced objects or equally likely selection. Under the original prompts, the average TVD across the twelve generators is 36.1, compared with a 99th percentile of 13.2 from matched finite samples of the reference distributions. In Table R3, Eleven of the twelve generators exceed their own reference 99th percentile. The result also holds with Qwen as the evaluator. When both the assumptions and full reference distributions are added to the prompts, all three tested generators remain above their reference 99th percentiles on these nine scenes.

Table R3. Average Calibration TVD (×100) versus the 99th percentile of matched reference samples (Ref. q99, Appendix D.5). Nine scenes: coins, dice, and draws. Avg. is taken over each model's passing scenes in the set.

PromptScene setEvaluatorModelsAvg.Ref. q99Above q99
OriginalNineGemini1236.113.211 / 12
OriginalNineQwen1240.811.612 / 12
+AssumptionsAll 25Gemini322.010.53 / 3
+AssumptionsNineGemini327.713.53 / 3

Together, these results show that the calibration gap is not explained by disagreement with the idealized assumptions. We will add the assumptions and derivations for all 25 Calibration scenes, together with this experiment, to the revised appendix.

W2 & Q2: Conditional evaluation and comparisons on shared scenes.

We thank the reviewer for these questions. We compute each scene's outcome distribution from readable, in-schema rollouts and include the scene in the average if at least 20 of its 50 rollouts meet these criteria. SPR reports the fraction of scenes that pass this threshold. We report conditional TVD separately because treating unreadable videos as an extra outcome can obscure differences in outcome bias. Consider a fair coin and two models that both produce unreadable videos half the time. Among readable videos, one model always produces heads, while the other produces heads and tails equally often. Their distributions over (heads, tails, unreadable) are (0.5,0,0.5)(0.5, 0, 0.5) and (0.25,0.25,0.5)(0.25, 0.25, 0.5). Both have a TVD of 0.5 against the reference (0.5,0.5,0)(0.5, 0.5, 0), whereas their conditional TVDs are 0.5 and 0. Reporting conditional TVD alongside SPR distinguishes outcome bias among readable rollouts from whether scenes meet the evaluation threshold.

The main calibration finding also holds when we account for scene composition. Appendix D.5 compares each model with reference samples matched to its passing scenes and readable counts. Every model's TVD exceeds the 99th percentile of this matched baseline. On the three Calibration scenes that pass for all twelve models, the average TVD is 27.97, and every model exceeds its own reference 99th percentile. These reference thresholds average 12.37 across models (Table R10 in our response to Reviewer xcNA).

For Q2, the conclusions about prompt engineering and coupled noise hold on scenes that pass under both the intervention and its Base condition (Table R4). Oracle PE lowers TVD and raises Coverage for every tested generator, whereas neither PE nor Native PE significantly lowers TVD for any tested generator. C2C raises Coverage for all three tested generators, with a statistically significant increase for Cosmos 3 Super I2V, consistent with Sec. 5.2. Its effect on TVD varies across generators.

Within a scene, readout failures bias the conditional distribution only if they depend on the outcome. To bound this effect, we recompute each generator's TVD on all 25 Calibration scenes. We keep the readable labels fixed and assign every rollout without a readable label to the outcome that minimizes TVD (Table R5). Even under this most favorable assignment, the average TVD across models is 14.19, above the 99th percentile of 7.55 from complete 50-rollout reference samples. Six of the twelve generators also exceed their own reference 99th percentiles. The remaining six have fewer readable rollouts, ranging from 332 to 766 of 1,250. This leaves more rollouts available for assignment and makes their bounds less informative. Assuming the readable labels are correct, readout failures alone cannot explain the average calibration gap.

Table R4. Change from Base on scenes passing in both conditions: Calibration TVD (×100, ↓) and Coverage (pp, ↑), with 95% paired scene-bootstrap intervals.

ModelInterventionCal. scenesΔTVD [95% CI]Cov. scenesΔCoverage [95% CI]
Wan2.2PE16−0.53 [−11.47, 8.63]23+0.66 [−11.95, 11.99]
Cosmos 3 Super I2VPE20+12.11 [2.55, 21.53]23+11.49 [0.71, 22.80]
MiniMax H3PE17+13.71 [4.56, 22.83]23+2.24 [−5.87, 10.50]
LTX-2.5 DistilledPE15+6.89 [−2.19, 16.25]22−1.02 [−7.95, 6.59]
Wan2.2Oracle PE15−13.47 [−22.75, −5.24]22+16.03 [3.12, 29.18]
Cosmos 3 Super I2VOracle PE20−8.44 [−17.61, −0.07]23+34.00 [21.26, 46.45]
MiniMax H3Oracle PE17−13.19 [−21.25, −5.84]23+33.45 [23.60, 42.36]
LTX-2.5 DistilledOracle PE15−9.93 [−16.79, −2.74]22+16.97 [4.49, 29.74]
Cosmos 3 Super I2VNative PE20+13.91 [6.79, 21.65]23−0.77 [−11.55, 10.53]
MiniMax H3Native PE17+15.83 [6.17, 26.12]23−8.51 [−17.18, 0.53]
Wan2.2C2C16+1.13 [−1.92, 4.13]22+6.41 [−0.16, 13.24]
LTX-2.3C2C6−9.65 [−13.59, −4.74]17+2.78 [−6.18, 12.22]
Cosmos 3 Super I2VC2C18−2.03 [−6.33, 2.28]23+8.63 [0.87, 16.61]

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Major Responses and Revisions

Dear Reviewers and Area Chairs,

Thank you for your time and constructive feedback. Below we summarize our main responses.

Strengths noted by the reviewers. PAWBench asks an under-studied question: given the same initial observation and action, do video generators reproduce the distribution of possible physical outcomes? (Yq64, 8GQk, Sbs8). Reviewers found the split between Calibration and Coverage well motivated, because the two expose different failures (8GQk, Sbs8). They considered the matched finite-sample analysis, alternative evaluator, human study, and intervention experiments substantive validation (Yq64, xcNA, Sbs8), found that the qualitative examples illustrate the failure modes well (8GQk), and found the paper clearly written (xcNA). Sbs8 noted that our responses and additional analyses address most of their concerns.

Main responses.

  1. Assumptions behind the reference probabilities (Yq64 W1/Q1, Sbs8 W3, xcNA Q3). We sort the assumptions of all 25 Calibration scenes into five types and give their derivations. Two analyses show that the deviation comes from the models rather than from these assumptions. When the prompt states the assumptions and the full reference distribution, three generators still reach an average TVD of 22.0, about twice the finite-sampling 99th percentile (10.5). On the nine scenes whose references are hardest to dispute (coins, dice, and draws), the twelve generators average a TVD of 36.1 against 13.2, and 11 of 12 exceed their own threshold.

  2. Conditional evaluation, scene composition, and readout failures (Yq64 W2/Q2, xcNA W2/Q2, Sbs8 W2). On the scenes that pass for all twelve models, every model exceeds its own 99th percentile. Paired comparisons on scenes that pass both with and without each intervention confirm the main conclusions of Sec. 5. We also report newly passing and newly failing scenes separately, along with Coverage at equal readable counts. If every rollout without a readable label is assigned to the outcome most favorable to the generators, the average TVD is 14.19, still above the 99th percentile of complete 50-rollout reference samples (7.55).

  3. Human validation of outcome readout (Yq64 Q3/W3, Sbs8 W1). Rejections largely coincide with videos that humans also cannot read: Gemini rejects 82.1% and Qwen 74.7% of the videos the human majority labeled "Cannot tell." As suggested, we collected ten human responses for every video in six representative 50-video galleries, rejected videos included. TVD computed from human labels still exceeds the finite-sampling 99th percentile, and readout errors lower Coverage rather than raise it.

  4. Real-world inputs (8GQk W1). We added five real photographs (two Calibration and three Coverage scenes), each with 50 rollouts from three generators. The failures seen on generated inputs recur: five of six Calibration cells exceed the finite-sampling 99th percentile. Replacing the background with GPT Image 2.6 changes little, so the findings do not depend on whether an image model produced the input.

  5. Uncertainty and rollout budget (8GQk W3, Q1). We added 95% intervals to Table 1. For all twelve generators, the lower end of the Calibration TVD interval stays above the finite-sampling 99th percentile. Full results for K = 10 to 50 show that the calibration gap does not close with more rollouts, while Coverage rises steadily with K.

  6. Controls and uncertainty in interventions (Yq64 W4). We rebuilt the three intervention pairs whose variants differed in background or apparatus appearance, and the conclusions hold in eight of nine model-pair combinations. We also computed paired uncertainty across all eight pairs and seventeen models. Responses to causal changes fall short of the required shift in 20 of 24 combinations. Non-causal changes shift the distribution beyond sampling variation in 25 of 45 video-generator combinations and 24 of 30 VLM combinations.

  7. Scope of Coverage, trajectory plausibility, and related work (xcNA W1.1-W1.3, 8GQk W2/Q2, Yq64 W3/W5). We clarify that Coverage measures support alignment and complements Calibration. We describe the separate trustworthiness audit, which checks action completion, object continuity, and physical process, and explain how PAWBench differs from CaliBench.

  8. Reproducibility (8GQk W4, xcNA Q3). Model rankings from the open-weight Qwen3.8-27B evaluator agree with those from Gemini (Spearman ρ = 0.84 for Calibration and 0.81 for Coverage). We will release all scenarios, the reference distributions with their derivations, rubrics, evaluation code, and evaluator configurations.

We would be glad to answer any remaining questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.