- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FPAbench, a benchmark that evaluates video world models at three levels: Signal Fidelity, Video Plausibility, and Future-State Alignment. The authors test 15 video generation models on 2,000 real-world video sequences across 13 physical event categories.
Critical strengths
-
This paper try to address an important problem in video world model evaluation.
-
The authors introduce a benchmark with real-world video sequences and a three-level evaluation strategy.
Critical weaknesses
-
I think the main idea is same with existing works, like WorldReasonBench, Physics-IQ Verified, and GAUGE. I do not see much novelty in this work.
-
The paper evaluates 15 video generation systems, but several recent open-source and close-source models are missing (like HunyuanVideo 1.5, LTX 2.5, Kling-Video, Seedance 2.0, Veo 3.1). This makes the reliability of the main conclusions less convincing.
[1] WorldReasonBench, https://arxiv.org/abs/2605.10434, NeurIPS 2026.
[2] Physics-IQ, https://ieeexplore.ieee.org/abstract/document/11492408, WACV 2026.
[3] GAUGE, https://arxiv.org/pdf/2608.05948
Questions for authors
-
Several recent works (e.g. WorldReasonBench [1], Physics-IQ Verified [2], and GAUGE [3]) have already explored similar problems. Could the authors explain more clearly what is new in FPAbench compared with these benchmarks?
-
The evaluated models seem somewhat outdated. Could the authors include more recent open-source and close-source models?
-
Using a VLM as a judge is already common in video evaluation. Could the authors provide a human study to verify the reliability of L2 and L3? I think it would be helpful to compare different and stronger VLM judges (e.g. GPT 6.1-sol/GPT 5.6-sol) with carefully designed prompts, to see whether the results remain consistent.
-
A single input image may have multiple physically plausible futures. How does the benchmark handle cases where a generated video is physically correct but different from the recorded ground truth?
-
The evaluation seems to use only five sampled frames. Is this enough to capture short physical events (e.g. collisions and contacts)?
[1] WorldReasonBench, https://arxiv.org/abs/2605.10434, NeurIPS 2026.
[2] Physics-IQ, https://ieeexplore.ieee.org/abstract/document/11492408, WACV 2026.
[3] GAUGE, https://arxiv.org/pdf/2608.05948
2 comments
The human study, judge reliability, alternative futures, and five-frame coverage
This second comment answers Q3 to Q5. The new evidence used here is a blinded human preference study: more than 2,300 judgments on anonymized rollouts, comprising 1,418 pairwise verdicts covering all 105 system pairs at each layer, 660 five-level ratings, and 255 per-case rankings. Pooled human rankings agree with our VLM judge at the system level (Kendall tau of 0.905 for L2 and 0.848 for L3). On the 25-pair core battery judged by every annotator, inter-annotator agreement is 0.928 and 0.935, and case-level agreement with the judge rises from about 45% on near-tied pairs to 95.7% (L2) and 87.9% (L3) on well-separated pairs. Five-level ratings give pooled quadratic-weighted kappa of 0.669 and 0.652; same-rater retest consistency is 95.6% and 98.0%. Judge re-annotation test-retest gives kappa of 0.980 (95.2% exact).
Q3 (human study; stronger closed judges). The human study is now done, with the numbers above, and we will add it to the paper. We cannot run GPT-class closed judges under the same standardized protocol because they do not provide frame-level access. We also note that judge strength is not automatic: in GenAI-Arena, GPT-4o used as a judge agrees with human arena votes on only 49.19% of comparisons (48.46% on video, with chance at 25%). Our validation therefore rests on the human study, on judge test-retest self-consistency, and on independent open tools, including human-anchored ones (the GenAI-Arena human ELO at 0.90, n=5; the original Physics-IQ ranking at 1.00, n=3; DINO and CLIP generated-versus-ground-truth feature chains at 0.94 and 0.91, n=15).
Q4 (a physically correct rollout that differs from the recorded truth). This case is exactly what the paired L2 and L3 design is built for, and the paper states it in the Problem Formulation and in Sec. 3: FPAbench does not assume the observed continuation is the unique valid one, and L3 is defined as alignment to one observed continuation, not universal validity. A rollout that is physically plausible but different scores high on L2, which never sees the reference, and lower on L3, so Delta (L2 minus L3) quantifies precisely this "plausible without matching the observed event" quantity on one rubric, scale, judge, and rollout. Humans show the same ordering on the five-level battery, with mean plausibility 3.69 versus alignment 2.79 on identical cards (judge 3.41 versus 2.96), so the dissociation is not an artifact of one model's rating style. Empirically the gap is not an ambiguity artifact either: it is about twice as large on state-transition categories, where the initial state and physical laws determine a specific outcome, than on stable scenes; and a constant-flow baseline that can only extend the input frame attains the best category-level L3 on two stable-scene categories (Perspective and Complex Light) while collapsing on transitions, the opposite of what an input-ambiguity artifact would produce. Humans judging the same single-frame mosaics rank systems consistently with the judge (tau 0.85 to 0.91), so the alignment axis is recoverable from the same evidence by human judgment.
Q5 (are five frames enough?). All systems commit to the same five-frame horizon sampled uniformly over the source clip, so coverage is matched across systems by design. Category construction and manual inspection ensure that the event, such as a collision, a contact, or a state change, is observable within that grid, and the judge always sees the full 1x5 and 2x5 mosaics rather than isolated frames. Two new pieces of evidence indicate that the frame set supports the judgments at issue: humans judge the same mosaics with high consistency (retest 95.6% to 98.0%, inter-annotator agreement 0.93), and the five-frame grid already exposes the layer dissociation the benchmark is designed to detect (15 of 15 systems, 190 of 195 pairs, group-stratified Delta). Longer horizons and multi-seed rollouts are listed as future extensions in the Conclusion.
Novelty relative to [1, 2, 3], and two newer open-source checkpoints
We thank the reviewer for the pointers and the concrete questions. This first comment answers Q1 (novelty relative to [1, 2, 3]) and Q2 (newer models). A second comment answers Q3 to Q5 with a new human study.
Q1 (what is new relative to [1][2][3]). We will add all three benchmarks to our comparison table. Our positioning is as follows.
Physics-IQ [2] is the closest protocol, since it also conditions on a single frame and compares the continuation with a real future. We report the overlap directly: on the systems covered by both, the original Physics-IQ ranking matches our layer scores exactly (Spearman correlation of 1.00, n=3). The difference lies in measurement rather than task. Physics-IQ scores continuations with signal-level metrics and human facet ratings, and it has no reference-free plausibility layer. FPAbench pairs a no-reference plausibility assessment (L2) with a reference-aligned assessment (L3) of the same rollout on the same rubric and scale, which turns the plausibility-alignment shortfall into a measured quantity (Delta) rather than a qualitative claim, and stratifies it by event structure: 0.156 on stable scenes versus 0.308 on state-transition events and 0.275 on long-horizon causal events, with L3 below L2 for 15 of 15 systems and 190 of 195 model-category pairs.
WorldReasonBench [1] (436 cases) conditions on an initial state plus an action or instruction, and evaluates against structured QA ground truth with reasoning-phase verification. FPAbench conditions on the image alone and compares rollouts to the observed future frames under a paired ordinal rubric (eight dimensions; entity, trajectory, interaction, and causal evolution on L3), over 2,000 sequences and 15 systems, with system-level external triangulation and, now, human calibration. Its reported finding, a persistent gap between visual plausibility and world reasoning, is convergent with our central result; the two benchmarks reach it by different measurement routes.
GAUGE [3] measures parameter-level physical-law consistency (accelerations, momentum transfer, oscillation timing) on five controlled rigid-body task families for six I2V models, alongside physics simulation engines. FPAbench measures observed-future alignment at benchmark scale (2,000 real sequences, 13 event categories, 15 systems and growing). The two granularities are complementary rather than competing.
None of [1, 2, 3] contains a reference-free layer paired with a reference layer on identical rollouts and scale, which is the mechanism behind the quantitative claims above.
Q2 (newer models). We extended coverage to the strongest currently deployable open-source checkpoints: HunyuanVideo-1.5 (100-rollout pilot) and LTX-Video-0.9.7-distilled (2,000 rollouts), evaluated under our external-scorer suite alongside the 15 submission systems. Both rank alongside the top canonical I2V models. HunyuanVideo-1.5 reaches 3.68 average VideoScore-v1.1 (3.76 on visual quality and consistency) and 0.72 joint pass rate on VideoPhy-2; LTX-Video-0.9.7-distilled reaches 3.74 and 0.67, against 3.72 and 0.57 for CogVideoX-5B and 3.55 and 0.53 for Wan2.2-5B. Rubric-level L2 and L3 annotation for these two checkpoints is near completion on an independently deployed instance of our teacher (more than 15,000 of 16,800 judgments), and we will report their full layer profile in the revision. Commercial closed-source generators cannot satisfy the standardized protocol: it requires frame-accurate extraction at the five evaluation timepoints, fixed-seed reproducibility, and watermark-free frames, none of which the APIs expose. This is stated in the Limitations, with a near-term extension plan.