acceptodds
Under review as a conference paper at ICLR 2027

From Signal Fidelity to Future-State Alignment: A Hierarchical Evaluation of Video World Models

Abstract

Video generation models are increasingly viewed as candidate world models, yet visually convincing rollouts do not necessarily predict how a depicted scene actually evolves. Existing evaluations often conflate frame-level signal similarity or no-reference video quality with predictive state understanding. This work introduces FPAbench (Fidelity, Plausibility, and Alignment benchmark), a hierarchical evaluation of video world models that separates three questions: Signal Fidelity (L1), measuring frame and motion-signal proximity to an observed future; Video Plausibility (L2), measuring the visual coherence, temporal smoothness, motion, and apparent physical validity of an unconstrained rollout; and Future State Alignment (L3), measuring whether the same rollout preserves entities, trajectories, spatial interactions, and causal state evolution relative to the observed future. FPAbench evaluates 15 video generation systems on 2,000 five-frame sequences spanning 13 physical-event categories. Across every evaluated system, L3 is lower than L2, and the same pattern holds in 190 of 195 model-category comparisons. The gap is largest for state-transition events involving gravity, collisions, fluids, deformation, and precarious configurations, whereas relatively stable scenes are easier. Motion dynamism is not predictive of future-state alignment. These results show that signal fidelity and plausible video appearance are necessary but insufficient evidence of world-model capability, motivating hierarchical evaluation from signals to event-specific future states in observed scenes.

Primary area:
datasets and benchmarks
Keywords:
video generation, world models, benchmark, evaluation, vision-language models, physical plausibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Qho9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FPAbench, a benchmark that evaluates video world models at three levels: Signal Fidelity, Video Plausibility, and Future-State Alignment. The authors test 15 video generation models on 2,000 real-world video sequences across 13 physical event categories.

Critical strengths
  1. This paper try to address an important problem in video world model evaluation.

  2. The authors introduce a benchmark with real-world video sequences and a three-level evaluation strategy.

Critical weaknesses
  1. I think the main idea is same with existing works, like WorldReasonBench, Physics-IQ Verified, and GAUGE. I do not see much novelty in this work.

  2. The paper evaluates 15 video generation systems, but several recent open-source and close-source models are missing (like HunyuanVideo 1.5, LTX 2.5, Kling-Video, Seedance 2.0, Veo 3.1). This makes the reliability of the main conclusions less convincing.

[1] WorldReasonBench, https://arxiv.org/abs/2605.10434, NeurIPS 2026.
[2] Physics-IQ, https://ieeexplore.ieee.org/abstract/document/11492408, WACV 2026.
[3] GAUGE, https://arxiv.org/pdf/2608.05948

Questions for authors
  1. Several recent works (e.g. WorldReasonBench [1], Physics-IQ Verified [2], and GAUGE [3]) have already explored similar problems. Could the authors explain more clearly what is new in FPAbench compared with these benchmarks?

  2. The evaluated models seem somewhat outdated. Could the authors include more recent open-source and close-source models?

  3. Using a VLM as a judge is already common in video evaluation. Could the authors provide a human study to verify the reliability of L2 and L3? I think it would be helpful to compare different and stronger VLM judges (e.g. GPT 6.1-sol/GPT 5.6-sol) with carefully designed prompts, to see whether the results remain consistent.

  4. A single input image may have multiple physically plausible futures. How does the benchmark handle cases where a generated video is physically correct but different from the recorded ground truth?

  5. The evaluation seems to use only five sampled frames. Is this enough to capture short physical events (e.g. collisions and contacts)?

[1] WorldReasonBench, https://arxiv.org/abs/2605.10434, NeurIPS 2026.
[2] Physics-IQ, https://ieeexplore.ieee.org/abstract/document/11492408, WACV 2026.
[3] GAUGE, https://arxiv.org/pdf/2608.05948

2 comments
AuthorsOpenReview ↗

Novelty relative to [1, 2, 3], and two newer open-source checkpoints

We thank the reviewer for the pointers and the concrete questions. This first comment answers Q1 (novelty relative to [1, 2, 3]) and Q2 (newer models). A second comment answers Q3 to Q5 with a new human study.

Q1 (what is new relative to [1][2][3]). We will add all three benchmarks to our comparison table. Our positioning is as follows.

Physics-IQ [2] is the closest protocol, since it also conditions on a single frame and compares the continuation with a real future. We report the overlap directly: on the systems covered by both, the original Physics-IQ ranking matches our layer scores exactly (Spearman correlation of 1.00, n=3). The difference lies in measurement rather than task. Physics-IQ scores continuations with signal-level metrics and human facet ratings, and it has no reference-free plausibility layer. FPAbench pairs a no-reference plausibility assessment (L2) with a reference-aligned assessment (L3) of the same rollout on the same rubric and scale, which turns the plausibility-alignment shortfall into a measured quantity (Delta) rather than a qualitative claim, and stratifies it by event structure: 0.156 on stable scenes versus 0.308 on state-transition events and 0.275 on long-horizon causal events, with L3 below L2 for 15 of 15 systems and 190 of 195 model-category pairs.

WorldReasonBench [1] (436 cases) conditions on an initial state plus an action or instruction, and evaluates against structured QA ground truth with reasoning-phase verification. FPAbench conditions on the image alone and compares rollouts to the observed future frames under a paired ordinal rubric (eight dimensions; entity, trajectory, interaction, and causal evolution on L3), over 2,000 sequences and 15 systems, with system-level external triangulation and, now, human calibration. Its reported finding, a persistent gap between visual plausibility and world reasoning, is convergent with our central result; the two benchmarks reach it by different measurement routes.

GAUGE [3] measures parameter-level physical-law consistency (accelerations, momentum transfer, oscillation timing) on five controlled rigid-body task families for six I2V models, alongside physics simulation engines. FPAbench measures observed-future alignment at benchmark scale (2,000 real sequences, 13 event categories, 15 systems and growing). The two granularities are complementary rather than competing.

None of [1, 2, 3] contains a reference-free layer paired with a reference layer on identical rollouts and scale, which is the mechanism behind the quantitative claims above.

Q2 (newer models). We extended coverage to the strongest currently deployable open-source checkpoints: HunyuanVideo-1.5 (100-rollout pilot) and LTX-Video-0.9.7-distilled (2,000 rollouts), evaluated under our external-scorer suite alongside the 15 submission systems. Both rank alongside the top canonical I2V models. HunyuanVideo-1.5 reaches 3.68 average VideoScore-v1.1 (3.76 on visual quality and consistency) and 0.72 joint pass rate on VideoPhy-2; LTX-Video-0.9.7-distilled reaches 3.74 and 0.67, against 3.72 and 0.57 for CogVideoX-5B and 3.55 and 0.53 for Wan2.2-5B. Rubric-level L2 and L3 annotation for these two checkpoints is near completion on an independently deployed instance of our teacher (more than 15,000 of 16,800 judgments), and we will report their full layer profile in the revision. Commercial closed-source generators cannot satisfy the standardized protocol: it requires frame-accurate extraction at the five evaluation timepoints, fixed-seed reproducibility, and watermark-free frames, none of which the APIs expose. This is stated in the Limitations, with a near-term extension plan.

AuthorsOpenReview ↗

The human study, judge reliability, alternative futures, and five-frame coverage

This second comment answers Q3 to Q5. The new evidence used here is a blinded human preference study: more than 2,300 judgments on anonymized rollouts, comprising 1,418 pairwise verdicts covering all 105 system pairs at each layer, 660 five-level ratings, and 255 per-case rankings. Pooled human rankings agree with our VLM judge at the system level (Kendall tau of 0.905 for L2 and 0.848 for L3). On the 25-pair core battery judged by every annotator, inter-annotator agreement is 0.928 and 0.935, and case-level agreement with the judge rises from about 45% on near-tied pairs to 95.7% (L2) and 87.9% (L3) on well-separated pairs. Five-level ratings give pooled quadratic-weighted kappa of 0.669 and 0.652; same-rater retest consistency is 95.6% and 98.0%. Judge re-annotation test-retest gives kappa of 0.980 (95.2% exact).

Q3 (human study; stronger closed judges). The human study is now done, with the numbers above, and we will add it to the paper. We cannot run GPT-class closed judges under the same standardized protocol because they do not provide frame-level access. We also note that judge strength is not automatic: in GenAI-Arena, GPT-4o used as a judge agrees with human arena votes on only 49.19% of comparisons (48.46% on video, with chance at 25%). Our validation therefore rests on the human study, on judge test-retest self-consistency, and on independent open tools, including human-anchored ones (the GenAI-Arena human ELO at 0.90, n=5; the original Physics-IQ ranking at 1.00, n=3; DINO and CLIP generated-versus-ground-truth feature chains at 0.94 and 0.91, n=15).

Q4 (a physically correct rollout that differs from the recorded truth). This case is exactly what the paired L2 and L3 design is built for, and the paper states it in the Problem Formulation and in Sec. 3: FPAbench does not assume the observed continuation is the unique valid one, and L3 is defined as alignment to one observed continuation, not universal validity. A rollout that is physically plausible but different scores high on L2, which never sees the reference, and lower on L3, so Delta (L2 minus L3) quantifies precisely this "plausible without matching the observed event" quantity on one rubric, scale, judge, and rollout. Humans show the same ordering on the five-level battery, with mean plausibility 3.69 versus alignment 2.79 on identical cards (judge 3.41 versus 2.96), so the dissociation is not an artifact of one model's rating style. Empirically the gap is not an ambiguity artifact either: it is about twice as large on state-transition categories, where the initial state and physical laws determine a specific outcome, than on stable scenes; and a constant-flow baseline that can only extend the input frame attains the best category-level L3 on two stable-scene categories (Perspective and Complex Light) while collapsing on transitions, the opposite of what an input-ambiguity artifact would produce. Humans judging the same single-frame mosaics rank systems consistently with the judge (tau 0.85 to 0.91), so the alignment axis is recoverable from the same evidence by human judgment.

Q5 (are five frames enough?). All systems commit to the same five-frame horizon sampled uniformly over the source clip, so coverage is matched across systems by design. Category construction and manual inspection ensure that the event, such as a collision, a contact, or a state change, is observable within that grid, and the judge always sees the full 1x5 and 2x5 mosaics rather than isolated frames. Two new pieces of evidence indicate that the frame set supports the judgments at issue: humans judge the same mosaics with high consistency (retest 95.6% to 98.0%, inter-annotator agreement 0.93), and the five-frame grid already exposes the layer dissociation the benchmark is designed to detect (15 of 15 systems, 190 of 195 pairs, group-stratified Delta). Longer horizons and multi-seed rollouts are listed as future extensions in the Conclusion.

Reviewer_eEyAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FPAbench, a benchmark that distinguishes signal fidelity (L1), reference-free video plausibility (L2), and alignment with an observed future (L3) when evaluating video world models. Its contributions include a three-level evaluation framework, a dataset of 2,000 five-frame real-world sequences covering 13 event categories, and a Puzzle Protocol that presents generated and reference sequences in temporally aligned mosaics. Models receive only the initial frame. L1 uses image and motion-signal metrics, while L2 and L3 use Qwen2.5-VL-72B to assess rollout plausibility and consistency with observed entities, trajectories, interactions, and event evolution. The empirical study evaluates 15 systems and reports lower L3 than L2 scores for every system and 190 of 195 model–category pairs, with larger gaps for state-transition events. These findings motivate reporting visual plausibility and observed-future prediction performance separately.

Critical strengths

1.Clear separation of evaluation targets. Distinguishing signal fidelity, video plausibility, and observed-future alignment helps prevent visual quality from being mistaken for predictive capability. 2.Diagnostic empirical coverage. Evaluating 15 systems on 2,000 real-world sequences across 13 event categories provides detailed insights into prediction failures that aggregate rankings may obscure.

Critical weaknesses

1.Limited independent validation of the scoring protocol. L2 and L3 rely on a single VLM without human calibration, leaving the reliability of model rankings and the plausibility–alignment gap insufficiently validated. 2.Limited empirical support for the incremental value. Although the paper explains its framework and protocol differences, the additional diagnostic value over existing evaluations is not sufficiently demonstrated.

Questions for authors

1.Scoring reliability: Given the absence of human calibration, what independent validation or robustness evidence supports the VLM scores and the interpretation of the L2–L3 gap? 2.Incremental diagnostic value: What additional failure modes do the hierarchy and Puzzle Protocol reveal beyond existing evaluations, and what comparative evidence supports this?

1 comment
AuthorsOpenReview ↗

Human validation of the scoring protocol and its incremental value

We thank the reviewer for the constructive summary and the two concrete questions. Both concerns, namely independent validation of the scoring protocol and demonstrated incremental value, are addressed by evidence completed after the submission deadline. We summarize it here and answer each question below.

New evidence.

  1. A blinded human preference study. Annotators blinded to system identity provided 1,418 pairwise verdicts spanning all 105 system pairs at each layer (rollouts anonymized, presentation order randomized), together with 660 five-level ratings and 255 per-case system rankings, more than 2,300 judgments in total. Pooled human preference rankings correlate with the VLM judge at the system level (Kendall tau of 0.905 for L2 and 0.848 for L3 across all 15 systems). On the 25-pair core battery judged by every annotator, inter-annotator agreement is 0.928 (L2) and 0.935 (L3). Case-level agreement with the judge, stratified by the pre-registered terciles of teacher score separation, rises from 45.5% (L2) and 43.6% (L3) on near-tied pairs, through 86.0% and 85.6% at medium separation, to 95.7% (L2) and 87.9% (L3) on well-separated pairs. The five-level battery (bad, poor, fair, good, excellent; one word per judgment plus a confidence distribution) agrees with the judge at pooled quadratic-weighted kappa of 0.669 (L2) and 0.652 (L3), and the same annotators reproduce 95.6% to 98.0% of their own answers on retest. On the 24 shared rating cards, humans rate plausibility above alignment on the identical scale (mean 3.69 versus 2.79), the same ordering the judge produces on these cards (3.41 versus 2.96).

  2. Judge self-consistency. Re-annotating the same rollouts in an independent run reproduces the judge's scores at quadratic-weighted kappa of 0.980 (95.2% exact), which bounds within-judge instability well below the between-system effects we interpret.

  3. External triangulation. Three mutually independent tool chains correlate with our layer scores. Reference-free feature chains (DINOv2 and CLIP, generated against ground truth) correlate with L3 at 0.94 and 0.91 (n=15, p<0.001). Literature anchors reproduce our ordering: the VBench-I2V i2v_score correlates with L3 at 1.00 (n=6, exact p=0.0028), while VBench's own total_score reaches only 0.77 (not significant) on the same systems; the GenAI-Arena human ELO correlates at 0.90 (n=5) and the original Physics-IQ ranking at 1.00 (n=3). Finally, an upgraded rubric re-annotation of all rollouts (system-level ranking correlates 0.991 with the submission) was crossed with 26 external metrics: AIGCBench image-video CLIP 0.939, VBench-I2V subject and background 0.86 and 0.88, VideoScore2-physics 0.90 against L2 and 0.75 against L3, and DOVER-technical 0.62.

Q1 (scoring reliability). The human study directly calibrates both the rankings and the interpretation of the L2 to L3 gap. Humans, judging the same single-frame mosaics under the same definitions, rank the systems as the judge does (tau 0.85 to 0.91), agree with it on individual pair verdicts at 88 to 96% wherever pairs are well separated, and rate plausibility above alignment on identical cards exactly as the judge does (3.69 versus 2.79; judge 3.41 versus 2.96). The gap itself is a paired measurement by construction: L2 and L3 share the rubric, scale, judge, and rollouts, and the human five-level battery rates both layers independently. We will add this study, together with the self-consistency and triangulation results, to the paper.

Q2 (incremental diagnostic value). The triangulation above is also the comparative evidence. Instruments that measure visual quality or aesthetics track L2 (VideoScore2-physics 0.90, DOVER-technical 0.62) but are substantially weaker or absent for L3 (the same tool at 0.75; general-quality Q-ReAlign at 0.07, not significant; aesthetics at 0.14, not significant; frame-level IQA not significant; VideoPhy-2 automatic scores not significant). In other words, existing quality machinery covers plausibility, and observed-future alignment is the axis those instruments do not measure, which is what the hierarchy is for. The paired protocol also surfaces failure modes that aggregate or quality scores do not: the plausibility-alignment gap is about twice as large on state-transition events (0.308) and long-horizon causal events (0.275) as on stable scenes (0.156, CI [0.150, 0.162]); L2 Motion Dynamism is negatively associated with L3 at the model level (Spearman -0.67), so more visible motion does not buy event alignment; and Causal Evolution collapses precisely on state-transition categories (Gravity Motion 0.013). We will add a comparison covering Physics-IQ, WorldReasonBench, and GAUGE, together with the human-study section, in the revision.

Reviewer_94LmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces a hierarchical evaluation framework (FPAbench) for video generation, which consists of three layers of metrics: Signal Fidelity (L1), Video Plausibility (L2), and Future-State Alignment (L3). L1 consists of some existing frame-level visual or motion-signal similarity metrics. L2 utilizes a Qwen2.5-VL-72B model to check the quality of generated frames without references, while L3 uses the same VL model to evaluate the similarity between generated and ground-truth frames. Besides, the authors collect a dataset containing 2,000 five-frame sequences and evaluate 15 video generation systems with the above metrics and dataset. The results show that all systems have a lower L3 score than L2, and the L2-L3 gap for state-transition events is larger than for stable visual scenes. Based on the gap, the authors claim that signal fidelity, visible motion, and plausible appearance are not enough to suggest that a generator predicts a specific event.

Critical strengths
  1. The topic is important. Since video generation models have been applied broadly, the evaluation of these models is necessary.

  2. The presentation is easy to follow. The figures in the paper are illustrative, and the paper is well structured.

Critical weaknesses
  1. The experiments are inadequate. First, the VLM judge is unverified. As written in the code, the prompt for the judge is short and lacks a detailed description, which may affect the performance of the judge. Because the judge has a direct impact on the experiment results, this reviewer considers that it’s important to guarantee the correctness of the judge. Second, some open-source SOTA models are not evaluated, such as LongCat-Video and HunyuanVideo-1.5. More powerful models are needed to evaluate and suggest the value of the proposed benchmark.

  2. The soundness and the novelty of the paper are weak. First, the novelty of the L1, L2, and the dataset is limited. L1 and L2 metrics are almost borrowed from prior work, while the dataset offers no particular or superior advantages compared with other datasets. Second, the design of L1 and L3 metrics is flawed. As claimed in lines 209-210, FPAbench uses the first sampled frame as the only model input, which provides limited information and can hardly guarantee the uniqueness of the answer. Therefore, comparing the generated rollout with one specific reference is practically meaningless. Third, the frame sampling and aligning methods are unclear. Since the authors compare sampled frames between two rollouts, the time points of different frames across rollouts should be aligned first to make an exact comparison. However, this paper does not present a valid measure to ensure the above alignment.

Questions for authors
  1. Why provide only the first frame as input for the target model? The information in one frame is probably insufficient for the model output to match the only reference. Therefore, the low scores on L3 are somewhat predictable and may not be entirely attributable to model capability.

  2. What’s the practical meaning of L3 metrics in terms of somewhat specific model performance? As mentioned in the first question, limited input information can hardly guarantee the uniqueness of the answer. This impairs the meaning of checking whether the generated rollout adheres to the set answer.

  3. For ModelScopeT2V, how to input an image to a text-to-video model? For RAFT, which usually requests two frames, how to generate the video with only the first frame? Because the input format requirements of these two systems contradict the claim “uses the first sampled frame as the only model input”, further explanation is needed for better technical correctness.

  4. Why not verify the correctness of the VLM judge? Instability or bias of the judge will directly affect the L2 and L3 scores. What’s more, the prompt for evaluation is simple and underspecified, which further raises concerns about the performance of the judge. Could the authors conduct a simple verification experiment to ensure the robustness of the main experiment results?

  5. Which protocol item do the commercial closed-source generators violate? Could they satisfy the protocol with minor adaptation, and if so, can a small-scale evaluation be reported? Why not compare the later open-source models? This reviewer wonders about the performance of the SOTA models on this benchmark.

  6. What’s the standard for sampling frames from the collected videos? How to align the original and generated frames to ensure time consistency? Because of the frame comparison across rollouts, most L1 and L3 metrics are sensitive to temporal offsets. Time point misalignment between two rollouts would penalize models for timing rather than physical understanding.

3 comments
AuthorsOpenReview ↗

On the single input frame and what L3 measures

We thank the reviewer for a careful reading, and for questions that exposed one sentence needing correction (addressed in our second comment) and several points where new evidence, completed after the submission deadline, directly answers the soundness concerns. This comment answers Q1 and Q2 on the single input frame.

New evidence used here (details in the later comments): a blinded human study, over 2,300 judgments on anonymized rollouts generated from a single input frame under the paper's mosaic protocol. System rankings agree with our VLM judge at Kendall tau of 0.905 (L2) and 0.848 (L3) across the 15 systems, and case-level agreement rises from about 45% on near-tied pairs to 95.7% and 87.9% on well-separated pairs.

Q1 and Q2 (one frame admits multiple plausible futures; is low L3 therefore predictable, and is L3 meaningful?). We agree that one frame admits multiple physically plausible continuations, and the paper says so twice: the Problem Formulation states that a single image can admit multiple physically plausible continuations and that L3 measures alignment to the observed continuation rather than universal physical validity, and the Dataset section states that the protocol does not assume the observed continuation is the unique physically valid one. The benchmark's answer to non-uniqueness is structural rather than rhetorical. L2 exists precisely to credit plausible but different rollouts, because its judge never sees the reference, so a rollout that is physically correct but different from the recording scores high on L2 and lower on L3, and Delta (L2 minus L3) measures that quantity on one rubric, scale, judge, and rollout. The claim attached to L3 is deliberately narrow: diagnostic alignment to the observed event, not a verdict on unobserved alternatives (Sec. 3).

The data also rule out the specific failure mode the reviewer predicts, namely that low L3 reflects input insufficiency rather than model capability:

  1. The gap is event-structured, not uniform. If one-frame ambiguity drove L3 down, the gap should be largest where futures are least determined. The opposite holds. Delta is 0.156 (95% CI [0.150, 0.162]) on stable visual scenes but 0.308 on state-transition and 0.275 on long-horizon causal events, the categories where the initial state plus physical laws determine a specific outcome such as a collision, a fall, or fluid behavior. L3 is low exactly where event prediction is required, not where ambiguity is highest.

  2. L3 is attainable from one frame when the future is visually determined. The constant-flow baseline, which can only warp the input frame, attains the best category-level L3 among all 15 systems on two of the four stable-scene categories (Perspective and Complex Light, Table 2). A trivial single-frame method succeeds on precisely the categories with the smallest gap and collapses on transitions. Input limitation cannot explain that pattern.

  3. Humans judge the same single-frame mosaics consistently. Under the same protocol and definitions, blinded humans rank the 15 systems as the judge does (tau 0.85 to 0.91), agree on pair verdicts from about 45% on near-tied pairs to 95.7% and 87.9% on well-separated ones, and reproduce the plausibility-above-alignment ordering on identical five-level cards (3.69 versus 2.79; judge 3.41 versus 2.96). The benchmark's central dissociation is therefore recoverable by human judgment from the same evidence, which indicates that the task is well-posed at the level the benchmark operates.

  4. The headline claims are coverage counts rather than averages driven by a few ambiguous cases: 15 of 15 systems and 190 of 195 model-category pairs show L3 below L2.

In sum, the benchmark does not claim that the observed future is the only valid one. It claims that failing to reproduce the observed event while scoring well on reference-free plausibility is diagnostic, and that claim is supported by the L2 and L3 measures, by Delta, by the group stratification, and now by the human study.

AuthorsOpenReview ↗

The ModelScopeT2V and RAFT baselines, and verification of the judge

This second comment answers Q3 (ModelScopeT2V and RAFT) and Q4 (verifying the VLM judge).

Q3a (how is an image given to a text-to-video model?). ModelScopeT2V is deliberately included as a text-to-video control. It does not receive the image; it receives a caption derived from the input image, as stated in the Evaluation Setup and in the Evaluated Systems paragraph ("a text-to-video control that receives a caption derived from the input image rather than direct image conditioning"). It probes the conditioning modality under the identical five-frame protocol, and its failed outputs are retained in reported coverage rather than selectively rerun. We will make this control role more prominent in the revision so that it cannot be read as an image-input claim.

Q3b (RAFT with only one frame?). The baseline is implemented as constant-flow extrapolation: the input frame is warped with a fixed, linearly scaled analytic motion field, using no observed future frames. The clause "estimated by RAFT on a synthetically shifted copy of that frame" over-specified the mechanism, and we will correct it in the revision. The supplement's system table already describes this baseline as a "constant-flow extrapolation baseline; fixed implementation settings", which remains accurate, and no reported number depends on the corrected clause. RAFT remains cited for the optical-flow tool family. We appreciate the reviewer catching the inconsistent reading.

Q4 (verify the judge; the prompt is short and underspecified). Three answers.

  1. The prompts are not underspecified. The released code (evaluate_l2_l3.py, included in the supplementary archive) contains the full templates. Each of the eight dimensions (four for L2, four for L3) is scored in an independent API call with explicit ordinal anchors (1.0, 0.5, and 0.0, each defined by concrete artifact criteria such as mosaic artifacts, flicker, or smear for L2, and entity, trajectory, interaction, and state-evolution criteria for L3), a strictness preamble, chain-of-thought-guided structured JSON output, robust parsing, retries, and resumable execution (paper: "Puzzle Protocol and Judge Safeguards"). The concision is by design: the anchors, rather than lengthy instructions, define the scale.

  2. Self-consistency. Re-annotating the same rollouts in an independent run reproduces the judge's scores at quadratic-weighted kappa of 0.980 (95.2% exact), which bounds within-judge instability well below the between-system effects we interpret.

  3. Human validation (new). Blinded annotators judged anonymized rollouts under the same mosaic protocol, more than 2,300 judgments in total. Pooled human rankings agree with the judge at the system level (tau of 0.905 for L2 and 0.848 for L3). On the 25-pair core battery judged by every annotator, inter-annotator agreement is 0.928 and 0.935, and human-judge case agreement rises from about 45% on near-tied pairs to 95.7% and 87.9% on well-separated pairs. Five-level ratings give pooled kappa of 0.669 and 0.652 and reproduce the plausibility-above-alignment ordering (3.69 versus 2.79; judge 3.41 versus 2.96). Same-rater retest consistency is 95.6% and 98.0%.

The layer scores are additionally corroborated by judge-free evidence: DINOv2 and CLIP generated-versus-ground-truth feature chains correlate with L3 at 0.94 and 0.91 (n=15, p<0.001), and the literature anchors hold (the VBench-I2V i2v_score against L3 at 1.00, n=6, exact p=0.0028; the GenAI-Arena human ELO at 0.90; the original Physics-IQ ranking at 1.00), so the main results do not hinge on the wording of one prompt.

AuthorsOpenReview ↗

Closed-source and newer open models, temporal alignment, and novelty

This third comment answers Q5 (closed-source and newer models), Q6 (frame sampling and alignment), and the novelty points in Critical Weakness 2.

Q5. Commercial closed APIs cannot satisfy the standardized rollout protocol: frame-accurate extraction at the five evaluation timepoints, fixed-seed reproducibility, and watermark-free frames are not exposed. This is stated in the Limitations with a near-term extension plan. For open-source systems we have added HunyuanVideo-1.5 (100-rollout pilot; 3.68 average VideoScore-v1.1 and 0.72 joint pass rate on VideoPhy-2) and LTX-Video-0.9.7-distilled (2,000 rollouts; 3.74 and 0.67), evaluated under our external-scorer suite alongside the 15 submission systems, against 3.72 and 0.57 for CogVideoX-5B and 3.55 and 0.53 for Wan2.2-5B. Their rubric-level L2 and L3 profile is being finalized on an independently deployed teacher instance (more than 15,000 of 16,800 judgments complete) and will be added in the revision.

Q6 (sampling standard; temporal alignment). Every sequence defines five evaluation timepoints uniformly spanning the source clip. Each system generates with its official or author-recommended configuration, and its rollout is resampled to the same five timepoints, so L1, L2, and L3 always compare matched temporal positions. In the L3 mosaic the ground-truth row and the generated row share those matched positions, with the input frame as the first column of the ground-truth row (Sec. 3). Because horizon and grid are identical for all systems under the paired design, residual misalignment within a timepoint is common-mode measurement noise rather than a per-model penalty. Per-system generation length, frame rate, and resampling are documented in the supplementary technical document, and manual inspection before inclusion ensures that the event is observable within the grid (Dataset section).

Novelty (Critical Weakness 2). We do not claim novelty for the L1 instruments: PSNR, SSIM, LPIPS, and optical-flow endpoint error are deliberately the field's standard fidelity floor, and the paper positions them as such ("the contribution is not the use of a VLM judge alone, but its role in maintaining the separation of evaluation layers"). The claimed contribution is the ordered three-layer protocol and the paired no-reference and reference measurement (Delta), which no prior instrument provides on identical rollouts. The triangulation above supports that the L3 axis is not covered by existing machinery: quality tools track L2 at 0.90 and 0.62, while aesthetics, general-quality, and frame-level IQA instruments are uncorrelated with L3 (0.14, 0.07, and not significant). For the dataset, 2,000 real sequences organized by event structure (13 categories in three diagnostic groups) is what makes the central dissociation measurable; the stable-versus-transition separation is constructed rather than incidental, and every sequence was manually inspected. We will extend the benchmark-comparison table with Physics-IQ, WorldReasonBench, and GAUGE in the revision.

All revisions (the human-study section, judge-reliability reporting, the corrected RAFT sentence, the extended comparison table, and the newer-checkpoint profile) will be marked in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.