acceptodds
Under review as a conference paper at ICLR 2027

VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement

Abstract

Video generative models are increasingly used to simulate physical interactions and support downstream decisions, yet their evaluators still tend to return scalar scores or free-text flaw lists. Such outputs make it difficult to tell which prompt requirement was satisfied, what evidence was resolved, and where physics laws were silently disobeyed. We present VeriPhy, a trace-first critic that evaluates generated video through executable, claim-bound evidence. VeriPhy decomposes the prompt into span-bound claims and a typed check plan. Then it composes a tool chain for each plan, and performs execution that records each check as measured, abstained, or errored. These records are finally assembled into verdicts that determine the physics alignment of each video. This trace supports diagnosis, learning, and refinement. Such a planning architecture also allows the introduction of in context learning capabilities, training free by prompt injection. VeriPhy demonstrated strong learning performance from human annotations, raising the flaws found on held-out clips significantly, and correlates more strongly with human semantic-adherence ratings than VideoScore2. Finally, VeriPhy can be directly wired back into the generation process for trace-guided prompt rewrite, and demonstrated better generation performance training free. Together, these results position executable evidence traces as an interface for auditing, improving, and using physical-video evaluators.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video generation evaluation, Physical plausibility, Agentic reasoning, Tool-augmented verification, Critic-guided refinement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_exdYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VeriPhy reframes video generation evaluation as an "executable evidence trace." It begins by extracting assertions, anchored to specific segments of the source text, from the prompt. These are compiled into a typed verification plan, which is then executed using a suite of tools (including segmentation, tracking, counting, depth estimation, OCR, and vision-language models). Each measurement is categorized as a valid assessment, an "unmeasurable" case, or an infrastructure error. These results are then aggregated, based on fixed rules, into assertion statuses and an overall video-level verdict. This same tracing mechanism is also leveraged for empirical distillation (without weight updates) and prompt rewriting, helping to diagnose evaluation failures and guide improvements in the generation process.

Critical strengths

Evaluation conclusions are fully traceable back to specific prompt segments, tool logs, and aggregation rules; the clear chain of evidence facilitates human review and failure attribution.

It explicitly distinguishes between valid negative findings, "unmeasurable" instances, and infrastructure errors, preventing cases where a measurement cannot be performed from being misconstrued as a "video generation error."

Critical weaknesses

The annotated dataset consists exclusively of videos containing defects. It is primarily designed for recall-based evaluation, making it difficult to measure false-positive rates or precision, which results in an incomplete assessment.

Components such as the planner, semantic checkers, verification models, and text-matching judges can introduce model bias, sensitivity to prompts, and inconsistent decision-making.

The verification phase is conservative, allowing only for the reclassification of "supported" claims to "contradicted", which may lead to missing instances where an initial "supported" judgment should have been corrected.

Questions for authors

Please refer to the weaknesses.

Reviewer_bmHpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VeriPhy evaluates generated videos by binding claims to prompt spans, compiling typed verification programs, recording measurement status and provenance, and aggregating evidence through three-valued rules. The same traces support failure attribution, instruction-based adaptation from annotated misses, and prompt rewriting for generation refinement.

Critical strengths

The evidence interface is a useful systems contribution. Binding claims to prompt spans and separating negative measurements from abstentions or infrastructure failures makes the evaluator easier to audit than a scalar score. Recorded pre-review and post-review states also support failure attribution.

The held-out lesson experiment reports a recall increase from 340/502 to 375/502 flaws, with prompt and clip overlap excluded between experience and evaluation data. Together with the VideoPhy-2 rank-correlation result, this provides evidence that the traces can support useful evaluator adaptation, beyond merely explaining outputs (Sections 4.3 and D).

Critical weaknesses

The evaluation does not yet isolate reliable gains in physical assessment. The main comparison reports 228/304 flaws for VeriPhy versus 222/304 for the single-pass backbone, but only VeriPhy includes the GPT-5.6 Sol review stage (Table 9). The 83 matches to review-stage findings are not necessarily unique additions; a matched pre-review total is missing under the R1 protocol. R5 uses a different scorer and denominator. Moreover, the core contains only seven physical-motion flaws, and false-positive behavior is not established: the 26% finding-to-annotation match rate is not a precision estimate (Appendices E–F). These issues limit attribution of the gain and the broader physical-reliability claim.

The refinement claim lacks clear independent quantitative support. VeriPhy's own judgments improve on 49/65 initially implausible acted clips, but the independent AutoRater shows little aggregate change: joint agreement is 22.8% before versus 23.1% after, and mean physical commonsense is 3.59 versus 3.58 (Table 17). The denominators differ, 600 versus 589. A paired analysis of the edited subset is needed to distinguish a useful feedback-loop demonstration from demonstrated improvement in video physics.

Questions for authors

Under the same clips and matching protocol, what are the results for VeriPhy without review and for the single-pass baseline with the same review model? Which findings are unique to the review stage?

Can an adjudicated sample of unmatched findings establish whether they are real additional flaws or false positives? For the lesson gain, would uncertainty estimated by resampling clips, rather than treating all flaws independently, change the conclusion?

On the 589 paired clips, and separately on the acted clips, what do the independent ratings show? Please reconcile those results with the stronger refinement language in the abstract.

Reviewer_wzkUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VeriPhy. It discards scalar scores and free‑text flaw lists. Its evidence‑driven interface logs tested requirements, checking steps and measurement results. This makes evaluator failures locatable and reusable: missed flaws become planner instructions, and cited failures become rewritten prompts. Held-out recall rises from 67.7% to 74.7% after adaptation, and rank agreement with human ratings improves, which supports the direction.

The method assembles existing models and tools. Its main contribution is a shared evidence record linking planning, measurement, verdicts, and feedback. Evidence for cost efficiency, physics-specific checking, and generation refinement stays weaker, and those parts of the claim should be read with that in mind.

Critical strengths

S1. The framework establishes an auditable evidence interface for generated-video evaluation. Each verdict is linked to a prompt span, claim, verification program, and evidence record. This allows users to trace decisions and identify failures in requirement decomposition, visual verification, or aggregation. The interface has practical value for analyzing evaluator behavior.

S2. The framework explicitly distinguishes negative measurements from unsuccessful measurements. The measured, abstained, and errored states separate valid results, insufficient evidence, and infrastructure failures while preserving unknown outcomes. Temporal ordering checks establish event occurrence before comparing time windows, ensuring that measurement failures and actual violations receive different treatment.

S3. Evidence-based diagnosis can produce effective planning instructions. On 306 held-out clips, adding 33 lessons increases the number of detected flaws from 340 to 375, raising recall from 67.7% to 74.7%. This supports using missed-flaw records to improve planning strategies without updating model weights.

S4. The method achieves competitive evaluation results while retaining traceable outputs. Development-core flaw recall is 75.0%, close to the same-backbone single-pass baseline’s 73.0%. On 200 VideoPhy-2 clips, its Spearman correlation with human semantic-adherence ratings is 0.465, compared with 0.327 for VideoScore2. These results suggest that structured evaluation can provide inspectable outputs while maintaining useful detection and ranking performance.

Critical weaknesses

W1. Each component builds on existing work; the real advance is evidence binding and workflow organization. But the paper does not separate what each design choice adds. Proposition 1 only proves monotonicity of the aggregation rules, so its theoretical scope stops at decision logic.

W2. The 75.0% recall includes a closed-source review stage that supplies 83 of 228 findings. Calls run 14.1 per clip versus one for the single-pass baseline, and adaptation raises claims per clip from 9.5 to 16.7. Nothing in the current results separates the contributions of traces, review, and finer checking.

W3. Physical failures are reported together with action, counting, and text failures, so the benefit of physical checks cannot be isolated. Single-annotator labels and a corpus where every clip carries a flaw also prevent any estimate of false positives or annotation consistency.

Questions for authors
  1. Which capability follows specifically from binding claims to prompt spans and recording measurement status, rather than from the tools themselves? A worked example from the existing traces would help clarify this distinction.

  2. Can the authors report results for the same configuration with the review stage removed? The current numbers mix the review model's findings with those of the trace-based checks.

  3. How much of the 9.5 to 16.7 increase in claims per clip, rather than the lessons themselves, drives the held-out gain? A comparison against simply asking for more claims would separate the two.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.