- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VeriPhy reframes video generation evaluation as an "executable evidence trace." It begins by extracting assertions, anchored to specific segments of the source text, from the prompt. These are compiled into a typed verification plan, which is then executed using a suite of tools (including segmentation, tracking, counting, depth estimation, OCR, and vision-language models). Each measurement is categorized as a valid assessment, an "unmeasurable" case, or an infrastructure error. These results are then aggregated, based on fixed rules, into assertion statuses and an overall video-level verdict. This same tracing mechanism is also leveraged for empirical distillation (without weight updates) and prompt rewriting, helping to diagnose evaluation failures and guide improvements in the generation process.
Critical strengths
Evaluation conclusions are fully traceable back to specific prompt segments, tool logs, and aggregation rules; the clear chain of evidence facilitates human review and failure attribution.
It explicitly distinguishes between valid negative findings, "unmeasurable" instances, and infrastructure errors, preventing cases where a measurement cannot be performed from being misconstrued as a "video generation error."
Critical weaknesses
The annotated dataset consists exclusively of videos containing defects. It is primarily designed for recall-based evaluation, making it difficult to measure false-positive rates or precision, which results in an incomplete assessment.
Components such as the planner, semantic checkers, verification models, and text-matching judges can introduce model bias, sensitivity to prompts, and inconsistent decision-making.
The verification phase is conservative, allowing only for the reclassification of "supported" claims to "contradicted", which may lead to missing instances where an initial "supported" judgment should have been corrected.
Questions for authors
Please refer to the weaknesses.