- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks at how auxiliary text (in particular execution logs) can bias multimodal verifiers in video generation agents. By keeping the video content constant and varying the supporting/contradicting text, they demonstrate that the judges can be led to rely on the agent's own reported actions rather than the actual video evidence. The main contributions are a controlled contamination audit, a manually labelled benchmark of 109 generated clips, an analysis of how verification errors can still undermine the repair loops even when you're not dealing with an adversarial planner and finally least privilege judging: only sending the evidence that is relevant to a particular requirement (e.g. frames for a visual check, traces for a process check) with OCR masking of the embedded captions.
Validation involves UCF101 videos, generated clips with incomplete actions, multi-shot consistency tests and simulated repair loops. Across three Qwen-VL judges the supporting logs lead to an increase in false acceptances; the instruction to ignore text only partially offsets this. The closed model judges are substantially less affected. In the 8B repair simulation the honest planner achieves 100% judge reported success but only 28% success according to human labels. Least privilege judging leads to an increase in actual success whilst maintaining high process check accuracy and requiring more regeneration. These results do support the idea of separating evidence by requirement type, albeit the validation is constrained by the small author labelled benchmark, the single open weight model family and the fact that the repair workflows are simulated rather than actual deployed workflows.
Critical strengths
Holding the video content fixed and varying the auxiliary text shows that execution logs can trump visual evidence. The large number of false acceptances across the three Qwen-VL judges, and the comparison against the less susceptible closed model judges, makes the result quite significant and appropriately model-specific. This is a very strong piece of evidence that evidence selection is a key design decision in the evaluation of video agents.
The repair-loop experiments demonstrate that contamination can actually damage an honest planner, yielding 100% judge-reported success and only 28% human-labelled success. Least privilege judging increases actual success in the tested configurations, whilst maintaining high process check accuracy and making the extra regeneration cost explicit. The loops are of course simulated but the link between diagnosis, practical consequences and mitigation is a very significant contribution to the paper.
Critical weaknesses
The central generated video benchmark has just 109 author labelled clips from a single generator, filters out the ambiguous cases and evaluates open weight judges from a single model family. Some of the closed model comparisons use dramatically smaller subsets (e.g. only 26 clips for Claude Opus 5). This establishes a very clear vulnerability in the tested configurations but provides very little support for the more general conclusion that open vs closed judges or learned trust in the tool logs is in any way the issue. This is important because the significance of the paper is very much predicated on how widely that failure generalises.
The repair loop commences from 36 known failures and models regeneration via sampling of the labelled clips with a pre-determined success probability. A large proportion of the outcome is governed by false acceptance prior to any repair so matching the ceiling (1-alpha_0) essentially just verifies the simulation's stopping rule rather than independently validating the repair dynamics. The reported success rates therefore do not demonstrate that least privilege judging provides the same benefit as actual regeneration, evolving traces and realistic planner decisions. An end to end experiment would be a significant step in supporting the core claim of practical benefit.
Questions for authors
How is the assumed 85% regeneration success probability implemented when sampling the finite, per-prompt clip pool? Can failed regenerations also be falsely accepted in later rounds? Please clarify why the observed best true pass rate consistently reaches (1-alpha_0), despite possible later-round false accepts and the four-round limit. This affects how much independent evidence the simulation provides for the mitigation’s effectiveness.