acceptodds
Under review as a conference paper at ICLR 2027

Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

Abstract

Agentic video-generation systems close a loop between a generator and a verifier: an LLM plans shots, calls a text-to-video model, and a multimodal judge decides whether the result satisfies the request. To diagnose where a long workflow fails, recent harnesses deliberately show the judge more than the video—the agent's execution trace, its plan, the narration it synthesized. We ask whether this auxiliary text moves the judge's verdict on purely visual requirements, holding the frames fixed. On a benchmark of 109 generated two-event clips with manual labels, in which the requested event is either visibly completed or visibly missing (`near-miss' failures, the common case in practice), a trace that reports a successful tool call makes three open-weight Qwen-VL judges (7B, 8B, 32B) accept – of the failures, up from – without text, and a contradicting trace makes them reject up to of correct clips; an instruction to “use only the frames” does not remove the effect. Frontier closed judges (GPT-5.4-mini, GPT-5.5, Claude Opus 5, and DeepSeek-V4 reading a vision tool's description) are essentially unmoved on the same clips, showing that the vulnerability is a property of the judge's learned trust in tool logs rather than of the task. Plan-derived text carries no clip-specific information, so it can only shift a judge's operating point, and in a repair loop that shift becomes a cap on the true pass rate that no repair policy can exceed; the cap matches simulation to two decimals. In the loop, contamination is exploited without any adversarial agent: an honest LLM planner that always regenerates ends with a judge pass rate of and a human-labelled pass rate of , and a pipeline in which a cheap checker writes its verdict into the trace launders that checker's errors into a stronger final judge ( false accepts). We propose least-privilege judging: every requirement declares the evidence type that can satisfy it, and the judge sees only that evidence—frames for visual requirements, the trace for process requirements, with OCR-masking for text burned into the frames. It restores the true pass rate to – at the (necessary) cost of actually regenerating failed clips, loses nothing on process-level checks, and is a zero-cost guarantee for any judge, including ones whose trust in the trace is unknown.

Primary area:
generative models
Keywords:
agentic video generation, MLLM-as-a-judge, verifier contamination, reward hacking, execution traces, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F4wbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at how auxiliary text (in particular execution logs) can bias multimodal verifiers in video generation agents. By keeping the video content constant and varying the supporting/contradicting text, they demonstrate that the judges can be led to rely on the agent's own reported actions rather than the actual video evidence. The main contributions are a controlled contamination audit, a manually labelled benchmark of 109 generated clips, an analysis of how verification errors can still undermine the repair loops even when you're not dealing with an adversarial planner and finally least privilege judging: only sending the evidence that is relevant to a particular requirement (e.g. frames for a visual check, traces for a process check) with OCR masking of the embedded captions.

Validation involves UCF101 videos, generated clips with incomplete actions, multi-shot consistency tests and simulated repair loops. Across three Qwen-VL judges the supporting logs lead to an increase in false acceptances; the instruction to ignore text only partially offsets this. The closed model judges are substantially less affected. In the 8B repair simulation the honest planner achieves 100% judge reported success but only 28% success according to human labels. Least privilege judging leads to an increase in actual success whilst maintaining high process check accuracy and requiring more regeneration. These results do support the idea of separating evidence by requirement type, albeit the validation is constrained by the small author labelled benchmark, the single open weight model family and the fact that the repair workflows are simulated rather than actual deployed workflows.

Critical strengths

Holding the video content fixed and varying the auxiliary text shows that execution logs can trump visual evidence. The large number of false acceptances across the three Qwen-VL judges, and the comparison against the less susceptible closed model judges, makes the result quite significant and appropriately model-specific. This is a very strong piece of evidence that evidence selection is a key design decision in the evaluation of video agents.

The repair-loop experiments demonstrate that contamination can actually damage an honest planner, yielding 100% judge-reported success and only 28% human-labelled success. Least privilege judging increases actual success in the tested configurations, whilst maintaining high process check accuracy and making the extra regeneration cost explicit. The loops are of course simulated but the link between diagnosis, practical consequences and mitigation is a very significant contribution to the paper.

Critical weaknesses

The central generated video benchmark has just 109 author labelled clips from a single generator, filters out the ambiguous cases and evaluates open weight judges from a single model family. Some of the closed model comparisons use dramatically smaller subsets (e.g. only 26 clips for Claude Opus 5). This establishes a very clear vulnerability in the tested configurations but provides very little support for the more general conclusion that open vs closed judges or learned trust in the tool logs is in any way the issue. This is important because the significance of the paper is very much predicated on how widely that failure generalises.

The repair loop commences from 36 known failures and models regeneration via sampling of the labelled clips with a pre-determined success probability. A large proportion of the outcome is governed by false acceptance prior to any repair so matching the ceiling (1-alpha_0) essentially just verifies the simulation's stopping rule rather than independently validating the repair dynamics. The reported success rates therefore do not demonstrate that least privilege judging provides the same benefit as actual regeneration, evolving traces and realistic planner decisions. An end to end experiment would be a significant step in supporting the core claim of practical benefit.

Questions for authors

How is the assumed 85% regeneration success probability implemented when sampling the finite, per-prompt clip pool? Can failed regenerations also be falsely accepted in later rounds? Please clarify why the observed best true pass rate consistently reaches (1-alpha_0), despite possible later-round false accepts and the four-round limit. This affects how much independent evidence the simulation provides for the mitigation’s effectiveness.

Reviewer_9TLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how execution logs and other additional text affect multimodal verifiers in video-generation agents. The authors find that several Qwen-VL models rely heavily on execution traces even if only making judgement on visual instances, while the tested closed-source models are much less affected. They further simulate a repair loop where incorrect verifier decisions prevent failed videos from being fixed. The proposed solution, least-privilege judging, restricts each verifier to the evidence relevant to its requirement. Experiments cover UCF101, a small generated-video benchmark, and simulated repair loops.

Critical strengths
  1. Evaluation by fixing video and changing text is a useful way to study whether verifiers actually rely on visual evidence. The reported effects are quite large.

  2. Studying verifier errors inside a repair loop is a good way for researching. A wrong acceptance can stop further repair, even when the agent itself behaves reasonably.

Critical weaknesses
  1. My major concern is that this paper doesn't show strong evidence for analyzing the influence of execution logs. In Appendix A, the static logs include an explicit event_check result, which claims that all events are present. This is quite different from an normal execution trace which report a successful call. If a model trusts an explicit but incorrect verification result, that is not surprising at all. The paper needs a proper ablation study on separating plans, tool status, and verification results.

  2. The closed-loop evaluation is quite limited. It simulates repair using only 36 failed clips and samples replacements from an existing pool. The bound (1−α0)(1-\alpha_0) means that clips accepted at round zero are never repaired. I don' t see any theoretical contribution here. Some quantitative results are also inconsistent. For example, the paper claims that cost-greedy achieves the same hacked rate within only about 1/5 cost, but Table 3 reports costs of 0.30 compared with 0.36. These issues make me less confident for the evaluation.

PS: I cannot tell whether an autonomous AI research or writing workflow was used, and using such tools would not itself be a problem. However, I believe the writing style of this paper has a "unique" distribution far beyond my prior "human writing" distribution. And the overstated contributions, basic inconsistencies makes me question how carefully the final manuscript was reviewed. This substantially weakens my confidence in the paper.

Questions for authors

See Weaknesses.

Reviewer_XexZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This study investigates whether a multimodal "judge" within a closed-loop agentic video generation system, given a fixed set of video frames, alters its assessment of purely visual requirements based on auxiliary text, such as execution trajectories, plans, or narrations. The authors compare several open-weight Qwen-VL models against state-of-the-art closed-source models using 109 human-labeled video clips, each containing two distinct events. They find that open-weight models are significantly influenced by tool-use trajectories (whether successful or contradictory), whereas state-of-the-art closed-source models remain largely unaffected. Furthermore, the authors analyze how this bias establishes an upper bound on success rates within self-correction loops and leads to the "whitewashing" of errors.

Critical strengths

By keeping video frames constant and utilizing human-labeled data, while comparing both open-weight and state-of-the-art closed-source models, the study effectively attributes observed effects to the judge’s reliance on tool-use logs rather than the task itself.

Beyond simply reporting error rates, the study provides a nuanced analysis of how plan-related text shifts the "operating point," creates a ceiling for success rates during iterative correction, and allows errors flagged by a "cheaper" (less capable) checker to be effectively "whitewashed" when passed to a more powerful judge.

Critical weaknesses

The “near-miss” benchmark is small, and the authors made the labels themselves. This limits both its statistical power and how objective the labels are.

Top closed-source models were tested on only part of the data because of budget limits. This makes the claim that they are “largely unaffected” less statistically strong.

The closed-loop correction experiments are simulations, not tests in a real-world setup. This limits how well the results apply outside the simulation.

The study looks only at the Qwen-VL family of open-weight models. So the findings cannot be applied to all open-weight models.

The “least-privilege judging” approach depends on correct typing of requirements. Also, the OCR-based masking method works only for “burned-in” (on-screen) subtitles. It cannot handle text that is mixed into the underlying pixels.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.