acceptodds
Under review as a conference paper at ICLR 2027

Where Did the Agent Go Wrong? Localizing—and Fixing—Failure Steps in Multi-Step Video-Editing Agent Traces

Abstract

Autonomous agents run long, tool-rich workflows, yet when a run fails, telling where it went wrong is hard: a syntactically valid trace can quietly drift off the user's goal. We study this failure-step localization problem in multi-step video editing, where one request expands into a chain of metadata probes, shot detections, clip cuts, and concatenations. We introduce AV-Fail, a benchmark that asks a judge to identify the earliest faulty step of an agent trace. AV-Fail couples a controlled track injecting one of five fault types (or none) for programmatic exact-step gold labels with a real track of genuine runs verified via multi-model human-in-the-loop review. A blind two-annotator study fixes the human ceiling at Cohen's (substantial), which no automatic judge reaches: the best single model attains only and different judges err in opposite directions, so automatic failure localization cannot stand in for human review. Across five judge models we find a sharp difficulty spectrum: mechanical faults are nearly solved (), a cumulative-budget fault is a systematic blind spot (), and clean traces trigger over-attribution (a false-positive rate); both effects are strongly model-dependent. Two lightweight mitigations each help but interfere when combined. We resolve this with Trace-State Prompting, which—wherever a constraint is expressible as a checkable predicate over the trace—computes that trace state deterministically and routes only the residual decision to the model, raising accuracy from to and closing the budget blind spot losslessly. On real traces it surfaces semantic failures that objective signals miss, showing off-goal drift—not crashes—is the dominant unhandled failure mode. The findings are not video-specific: ported to a data-pipeline (ETL) agent domain, the same ordering recurs and the protocol again lifts accuracy from to . AV-Fail provides a reproducible lens on a capability that agentic evaluation has so far left largely unmeasured.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, failure localization, tool-using agents, process evaluation, benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ksGbOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AV-FAIL, a benchmark for identifying the earliest faulty step in tool-using agent traces. It combines a controlled video-editing track containing five injected fault types and a no-fault condition with a pool of 1,848 real agent runs. The authors evaluate several LLM judges and identify difficulties with cumulative-budget violations and false positives on clean traces. They propose Trace-State Prompting (TSP), which applies deterministic constraint checks before routing unresolved cases to an LLM judge. Experiments cover video-editing traces and an additional ETL domain.

Critical strengths

The evaluation problem is relevant. Identifying where an agent first goes wrong can provide useful information for debugging. Including a no-fault condition also allows the benchmark to measure incorrect attribution on successful runs.

The controlled experiments support useful diagnostic comparisons. Fault injection provides known injection locations, and the breakdown by fault type exposes differences that aggregate accuracy would obscure. The paper also reports the adverse effect of the no-fault gate on subtitle-drift detection.

The proposed approach is practical and shows measurable gains. Deterministic verification is a reasonable way to handle explicit, machine-checkable constraints. TSP improves overall accuracy on the controlled benchmark, and the additional experiments with different generators and an ETL domain broaden the evaluation.

Critical weaknesses

Unclear methodological contribution. TSP combines predefined constraint checks with an LLM judge. Comparisons with judges equipped with arithmetic or code-execution tools are needed to isolate the benefit of the proposed routing design.

Overstated generalization. Figure 5 reports S5 baseline accuracy of 0.90, exceeding S1 and S2, despite claiming that the difficulty ordering recurs. This weakens the claim of a persistent budget-reasoning blind spot.

Limited validation on real traces. Appendix H states that Stage 1 becomes a no-op without machine-readable constraints, leaving its practical benefit unverified. The selectively adjudicated sample also cannot establish the dominant failure mode across the full dataset.

Inconsistent reporting. Section 3.1 states 300 tasks per condition, while Appendix K reports 300 in total. The reported 57 semantic failures also need reconciliation with the 53 confirmed cases among 54 adjudicated examples in Section 6.2.

Questions for authors

How does TSP compare with a judge equipped with arithmetic or code-execution tools under the same inputs? This comparison would help isolate the contribution of the routing design.

Why does S5 baseline accuracy increase from 0.30 on template-based ETL traces to 0.90 on agent-authored traces? Does this suggest that the claimed budget-reasoning blind spot depends on trace construction?

Reviewer_wmYDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies failure-step localization in video-editing agents. The authors introduce AV-FAIL, a benchmark that evaluates whether an LLM judge can identify the earliest step where the execution goes wrong. They find that LLMs can recognize simple local errors but struggle when constraints become harder to track. They also sometimes report mistakes when there are none. To address these problems, the authors propose Trace-State Prompting, which verifies explicit requirements using deterministic code and relies on an LLM to judge the rest. The method gives better results on their benchmark. However, it is less effective when the requirements are not clearly stated and are more implicit.

Critical strengths

The paper addresses an important problem of identifying the faulty step instead of only checking whether the task was successful. The fact that the authors also include traces without any mistakes and test their method on a task other than video editing (ETL) is a strength. This makes the benchmark more useful for understanding agent failures and provides some evidence that the method can work on different tasks.

Critical weaknesses
  1. To show generalizability, the authors also test their method on an ETL task. However, video editing and ETL are similar in terms of the types of constraints involved, such as ordering. It would be useful to evaluate the method on a different type of task, like web navigation. It would provide stronger evidence that the method can generalize to different agent tasks.

  2. Although their method improves the results over other methods, the underlying idea is simple. It combines rule-based checks with an LLM as a judge. This limits the novelty of the proposed method.

Questions for authors
  1. How could Trace-State Prompting be applied to other types of agent tasks, such as web navigation, where the constraints may be different from those in video editing and ETL?
  2. What is the main novelty of Trace-State Prompting beyond combining rule-based checks with an LLM judge?
Reviewer_PmGmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission studies failure-step localization in agent traces: given a trace from a multi-step video-editing agent, decide whether the run failed and name the earliest faulty step. The authors build AV-FAIL: on the controlled track a deterministic injector inserts one of five fault types (S1–S5), or none (T0), into valid traces, gold labels come directly from the injection point, and scoring is exact-step; the real track (P1-B) collects 1,848 genuine runs and produces reference labels through two-judge prelabeling, a 268-run adjudication pilot, and human spot-checks. Across five judge models the difficulty spectrum is steep: mechanical faults are near-solved (S2 and S3 at 1.00), the cumulative-budget fault S5 drops to 0.20, and judges invent a culprit on 0.35 of clean traces (Figure 3a). Two lightweight interventions (a budget scaffold and a no-fault gate) each help but cancel each other when combined, so the authors propose Trace-State Prompting (TSP), which computes checkable trace state deterministically outside the model and routes only the residual decision to it, lifting accuracy from 0.775 to 0.872 and replicating on an ETL agent domain (0.658 to 0.883). On the real track the protocol surfaces 57 semantic failures that objective signals miss.

Critical strengths
  1. The core design of the benchmark is good: programmatic gold labels plus a T0 no-fault negative control. Gold steps come directly from the injector and need no human annotation, and T0 turns “the judge invents a culprit on a clean trace” into a directly measurable false-positive rate (0.35, Figure 3a), a failure mode that success-rate-only evaluation cannot see. I expect this protocol (injectors, judge harness, negative control) to be reused by later work, and it is the part of the paper most worth keeping.
Critical weaknesses
  1. The gain of TSP comes mostly from one deterministic rule, and the method contribution is inflated. In Table 3 the total improvement is +0.097 (0.775 to 0.872), of which S5 alone accounts for +0.58 (0.31 to 0.89); the Stage 1 check for S5 just recomputes the cumulative duration against spec.max total ms (Section 5), the same quantity the injector writes into the trace (Appendix B). The paper lacks a key ablation: Stage 1 attached directly to the Base judge without the gate; by my rough arithmetic from Table 3 that variant lands about where TSP (0.872) does, in which case what TSP really adds is only separating the gate from the scaffold. So most of the 0.872 comes from the checker, not from a new method.

  2. The generalization claims outrun the evidence. Section 4.1 asserts the findings “reflect a property of the judge models, not of any generator”, yet in Table 7, with DeepSeek or Qwen as the generator, T0 is 1.00 for both (0.86 with Claude), so the false-positive rate clearly varies with the generator; the cross-domain replication (Table 9) also holds only on the templated ETL suite, and on agent-authored ETL traces the S5 baseline is already 0.90 (Figure 5), so the hardest condition does not replicate on more realistic traces. I therefore discount both the cross-domain claim and the judge-property claim in Section 6; the “first benchmark” in the contribution list also needs narrowing, since ProcessBench already requires outputting the earliest erroneous step.

  3. The reference labels on the real track (P1-B) are weakly validated. Of the 1,848 runs only 116 end with a valid adjudication (Table 5); the adjudicator is an LLM that sees both judge verdicts (Appendix D), humans only spot-check, and step-level localization accuracy against the reference labels is never reported. Table 5 does not add up: 82 (adjudicator agreed with both judges) + 11 (overrode both) + 5 (sided with the anchor) = 98, which does not match 116; Figure 2 is labeled n=300/type and Section 3.1 says each condition is evaluated over 300 tasks, yet Table 2 has only 250 and 249 records for S2 and S3. So conclusions drawn from the real track, such as the 57 semantic failures, carry limited credibility.

Questions for authors
  1. [Clarification] Can you add the ablation of Stage 1 attached directly to the Base judge without the gate and report its total accuracy? How far is it from TSP's 0.872?
  2. [Clarification] In Table 5, how do 82+11+5=98 correspond to the 116 valid adjudications, and what happened to the remaining 18 runs?
  3. [Clarification] On the real track, what is the step-level localization accuracy against the adjudicated labels? If it was not computed, why not?
  4. [Clarification] In Table 7, T0 is 1.00 for both the DeepSeek and Qwen generators. How is this compatible with the claim that the findings reflect judge properties rather than the generator?
  5. [Suggestion] Would you narrow the “first benchmark” claim, state the difference from ProcessBench, and scope the cross-domain replication to the templated ETL suite?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.