- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces AV-FAIL, a benchmark for identifying the earliest faulty step in tool-using agent traces. It combines a controlled video-editing track containing five injected fault types and a no-fault condition with a pool of 1,848 real agent runs. The authors evaluate several LLM judges and identify difficulties with cumulative-budget violations and false positives on clean traces. They propose Trace-State Prompting (TSP), which applies deterministic constraint checks before routing unresolved cases to an LLM judge. Experiments cover video-editing traces and an additional ETL domain.
Critical strengths
The evaluation problem is relevant. Identifying where an agent first goes wrong can provide useful information for debugging. Including a no-fault condition also allows the benchmark to measure incorrect attribution on successful runs.
The controlled experiments support useful diagnostic comparisons. Fault injection provides known injection locations, and the breakdown by fault type exposes differences that aggregate accuracy would obscure. The paper also reports the adverse effect of the no-fault gate on subtitle-drift detection.
The proposed approach is practical and shows measurable gains. Deterministic verification is a reasonable way to handle explicit, machine-checkable constraints. TSP improves overall accuracy on the controlled benchmark, and the additional experiments with different generators and an ETL domain broaden the evaluation.
Critical weaknesses
Unclear methodological contribution. TSP combines predefined constraint checks with an LLM judge. Comparisons with judges equipped with arithmetic or code-execution tools are needed to isolate the benefit of the proposed routing design.
Overstated generalization. Figure 5 reports S5 baseline accuracy of 0.90, exceeding S1 and S2, despite claiming that the difficulty ordering recurs. This weakens the claim of a persistent budget-reasoning blind spot.
Limited validation on real traces. Appendix H states that Stage 1 becomes a no-op without machine-readable constraints, leaving its practical benefit unverified. The selectively adjudicated sample also cannot establish the dominant failure mode across the full dataset.
Inconsistent reporting. Section 3.1 states 300 tasks per condition, while Appendix K reports 300 in total. The reported 57 semantic failures also need reconciliation with the 53 confirmed cases among 54 adjudicated examples in Section 6.2.
Questions for authors
How does TSP compare with a judge equipped with arithmetic or code-execution tools under the same inputs? This comparison would help isolate the contribution of the routing design.
Why does S5 baseline accuracy increase from 0.30 on template-based ETL traces to 0.90 on agent-authored traces? Does this suggest that the claimed budget-reasoning blind spot depends on trace construction?