- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
DeFA is a dependency-guided failure attribution framework for LLM agent trajectories. An LLM induces protocol rules that a deterministic interpreter executes to recover protocol edges, and LLM-identified semantic dependencies complete an event dependency graph. Candidate error events are recalled per segment and traced to their sources and effects in a failure propagation graph, from which an adjudicator selects the decisive step and error category; the responsible agent is that step's actor. Long trajectories are segmented, with summaries of the other segments as context. With three backbones and seven baselines, DeFA achieves the best exact-step accuracy on Who&When (+7.6–9.8 points over the strongest baseline) and on a 300-trajectory text subset of Who&When Pro (+4.0–8.3), the best or tied-best responsible-agent accuracy, the best failure-mode accuracy among taxonomy-aligned methods, and gains on 100 image and video trajectories. Ablations support segmentation and both graphs, and using DeFA's diagnoses in Trace2Skill improves SpreadsheetBench-Verified accuracy by 6–15 points.
Critical strengths
-
Consistent gains on the hardest part of the task. Exact-step accuracy improves for all three backbones on both text benchmarks, with further gains on a multimodal subset, against a recent baseline set run with matched backbones and, on the text benchmarks, matched greedy decoding.
-
Mechanistic support from the ablations. Removing only the propagation edges while holding the adjudication set fixed (coverage unchanged at 83.2%) lowers exact-step accuracy by 12.5 points, indicating that propagation relations, not just candidate recall, drive the gain. English translations of DeFA's system prompts, a worked example, and released code support reproducibility.
Critical weaknesses
-
No uncertainty quantification for a multi-stage prompting pipeline. No result reports variance, confidence intervals, or paired tests, and the number of runs is not stated. On Who&When, one trajectory is 0.54 points; multimodal agent accuracy uses 17 trajectories, where the lead is one or two trajectories; the Trace2Skill result uses 100 test tasks; and every ablation uses one backbone on one benchmark. Such pipelines are sensitive to prompt and sampling variation even at temperature 0 with hosted APIs, so the size and robustness of the gains are not established. This is the main reason my rating is below acceptance.
-
Incomplete comparison and protocol specification. The paper cites but does not compare with AgenTracer (ICLR 2026), a published attributor trained for this task, and does not state whether the baselines receive the ground-truth answer (DeFA's stage inputs suggest that DeFA does not), a setting the original Who&When evaluation reports separately. Gains are also concentrated in exact-step accuracy (DeFA is not best on Step ±1/±3 with DeepSeek-V4-Pro and Qwen3.8-Max on Who&When Pro), and with GLM-5.2 its accuracy is 17.2% on the longest Who&When trajectories (33–130 steps, n = 29; Fig. 2a), so the benefit for long executions is unclear.
Questions for authors
- Please report means and 95% intervals over repeated runs, or paired tests against the strongest baseline, for the main tables, and repeat the key ablation (removing propagation edges) on a second backbone and on Who&When Pro.
- Do the baselines receive the ground-truth answer on Who&When? Results under both settings would help.
- How does DeFA compare with AgenTracer on Who&When?