acceptodds
Under review as a conference paper at ICLR 2027

DeFA: Dependency-Guided Failure Attribution for LLM Agents

Abstract

Errors in LLM agent executions and their visible consequences can be separated by many steps, making decisive-error localization a matter of understanding both step content and step dependencies. We introduce DeFA, a dependency-guided framework for agent failure attribution. DeFA first combines protocol relations and semantic dependencies into an event dependency graph spanning the trajectory. It then identifies events that may violate task requirements and traces their sources and subsequent effects to construct a failure propagation graph. Finally, DeFA uses step evidence and the steps' roles in failure propagation to identify the decisive error, responsible agent, and error category. To support long trajectories, DeFA partitions executions into segments and combines the current segment's detailed content with summaries of the other segments, giving local diagnosis access to global execution context. Across Who&When and the Who&When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro. Further experiments on image and video trajectories demonstrate its applicability to multimodal failure attribution. Ablations support the contributions of segmentation, the event dependency graph, and the failure propagation graph. Using DeFA's diagnostic feedback for skill evolution in Trace2Skill improves downstream task accuracy by 6–15 percentage points over the native pipeline, showing that the diagnoses can also support agent improvement on subsequent tasks. Code is available at https://anonymous.4open.science/r/defa-code-FA22/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, failure attribution, root cause localization, dependency graphs

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yNASOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

DeFA is a dependency-guided failure attribution framework for LLM agent trajectories. An LLM induces protocol rules that a deterministic interpreter executes to recover protocol edges, and LLM-identified semantic dependencies complete an event dependency graph. Candidate error events are recalled per segment and traced to their sources and effects in a failure propagation graph, from which an adjudicator selects the decisive step and error category; the responsible agent is that step's actor. Long trajectories are segmented, with summaries of the other segments as context. With three backbones and seven baselines, DeFA achieves the best exact-step accuracy on Who&When (+7.6–9.8 points over the strongest baseline) and on a 300-trajectory text subset of Who&When Pro (+4.0–8.3), the best or tied-best responsible-agent accuracy, the best failure-mode accuracy among taxonomy-aligned methods, and gains on 100 image and video trajectories. Ablations support segmentation and both graphs, and using DeFA's diagnoses in Trace2Skill improves SpreadsheetBench-Verified accuracy by 6–15 points.

Critical strengths
  1. Consistent gains on the hardest part of the task. Exact-step accuracy improves for all three backbones on both text benchmarks, with further gains on a multimodal subset, against a recent baseline set run with matched backbones and, on the text benchmarks, matched greedy decoding.

  2. Mechanistic support from the ablations. Removing only the propagation edges while holding the adjudication set fixed (coverage unchanged at 83.2%) lowers exact-step accuracy by 12.5 points, indicating that propagation relations, not just candidate recall, drive the gain. English translations of DeFA's system prompts, a worked example, and released code support reproducibility.

Critical weaknesses
  1. No uncertainty quantification for a multi-stage prompting pipeline. No result reports variance, confidence intervals, or paired tests, and the number of runs is not stated. On Who&When, one trajectory is 0.54 points; multimodal agent accuracy uses 17 trajectories, where the lead is one or two trajectories; the Trace2Skill result uses 100 test tasks; and every ablation uses one backbone on one benchmark. Such pipelines are sensitive to prompt and sampling variation even at temperature 0 with hosted APIs, so the size and robustness of the gains are not established. This is the main reason my rating is below acceptance.

  2. Incomplete comparison and protocol specification. The paper cites but does not compare with AgenTracer (ICLR 2026), a published attributor trained for this task, and does not state whether the baselines receive the ground-truth answer (DeFA's stage inputs suggest that DeFA does not), a setting the original Who&When evaluation reports separately. Gains are also concentrated in exact-step accuracy (DeFA is not best on Step ±1/±3 with DeepSeek-V4-Pro and Qwen3.8-Max on Who&When Pro), and with GLM-5.2 its accuracy is 17.2% on the longest Who&When trajectories (33–130 steps, n = 29; Fig. 2a), so the benefit for long executions is unclear.

Questions for authors
  1. Please report means and 95% intervals over repeated runs, or paired tests against the strongest baseline, for the main tables, and repeat the key ablation (removing propagation edges) on a second backbone and on Who&When Pro.
  2. Do the baselines receive the ground-truth answer on Who&When? Results under both settings would help.
  3. How does DeFA compare with AgenTracer on Who&When?
Reviewer_LWHTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates failure attribution in LLM/VLM agent. It focuses on establishing connections between execution actions and key events (decisive steps) to provide more interpretable explanations of execution failures. To achieve this, the authors construct an event dependency graph and employ an LLM-based adjudicator to identify agent failures, decisive errors, responsible agents, and error categories.

Critical strengths

The paper presents an interesting and reasonable research idea.

The manuscript is generally well written and clearly structured.

The experimental evaluation examines the proposed approach from multiple perspectives and provides supporting evidence for its effectiveness.

Critical weaknesses

See Questions

Questions for authors
  1. Novelty and comparison with related work: How does the proposed approach differ from [R1]? Although [R1] focuses on LLM-based multi-agent systems rather than the broader LLM/VLM setting, the underlying idea appears quite similar. In particular, [R1] does not rely on an LLM as an adjudicator, which seems appealing in terms of methodological simplicity and potentially lower inference costs. Could the authors clarify the key technical differences and explain the distinctive technical contributions of DeFA compared with [R1], especially regarding to the construction of the directed acyclic graph?

  2. LLM adjudicator: Which specific LLM is used as the adjudicator? How sensitive are the attribution results to the choice of model?

  3. Multimodal evaluation (Section 4.5): My understanding is that Who&When Pro already contains image and video data. Why, then, does DeFA need to construct multimodal inputs? Could the authors clarify how these inputs are constructed, how they differ from the original benchmark observations, and whether any information is transformed or omitted during this process?

  4. Integration with Trace2Skill (Section 4.7): How is DeFA integrated into Trace2Skill? Could the authors provide more details about how the failure attribution results are used by the downstream framework?

  5. Out-of-distribution generalisation (Figure 3): Could the authors provide a more detailed explanation of the OOD results shown in Figure 3? In particular, how should these results be interpreted in terms of the generalisation capabilities and limitations of DeFA?

  6. Comparison of DeFA variants (Table 4 and Figure 3): Are there any experimental results comparing the different DeFA variants presented in Table 4 on the tasks evaluated in Figure 3? Such a comparison would help clarify how the different design choices affect performance across these tasks.

References

[R1] https://arxiv.org/pdf/2608.18575

Reviewer_gmTgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses llm agent failures attribution with DeFA,a LLM workflow that attributes agent failures to a decisive step. DeFA first builds an event dependency graph from protocol relation and semantic edges. Using this graph, it flags suspicious steps and traces their sources and effects into a failure propagation graph. An adjudicator picks this graph's earliest step whose correction would avert the failure, and labels its error mode. According to experiment results, DeFA beats seven baselines by 4–10 pp in exact-step accuracy and improves Trace2Skill by 6–15 pp.

Critical strengths
  1. The gains are consistent and stable. The paper compares four recent attribution systems. Besides the three Who&When judges, the baselines include four recent attribution systems: AgentRx, DeepDebug, ErrorProbe and CockpitHAT. DeFA has the best exact-step accuracy in all eight benchmark (Tables 1–3), leading the strongest baseline. The baselines look faithfully implemented.

  2. Ablation is well designed. The last variant in Table 4 removes only the propagation edges and keeps the same steps for adjudication. This separates which steps the adjudicator considers from what it is told about how they relate. Exact-step accuracy drops by 12.5 pp, which is the paper's most direct evidence that this relational information matters.

Critical weaknesses
  1. The paper does not show that dependency tracing is why DeFA works

The motivation is that existing attributors miss the decisive error because it lies many steps before its symptom (L040–045). The paper never measures how far gold errors are from the final failure, whether baselines pick downstream symptom steps, or whether DeFA's gains concentrate where this distance is large.

  1. Separate dependency guidance from extra LLM analysis Each propagation edge comes with an LLM-written explanation of what the source step passed on and how the target step used it (L207–208). Then the adjudicator reads these explanations along with the edges. In effect, the graphs give the adjudicator two extra LLM analysis calls per segment. Do the graph gains in Table 4 come from following dependencies, or simply from this extra analysis? A second confound is that only DeFA's adjudicator is given the benchmark's definition of the decisive error (L1190–1231).
Questions for authors
  1. where the remaining errors come from: Coverage is 83.2%, but exact-step accuracy is 52.7% (Table 4). In about 30% of trajectories, therefore, the decisive step is in the adjudication set but is not chosen. How do DeFA's failures split between missed candidates, wrong adjudication and wrong graphs? Does the "earliest step" rule push predictions too early?

  2. Does the gain come from the graph or from the text on its edges? L412 shows that removing the propagation edges also removes their LLM-written explanations, and DeFA without both graphs (36.4%) scores below four baselines. What happens if the explanations are kept but the edges are dropped or randomly rewired? How does All-at-Once perform with DeFA's adjudication prompt?

  3. Do the graphs help more when the error and its symptom are far apart? This distance is the paper's motivation. An ablation split by this distance, or at least by trajectory length, would test the motivation directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.