acceptodds
Withdrawn from ICLR 2027

DAAF: From Failure Localization to Editable System Assets in LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deployed LLM agents increasingly rely on persistent, versioned system assets such as routing rules, knowledge segments, prompt instructions, and reusable skills. Failure-localization methods can identify where an error manifests in an agent or execution trace, but repair requires a different decision: *which editable system asset should be changed, and is that change expected to improve the task outcome?* We study this gap through component-attribute failure attribution, where diagnosis targets versioned, addressable items rather than execution locations. We propose the **Detection-Aware Attribution Framework (DAAF)**, which learns the effects of valid attribute replacements and amortizes this intervention evidence into deployment-time diagnosis. DAAF combines sparse and noisy failure signals to decide whether intervention is warranted, learns component-type-conditioned replacement effects from controlled replays evaluated by executable task outcomes, and shares supervision across requests with compatible intervention responses. At diagnosis time, DAAF uses only the observed execution, registered candidates, and available failure signals; it requires neither counterfactual replay nor task reward and returns `no_change`, a repair target, or an unresolved decision when evidence is insufficient. On held-out τ²-bench Telecom tasks, DAAF achieves 80.72% attribute Hit@1, recovers 62.65% of failed executions while limiting clean-task regression to 3.23%, and reaches 71.93% overall task success. These results show that intervention-grounded attribute attribution can connect failure localization to executable system repair.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, failure attribution, agent debugging, counterfactual intervention, system repair

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LkeiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Detection-Aware Attribution Framework (DAAF), which is a framework that addresses the problem of which system assets should be changed upon an agent failure. The authors register five editable assets in a telecom support agent, consisting of two prompts, a knowledge segment, a runtime setting and a support procedure. Each has replacements written in advance and DAAF chooses between these replacements. They find that this generally helps support agents recover in the telecom setting.

Critical strengths

The problem that this paper studies is practical and relevant. The paper is written relatively clearly and the problem setup is also stated clearly. The problem of selecting a concrete system asset to repair makes sense. The result is promising, though the restricted edit menu and missing baselines limit what can be concluded about the method’s broader value.

Critical weaknesses

The evaluation setting is limited. The proposed method is only tested on tau-bench Telecom tasks, and the alternatives for editable assets are all hand-written. Extension to other agentic settings or support tasks, with larger and less curated candidate sets, would help establish generality. It would also be useful to compare against simpler policies such as simply applying the best fixed edit on all eval tasks to see when request-specific decisions are needed.

The practical advantage of this system over harness optimization and self-correction methods is not clear. DAAF learns to select among pre-specified replacements using a substantial offline replay budget, but the evaluation does not compare it with using that budget to optimize a single system configuration, for example with a prompt optimizer such as GEPA, or with an agent that corrects its execution using the same budget. A budget-matched comparison with direct evolution methods or simply giving the agent a higher step count to correct mistakes would make this a lot stronger.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer Lkei

We thank Reviewer Lkei for the review and for recognizing the practical relevance of identifying a concrete system asset for repair.

Evaluation scope and fixed-edit controls. The current study evaluates executable repairs on Telecom tasks using five pre-registered asset replacements. This controlled setting supports direct measurement of recovery and unintended regressions, but the compact candidate space and constrained repair interface do not sufficiently test the approach across more complex agent configurations. We agree that a development-selected best fixed edit, larger and less curated candidate sets, and additional environments are important for determining when request-specific repair decisions provide value beyond a shared system change.

Global optimization and self-correction. We agree that the practical trade-offs between attribution-guided repair, global system optimization, and agent self-correction deserve explicit study. Budget-matched comparisons with approaches such as GEPA-style configuration optimization and additional-step self-correction would help establish when targeted repair selection is useful. These approaches make different decisions, so a meaningful comparison should account for offline and online costs and evaluate actual task recovery, regressions, and overall success. The present experiments do not yet provide this comparative evidence.

We have decided to withdraw the submission to refine the method and establish a broader and more demanding evaluation, rather than make incremental changes to the current experimental setup. We appreciate your suggestions and will use them to guide the next stage of the work.

Reviewer_7yTCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Paper Summary

The paper studies how to repair LLM agents that fail because of a shared system asset, such as a prompt, a routing rule, a knowledge segment, or a reusable skill. Existing failure-localization methods say where in a trajectory an error appears. They do not say which persistent asset to change. The paper proposes DAAF, which treats repair as a choice among a fixed registry of editable assets, each with a pre-written replacement. DAAF has three learned parts. A detector combines weak failure signals (trajectory anomalies, user feedback, disagreement across repeated runs) and keeps track of which signals are missing. An attribution model learns the signed effect of each replacement from controlled replays run during training, where edits are applied in sampled orders and scored by the official task evaluator. A joint objective shares evidence across requests that respond to interventions in the same way. At test time DAAF uses only the observed trajectory, the registry, and the available signals, and returns no-change, one repair, or unresolved. Experiments use tau^2-bench Telecom with GPT-5.6 as the agent and user simulator. The registry has five assets. On 114 held-out tasks (83 failures, 31 successes), DAAF reaches 80.72% attribute Hit@1 versus 40.96% for an LLM that reads the same trace and registry. Executed repair recovers 62.65% of failures versus 35.74%, with clean-task regression of 3.23% versus 19.35%, and overall success of 71.93% versus 47.95%. Composed edits change outcomes in 31.02% of replay steps versus 11.36% for isolated edits. A second-edit ranking derived from training replays transfers to held-out tasks, with the top candidate adding 18.88 points of recovery. A trajectory-level front end also scores above AgentDebugX on the Who&When benchmark.

Critical strengths

Strengths

  1. The framing is useful and new. Most attribution work points to a step or an agent. This paper points to a versioned object that someone can actually edit and re-test. That is the decision a maintainer needs.
  2. The evaluation goes all the way to executed repair with the official task evaluator. Recovery, regression, and overall success are measured on real reruns, not on agreement with labels.
  3. The gains are large on every metric: Hit@1 nearly doubles, recovery rises from 35.74% to 62.65%, and clean-task regression falls from 19.35% to 3.23%. The method repairs more while breaking less.
  4. The controls are the right ones. A detection-only model with the same pipeline recovers just 4.42%. Attaching the effect predictor to the plain LLM gives 34.94% Hit@1. Uniform choice gives 37.35%. These show that neither detection alone nor better scoring alone explains the result.
  5. The interaction analysis is a real finding. Isolated edits change outcomes in 11.36% of replay steps, composed edits in 31.02%. Single-edit probing misses 88.54% of the effective task-attribute pairs. This argues against the common practice of testing one fix at a time.
  6. The transfer test is clean. The ranking of second edits comes only from training replays, and the top candidate adds 18.88 recovery points on held-out tasks with a confidence interval that excludes zero.
  7. The statistics and reporting are careful: three matched execution seeds shared across methods, paired bootstrap intervals, a disjoint development set for all thresholds, and a full configuration table. The paper also admits where a claim did not hold, such as missingness-aware encoding matching plain mean imputation.
Critical weaknesses

Weaknesses

  1. The whole executable repair result comes from one environment, Telecom, with 114 tasks and 83 failures. Who&When is only a localization check with a different model and no repair. Clean regression of 3.23% is one task out of 31. The paper needs a second environment with editable assets, for example Retail or Airline in the same benchmark.
  2. The registry has only five assets, each with one pre-written replacement. DAAF therefore chooses among five hand-made fixes, plus no-change. Chance already gives 37.35% Hit@1. The paper should report how Hit@1 and recovery change as the registry grows to 20 or 50 candidates, including decoys that have no effect.
  3. The replacements were written by the authors for this benchmark. The prompt replacement, for example, adds diagnostic ordering and user verification, which are exactly the known failure causes in Telecom. The gain may come from good replacement content, not from good diagnosis. A control that applies the single best replacement to every failure, with no diagnosis, would separate the two. The second-edit result, where one prompt edit adds 18.88 points everywhere, suggests this control could be strong.
  4. There is no comparison with any prior intervention-based method. CausalFlow, DoVer, CAR, AgentDebug, and AgentTracer all use counterfactual replay or targeted intervention and are cited as the closest work, but none is run. The only baselines are two prompted versions of GPT-5.6. Run at least one of these methods on the same registry.
  5. The training and test tasks share the same five assets and the same replacements. The attribution model may simply learn a fixed per-asset prior, such as "the prompt instruction usually helps". The paper should show the distribution of DAAF's chosen edits across the 83 failures and report Hit@1 on an asset that was held out of training.
  6. The same replay procedure produces both the training targets and the evaluation repair labels, and the same GPT-5.6 model is the agent, the user simulator, and the LLM baseline. Attribute Hit@1 therefore measures agreement with the method's own supervision source. Use a different model for the baseline and an independent labeling route for at least a subset of tasks.
  7. The method is heavy for what it decides. It needs 5,280 controlled executions, a weak-supervision pipeline, Shapley-style marginal estimates, type-specific heads, and a graph regularizer, to pick one of five options. The two propositions are standard concentration bounds that are not used to set any budget. The paper should show which parts matter with a component ablation, as the detector-only and LLM+CTCA controls do not cover self-consistency, type-specific heads, or the missingness encoding.
Questions for authors

I have no questions

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7yTC

We thank Reviewer 7yTC for the detailed assessment and for recognizing the value of directing failure attribution to versioned assets that can be edited and re-evaluated. We also appreciate your careful examination of the executed repair metrics, intervention analysis, and evaluation protocol. We take the presentation feedback seriously and see room to make the roles of detection, effect prediction, and repair selection clearer.

Evaluation scope, candidate design, and asset priors (W1–W3, W5). The reported repair results are obtained on held-out Telecom tasks using the official task evaluator, with 83 natural failures and 31 natural successes. The five replacement candidates were fixed before evaluation, and their design did not use held-out evaluation failure trajectories or hidden task answers. This setting supports controlled comparisons on unseen requests, but it does not establish generalization to new environments or previously unseen assets. We agree that larger registries with ineffective candidates, fixed-edit controls, and explicit per-asset-prior comparisons would make the added value of request-conditioned attribution clearer. Reporting the distribution of selected assets and testing held-out assets are also useful ways to examine this distinction.

Intervention baselines and evaluation independence (W4, W6). The experiments evaluate executed repairs with the official binary task evaluator, and the repair labels are validated independently of DAAF's predictions, as described in Appendix E. This prediction-independent validation is an important part of the existing protocol, although it does not by itself establish robustness across alternative labeling procedures or model-role choices. We agree that additional checks using different model-based baselines or an independent labeling route would strengthen the analysis. We also see value in comparing with suitably adapted intervention-based methods, such as CAR or DoVer, under a shared repair interface and clearly specified computational budgets.

Cost, ablations, and method clarity (W7). We agree that the 5,280 offline replay executions should be evaluated against simpler alternatives. The replay statistics characterize interactions, but do not replace isolated-versus-Shapley or μ = 0 ablations. The contribution of type-specific heads and the handling of missing failure signals also warrant clearer analysis. In particular, the current paper reports that missingness-aware encoding matches mean imputation in the evaluated settings, so a stronger advantage over that alternative should not be inferred. We will also improve the presentation of the method's individual components and the relationship between offline supervision and online decisions.

We have decided to withdraw this version to allow a more substantial refinement of the method and evaluation rather than make limited changes during the discussion period. Our goal is to test the approach in a more informative setting and better establish both its advantages and its limitations. We are grateful for the detailed and actionable feedback.

Reviewer_oUG6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies component-level failure attribution for LLM agents. It moves the diagnosis target from the location of an error in the execution trace to versioned, replaceable registered system assets in five roles (Prompt, Knowledge, Harness, Skill, Node), and defines attribution as the signed change in task outcome from replacing a registered asset. The method has three parts. MSWSD combines weak signals such as trajectory anomalies, visible user feedback, and disagreement across repeated runs to decide whether intervention is warranted. CTCA runs controlled replays with the official binary task evaluator in composed replacement settings and trains component-type-conditioned effect predictors on Shapley-style marginal effects. Joint learning adds a cross-request consistency regularizer that shares repair evidence across requests with compatible intervention responses; diagnosis uses no online replay and no task reward and returns no_change, a single repair target, or an unresolved decision. Experiments run on all 114 held-out τ²-bench Telecom default tasks (83 natural failures, 31 natural successes, with GPT-5.6-luna for execution and diagnosis) and report attribute Hit@1 of 80.72%, Failure Recovery of 62.65%, Clean Regression of 3.23%, and Overall Success of 71.93% (Table 2).

Critical strengths

The paper reframes attribution as deciding which registered asset to change and whether that change improves the task outcome, rather than localizing the execution step where a failure appears. The output of diagnosis then maps directly to an object that can be replaced and re-evaluated. This reframing shapes the entire experimental design and is the most valuable judgment in the paper.

The protocol is rigorous. Both attribution supervision and repair evaluation use the official binary task evaluator rather than an LLM judge, side effects are measured separately through Clean Regression, repair results are reported over three matched execution repeats with paired bootstrap confidence intervals, and repair labels are validated independently of method predictions (Appendix E, G.3). These choices make the comparison in Table 2 credible, and they carry real weight in my judgment of the paper.

Critical weaknesses

The paper lacks a request-independent static baseline, so the current results do not demonstrate attribution ability. In Appendix Table 10, applying Skill alone changes the task outcome on 52.20% of replacement steps, while the other four attributes change it on only 0.53%–1.25% of standalone steps. Moreover, uniformly appending Prompt: instruction to all tasks after Full DAAF raises Failure Recovery from 62.65% to 81.53%, higher than the edit DAAF itself selects (Table 3). If a fixed policy such as "always change Skill", which never looks at the request, collects comparable gains, then the main tables measure the existence of one useful replacement in the system, not the ability of DAAF to attribute per request.

The two core designs have no ablations. The interaction-aware Shapley target (Eq. 5, Section 3.3) and the consistency regularizer (Eq. 10, μ=0.1 in Table 7) are the two designs that separate DAAF from a plain effect regressor, yet the paper contains neither a comparison between an isolated-edit target and the Shapley target nor a result with μ=0. The composed-effect statistics in Section 4.3 only show that interactions exist in the replay corpus; they do not show how much each of the two designs contributes.

Questions for authors

[Suggestion] Add a request-independent static baseline such as "always change Skill". If DAAF is clearly better, I will raise my score. [Clarification] Who designed the five registered replacements and on what basis? Did the design process look at the failure trajectories of the evaluation tasks?

1 comment
AuthorsOpenReview ↗

Response to Reviewer oUG6

We thank Reviewer oUG6 for the thoughtful and constructive review. We particularly appreciate your recognition of the central motivation of the paper: moving from locating errors in execution traces to identifying persistent system assets that can be modified and re-evaluated. We are also grateful for your positive assessment of our outcome-based evaluation protocol.

Request-independent baseline. We agree that a static policy, such as always applying the Skill replacement or using the best fixed edit selected on development data, is an important comparison for establishing the additional value of request-conditioned attribution. The present results demonstrate executable repair outcomes on held-out Telecom tasks, but they do not fully separate this value from the benefit of the registered replacement content. For clarity, the 52.20% standalone Skill statistic in Table 10 measures changes in task outcomes, not failure recovery. Similarly, the 81.53% recovery in Table 3 is obtained by adding a second edit after Full DAAF, rather than by applying a fixed edit alone. These results motivate, but do not substitute for, a direct static-policy comparison.

Method ablations. We agree that the composed-replay analysis establishes the presence of interactions, but does not by itself quantify the respective contributions of Shapley-style targets and the cross-request consistency regularizer. Comparisons with isolated-edit targets and with the consistency weight set to μ = 0 are important next steps.

Replacement design. The five registered replacements were designed by the authors and fixed before evaluation. They represent different intervention mechanisms associated with the registered assets, including diagnostic ordering and verification in prompts, evidence handling in knowledge, execution settings in the harness, and workflow structure in the support skill, as described in Appendix B, Table 6. We confirm that we did not inspect the failure trajectories or hidden task answers of the held-out evaluation tasks when designing or selecting these replacements. The compact registry provides a controlled evaluation of executable repair, while leaving the method's behavior under larger and more varied asset spaces to be studied.

After considering the feedback, we have decided to withdraw this submission to refine the method and develop a broader and more demanding evaluation, with stronger fixed-policy comparisons and targeted ablations. We appreciate the specific suggestions, which will help us assess the approach more thoroughly in a future version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.