- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the Detection-Aware Attribution Framework (DAAF), which is a framework that addresses the problem of which system assets should be changed upon an agent failure. The authors register five editable assets in a telecom support agent, consisting of two prompts, a knowledge segment, a runtime setting and a support procedure. Each has replacements written in advance and DAAF chooses between these replacements. They find that this generally helps support agents recover in the telecom setting.
Critical strengths
The problem that this paper studies is practical and relevant. The paper is written relatively clearly and the problem setup is also stated clearly. The problem of selecting a concrete system asset to repair makes sense. The result is promising, though the restricted edit menu and missing baselines limit what can be concluded about the method’s broader value.
Critical weaknesses
The evaluation setting is limited. The proposed method is only tested on tau-bench Telecom tasks, and the alternatives for editable assets are all hand-written. Extension to other agentic settings or support tasks, with larger and less curated candidate sets, would help establish generality. It would also be useful to compare against simpler policies such as simply applying the best fixed edit on all eval tasks to see when request-specific decisions are needed.
The practical advantage of this system over harness optimization and self-correction methods is not clear. DAAF learns to select among pre-specified replacements using a substantial offline replay budget, but the evaluation does not compare it with using that budget to optimize a single system configuration, for example with a prompt optimizer such as GEPA, or with an agent that corrects its execution using the same budget. A budget-matched comparison with direct evolution methods or simply giving the agent a higher step count to correct mistakes would make this a lot stronger.
Questions for authors
N/A
Response to Reviewer Lkei
We thank Reviewer Lkei for the review and for recognizing the practical relevance of identifying a concrete system asset for repair.
Evaluation scope and fixed-edit controls. The current study evaluates executable repairs on Telecom tasks using five pre-registered asset replacements. This controlled setting supports direct measurement of recovery and unintended regressions, but the compact candidate space and constrained repair interface do not sufficiently test the approach across more complex agent configurations. We agree that a development-selected best fixed edit, larger and less curated candidate sets, and additional environments are important for determining when request-specific repair decisions provide value beyond a shared system change.
Global optimization and self-correction. We agree that the practical trade-offs between attribution-guided repair, global system optimization, and agent self-correction deserve explicit study. Budget-matched comparisons with approaches such as GEPA-style configuration optimization and additional-step self-correction would help establish when targeted repair selection is useful. These approaches make different decisions, so a meaningful comparison should account for offline and online costs and evaluate actual task recovery, regressions, and overall success. The present experiments do not yet provide this comparative evidence.
We have decided to withdraw the submission to refine the method and establish a broader and more demanding evaluation, rather than make incremental changes to the current experimental setup. We appreciate your suggestions and will use them to guide the next stage of the work.