- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies failure attribution for skill-augmented LLM agents, distinguishing defects in supplied guidance from errors in its use. It introduces six fault families and SkillAttributionBench, comprising 194 human adjudicated trajectories across 30 task groups. The authors train SkillTracer, a 9B model, using supervised fine-tuning and GRPO on disjoint task groups. SkillTracer improves family pass@1 from 29.95% to 41.28%. A separate experiment uses human adjudicated diagnoses to guide single step skill edits, increasing task success from 43.3% to 63.3% across ten tasks.
Critical strengths
1.The research problem holds clear value for system maintenance: whether a failed execution calls for changing skill access, revising skill content, or correcting the agent’s use of adequate guidance. Organizing the taxonomy around these interventions gives the categories a clear practical purpose. 2.The benchmark draws on failures generated by multiple agent models and harnesses across several task domains. Independent annotation by three annotators, followed by adjudication, and task-group-disjoint training and evaluation are sensible design choices. The improvement over the base model also provides initial evidence that this diagnostic capability can be learned.
Critical weaknesses
1.The evaluation does not directly verify identification of the failure-producing decision. Family Accuracy and Anchor-F1 measure category agreement and quotation overlap, while the output contains neither a decision location nor trajectory-level evidence. A prediction could therefore match the gold category and rule while identifying the wrong causal decision. This leaves the stronger claim of tracing failures to their source insufficiently validated.
2.The repair experiment does not isolate the incremental value of explicit attribution. Kimi-K3 edits skills using human-adjudicated diagnoses, but the only comparison is against unmodified skills. Without a matched direct repair baseline receiving the same failure trace and editing budget, the reported 20-point gain cannot be attributed specifically to the supplied diagnosis. SkillTracer predictions are also not used, leaving the trained model’s contribution to downstream repair untested.
Questions for authors
1.How is correct identification of the decisive action verified when the required output contains only a fault family and rule quotations? Have you audited whether correct category predictions are supported by correct trajectory-level causal explanations?
2.On the existing ten repair tasks, could you compare direct one-edit repair, repair guided by human diagnoses, and repair guided by SkillTracer predictions?
A: Thank you for the comment. These two concerns point to the same issue, and we answer them together. Your question is why the combination of "category + skill-rule citation" can, to a large extent, pin down the true root cause. The key is that every component of this combination demands causal-level judgment. Our six-family criteria are defined by the decisive facts on the causal chain. A piece of skill text almost always influences only one specific decision, and different decisions in a trajectory are influenced by different skill texts; therefore the rule text located by the anchor itself corresponds to a specific decision point. Combined with the error category's further determination of "what causal role that text played in that decision," the two together essentially determine the location of the error's root in the current trajectory.
We also conducted a further validation: a manual audit of Kimi-K3's correct predictions. Among samples where the predicted family was exactly correct and the evidence citations highly overlapped with the adjudicated evidence (Anchor-F1 ≥ 0.8), we drew a stratified sample of 20 instances covering four families — M-unused, M-override, R-unspecified, and R-harmful — and the authors manually checked the model's full causal reasoning chain item by item. In all 20 cases, the causal explanation in the model's reasoning chain correctly located the failure-causing decision, consistent with human attribution. This shows that the current "category + rule citation" output does, in practice, pin down the true root cause, rather than merely happening to guess the right label.
A: Thank you for the suggestion. During the rebuttal we conducted the corresponding comparison experiments. We have completed all three settings you requested, with 6 runs per setting per task; the original and human-attribution settings merge the paper's 3 runs with 3 new runs from this round. Regarding the setup, in all settings the skill edit is produced by Kimi-K3, and each repaired skill is then tested by re-running the task with a DeepSeek-V4-Flash agent in the OpenCode harness. Results are as follows.
As can be seen, direct repair outperforms the original skills, and both attribution-guided repairs clearly outperform direct repair — this directly isolates the incremental value of explicit attribution.
We checked the diagnoses actually used in the SkillTracer setting one by one: for the 10 tasks, only 3 of its family labels matched the human adjudication, yet repair effectiveness was not impaired. There are two reasons. On one hand, the current repair mode channels every error type toward editing the skill text; on the other hand, what the repair model consumes is the causal explanation and evidence anchors in the diagnosis, and the repair model itself is fairly capable (the repair model used is Kimi-K3). Even when the family is judged wrong, the root-cause explanation in the reasoning chain can usually still steer the repair model toward the correct skill file and the problematic rule. This is consistent with the paper's design — repair value is carried mainly by the explanation and anchors — and it also shows that SkillTracer's diagnoses are more usable in the repair sense than its family-label accuracy alone would suggest.
Here is a concrete example illustrating the value of attribution. lab-unit-harmonization is a lab-unit conversion task whose verifier requires every lab value to fall exactly within the given reference range. In this failed trajectory, the agent set its own 6% tolerance and rewrote the lower bound of the reference range, causing the final acceptance check to fail. The no-attribution repair guessed the root cause to be "incomplete unit conversion" and went on to modify the conversion logic; all three post-repair runs failed (0/3). The attribution-guided repair knew the root cause lay in boundary determination, so it explicitly required strict range checking: values near a boundary must be corrected to the exact boundary value (e.g., 0.49 to 0.50), and the model is not allowed to use its own tolerance standard. All three post-repair runs passed (3/3).