acceptodds
Under review as a conference paper at ICLR 2027

When and How Do Skills Fail Agents? Tracing Failures in Skill-Augmented Agent Trajectories to Their Source

Abstract

Skills have become an important mechanism for improving the ability of large language model (LLM) agents to perform complex tasks. When a skill-augmented execution fails, however, the failure signal itself says little about the cause: a defective skill and an agent's misuse of a correct skill produce the same outcome but demand opposite repairs. This makes failure attribution over long-horizon trajectories of skill-augmented agents an important research problem. To study this problem, we develop a fine-grained attribution taxonomy and construct a human-annotated benchmark containing 194 trajectories from 30 task groups to evaluate the attribution capabilities of leading models. Our evaluation shows that many mainstream reasoning models have not yet reached a practically useful level of attribution performance. To further advance research on this capability, we propose SkillTracer, a 9B model specialized for skill attribution. SkillTracer improves attribution pass@1 by 11.3 points over its base model and outperforms frontier models evaluated without explicit reasoning at a fraction of their inference cost. Benchmark and code are available at https://anonymous.4open.science/r/SkillAttribution-5906.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WFivOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies failure attribution for skill-augmented LLM agents, distinguishing defects in supplied guidance from errors in its use. It introduces six fault families and SkillAttributionBench, comprising 194 human adjudicated trajectories across 30 task groups. The authors train SkillTracer, a 9B model, using supervised fine-tuning and GRPO on disjoint task groups. SkillTracer improves family pass@1 from 29.95% to 41.28%. A separate experiment uses human adjudicated diagnoses to guide single step skill edits, increasing task success from 43.3% to 63.3% across ten tasks.

Critical strengths

1.The research problem holds clear value for system maintenance: whether a failed execution calls for changing skill access, revising skill content, or correcting the agent’s use of adequate guidance. Organizing the taxonomy around these interventions gives the categories a clear practical purpose. 2.The benchmark draws on failures generated by multiple agent models and harnesses across several task domains. Independent annotation by three annotators, followed by adjudication, and task-group-disjoint training and evaluation are sensible design choices. The improvement over the base model also provides initial evidence that this diagnostic capability can be learned.

Critical weaknesses

1.The evaluation does not directly verify identification of the failure-producing decision. Family Accuracy and Anchor-F1 measure category agreement and quotation overlap, while the output contains neither a decision location nor trajectory-level evidence. A prediction could therefore match the gold category and rule while identifying the wrong causal decision. This leaves the stronger claim of tracing failures to their source insufficiently validated.

2.The repair experiment does not isolate the incremental value of explicit attribution. Kimi-K3 edits skills using human-adjudicated diagnoses, but the only comparison is against unmodified skills. Without a matched direct repair baseline receiving the same failure trace and editing budget, the reported 20-point gain cannot be attributed specifically to the supplied diagnosis. SkillTracer predictions are also not used, leaving the trained model’s contribution to downstream repair untested.

Questions for authors

1.How is correct identification of the decisive action verified when the required output contains only a fault family and rule quotations? Have you audited whether correct category predictions are supported by correct trajectory-level causal explanations?

2.On the existing ten repair tasks, could you compare direct one-edit repair, repair guided by human diagnoses, and repair guided by SkillTracer predictions?

1 comment

Q1: How is correct identification of the decisive action verified when the required output contains only a fault family and rule quotations? Have you audited whether correct category predictions are supported by correct trajectory-level causal explanations?

A: Thank you for the comment. These two concerns point to the same issue, and we answer them together. Your question is why the combination of "category + skill-rule citation" can, to a large extent, pin down the true root cause. The key is that every component of this combination demands causal-level judgment. Our six-family criteria are defined by the decisive facts on the causal chain. A piece of skill text almost always influences only one specific decision, and different decisions in a trajectory are influenced by different skill texts; therefore the rule text located by the anchor itself corresponds to a specific decision point. Combined with the error category's further determination of "what causal role that text played in that decision," the two together essentially determine the location of the error's root in the current trajectory.

We also conducted a further validation: a manual audit of Kimi-K3's correct predictions. Among samples where the predicted family was exactly correct and the evidence citations highly overlapped with the adjudicated evidence (Anchor-F1 ≥ 0.8), we drew a stratified sample of 20 instances covering four families — M-unused, M-override, R-unspecified, and R-harmful — and the authors manually checked the model's full causal reasoning chain item by item. In all 20 cases, the causal explanation in the model's reasoning chain correctly located the failure-causing decision, consistent with human attribution. This shows that the current "category + rule citation" output does, in practice, pin down the true root cause, rather than merely happening to guess the right label.

Q2: On the existing ten repair tasks, could you compare direct one-edit repair, repair guided by human diagnoses, and repair guided by SkillTracer predictions?

A: Thank you for the suggestion. During the rebuttal we conducted the corresponding comparison experiments. We have completed all three settings you requested, with 6 runs per setting per task; the original and human-attribution settings merge the paper's 3 runs with 3 new runs from this round. Regarding the setup, in all settings the skill edit is produced by Kimi-K3, and each repaired skill is then tested by re-running the task with a DeepSeek-V4-Flash agent in the OpenCode harness. Results are as follows.

SettingPass rate (6 runs)
Original skills43.3% (26/60)
Direct repair (no diagnosis)60.0% (36/60)
Repair guided by SkillTracer predictions66.7% (40/60)
Repair guided by human diagnoses68.3% (41/60)

As can be seen, direct repair outperforms the original skills, and both attribution-guided repairs clearly outperform direct repair — this directly isolates the incremental value of explicit attribution.

We checked the diagnoses actually used in the SkillTracer setting one by one: for the 10 tasks, only 3 of its family labels matched the human adjudication, yet repair effectiveness was not impaired. There are two reasons. On one hand, the current repair mode channels every error type toward editing the skill text; on the other hand, what the repair model consumes is the causal explanation and evidence anchors in the diagnosis, and the repair model itself is fairly capable (the repair model used is Kimi-K3). Even when the family is judged wrong, the root-cause explanation in the reasoning chain can usually still steer the repair model toward the correct skill file and the problematic rule. This is consistent with the paper's design — repair value is carried mainly by the explanation and anchors — and it also shows that SkillTracer's diagnoses are more usable in the repair sense than its family-label accuracy alone would suggest.

Here is a concrete example illustrating the value of attribution. lab-unit-harmonization is a lab-unit conversion task whose verifier requires every lab value to fall exactly within the given reference range. In this failed trajectory, the agent set its own 6% tolerance and rewrote the lower bound of the reference range, causing the final acceptance check to fail. The no-attribution repair guessed the root cause to be "incomplete unit conversion" and went on to modify the conversion logic; all three post-repair runs failed (0/3). The attribution-guided repair knew the root cause lay in boundary determination, so it explicitly required strict range checking: values near a boundary must be corrected to the exact boundary value (e.g., 0.49 to 0.50), and the model is not allowed to use its own tolerance standard. All three post-repair runs passed (3/3).

Reviewer_5QyxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to identify whether a skill-augmented agent fails because a skill was not loaded, because the skill is incomplete or harmful, or because the agent used it incorrectly. It introduces a six-category taxonomy, a benchmark with 194 human-adjudicated trajectories, and SkillTracer, a 9B model that improves pass@1 over its base model by 11.3 points. A small repair study also shows that human-provided diagnoses can improve task success after one skill edit.

Critical strengths
  1. The paper defines a clear and useful diagnostic task. Different failure sources require different fixes, and the proposed taxonomy makes this distinction explicit.
  2. The benchmark keeps the task, skill files, skill-access records, full trajectory, and verifier feedback. This gives models enough context to make a more informed diagnosis than one based only on the final error.
Critical weaknesses
  1. The repair study doesn't test the paper's main claim. The motivation is that skill defects and agent misuse need opposite repairs, but the study only allows skill edits and never says which families the ten tasks belong to. The gain from 13/30 to 19/30 also isn't significant (p ≈ 0.2), and there is no control where the skill is edited without the attribution label. The original skills already pass 43% of fresh runs on these "failed" tasks.
  2. SkillTracer's training labels come from GPT-5.6-sol, which scores only 51.6 to 56.9 pass@1 on this same benchmark. The two-stage pipeline is never checked against human labels, so we don't know how noisy D_nat is. This also conflicts with the AI use statement that LLMs were not used as autonomous annotators.
  3. The 32K compression keeps the "adjudicated evidence spans" (L365-366), so it knows where the gold evidence is. A deployed attributor wouldn't have that. The compression check in Table 5 also runs on a much shorter subset than the full benchmark (41.6K mean full input vs. a 135K benchmark mean). So neither Figure 4 nor Appendix B shows that length and compression don't matter.
  4. The main numbers have no reference points. With six families, random guessing gives about 16.7% pass@1 and about 52% pass@4, and Qwen3-8B and Qwen3.5-9B sit close to that. The paper reports no label distribution or confidence intervals; with 192 items, pass@1 has roughly a ±7 point 95% interval. It also gives no human ceiling: annotators disagree 17.7% to 26.6% pairwise, but there is no per-annotator accuracy against the final labels.
  5. SkillTracer is oversold. Anchor-F1 goes from 12.36 to 14.09, which is still below every thinking model, even though anchors carry 40% of the reward. SkillTracer writes an explanation before its JSON, while the no-thinking baselines are told to output only JSON. And 62.5% is called not reliable enough to replace humans (L411-413), while 41.3% is called "the practical operating point" (L443-444).
  6. The taxonomy is closed-world. There is no "not skill-related" label, and the prompt asks for "the earliest skill-related decision" (Table 6). So the benchmark can't measure false attributions on failures that have nothing to do with skills, and in a repair loop those failures would still lead to skill edits.
  7. Related work misses close prior work. Aegis (ICLR 2026) already builds attribution training data by injecting faults into successful runs and trains with partial-credit RL. SkillAxe and SkillTriage separate skill-side from agent-side causes in some form. The name SkillTracer is also taken by a KDD 2026 paper on skill failure attribution.
Questions for authors
  1. 30 hours per annotator over 194 trajectories works out to about 9 minutes each, at a 135K-token mean. What did annotators actually read, and were they shown any LLM pre-labels?
  2. Task groups are disjoint, but are the skills themselves disjoint between training and the benchmark? SkillsBench tasks often share generic skills such as document or spreadsheet handling.
  3. Section 4.1 only describes removing, weakening or replacing rules. How are S examples built in D_inj, and where do the M-unused and M-override training examples come from?
  4. Is Anchor-F1 counted only when the predicted family is correct? Or can a model get anchor credit while choosing the wrong family?
  5. Was Qwen3.5-9B run with thinking enabled? How was the $0.50 cost for SkillTracer computed, compared with the API prices used for the other models?
7 comments

W1: The repair study doesn't test the paper's main claim. The motivation is that skill defects and agent misuse need opposite repairs, but the study only allows skill edits and never says which families the ten tasks belong to. The gain from 13/30 to 19/30 also isn't significant (p ≈ 0.2), and there is no control where the skill is edited without the attribution label. The original skills already pass 43% of fresh runs on these "failed" tasks.

A: Thank you for the detailed analysis.

The 10 tasks contain 20 failed trajectories — 4 C-absent, 5 R-unspecified, 1 R-harmful, 7 M-unused, and 3 M-override — and trajectories of the same task may exhibit different error types. For the M-family failures, model-level repair cannot be done inside the repair loop (we cannot modify the agent model itself), so under the constraint that only skills can be edited, the only available intervention is to emphasize and repeat the rule in the relevant skill sections.

Regarding the original skills' 43% pass rate on new runs: this reflects the true nature of agent failure. Errors are inherently diverse and sporadic — re-running the same task, the agent does not necessarily fail, and even when it fails, it does not necessarily make the same error as in the observed trajectory. Our repair targets the specific error in the observed trajectory; it cannot, and does not attempt to, eliminate all possible failure modes on that task. Hence the baseline's natural pass rate does not contradict the effectiveness of repair.

Regarding insufficient power and the missing control, during the rebuttal we completed a three-way comparison on the same batch of repair candidates, with 6 runs per setting per task (skill edits produced by Kimi-K3, and repaired skills tested by re-running the task with a DeepSeek-V4-Flash agent in the OpenCode harness): the original and with-attribution settings merge the paper's 3 runs with 3 new runs from this round, and the no-attribution setting consists of 6 newly executed runs. Results are as follows.

SettingPass rate (6 runs)
Original skills43.3% (26/60)
Repair without attribution60.0% (36/60)
Repair with attribution (adjudicated labels)68.3% (41/60)

As the results show, repair with attribution is clearly higher than both repair without attribution and the original skills. This directly answers the "control without attribution labels": the attribution diagnosis yields an increment of 8.3 points. Regarding the p ≈ 0.2 concern, the original 3-run experiment was indeed underpowered. Doubling the number of runs does not change the conclusion, the effect is consistent in direction and stable in size across both rounds (+20 points at 3 runs, +25 points at 6 runs). Meanwhile, the re-run original setting matches the paper's Figure 5 exactly (13/30), confirming that this round's execution protocol is faithful to the original experiment.

W2: SkillTracer's training labels come from GPT-5.6-sol, which scores only 51.6 to 56.9 pass@1 on this same benchmark. The two-stage pipeline is never checked against human labels, so we don't know how noisy D_nat is. This also conflicts with the AI use statement that LLMs were not used as autonomous annotators.

A: Thank you for raising this; we address each point in turn.

Regarding the AI-usage statement, there is no conflict. The first sentence of the statement explicitly discloses the use of an LLM for training-set annotation. The sentence "They were not treated as autonomous annotators" is used together with the immediately following "Final benchmark labels were human adjudicated," and refers to the benchmark, all final benchmark labels were human-adjudicated. This may be a conflation of the training set with the benchmark. We acknowledge that the sentence's reference could be clearer, and the revision will state training-set annotation and benchmark annotation in separate, explicit passages.

Regarding the quality of the training labels, they are produced by a two-stage automated pipeline: the first stage proposes candidate families and evidence anchors, and the second performs a root-cause-oriented review (CausalReview), confirming or correcting each label. Samples still uncertain after review (along with infrastructure failures and invalid tasks) are discarded (⊥ in Eq. (4)) — the pipeline keeps a label only when it can verify it, and only this verified subset enters D_nat. The teacher model's single-pass pass@1 (51.6–56.9) therefore cannot be directly used to estimate the annotation accuracy of this pipeline.

Ultimately, the quality of this data is reflected in the training outcome, SkillTracer improves over its base by 11.3 points on the benchmark. Weakly supervised data with excessive noise could not produce such an improvement; this gain is direct evidence of the data's validity.

W3: The 32K compression keeps the "adjudicated evidence spans" (L365-366), so it knows where the gold evidence is. A deployed attributor wouldn't have that. The compression check in Table 5 also runs on a much shorter subset than the full benchmark (41.6K mean full input vs. a 135K benchmark mean). So neither Figure 4 nor Appendix B shows that length and compression don't matter.

A: Thank you for the challenge, but the premise here is inconsistent with our design intent. Our purpose in compressing trajectories is to control the cost of evaluating and applying this benchmark: the benchmark's raw trajectories average 135K tokens, with the longest exceeding 4M, making direct evaluation both expensive and beyond the context of most models. We therefore provide a 32K compressed evaluation version of the benchmark. We want to emphasize that we are not proposing a lossless compression scheme, compression is part of the benchmark construction process, and compressing trajectories at construction time — when the labels are already known — is entirely legitimate. Its validity is directly verified by the controlled experiment in Table 5: three models show essentially unchanged pass@1 before and after compression (means 51.9 vs. 52.2) at roughly half the input cost, indicating that compression does not distort attribution difficulty.

In actual deployment, an attributor can simply take the raw trajectory as input, or apply only mechanical compression to repetitive tool observations. The 32K version is merely a standardized option we provide to reduce evaluation cost; it is not a component of the attribution method.

You are correct that Table 5 runs on a shorter subset: it covers only some instances, so its full-input mean of 41.6K is not directly comparable to the benchmark mean of 135K (which is pulled up by outlier trajectories as long as 4.37M tokens; the median is 49K). To fill this gap, we recomputed the compression comparison over all evaluation instances (32K version vs. raw trajectories, covering the full benchmark including the longest trajectories):

Model32K pass@1Raw-trajectory pass@1
GLM-5.3-Flash55.255.6
DeepSeek-V4-Flash48.745.4
Kimi-K2.648.446.2
Mean50.849.1

The pre/post-compression differences for all three models are within ±3.3 points (mean difference −1.7 points), confirming that compression indeed does not distort attribution difficulty. As for whether trajectory length itself causes measurement distortion, Figure 4 already answers this: under a uniform 32K input, grouped by source-trajectory length, the accuracy of three representative reasoning models varies by no more than 3.3 points, and the short-trajectory group's accuracy remains far from saturation (<65%). If difficulty came mainly from length, the short-trajectory group should be markedly easier — but it is not.

W4: The main numbers have no reference points. With six families, random guessing gives about 16.7% pass@1 and about 52% pass@4, and Qwen3-8B and Qwen3.5-9B sit close to that. The paper reports no label distribution or confidence intervals; with 192 items, pass@1 has roughly a ±7 point 95% interval. It also gives no human ceiling: annotators disagree 17.7% to 26.6% pairwise, but there is no per-annotator accuracy against the final labels.

A: Thank you for the suggestions.

Regarding the label distribution, the full benchmark released in the anonymous repository contains per-instance gold labels, so the distribution can be directly recomputed. Among the 192 evaluation instances: S 46 (24.0%), M-unused 43 (22.4%), R-harmful 34 (17.7%), M-override 30 (15.6%), R-unspecified 27 (14.1%), C-absent 16 (8.3%). Under this distribution, a random single-label baseline is about 16.7%, and a majority-class baseline (always predicting S) is about 23.4%. Qwen3-8B's 17.71% is indeed below the majority-class baseline, but this precisely demonstrates the absence of attribution ability in small models. Qwen3.5-9B's 29.95% is already clearly above the majority-class baseline, and SkillTracer's 41.28% is significantly higher than both, hardly "close to random." The revision will add the distribution table and these two baseline rows to the main text.

Regarding confidence intervals, pass@1 is the average correctness over 4 samples per instance. We computed bootstrap 95% confidence intervals at the instance level (rather than treating the 768 samples as i.i.d.), and the magnitude matches your estimate of ±7 points:

Modelpass@1Instance-level 95% CI (bootstrap)
Qwen3-8B17.71[13.7, 22.1]
Qwen3.5-9B base29.95[25.0, 34.9]
+ EG-SFT34.38[29.6, 39.1]
SkillTracer41.28[35.6, 47.3]
GLM-5.3-Flash55.21[49.0, 61.2]
GPT-5.6-sol (high)56.90[51.0, 62.6]
Kimi-K362.50[56.5, 68.5]

Note that SkillTracer's interval does not overlap with its base's ([35.6, 47.3] vs. [25.0, 34.9]), so the +11.3-point improvement is significant under an instance-level test. We also checked the stability of the evaluation protocol itself. The standard deviation across 4 repeated evaluation rounds is within 2.5 points for all 14 configurations (see the table below), and the inter-model gaps in Table 2 are far larger than this round-to-round fluctuation.

Modelpass@1 meanStd across 4 rounds (pt)
Kimi-K2.648.441.95
DeepSeek-V4-Flash48.702.35
GLM-5.250.782.50
GLM-5.3-Flash55.212.08
GPT-5.6-sol (low)51.562.25
GPT-5.6-sol (high)56.901.97
Kimi-K362.502.21
DeepSeek-V4-Flash (no-thinking)32.421.56
GLM-5.2 (no-thinking)26.951.07
Kimi-K2.6 (no-thinking)37.242.10
Qwen3-8B17.711.65
Qwen3.5-9B29.951.62
+ EG-SFT34.382.21
SkillTracer41.282.49

Regarding the human ceiling, the three annotators' accuracies against the final adjudicated labels are 82.1%, 86.4%, and 94.4% (mean 87.6%), which provides an approximate human-ceiling reference. The strongest model, Kimi-K3 at 62.5%, still falls clearly short of even the lowest of the three.

W5: SkillTracer is oversold. Anchor-F1 goes from 12.36 to 14.09, which is still below every thinking model, even though anchors carry 40% of the reward. SkillTracer writes an explanation before its JSON, while the no-thinking baselines are told to output only JSON. And 62.5% is called not reliable enough to replace humans (L411-413), while 41.3% is called "the practical operating point" (L443-444).

A: Thank you for the comment. SkillTracer's absolute performance is indeed still far from practical (41.3% pass@1, 14.09 Anchor-F1), and the paper explicitly says so. We have never claimed that it reaches the level of thinking models. For the same reason, the two quoted statements are not in tension. "Practical operating point" refers to cost-effectiveness at a given budget, not to replacing human adjudication.

The value of this work lies in another dimension, attribution cost. At roughly $0.5 per evaluation, SkillTracer achieves 41.3% pass@1 and surpasses frontier models evaluated in non-explicit-reasoning mode, the cheapest of which costs an order of magnitude more. Meanwhile, the +11.3-point gain over the base shows that our data pipeline, dual-source causal supervision combined with an evidence-aligned reward, works, and that attribution ability is trainable even at the 9B scale. We also acknowledge that the gain is bounded by base capability. The Qwen3.5-9B base starts at only 29.95%, so +11.3 points already approaches what a small model can reach. A future 27B version should close much of the remaining gap while retaining the cost advantage.

Reviewer_cuU5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an attribution taxonomy and human annotated benchmark to attribute agentic failiures to skills across selection, content, and incorrect use. The paper introduces a 9B specialized attribution model trained using SFT and RL, outperforming frontier models at a fraction of the cost.

Critical strengths

The paper evaluates on a wide and diverse range of recent models and provides the benchmark and code for reproducibility

Critical weaknesses
  1. The results do not establish that SkillTracer provides actionable diagnoses as in Figure 5 the reported results and gains are not reported to come from SkillTracer predictions.
  2. The paper lacks analysis in terms of the causal validity of the attribution labels as they primarly determine root cause through human reconstruction.
  3. The paper lacks much discussion and evaluation in terms of what happens when the model encounters a failiure unrelated to any suppled skill.
Questions for authors

Could the authors please address the reported weaknesses, and I would be happy to consider raising my score?

2 comments

W1: The results do not establish that SkillTracer provides actionable diagnoses as in Figure 5 the reported results and gains are not reported to come from SkillTracer predictions.

A1: Thank you for the comment. Figure 5 evaluates the benchmark itself, whether human-adjudicated diagnoses provide an actionable interface from failure to repair. SkillTracer's own claim rests on the attribution accuracy in Table 2, and using adjudicated labels in Figure 5 is deliberate, so that the outcome cannot be confounded by the model's imperfect predictions.

That said, the submitted version indeed did not directly use SkillTracer's predictions to guide repair. We have therefore added this experiment in this round, prompted by the closely related concerns W2/Q2 of Reviewer WFiv, on the same 10 repair tasks, we added a repair setting guided by SkillTracer's predictions, with 6 runs per setting. Results are as follows (the original-skill and human-attribution settings merge the paper's 3 runs with 3 new runs from this round; see our response to Reviewer WFiv W2/Q2 for details):

SettingPass rate (6 runs)
Original skills43.3% (26/60)
Direct repair (no diagnosis)60.0% (36/60)
Repair guided by SkillTracer predictions66.7% (40/60)
Repair guided by human diagnoses68.3% (41/60)

Repair guided by SkillTracer's predictions (66.7%) is clearly higher than direct repair without a diagnosis (60.0%) and close to the level of repair guided by human diagnoses (68.3%). This directly demonstrates that SkillTracer's diagnoses are themselves actionable: the causal explanations carried in its reasoning chain and the evidence anchors it outputs are sufficient to steer the repair model toward the correct skill file and the problematic rule, thereby closing the loop between the trained model and the benchmark.

W2: The paper lacks analysis in terms of the causal validity of the attribution labels as they primarly determine root cause through human reconstruction.

A2: Thank you for the suggestion. You point out that root causes are primarily determined through human reconstruction, and we agree this warrants a more direct test. The strongest validation of the causal validity of attribution labels is intervention: if a label locates a true root cause, then a targeted intervention following that label should produce observable improvement, whereas an intervention based on a wrong label should be hard to make work. This is exactly the design of Figure 5: make at most one edit to the skills according to the adjudicated label. The paper reports pass rate improving from 43.3% to 63.3% (3 runs each); in this round we expanded the number of runs to 6 per setting per task.

Meanwhile, in practice we found that post-repair success is itself a limited metric: errors occur randomly and diversely (re-running the same task does not necessarily fail, and a failure does not necessarily repeat the same error — which is exactly why re-runs of the original skills also achieve a non-trivial pass rate). Combined with task complexity, a single atomic edit, and the repair model's limited capability, a correct diagnosis does not guarantee that one repair attempt succeeds. We therefore additionally define a process-progress metric: each task is paired with a fixed table of process stages (the required intermediate steps specified by the acceptance criteria or task instructions), with manual verification judging, stage by stage for each trajectory, whether the stage was "completed and correct"; each judgment must quote the trajectory text and is programmatically cross-checked. Process progress is the fraction of stages (0–1) that the trajectory advances through correctly, and the metric itself is auditable.

Results of the two metrics on the 10 tasks of the repair experiment are as follows (the stage tables themselves were validated against passing trajectories):

MetricOriginal skillsRepair guided by adjudicated labels
Pass rate (6 runs)43.3% (26/60)68.3% (41/60)
Process progress (0–1)0.7730.903

For every task where the pass rate improved, process progress also increased. More telling are the tasks where the pass rate stayed flat: for example, on energy-unit-commitment the pass rate did not change, yet process progress under the repair setting rose from 0.233 to 0.533 — i.e., after repair, executions advanced through and correctly completed more of the required stages within the same time budget. This shows that the attribution labels do carry the causal information needed for repair, even where the final pass rate, bounded by randomness, fails to reflect it, the repaired execution process is systematically closer to the correct solution of the task.

W3: The paper lacks much discussion and evaluation in terms of what happens when the model encounters a failiure unrelated to any suppled skill.

A3: Thank you for the suggestion. We did consider this question when constructing the benchmark. On one hand, in our attribution practice we found that such failures are almost never seen; on the other hand, for skill-guided complex tasks, we expect skills to provide adequate guidance given the agent's current capabilities, so we deliberately removed this category during benchmark construction. The reasons can be summarized in three points. First, skill-unrelated errors are mainly infrastructure problems: runs terminated early due to environment/infrastructure issues, or runs with invalid tasks/verifiers, can be directly identified and filtered, and do not require model evaluation. Second, failures that appear "skill-unrelated" have in fact already been absorbed by the taxonomy. The C-absent family describes exactly the situation where "no reusable rule exists in the consulted skills to guide this decision" — i.e., failure caused by missing guidance. Many failures that intuitively seem "unrelated to skills," once traced back to the failure-causing decision, turn out to fall under "the skill did not cover that decision" (C-absent) or "the skill provided wrong/incomplete guidance" (the R families). Third, current agents' basic capabilities are already strong, and low-level errors are rare. Among failures on these skill-guided complex tasks, pure execution-level low-level mistakes are uncommon; a failure can almost always be traced to a failure-causing decision at one of three levels: skill selection, skill content, or skill usage. This is precisely why we built the taxonomy on these three repair actions: it covers the actual distribution of failures in practice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.