acceptodds
Under review as a conference paper at ICLR 2027

Not All Skills Help: Measuring and Repairing Agent Skill Libraries

Abstract

LLM agents can improve without weight updates by accumulating natural-language skills, but a plausible instruction need not help the task on which it is used. We propose ASSAY, a framework that connects skill-library repair and task-specific selection through shared execution measurements. Randomized masking estimates each skill's marginal causal effect under sampled skill contexts across development tasks. The resulting task-indexed profiles guide conditional rewrites offline and are combined over nearby development tasks to select skills for each test instruction. Task similarity retrieves development evidence; execution-based scores determine which eligible skills to suppress. On GPT-5.1 / AppWorld *test_normal*, ASSAY achieves 77.4% completion versus 70.8% for ACE with the same templates. On the same restructured library, task-conditioned masking outperforms count-matched random and global-score masking by 7.4 and 4.2 percentage points, respectively. Independently regenerating attribution yields 76.8%. Across seven models and four providers, the complete pipeline improves over the respective baselines on both AppWorld splits and for five of seven models on -bench retail. GPT-5.1 improves from 49.9% with ACE to 66.4% on *test_challenge*. These results establish the practical value of connecting skill generation to empirical curation and task-specific use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LM agents, skill learning, agent memory, causal attribution, skill curation, test-time adaptation, tool-use agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BqPxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ASSAY measures how including individual instructions affects an agent's task success under randomized skill masks. The resulting task-indexed scores guide offline library restructuring and task-dependent filtering at inference. For a new task, the method combines scores from similar development tasks rather than observing the new task's reward. The evaluation covers seven models on AppWorld and τ-bench retail, with detailed controls on GPT-5.1/AppWorld.

Critical strengths

The paper identifies a useful limitation of accumulated skill libraries: a generally relevant instruction can still impair execution on a particular task. Measuring outcomes provides a concrete basis for deciding which instructions to retain. The estimator has a clear interpretation under the specified mask distribution, and the paper distinguishes it from both Shapley attribution and full-library leave-one-out effects.

The matched controls are persuasive. On GPT-5.1/AppWorld test_normal, ASSAY reaches 77.4% versus 70.8% for ACE with the same five templates. On the same restructured library and with matched removal counts, random masking reaches 70.0%, global-score masking 73.2%, and task-conditioned masking 77.4% (Appendix F). These comparisons support a contribution beyond supplying templates or merely shortening the prompt. Fresh masks and three resampled development sets provide further evidence that the result is not peculiar to one attribution run. I also appreciate the explicit reporting of noisy cell estimates and unchanged τ-bench results in two model settings.

Critical weaknesses
  1. The contribution of library repair is not separated from the contribution of selection. ACE with the same templates reaches 70.8% (Table 5), while the restructured library without masking reaches 69.6% (Table 6). This does not show that repair is harmful: restructuring may make subsequent selection more effective. But the comparison needed to establish that interaction is missing—task-conditioned masking applied directly to the original ACE-plus-template library. Figure 1 demonstrates such masking for one task, which reinforces the relevance of this control without answering it at dataset level. The current evidence supports the selection component more clearly than it supports the full measurement-and-repair pipeline.

  2. The uncertainty of the key task-conditioned versus global-score comparison is not directly reported. Table 7 differs by seven successes out of 168, but the paired disagreement counts are not given. The McNemar test in Appendix I.1 compares two ASSAY runs and answers a different question. The development-set repetitions are reassuring; paired uncertainty for the existing matched-control outcomes would show how strongly the main result supports task-dependent selection on this evaluation set.

Questions for authors
  1. What is the result of applying the same task-conditioned selection procedure to ACE plus the same templates, without split/retire/merge? Comparing this with the full pipeline would clarify whether repair improves the final decisions.

  2. Is there evidence that negatively scored skills also hurt when evaluated against a background closer to deployment, with protected entries always present? A limited check on selected skills would help connect the measured effect to the removal decision.

  3. Could you report the paired discordant counts or a paired confidence interval for ASSAY versus global-score masking, and for ASSAY versus ACE plus templates? Those comparisons would be more informative for the main claims than another test of agreement between ASSAY runs.

3 comments
AuthorsOpenReview ↗

To Reviewer BqPx: Additional Evidence on Repair, Selection, and Paired Outcomes

Thank you for identifying the missing control needed to establish the contribution of repair! We have completed this experiment and the paired analyses you requested.

1. Repair improves upon direct selection on the original library.

We apply the same attribution-guided selection procedure directly to ACE with identical templates, preserving the original attribution rows, selection hyperparameters, and fallback rule.

LibraryWithout selectionWith selection
Original ACE + templates119/168 (70.8%)121/168 (72.0%)
Restructured117/168 (69.6%)130/168 (77.4%)

Full ASSAY solves nine additional tasks relative to direct selection: 12 ASSAY-only versus three direct-selection-only successes. The improvement is +5.36 percentage points, with a scenario-clustered 95% confidence interval of [+1.19, +9.52].

This fills the missing experimental condition: with the selection procedure held fixed, restructuring improves the resulting pipeline. We distinguish this incremental contribution from the repair–selection interaction, whose estimated effect is positive but whose interval includes zero.

2. Paired uncertainty for the principal comparisons.

ComparatorASSAY-only / comparator-onlyImprovement, pp [95% CI]
ACE + identical templates16 / 5+6.55 [1.79, 11.31]
Global-score masking10 / 3+4.17 [0.00, 8.33]

These intervals resample complete scenarios, retaining paired outcomes and dependencies among task variants. They directly quantify uncertainty for the requested comparisons, rather than agreement between two ASSAY runs.

3. Negative scores predict harmful effects under deployment backgrounds.

We performed single-skill inclusion/removal interventions on the same 40 eligible tasks, keeping all other deployed instructions fixed and protected entries present. Removing strongly negative, near-threshold, and nonnegative skills yields net benefits of +25, 0, and −15 percentage points, respectively.

The within-task negative-versus-nonnegative contrast is +40 points [95% scenario-clustered CI: +12.20, +69.05]. Because each task contributes all three interventions, this comparison controls for differences in task composition. The result applies to the audited population satisfying all three score-stratum requirements.

Together, these experiments supply the missing evidence for repair’s incremental contribution and directly connect attribution predictions to execution consequences under deployment conditions. Thank you again!

I appreciate the authors’ response and their answers to my questions. However, I still believe my initial overall rating reflects my assessment of the paper, and therefore I would like to keep it.

AuthorsOpenReview ↗

Clarification on Remaining Concerns

Thank you very much for following up and for considering our response! We would appreciate any further guidance on concerns that remain after the additional experiments. We have added the requested original-library selection control, paired uncertainty analyses, and deployment-background interventions. In particular, the new control shows a +5.36 pp benefit from repair with selection held fixed (95% CI: [1.19, 9.52]). Could you help us understand where you find the evidence still insufficient? This would help us focus our remaining revisions. Thank you again for your time and feedback!

Reviewer_r7dfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ASSAY estimates task-specific skill effects with randomized masking, then uses those estimates to restructure the library and choose which skills to deploy. It improves performance in a controlled AppWorld experiment and reports further cross-model and transfer results, along with detailed uncertainty diagnostics.

Critical strengths

The paper studies a real practical issue: adding more skills can introduce advice that is harmful or only useful in certain contexts. Randomized masking gives more information than simple usage counts, and I appreciated that the authors report the weak points of the estimator quite openly.

Critical weaknesses

The strongest cell-level measurement claim does not fit the diagnostics very well. With K=12 masks, power for a +/-0.30 effect is 38.5%, fresh-mask cell correlation is 0.34, and only 4.2% of the selected masking predictions have a 95% interval wholly below the threshold. The last number is not a measure of decision accuracy, as the paper correctly notes, but it does show wide uncertainty around individual effect estimates. The stable downstream result supports the overall intervention more than reliable causal attribution for each task-skill cell.

It is also unclear whether the causal matrix is needed compared with a simpler task-conditioned selector using the same library and budget. Accepted split variants inherit transformed scores from their parent instead of being measured directly, and they are protected from later retirement or masking. The seven-model and tau-bench results also do not use the same matched decomposition as the main GPT-5.1 experiment. This makes both the novelty and the causal interpretation weaker than the paper suggests.

Questions for authors

Please compare with a semantic or learned task-conditioned skill selector that uses the same library and budget but does not build the causal matrix. This would show whether the gain comes from randomized attribution or from ordinary task-specific retrieval and pruning.

Please measure accepted split variants again on held-out development tasks and keep them eligible for retirement or masking. It would also help to test larger attribution budgets and decisions filtered by confidence intervals. The causal wording should be adjusted to match the reliability shown by these diagnostics.

3 comments
AuthorsOpenReview ↗

To Reviewer r7df: Attribution Reliability and Independent Validation of Split Variants

Thank you for highlighting the distinction between individual estimation precision and reliable deployment decisions! We address your concerns with competitive selectors, independent rewrite evaluation, and direct deployment interventions.

1. Attribution predictions guide effective deployment interventions.

Randomization identifies the marginal causal effect under the specified mask distribution (§2.1); finite K limits estimation precision. To test the resulting decisions directly, we performed single-skill interventions on the same 40 eligible tasks, holding the surrounding deployed context fixed.

Removing strongly negative, near-threshold, and nonnegative skills yields net benefits of +25, 0, and −15 percentage points. The within-task negative-versus-nonnegative contrast is +40 points [95% scenario-clustered CI: +12.20, +69.05]. Thus, predicted effect groups distinguish beneficial from harmful removal interventions under matched task contexts.

2. Comparison with task-conditioned selectors.

Using the same restructured library, protected entries, and per-task removal counts, semantic selection solves 120/168, GPT-5.1-based selection 124/168, and ASSAY 130/168 tasks. The LLM selector evaluates applicability, redundancy, conflicts, and execution overhead. Independently dev-calibrating its retention budget yields 125/168.

ASSAY achieves the highest observed completion; the matched-count LLM comparison is +3.57 points [95% scenario-clustered CI: −0.60, +7.74]. The intervention audit supplies complementary evidence that the attribution signal predicts execution consequences.

3. Accepted rewrites generalize beyond their development gate.

Across 108 task–parent comparisons from 36 scenarios, excluded from attribution and gate decisions, nine audited conditional rewrites improve completion from 68/108 to 78/108: +9.26 points [1.85, 16.67]. Six rewrites improve, two are unchanged, and one worsens; gains concentrate in trigger-applicable contexts.

We also remeasured accepted variants through 180 executions on another 15 development tasks, keeping nonsplit instructions fixed and templates present. Variants use retrieval over the new development set. Making them eligible for masking yields 128/168, versus 130/168 with protection: a protected-minus-unprotected difference of +1.19 points [−2.98, +5.36]. Allowing independently remeasured variants to be masked does not yield a detectable improvement over the submitted protection policy. This tests masking protection, with retirement held fixed.

4. Attribution budget and confidence filtering.

The larger-budget experiments already appear in §3.2 and Appendix I.3, Table 10: K=12/24/48 yields 130/131/128 successes. Appendix I.5 reports confidence-interval diagnostics. The deployed rule uses aggregated point estimates rather than confidence-bound filtering; the new interventions directly evaluate that rule’s behavioral utility.

These additions address the reliability concerns separately: accepted rewrites improve execution beyond their construction tasks, and attribution predictions distinguish deployment interventions with different consequences. Thank you again!

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your thoughtful feedback! As the discussion period is nearing its end, we would appreciate hearing whether our response and additional experiments address your concerns, or if there are any remaining points you would like us to clarify. We would be happy to follow up. Thank you so much for your time and consideration!

AuthorsOpenReview ↗

Follow-up before the discussion deadline

Dear Reviewer r7df, With the discussion period ending soon, we would be grateful if you could share your thoughts on our response and additional experiments when you have a chance. Please let us know if any concerns remain, and we would be more than happy to clarify them before the deadline. Thank you very much for your time and consideration!

Reviewer_x1BROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ASSAY: a causal attribution and skill-repair framework to tackle the problems that come with skill accumulation in LLM agents. ASSAY has three main stages: Causal attribution which estimates the contributions of every skill with respect to each task followed by library restructuring phase where the existing skills are split, retired or merged based on these estimates. Finally during the inference, similar development tasks are retrieved and their attribution scores are used to decide which skills have to be masked. The paper shows improvements over ReAct on seven models on AppWorld, and over existing skill-based methods (ACE, CUGA, Gupta et al.,) for four models. On t-bench, it improves over the unaugmented baseline for five of seven models.

Critical strengths
  1. The framework looks at the effect of the skill on task success rate by randomised masking rather than just looking at the task success with the full library or semantic relevance. This is an interesting approach for skill selection and fragments inter-skill dependency.
  2. The improvements over the baselines are significant for AppWorld and the paper evaluates 7 models. Shows similar pattern for a subset of models for another dataset.
  3. This paper also includes comparisons against random masking and global score masking. These show that the task based skill selection is better than removing the skills with low global importance.
Critical weaknesses
  1. The framework requires 180 trajectories/roll-outs (in default setting) for causal attribution. It seems very costly especially in long horizon tasks. Cost comparison with methods like ACE would also be helpful. A lot of recent work on online/offline skill based learning is reporting cost vs. improvement tradeoff to give a sense of moving to a bigger model vs relying on skills.
  2. During the inference, the framework uses embeddings to retrieve the topk (k=8) semantically similar tasks from dev set to the test task and transfers their attribution scores to the test task. But semantic similarity may not capture the functional similarity and usefulness. Given only 15 developmental tasks are used for the attribution, makes it a bigger problem?
  3. The main table uses bare ReAct and ACE, CUGA, Gupta et al., as the baselines. Masking ablations compare against random and global score masking. Simple approaches like letting the llm to select the relevant skills or using embeddings to retrieve relevant skills help to understand if causal attribution is even necessary.
  4. The causal attribute estimates the marginal effect of each skill by averaging over the randomised samples. But the skills retained during the inference might form different combinations. A skill that appears harmful with random subsets may not be harmful when used with the rest of the library. The paper doesn't tell how well these marginal effects correspond to the actual effects of removing skills from the deployed library.
  5. The attribution is fixed to a fixed library and model. But methods like ACE produce libraries that can be continually updated. It's unclear how the attribution could be rerun without running the whole pipeline again.
  6. Adding templates to ACE improves the performance by 3.5% on GPT 5.1 (fig3). The other six models on the app world and t-bench doesn't have access to templates for the baselines. It's unclear how much of the gains are from the template alone and ASSAY itself.
Questions for authors
  1. How does the method behave when the test task is functionally different from the development tasks? Not just the APIs from apps are different.
  2. Why are the gains not that significant in T-bench scenario? Can you elaborate more on T-bench results? Especially GPT-* models. 4.1 nd 4o has gains but 5.2 has no gain.
  3. How much of the reported gains (except for GPT5.1 on app world test_normal) are from the template alone?
3 comments
AuthorsOpenReview ↗

To Reviewer x1BR: Additional Selector Baselines, Template Controls, and Cost Analysis

Thank you for the questions about simpler selection methods, deployment validity, templates, and computational cost! We added targeted controls for each.

1. Competitive task-conditioned selectors.

With the same restructured library, protected entries, execution settings, and per-task removal counts:

SelectorSuccessful tasks
Semantic similarity120/168
GPT-5.1-based selection124/168
ASSAY130/168

The LLM selector considers applicability, redundancy, conflicts, and unnecessary execution steps. Independently dev-calibrating its retention budget yields 125/168. ASSAY solves six more tasks than matched-count LLM selection: +3.57 points [95% scenario-clustered CI: −0.60, +7.74].

2. Effects under deployment backgrounds.

On the same 40 eligible tasks, we performed single-skill inclusion/removal interventions with all other deployed instructions fixed. Removing strongly negative, near-threshold, and nonnegative skills yields net benefits of +25, 0, and −15 points. The within-task negative-versus-nonnegative contrast is +40 points [95% scenario-clustered CI: +12.20, +69.05]. This directly tests whether the attribution signal informs removal decisions under deployment conditions.

3. Additional template-matched controls.

Each baseline receives the same upstream learned library and operational templates:

SettingMatched baselineASSAY
DeepSeek-V3.2 / AppWorld134/168140/168
Gemini 2.5 Pro / τ-bench79/11585/115
GPT-5.1 / τ-bench71/11572/115

These extend the original GPT-5.1/AppWorld matched-template comparison. The additional estimates are positive but individually uncertain. GPT-5.1’s small τ-bench difference confirms that the benefit depends on the model–environment combination; these experiments establish the pattern rather than its cause.

4. Complete cost accounting.

At reference rates of USD 2 per million input tokens and USD 8 per million output tokens, attribution, rewriting, and gate evaluation add USD 72.50 in offline model costs beyond source-library construction. Deployment over 168 tasks costs USD 35.00 versus USD 32.32 for ACE + templates, while completion increases from 119 to 130 tasks.

The offline overhead amortizes to USD 0.43 per task over 168 deployments, or USD 0.073 over 1,000; local compute is excluded. Given the same repaired library, LLM selection costs USD 39.61 online. Both selectors inherit the repair cost.

5. Functional transfer and evolving libraries.

Development-set resampling preserves the selection advantage (Appendix I.2), and Appendix I.7 characterizes semantic coverage. Transfer to functionally dissimilar tasks remains untested. The current method evaluates fixed libraries; changing skills requires refreshing attribution evidence rather than assuming previous estimates remain valid.

The added controls isolate benefits beyond templates and relevance ranking, validate deployment interventions, and make the cost–performance trade-off explicit. Thank you again!

Reviewer_x1BROpenReview ↗

Thank you for adding additional details

Thank you for adding additional analysis on deployment interventions, template controls and cost accounting.

I still have concerns about:

  1. ASSAY’s generalisation to functionally dissimilar tasks remains untested. And evolving skill libraries would require refreshing the attribution estimates. The authors acknowledge both limitations in point 5.

  2. The difference between GPT 5.1 based task selection (124/168) and ASSAY (130/168) is small. This still doesn't answer about how much causal attribution contributes beyond simpler selection methods.

  3. Authors acknowledge the varying improvements on t-bench but do not explain why some models benefit significantly less than others.

The two main limitations I see are functional transfer and handling evolving libraries. Apart from these, the rebuttal has addressed several of my strongest concerns. Given the additional experiments and clarifications, I am inclined to increase my score.

AuthorsOpenReview ↗

Thank you for your reassessment and further clarifications

Thank you very much for your thoughtful reassessment and increased score! We sincerely appreciate your constructive feedback.

We would also like to offer a few clarifications in response to your remaining concerns:

  • Transfer and updates: Our results support held-out-task performance. Library changes require refreshed attribution; functionally dissimilar transfer requires separate evaluation.
  • Attribution’s contribution: With the library and removal counts fixed, attribution-based selection completes six additional tasks (+3.57 percentage points). Deployment interventions also validate the scores’ predictive value for removal effects.
  • Model variation: Executors may differ in how skills help or interfere with their decisions. We will distinguish this interpretation from experimentally established causes.

We will clarify these points in the revision. Thank you again for your careful consideration and encouragement!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.