acceptodds
Under review as a conference paper at ICLR 2027

When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Use

Abstract

Agent skills are expected to improve task performance. Yet we find that they often provide no benefit, and can even hurt performance while incurring additional token costs. Can we predict whether a skill will help before the agent acts? We introduce SkillDelta, a framework for predicting task-conditional skill gains from paired executions of the same agent with and without the skill. A local predictor transfers these historical gains to new tasks without retraining the agent. Under explicit transfer assumptions, our analysis links support coverage, representation mismatch, and execution noise to prediction error and decision regret. Across five benchmarks and three target agents, paired history improves observed-gain ranking over skill-assisted outcomes alone in 12 of 15 settings. At matched expected skill-use rates, SkillDelta improves success over random activation in all 15 settings, with an average absolute gain of 4.3%. Most of this advantage comes from allocation across task groups. Evidence for additional within-group selection value is strongest on ToolQA and weaker elsewhere. Code and reproduction materials are available at [this anonymous repository](https://anonymous.4open.science/r/SkillDelta-8DC4/).

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent skills, task-conditional gain prediction, selective prediction, skill gating

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gTKuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkillDelta, a method that uses past skill execution outcomes on similar tasks to estimate whether a skill will benefit the current task, thereby improving the efficiency of skill use. The experiments demonstrate that SkillDelta can selectively invoke skills and reduce some of the overhead associated with extensive skill use.

Critical strengths
  • The paper addresses an important problem: skill use can substantially affect agent performance. Useful skills can improve performance, whereas ineffective skills may introduce unnecessary overhead.
  • The proposed method is simple and easy to implement.
Critical weaknesses
  • Practicality of the problem setting. My primary concern is that the proposed method relies on a historical database to estimate skill utility. Constructing this database requires evaluating candidate skills across a substantial number of tasks and measuring the performance difference between using and not using each skill. These paired evaluations can be costly. In typical agent usage, such a database is unlikely to emerge naturally without a deliberate data collection effort. The practicality and cost of obtaining this historical information therefore require further justification.
  • Insufficient baseline discussions or comparisons. Although the paper provides fairly comprehensive hyperparameter sensitivity analyses, the comparisons with existing methods are less convincing. The main baseline in the paper is Self-judge, while comparisons with relevant work on skill or memory retrieval, such as [1–3], are missing. Although these studies address somewhat different settings, their methods and objectives are sufficiently related to warrant discussion and, where feasible, empirical comparison. In particular, when no historical database is available, directly assessing skill or memory utility on the target task, as these methods do, seems a more natural approach.
  • Unclear overall performance advantage. The proposed method reduces token consumption to some extent, but it also achieves a lower final success rate than other baselines that use skills. These results appear to represent different operating points on a cost–performance trade-off, and do not establish that one approach is unequivocally better. Moreover, when a historical database is not already available, constructing it also incurs substantial token costs. The overall savings are therefore difficult to assess without accounting for this initial investment. In addition, the comparisons under the same setting in Table 4 show only limited advantages for the proposed method.

[1] Live-Evo: Online Evolution of Agentic Memory from Continuous Feedback

[2] MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning

[3] MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

Questions for authors
  • Under what practical circumstances is the assumed historical database readily available, making the proposed setting realistic?
  • If no such database exists, what advantages does SkillDelta offer over methods that assess skill utility directly on the target task?
  • What is the cost of constructing the historical database from scratch, and how many subsequent tasks are needed to offset that cost through reduced token consumption?
4 comments
AuthorsOpenReview ↗

Reusable Pre-Execution Skill Selection via Shared History

Thank you for highlighting the practical source of historical evidence, related baselines, and the cost–performance interpretation. SkillDelta addresses one specific decision: before execution, determine whether a supplied skill is likely to improve the current task. It is intended as a reusable layer for recurring use of public skills, not as a cold-start solution for every one-off task.

1. When is historical evidence practical? For a public skill, paired history need not be rebuilt privately by every user. A shared pool can collect runs with and without the skill, record agent, skill-version, and task metadata, and reuse compatible evidence across later tasks and users. This amortizes the initial cost and accumulates evidence about when a skill helps, has no effect, or causes harm. For a one-off task with no history, direct assessment or a default policy remains a reasonable cold-start choice.

2. What does SkillDelta add without a history pool? Direct target-task assessment is a natural fallback when no evidence exists. Once similar paired outcomes are available, SkillDelta reuses them before execution instead of adding a model call for every task. Self-judge performs target-task Use/Skip assessment, while Live-Evo, MemSifter, and MemRL focus on memory evolution, retrieval, or runtime feedback. They are complementary, but do not directly estimate the paired gain of the supplied skill. We will clarify this scope and relation in the revision.

3. Construction cost and overall value. Under conservative single-deployer accounting, token-only break-even requires 1,421–278,909 future tasks (median 13,776). This excludes the savings from shared construction and costs for embedding, retrieval, storage, and refresh. Once history is available, the default policy reduces target-task execution tokens by 20.8% relative to Always-on; at a matched skill-use rate, it improves success by 4.33% over random activation. Thus, SkillDelta exposes a configurable success–cost operating point rather than claiming universal dominance. Table 4 likewise shows that transfer quality depends on the representation and benchmark.

Summary. SkillDelta formalizes a missing capability in skill-enabled agents: deciding whether to use a supplied skill before execution. Paired history makes this decision observable and reusable, while revealing when the signal is useful and when its construction cost is justified.

Thank you for the rebuttal. I agree that maintaining a server-side history pool shared across users is an effective way to reduce amortized cost. Could the authors clarify whether the server builds this pool proactively or collects histories passively from user requests?

If the server builds the pool proactively, how does it ensure broad coverage of skill–task pairs, and what is the approximate cost of building the pool? If the pool is built passively, how do the authors ensure that it contains both requests where a skill was used and requests where it was not?

AuthorsOpenReview ↗

Reply to Reviewer gTKu

Thank you for the follow-up. Our intended deployment uses a hybrid construction strategy. The history pool grows passively from user-contributed executions, while targeted paired audits bootstrap new skills and fill missing coverage.

1. Proactive coverage

When proactive collection is needed, the server maintains a coverage matrix indexed by agent, skill version, and task family. It samples tasks from under-covered cells and evaluates both Use-skill and Skip-skill conditions. The cited SkillTester work addresses the upstream problem of constructing broad and systematic evaluation suites for a skill. SkillDelta addresses the downstream decision problem: given a concrete target task and a supplied skill, estimate its incremental benefit and decide Use or Skip before execution.

2. Proactive cost and observed effect

In our DeepSeek-V4.1-Flash configuration, two paired tasks per skill family required approximately 1.2×1041.2\times10^4 tokens, or $0.0018 per skill. This operating point improved absolute success by 4.89% over matched-rate random activation and reduced execution-token use by 6.16% relative to Always-on.

In the support-size analysis, five to ten paired tasks per skill produced approximately 11.6%–13.0% higher success and 9.2%–9.8% lower token use on ToolQA, and 12.5%–15.5% higher success and 31.2%–35.2% lower token use on MedCalc-Bench. The corresponding construction cost is approximately 30,000–61,000 tokens, or USD 0.004–0.009 per skill.

3. Passive collection and Use/Skip coverage

Passive Use-skill logs alone do not provide paired gain evidence. We therefore treat a record as paired evidence only when comparable Use-skill and Skip-skill outcomes are available. When one condition is missing, the pool can request an opt-in paired replay or schedule a small controlled audit for that skill and task family. Use-only and Skip-only records remain available for monitoring, while paired records provide the gain labels used by SkillDelta.

This design lets the history pool grow from real usage, fills coverage gaps with targeted paired evaluations, and supports reusable pre-execution Use/Skip decisions.

Thank you for the reply. I have no further questions. After considering the rebuttal, I would rate the paper a 2.5 on the current 1–4 scale. Proactively building the history pool does incur some cost, but it offers a relatively simple approach to skill evaluation. I am therefore raising my score to 3. The area chair may use either 2.5 or 3 when weighing my assessment.

Reviewer_QhjsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SkillDelta, which predicts whether a supplied skill will help an agent using paired skill-on/off outcomes from similar historical tasks. Across five benchmarks and three agents, it improves success over matched-rate random activation and reduces execution costs relative to always using skills. Most gains arise from allocation across task groups.

Critical strengths
  • Clearly distinguishes incremental skill benefit from skill-assisted success.
  • Provides a simple, interpretable method without agent retraining.
  • Connects prediction error to coverage, representation mismatch, and execution noise.
  • Evaluates multiple benchmarks and agents, with useful controls and transparent analysis of limitations.
Critical weaknesses
  • Limited evidence for reliable task-level selection. Most gains come from allocation across task groups, while within-group benefits are largely confined to ToolQA. The significantly negative within-group result on Qwen–BigCodeBench further raises concerns about the robustness of fine-grained gain prediction.

  • The success–cost trade-off requires stronger justification. The reported 20.8% execution-token savings come with a 0.92-percentage-point average success reduction, with lower success than Always-on in 12 of 15 settings. This is not a uniformly beneficial improvement; its practical value depends on application-specific costs of failure. Comparisons at matched success levels would be more informative.

  • Deployment costs are incompletely accounted for. Reported savings exclude history collection, embedding, and retrieval. An end-to-end cost analysis, including the reuse needed to amortize the paired-history bank, is necessary to establish practical efficiency.

  • Generalization remains insufficiently established. Although the evaluation covers five benchmarks and three agents, the main results rely on within-inventory leave-one-task-out evaluation. The prospective studies use only one agent and do not test unseen skills, leaving robustness to new task distributions and changes in agents or skills unclear.

Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

From Historical Evidence to Prospective Skill Selection

Thank you for highlighting the limits of fine-grained selection, cost accounting, and generalization. SkillDelta addresses a specific decision problem: whether a supplied skill is worth using before execution, based on its incremental gain rather than skill-assisted success.

1. Selection granularity. At matched skill-use rates, SkillDelta improves success over random activation by 4.33%. The decomposition attributes 3.72% to allocation across task groups and 0.61% to within-group refinement. Fine-grained selection is conditional: it is positive in only 5 of 15 panels, reliable on ToolQA, and negative on Qwen–BigCodeBench, where sparse support makes local prediction unreliable. We will present group allocation as the robust result and local refinement as an applicability boundary.

2. Success–cost trade-off. SkillDelta does not dominate Always-on. The default policy uses 20.8% fewer execution tokens with 0.92% lower absolute success, and has lower success than Always-on in 12 of 15 settings. The threshold nevertheless provides a controllable operating point. In the existing retrospective sweeps, all 15 panels contain a point with positive token saving at or above the Always-on success level; the maximum saving at that level averages 9.79% (median 4.54%). These are retrospective diagnostics, not independent deployment guarantees. Our better-supported claim is that, at a fixed skill-use rate, SkillDelta selects better tasks than random activation.

3. Deployment cost. Reported savings exclude history collection, embedding, retrieval, and refresh. For public skills, a shared, versioned history pool could collect paired executions from multiple users and amortize this cost across later users and compatible tasks. This shared-pool model is a deployment proposal, not an evaluated result.

4. Prospective evaluation. We added a prospective support-to-audit test using DeepSeek-V4.1-Flash, a model version outside the main evaluation. The gate uses 101 valid support pairs and is evaluated on 197 strictly held-out audit pairs. At a 15.7% use rate, it achieves 91.37% success versus 86.48% for matched-rate random activation (+4.89%, family-cluster bootstrap 95% interval [0.56%, 10.34%]). Always-on reaches 92.89%, while SkillDelta uses 6.16% fewer tokens and has 1.52% lower absolute success. Invalid or empty outputs accounted for 11 support trajectories (5 no-skill, 6 skill) and 27 audit trajectories (18 no-skill, 9 skill). Counting them as failures leaves a positive 5.83% advantage over matched-rate random activation, although the operating point shifts to 3.86% token saving and 2.73% lower success than Always-on. This test covers one agent; unseen skills and new task distributions remain untested.

Overall, the evidence supports SkillDelta as a pre-execution selection signal whose clearest benefit is gain-based allocation with a controllable cost–success trade-off.

Reviewer_FkaPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when a supplied skill improves a fixed agent's task performance. The authors propose SkillDelta, which predicts this benefit by averaging gains from similar historical tasks executed with and without the skill, and uses the prediction to decide whether to enable it. The paper provides an error analysis and evaluates prediction quality, selective skill use, and the effects of historical evidence across several benchmarks and agents.

Critical strengths
  1. The paper asks a useful and well-motivated question: whether a skill actually improves an agent's performance, rather than whether the agent can succeed with it. The paired-execution setup directly addresses this distinction and provides a sensible basis for deciding when to use skills.

  2. The evaluation spans several benchmarks and agents and goes beyond reporting aggregate performance. The analyses of task-group allocation, local selection, and historical evidence provide useful insights into where the approach works and where it remains limited. The resulting picture of selective skill use is informative and appropriately qualified.

Critical weaknesses
  1. The theoretical analysis provides limited guidance on when skill-gain prediction is reliable. Its usefulness depends on how well the task representation preserves differences in skill benefit, but the experiments do not directly assess the size of this mismatch. The analysis identifies sources of error without establishing which task settings support reliable local transfer.

  2. The incremental contribution of local task-level prediction remains unclear outside ToolQA. Most of the reported selection advantage comes from allocating skill use across task groups, and the existing group-level controls operate at different success-cost trade-offs. The evidence therefore does not clearly isolate the benefit of local prediction over a simpler group-level policy across domains.

  3. The self-judgment baseline could be stronger. SkillDelta outputs a continuous gain score, whereas the self-judge produces only a binary use/skip decision and largely defaults to using the skill. The AUROC comparison therefore reflects differences in score granularity as well as predictive ability. A self-judge that outputs confidence or estimated gain would provide a more informative comparison.

Questions for authors
  1. Can the empirical analysis relate prediction reliability more directly to representation distance and mismatch, helping distinguish when local transfer works well from when it does not?

  2. Beyond ToolQA, does local prediction consistently improve on a simple group-level policy using the same historical evidence at comparable skill-use rates or execution budgets?

  3. Would a confidence-based or explicit gain-estimation self-judge provide a more competitive baseline?

1 comment
AuthorsOpenReview ↗

Empirical Evidence for Gain-Based Skill Selection

Thank you for the careful and constructive review. The central contribution of SkillDelta is to formulate skill use as an ex-ante decision problem: using paired skill-on/off outcomes, it estimates a skill’s incremental gain and decides whether to Use or Skip it before execution, rather than treating skill-assisted success as evidence that the skill is useful. The analyses below clarify when this signal is reliable, show its value beyond family-level policies and Self-Judge, and identify representation quality and within-group gain variation as the conditions that determine its benefit.

  1. Representation distance and prediction reliability. In an exploratory analysis of frozen k=6 predictions on 120 randomly sampled tasks per benchmark across three models, local transfer worked best when neighbors were close and their observed gains agreed. In this group, local prediction reduced gain MAE relative to the target-excluded family mean by 0.075 on ToolQA and 0.043 on MedCalc-Bench. High neighbor-gain disagreement was associated with larger prediction errors even among nearby tasks. The disagreement is a proxy for representation mismatch and also reflects execution noise.

  2. Value of task-level selection. We compared the paper's fixed k=6 SkillDelta policy with a family-mean policy across all 15 model-benchmark settings. Both used the same historical paired outcomes, with each target excluded from its own support, and matched the number of skill activations. Outside ToolQA, SkillDelta improved success by 1.32 percentage points on average, with paired 95% task-bootstrap intervals above zero in 6 of 12 settings.

  3. Confidence-based Self-Judge baseline. We tested a continuous Self-Judge that asks DeepSeek-V4.1-Flash to estimate the probability of completing the task correctly with the supplied skill. We use this probability directly to rank tasks and evaluate the ranking against the archived paired execution outcomes. We sampled 60 tasks from each of the five DeepSeek benchmarks. Of the 300 sampled tasks, 277 had valid scores for all methods in the AUROC comparison. The benchmark-macro AUROC was 0.496 for binary Self-Judge, 0.480 for continuous Self-Judge, and 0.690 for SkillDelta. The continuous judge was more competitive on ToolQA, where it raised AUROC from 0.457 to 0.618, while SkillDelta achieved 0.749 on the same tasks. This shows that the additional score resolution can help on individual benchmarks, and SkillDelta retained the higher AUROC on each benchmark in our sample. Both Self-Judges tended toward Always-on. The binary judge enabled skills on over 91% of tasks in 14 of 15 settings, and a 0.5 threshold on the continuous scores would activate skills on 286 of 290 tasks. On the 290 tasks with valid continuous scores, total token use relative to Always-on increased by 53.5% for binary Self-Judge and 72.5% for continuous Self-Judge, and decreased by 13.4% for SkillDelta, averaging percentage changes equally across benchmarks. Both Self-Judge totals include the routing call and task execution. SkillDelta achieved higher task success on three of the five benchmarks and used fewer total tokens on all five.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.