- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SkillDelta, a method that uses past skill execution outcomes on similar tasks to estimate whether a skill will benefit the current task, thereby improving the efficiency of skill use. The experiments demonstrate that SkillDelta can selectively invoke skills and reduce some of the overhead associated with extensive skill use.
Critical strengths
- The paper addresses an important problem: skill use can substantially affect agent performance. Useful skills can improve performance, whereas ineffective skills may introduce unnecessary overhead.
- The proposed method is simple and easy to implement.
Critical weaknesses
- Practicality of the problem setting. My primary concern is that the proposed method relies on a historical database to estimate skill utility. Constructing this database requires evaluating candidate skills across a substantial number of tasks and measuring the performance difference between using and not using each skill. These paired evaluations can be costly. In typical agent usage, such a database is unlikely to emerge naturally without a deliberate data collection effort. The practicality and cost of obtaining this historical information therefore require further justification.
- Insufficient baseline discussions or comparisons. Although the paper provides fairly comprehensive hyperparameter sensitivity analyses, the comparisons with existing methods are less convincing. The main baseline in the paper is Self-judge, while comparisons with relevant work on skill or memory retrieval, such as [1–3], are missing. Although these studies address somewhat different settings, their methods and objectives are sufficiently related to warrant discussion and, where feasible, empirical comparison. In particular, when no historical database is available, directly assessing skill or memory utility on the target task, as these methods do, seems a more natural approach.
- Unclear overall performance advantage. The proposed method reduces token consumption to some extent, but it also achieves a lower final success rate than other baselines that use skills. These results appear to represent different operating points on a cost–performance trade-off, and do not establish that one approach is unequivocally better. Moreover, when a historical database is not already available, constructing it also incurs substantial token costs. The overall savings are therefore difficult to assess without accounting for this initial investment. In addition, the comparisons under the same setting in Table 4 show only limited advantages for the proposed method.
[1] Live-Evo: Online Evolution of Agentic Memory from Continuous Feedback
[2] MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning
[3] MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory
Questions for authors
- Under what practical circumstances is the assumed historical database readily available, making the proposed setting realistic?
- If no such database exists, what advantages does SkillDelta offer over methods that assess skill utility directly on the target task?
- What is the cost of constructing the historical database from scratch, and how many subsequent tasks are needed to offset that cost through reduced token consumption?
4 comments
Thank you for the rebuttal. I agree that maintaining a server-side history pool shared across users is an effective way to reduce amortized cost. Could the authors clarify whether the server builds this pool proactively or collects histories passively from user requests?
If the server builds the pool proactively, how does it ensure broad coverage of skill–task pairs, and what is the approximate cost of building the pool? If the pool is built passively, how do the authors ensure that it contains both requests where a skill was used and requests where it was not?
Reply to Reviewer gTKu
Thank you for the follow-up. Our intended deployment uses a hybrid construction strategy. The history pool grows passively from user-contributed executions, while targeted paired audits bootstrap new skills and fill missing coverage.
1. Proactive coverage
When proactive collection is needed, the server maintains a coverage matrix indexed by agent, skill version, and task family. It samples tasks from under-covered cells and evaluates both Use-skill and Skip-skill conditions. The cited SkillTester work addresses the upstream problem of constructing broad and systematic evaluation suites for a skill. SkillDelta addresses the downstream decision problem: given a concrete target task and a supplied skill, estimate its incremental benefit and decide Use or Skip before execution.
2. Proactive cost and observed effect
In our DeepSeek-V4.1-Flash configuration, two paired tasks per skill family required approximately tokens, or $0.0018 per skill. This operating point improved absolute success by 4.89% over matched-rate random activation and reduced execution-token use by 6.16% relative to Always-on.
In the support-size analysis, five to ten paired tasks per skill produced approximately 11.6%–13.0% higher success and 9.2%–9.8% lower token use on ToolQA, and 12.5%–15.5% higher success and 31.2%–35.2% lower token use on MedCalc-Bench. The corresponding construction cost is approximately 30,000–61,000 tokens, or USD 0.004–0.009 per skill.
3. Passive collection and Use/Skip coverage
Passive Use-skill logs alone do not provide paired gain evidence. We therefore treat a record as paired evidence only when comparable Use-skill and Skip-skill outcomes are available. When one condition is missing, the pool can request an opt-in paired replay or schedule a small controlled audit for that skill and task family. Use-only and Skip-only records remain available for monitoring, while paired records provide the gain labels used by SkillDelta.
This design lets the history pool grow from real usage, fills coverage gaps with targeted paired evaluations, and supports reusable pre-execution Use/Skip decisions.
Thank you for the reply. I have no further questions. After considering the rebuttal, I would rate the paper a 2.5 on the current 1–4 scale. Proactively building the history pool does incur some cost, but it offers a relatively simple approach to skill evaluation. I am therefore raising my score to 3. The area chair may use either 2.5 or 3 when weighing my assessment.
Reusable Pre-Execution Skill Selection via Shared History
Thank you for highlighting the practical source of historical evidence, related baselines, and the cost–performance interpretation. SkillDelta addresses one specific decision: before execution, determine whether a supplied skill is likely to improve the current task. It is intended as a reusable layer for recurring use of public skills, not as a cold-start solution for every one-off task.
1. When is historical evidence practical? For a public skill, paired history need not be rebuilt privately by every user. A shared pool can collect runs with and without the skill, record agent, skill-version, and task metadata, and reuse compatible evidence across later tasks and users. This amortizes the initial cost and accumulates evidence about when a skill helps, has no effect, or causes harm. For a one-off task with no history, direct assessment or a default policy remains a reasonable cold-start choice.
2. What does SkillDelta add without a history pool? Direct target-task assessment is a natural fallback when no evidence exists. Once similar paired outcomes are available, SkillDelta reuses them before execution instead of adding a model call for every task. Self-judge performs target-task Use/Skip assessment, while Live-Evo, MemSifter, and MemRL focus on memory evolution, retrieval, or runtime feedback. They are complementary, but do not directly estimate the paired gain of the supplied skill. We will clarify this scope and relation in the revision.
3. Construction cost and overall value. Under conservative single-deployer accounting, token-only break-even requires 1,421–278,909 future tasks (median 13,776). This excludes the savings from shared construction and costs for embedding, retrieval, storage, and refresh. Once history is available, the default policy reduces target-task execution tokens by 20.8% relative to Always-on; at a matched skill-use rate, it improves success by 4.33% over random activation. Thus, SkillDelta exposes a configurable success–cost operating point rather than claiming universal dominance. Table 4 likewise shows that transfer quality depends on the representation and benchmark.
Summary. SkillDelta formalizes a missing capability in skill-enabled agents: deciding whether to use a supplied skill before execution. Paired history makes this decision observable and reusable, while revealing when the signal is useful and when its construction cost is justified.