acceptodds
Under review as a conference paper at ICLR 2027

Skill: Skill Evolution as Policy Optimization

Abstract

Skills are increasingly generated and refined automatically, but the dominant sequential-refinement paradigm—execute the current skill, reflect on its rollout, and rewrite the whole skill—often degrades over rounds. We trace this failure to three missing ingredients from reinforcement learning: a baseline for the return, credit assignment to individual design choices, and a replay buffer that preserves evaluated skills and trajectories. These mechanisms are not optional heuristics: without a return baseline, dimension-level credit assignment, and a replay buffer, the effect of any design choice is unidentifiable from a single rollout. We therefore reframe skill evolution as optimizing the policy that writes skills, not the skill document itself. The actions are discrete design choices within a skill, and the policy parameter is a growing context of evaluated skills, trajectories, returns, and conclusions. We propose Skill, which performs controlled one-dimension-at-a-time comparisons: alternatives in a group differ only in the chosen dimension, so the rest of the group serves as a baseline; the return gap is credited to that dimension; and every evaluated skill remains available as a reusable control. The final skill is synthesized from choices that won such comparisons, while unsupported or null dimensions are removed. On SkillLearnBench, across three LLMs, Skill achieves 52.0% average accuracy, 9.4 points above the no-skill agent and 11.1 points above the strongest skill-generation baseline, and improves held-out transfer and judged skill completeness/determinism. Ablations confirm the contribution of controlled comparison, control reuse, and synthesis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent skills, LLM agents, skill evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wW65OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that sequential skill refinement degrades over iterations. Reasons are missing baselines and credit assignment issues. It frames skill evolution as optimizing the context of an LLM skill generator - effectively RL of a policy. In each round the generator picks an unresolved design dimension, builds skills that differ only on that dimension, and runs each one once on a seed instance. It checks and logs the score gap in that dimension and conclusion. Finally it synthesises only supported choices and drops dimensions that had no impact. On SkillLearnBench with three LLMs, the method beats a set of baselines across instances. The method further scores higher on LLM-judge skill completeness and determinism.

Critical strengths

The core mechanism is simple and adds limited computational cost. It is straightforward to vary one dimension per step and reuse old skills as controls for credit assignment in larger rewrites. It is general in that it should work with any frozen agent. The additional held-out evaluation and excluding the seed instance has only little impact and makes it likely that gains are not based on overfitting.

Critical weaknesses

The comparison isn’t exactly fair on compute or information IIUC. There seems to be no budget-matched (rollouts and verified feedback) search baseline (eg best-of-N), so the paper can’t separate the exact method from more samples and scoring.

For stronger reliability, the paper needs to move away from single stochastic rollouts on single instances: multiple seeds and confidence intervals. The ablations would also benefit from expanding beyond 5 tasks with one model and adding task accuracy (instead of using the judge). The judge itself has some issues with human-authored skills scoring lower than Skill2 on the judge (while having better accuracy).

Questions for authors

How does Skill2 compare with a best-of-N baseline (or evoskill) with the same rollout budget and verifier access? How much variance is there across seeds? Please report CIs, especially where the gain is small. Can you report task accuracy not just judge quality across ablations? Discuss why not if not possible. Does the method keep its advantage on harder/more stochastic environments? How does increased rollout noise affect this? Existing work: please contextualise with prompt optimization (aPSF) for single factor changes and credit assignment and context as policy parameter in training-free GRPO. Minimal: discuss relation, best: rebuttal response on if these are suited as baselines. If yes: please add.

5 comments
AuthorsOpenReview ↗

Response to Reviewer wW65 (1)

We appreciate your feedback. We would like to clarify the points you raised as follows:

Response to W1 & Q1

How does Skill$^2$ compare with a best-of-N baseline (or evoskill) with the same rollout budget and verifier access?

Our base baselines follow SkillLearnBench [5], which shows that Self-Feedback degrades as the number of refinement rounds grows (accuracy rises after the first revision and then drops sharply; [5], Sec. 5.4), so we adopt its original setting (K=2; [5], Sec. 4.1). For fairness, we evaluate four further baselines (three iterative skill-evolution methods and TF-GRPO), SkillOpt [1], SkillGrad [2], EvoSkill [3] and TF-GRPO [4], with the same rollout budget as Skill$^2$ (B=10 on the seed instance).

Accuracy on all instances / held-out instances (%) with Gemini-3.8-Flash, GPT-5.6-Luna and DeepSeek-V4-Flash, and Exec, the executability score from SkillLearnBench's LLM judge, computed as the average of its four original sub-metrics (completeness, determinism, consistency, usability; rescaled to 0–100; [5], Sec. 3.2.1):

MethodGemini (All/held-out/Exec)Luna (All/held-out/Exec)DeepSeek (All/held-out/Exec)
No Skill39.8 / 41.0 / –56.0 / 53.0 / –29.7 / 28.5 / –
One-Shot [5]55.7 / 58.5 / 50.743.2 / 44.5 / 59.130.5 / 27.5 / 54.0
Self-Feedback [5]59.5 / 57.5 / 49.243.5 / 42.0 / 48.730.8 / 33.0 / 47.9
SkillOpt [1]47.0 / 46.5 / 50.746.5 / 42.5 / 69.245.8 / 46.2 / 62.7
SkillGrad [2]46.0 / 45.0 / 49.649.2 / 46.8 / 56.535.0 / 36.0 / 52.8
EvoSkill [3]42.8 / 43.5 / 47.650.8 / 49.2 / 64.640.2 / 46.0 / 64.3
TF-GRPO [4]36.0 / 36.0 / 37.852.7 / 54.0 / 37.843.0 / 39.8 / 40.2
Skill$^2$75.7 / 74.8 / 53.060.5 / 62.5 / 61.547.2 / 46.5 / 74.5
Human-Authored (ref.)81.3 / 80.0 / 54.179.2 / 80.0 / 54.158.5 / 56.8 / 54.1

[1] Y. Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904, 2026.
[2] H. Wang et al. SkillGrad: Optimizing Agent Skills Like Gradient Descent. arXiv:2605.27760, 2026.
[3] S. Alzubi et al. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766, 2026.
[4] Y. Cai et al. Training-Free Group Relative Policy Optimization. arXiv:2510.08191, 2025.
[5] S. Zhong et al. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. arXiv:2604.20087, 2026.

AuthorsOpenReview ↗

Response to Reviewer wW65 (2)

Response to W2 & Q2

How much variance is there across seeds? Please report CIs, especially where the gain is small.

Seed variance. Our single-trial protocol follows SkillLearnBench [1]. Its authors re-ran one representative instance per task three times for each method and generation LLM; accuracy varied with a standard deviation of 13–17% for generated skills, and the ordering of method means matched the single-trial results. This variation comes from individual executions ([1], Appendix E). Our CIs below are computed with a two-stage cluster bootstrap that resamples both tasks and instances within each task, so they already account for this instance-level variance. Repeating the full search under multiple seeds is costly (B=10 containerized agent rollouts per task and backbone), but the three backbones act as independent replicates with consistent gains. Appendix B reports these CIs, pooled and per backbone, together with the pooled paired Wilcoxon tests.

Confidence intervals. Each (method, backbone, task) yields one optimized skill, evaluated on multiple instances of the task. We report 95% CIs from a two-stage cluster bootstrap (resampling tasks, then instances within each task; 10,000 draws) and paired Wilcoxon signed-rank tests over the 20 tasks.

Pooled over the three backbones, Skill$^2$ significantly outperforms every baseline:

Skill$^2$ vs.Δ (pts)95% CIWilcoxon p
No Skill+19.3[8.1, 29.8]0.003
One-Shot+18.0[5.6, 30.4]0.008
Self-Feedback+16.5[5.2, 27.6]0.009
SkillOpt+14.7[6.1, 23.3]0.003
SkillGrad+17.7[8.4, 26.9]0.001
EvoSkill+16.5[4.7, 28.3]0.009
TF-GRPO+17.2[6.6, 27.9]0.006

Against the strongest baseline chosen separately for each backbone, the pooled gain is +8.5 [-0.1, 16.8] (task-level Wilcoxon p=0.031). Per backbone the gain is positive but the CIs are wide: Gemini-3.8-Flash +16.2 [-2.8, 35.5], GPT-5.6-Luna +7.8 [-12.0, 26.0], DeepSeek-V4-Flash +1.4 [-15.8, 19.7].

[1] S. Zhong et al. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. arXiv:2604.20087, 2026.

Response to W3 & Q3

Can you report task accuracy not just judge quality across ablations? Discuss why not if not possible.

We now report task accuracy for all ablations on all 20 tasks with GPT-5.6-Luna, with the same rollout budget as full Skill$^2$:

VariantAll (%)Held-out (%)
Skill$^2$ (full)60.562.5
w/o Controlled Comparison53.5 (−7.0)54.3 (−8.2)
w/o Control Reuse51.2 (−9.3)50.3 (−12.2)
w/o Synthesis52.8 (−7.7)50.0 (−12.5)

Removing any component lowers task accuracy by 7–10 points on all instances and by 8–13 points on held-out instances, so each component contributes to actual task performance.

AuthorsOpenReview ↗

Response to Reviewer wW65 (3)

Response to Q4

Does the method keep its advantage on harder/more stochastic environments? How does increased rollout noise affect this?

Yes. To verify this, we split the 20 SkillLearnBench [1] tasks by difficulty and take the 10 hardest ones, i.e., those with the lowest No-Skill accuracy averaged over the three backbones (all ≤ 38.9%). Accuracy on these hard tasks:

MethodGeminiLunaDeepSeekAvg
No Skill7.324.715.315.8
One-Shot27.719.024.323.7
Self-Feedback30.017.021.022.7
SkillOpt10.724.326.720.6
SkillGrad14.024.722.020.2
EvoSkill11.733.020.721.8
TF-GRPO11.033.020.721.6
Skill$^2$57.337.034.042.8

On hard tasks, Skill$^2$ leads the strongest baseline on every backbone (+27.3/+4.0/+7.3 on Gemini-3.8-Flash/GPT-5.6-Luna/DeepSeek-V4-Flash), and its average margin grows to +12.9 points (95% CI [0.3, 24.1]).

As for increased rollout noise, its main risk is that a single noisy rollout leads to a wrong design conclusion. Skill$^2$ does not treat such conclusions as final: later rounds carry earlier-decided values into new skills, so a conclusion drawn from one noisy run is tested again and can be revised if contradicted. See the following case study:

Case study (backbone: Gemini, task: python-scala-translation): in round 2, the companion_module_object entry point failed (0.0) and would have been discarded by a single-shot comparison. In round 6 the same entry point was adopted again on a newer baseline and passed (1.0). The final synthesized skill adopts it. Higher noise thus mainly costs extra rounds to correct, rather than permanently misleading the search.

[1] S. Zhong et al. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. arXiv:2604.20087, 2026.

Response to Q5

Existing work: please contextualise with prompt optimization (aPSF) for single factor changes and credit assignment and context as policy parameter in training-free GRPO. Minimal: discuss relation, best: rebuttal response on if these are suited as baselines. If yes: please add.

Training-Free GRPO [1] It is already included as a baseline with the same rollout budget (Gemini-3.8-Flash/GPT-5.6-Luna/DeepSeek-V4-Flash: 36.0/52.7/43.0 vs. Skill$^2$ 75.7/60.5/47.2).

aPSF [2] also uses single-factor edits with credit assignment, but it is designed for static prompts with a fixed validation set and deterministic execution, so it does not fit our single-instance, noisy agentic setting. Counterfactual credit assignment is only one part of Skill$^2$: with GPT-5.6-Luna on 20 tasks, removing controlled comparison, control reuse, or synthesis each lowers accuracy by 7–10 points according to our ablations.

[1] Y. Cai et al. Training-Free Group Relative Policy Optimization. arXiv:2510.08191, 2025.
[2] H. Liu et al. Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs. ACL 2026, arXiv:2604.06699.

Reviewer_N2R2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an autonomous skill evolution method for large language models. The method uses an LLM to generate multiple skills, evaluates these skills, and stores the results in a Replay Buffer to form historical memory. New skills are then generated based on the feedback, and optimized skills are obtained through iterative refinement.

Critical strengths

Major Strengths The paper achieves substantial performance improvements compared with recent work.

Minor Strengths The paper provides a comprehensive survey and detailed discussion of recent related work.

Critical weaknesses
  1. The paper claims to optimize the policy responsible for generating Skills. However, despite establishing an MDP formulation, reinforcement learning is not adopted, and the skill-generating model is never updated. This makes the claimed policy optimization unclear and weakens the methodological contribution of the paper.

  2. The experimental comparisons are insufficient, particularly the lack of comparisons with relevant methods such as SkillOpt and SkillGrad. This makes it difficult to establish the effectiveness and novelty of the proposed method.

[1] Yang, Yifan, et al. "SkillOpt: Executive Strategy for Self-Evolving Agent Skills." arXiv preprint arXiv:2605.23904 (2026). [2] Wang, Hanyu, et al. "SkillGrad: Optimizing Agent Skills Like Gradient Descent." arXiv preprint arXiv:2605.27760 (2026).

Questions for authors
  1. Why is reinforcement learning not adopted despite the MDP formulation, and why is rollout feedback instead used as prompts?

  2. Given that the skill-generating model is never updated, where does the claimed policy improvement actually come from?

  3. What is the specific role of using advantage information as a prompt for the LLM? Can the authors provide additional ablation studies to demonstrate its effectiveness?

  4. Can the authors provide additional experimental comparisons with SkillOpt and SkillGrad?

4 comments
AuthorsOpenReview ↗

Response to Reviewer N2R2 (1)

We appreciate your feedback. We would like to clarify the points you raised as follows:

Response to W1 & Q1

Why is reinforcement learning not adopted despite the MDP formulation, and why is rollout feedback instead used as prompts?

We appreciate the opportunity to clarify how our framework relates to reinforcement learning.

Skill$^2$ performs RL over the textual parameter HH. Skill$^2$ is a reinforcement learning method under the standard definition of RL as optimizing a policy through environmental trial and error to maximize expected return [1]. In our formulation, the policy is the skill generator Gθ,H\mathcal{G}_{\theta,H}, an LLM from which each new candidate skill is sampled:

Snew∼Gθ,HS_{\mathrm{new}} \sim \mathcal{G}_{\theta,H}

An LLM policy is governed by two sets of parameters: its underlying neural network weights θ\theta and its textual parameter HH. Instead of updating model weights via gradient ascent, Skill$^2$ optimizes the textual parameter HH under the objective:

H⋆=arg⁡max⁡HES∼Gθ,H[J(S)]H^{\star} = \arg\max_{H} \mathbb{E}_{S \sim \mathcal{G}_{\theta,H}} [ J(S) ]

Thus, Skill$^2$ performs RL: it samples actions from a policy, receives environmental returns, and updates the policy to increase expected return, with the update applied to the textual parameter HH rather than the weights θ\theta.

Why optimize the textual parameter instead of the weights.

  • Weight updates do not match the setting of skills. While weight-level reinforcement learning on θ\theta is traditionally reserved for the post-training stage, Skill$^2$ treats skills as modular artifacts that a frozen agent loads at inference time to improve performance on downstream tasks [2]. Consequently, optimizing the skill generation policy through weight-level RL is ill-suited for this setting. Furthermore, weight-level RL cannot be applied to closed models such as Gemini-3.8-Flash and GPT-5.6-Luna, which are accessible only through inference APIs that restrict access to their proprietary weights θ\theta [3].
  • Optimizing a policy through its textual parameter is established. Formulating policy optimization over the textual parameter rather than the weights has precedent in recent literature. Because an agent's behavior depends jointly on its weights θ\theta and its textual parameter HH, updating HH directly changes the induced policy. This mechanism underlies Reflexion, which steers agent behavior through linguistic feedback [4], GEPA, which evolves prompts through natural-language reflection and outperforms weight-level RL with GRPO [5], and Agentic Context Engineering (ACE), which improves agents by evolving their textual parameter [6].

[1] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction (2nd ed.). MIT Press, 2018.
[2] Anthropic. Equipping Agents for the Real World with Agent Skills. Anthropic Engineering Blog, 2025.
[3] A. Zhao et al. ExpeL: LLM Agents Are Experiential Learners. AAAI, 2024. arXiv:2308.10144.
[4] N. Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS, 2023. arXiv:2303.11366.
[5] L. A. Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR, 2026. arXiv:2507.19457.
[6] Q. Zhang et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. ICLR, 2026. arXiv:2510.04618.

AuthorsOpenReview ↗

Response to Reviewer N2R2 (2)

Response to Q2

Given that the skill-generating model is never updated, where does the claimed policy improvement actually come from?

The skill generator is a policy parameterized jointly by frozen weights θ\theta and the textual parameter HH, generating candidate skills according to Snew∼Gθ,HS_{\mathrm{new}} \sim \mathcal{G}_{\theta,H}. The policy improvement in Skill$^2$ comes from updating the textual parameter HH of the skill generation policy.

In Skill$^2$, we formulate skill evolution itself as an MDP, where the objective is to optimize the policy that authors skills. The textual parameter HH is updated across rounds. Instead of rewriting the skill document in a single unconstrained step, each round updates HH with the advantage of specific design alternatives. For example, HH may record which choice has worked better on a design dimension, together with the skills, trajectories and returns behind that conclusion. Because the generator conditions on HH when producing future skills, updating HH shifts the generation distribution toward high-advantage skill designs and away from previously observed failure modes, yielding policy improvement without modifying the underlying model weights.

Response to Q3

What is the specific role of using advantage information as a prompt for the LLM? Can the authors provide additional ablation studies to demonstrate its effectiveness?

In Skill$^2$, a skill is a sequence of actions, one design choice per dimension (Eq. 4). In reinforcement learning, a policy is improved by giving each action an advantage, i.e., how much better it does than a baseline, which separates the effect of the action from everything else that affects the return [1]. Skill$^2$ obtains this baseline through controlled comparison: the members of a group share every design choice except the one on dimension mtm_t, so the control skill serves as the baseline. For a member Si=Sctrl[mt←xi]S_i = S_{\text{ctrl}}[m_t \leftarrow x_i], the advantage is Ai=r(Si)−r(Sctrl)A_i = r(S_i) - r(S_{\text{ctrl}}), which credits the score difference to that single choice (Sec. 3.3).

Without the advantage, the generator sees only each skill's absolute score and its single trajectory, which mixes the effect of a design choice with task difficulty and execution noise, so it cannot judge how much that choice is worth. With it, each round writes the advantages of the compared choices into HH, so the next skill sampled from the updated policy, Snew∼Gθ,HS_{\text{new}} \sim \mathcal{G}_{\theta,H}, tends to keep high-advantage designs.

To isolate the role of the advantage, we ablate it. The w/o Controlled Comparison variant removes the advantage: candidates are proposed freely instead of differing in one dimension, so no control exists, and the generator sees each candidate's absolute score. Both versions use the same rollout budget on all 20 SkillLearnBench tasks with GPT-5.6-Luna.

Variant (GPT-5.6-Luna, 20 tasks)All (%)Held-out (%)
Skill$^2$ (full)60.562.5
w/o Controlled Comparison53.5 (−7.0)54.3 (−8.2)

Removing the advantage signal lowers accuracy by 7.0 points on all instances and by 8.2 points on held-out instances.

[1] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018.

AuthorsOpenReview ↗

Response to Reviewer N2R2 (3)

Response to Q4

Can the authors provide additional experimental comparisons with SkillOpt and SkillGrad?

Our base baselines follow SkillLearnBench [5], which shows that Self-Feedback degrades as the number of refinement rounds grows (accuracy rises after the first revision and then drops sharply; [5], Sec. 5.4), so we adopt its original setting (K=2; [5], Sec. 4.1). For fairness, we evaluate four further baselines (three iterative skill-evolution methods and TF-GRPO), SkillOpt [1], SkillGrad [2], EvoSkill [3] and TF-GRPO [4], with the same rollout budget as Skill$^2$ (B=10 on the seed instance).

Accuracy on all instances / held-out instances (%) with Gemini-3.8-Flash, GPT-5.6-Luna and DeepSeek-V4-Flash, and Exec, the executability score from SkillLearnBench's LLM judge, computed as the average of its four original sub-metrics (completeness, determinism, consistency, usability; rescaled to 0–100; [5], Sec. 3.2.1):

MethodGemini (All/held-out/Exec)Luna (All/held-out/Exec)DeepSeek (All/held-out/Exec)
No Skill39.8 / 41.0 / –56.0 / 53.0 / –29.7 / 28.5 / –
One-Shot [5]55.7 / 58.5 / 50.743.2 / 44.5 / 59.130.5 / 27.5 / 54.0
Self-Feedback [5]59.5 / 57.5 / 49.243.5 / 42.0 / 48.730.8 / 33.0 / 47.9
SkillOpt [1]47.0 / 46.5 / 50.746.5 / 42.5 / 69.245.8 / 46.2 / 62.7
SkillGrad [2]46.0 / 45.0 / 49.649.2 / 46.8 / 56.535.0 / 36.0 / 52.8
EvoSkill [3]42.8 / 43.5 / 47.650.8 / 49.2 / 64.640.2 / 46.0 / 64.3
TF-GRPO [4]36.0 / 36.0 / 37.852.7 / 54.0 / 37.843.0 / 39.8 / 40.2
Skill$^2$75.7 / 74.8 / 53.060.5 / 62.5 / 61.547.2 / 46.5 / 74.5
Human-Authored (ref.)81.3 / 80.0 / 54.179.2 / 80.0 / 54.158.5 / 56.8 / 54.1

[1] Y. Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904, 2026.
[2] H. Wang et al. SkillGrad: Optimizing Agent Skills Like Gradient Descent. arXiv:2605.27760, 2026.
[3] S. Alzubi et al. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766, 2026.
[4] Y. Cai et al. Training-Free Group Relative Policy Optimization. arXiv:2510.08191, 2025.
[5] S. Zhong et al. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. arXiv:2604.20087, 2026.

Thank you to the authors for their response, which has alleviated some of my concerns about the experiments. If I considered only the method, I would regard this as perhaps a borderline paper, given its limited novelty but encouraging experimental results. However, I cannot give a higher score because the writing makes the paper extremely difficult for me to evaluate. I can provide the following examples:

The pseudocode contains puzzling expressions such as GEN.CHOOSEDIMENSION and GEN.COMPARE.

The authors mention "Three of its principles" in the introduction, but despite reading through the surrounding paragraphs, I could not find where these principles were introduced.

Sentences such as "Otherwise the generator writes two new skills. Sequential refinement cannot reuse experience in this way, because its feedback in Eq. 2 describes a skill that no longer exists once it is patched." make it almost impossible for readers to understand what the authors are trying to convey.

Similar difficult-to-read expressions appear throughout the manuscript. Furthermore, the figures are filled with strange wording and symbols, such as "r cannot be attributed to any single action" in Figure 1. These difficult-to-understand expressions make me suspect that the authors used AI to write the entire manuscript without properly revising it.

Reviewer_fomxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Skill², a framework that views agent skill evolution as optimizing the policy that generates skills. The method compares skill variants that differ along a single design dimension and uses execution feedback to assess the corresponding design choices. Evaluated skills, trajectories, scores, and comparison conclusions are retained to inform subsequent comparisons, and a final skill is synthesized from the accumulated evidence. Experiments on SkillLearnBench across three LLMs report improvements over no-skill, one-shot generation, and self-feedback baselines in both overall and held-out task accuracy.

Critical strengths
  1. The paper addresses a practical problem: automatically improving agent skills through execution feedback.
  2. The method is simple and interpretable, using single-dimension comparisons and reusing past evaluations.
  3. Experiments across three LLMs show improvements over the evaluated automated baselines, including on held-out instances.
Critical weaknesses
  1. The main comparison does not match rollout budgets or verifier feedback: Skill² has a 10-rollout budget, while Self-Feedback uses two rounds without verifier feedback. This makes the proposed mechanism’s contribution difficult to isolate.
  2. The ablations cover only five tasks with one model and report LLM-judged skill quality without task accuracy, leaving the components’ contributions to actual performance unclear.
  3. During skill search, each new candidate is evaluated once on the seed instance. Although execution trajectories also inform the comparisons, the stability of the resulting dimension-level conclusions across repeated executions is not evaluated.
Questions for authors
  1. How does Skill² compare with a refinement baseline given the same rollout budget and access to verifier scores?
  2. Could you report held-out task accuracy for the existing ablations to clarify whether the components improve actual task performance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fomx (1)

We appreciate your feedback. We would like to clarify the points you raised as follows:

Response to W1 & Q1

How does Skill$^2$ compare with a refinement baseline given the same rollout budget and access to verifier scores?

Our base baselines follow SkillLearnBench [5], which shows that Self-Feedback degrades as the number of refinement rounds grows (accuracy rises after the first revision and then drops sharply; [5], Sec. 5.4), so we adopt its original setting (K=2; [5], Sec. 4.1). For fairness, we evaluate four further baselines (three iterative skill-evolution methods and TF-GRPO), SkillOpt [1], SkillGrad [2], EvoSkill [3] and TF-GRPO [4], with the same rollout budget as Skill$^2$ (B=10 on the seed instance).

Accuracy on all instances / held-out instances (%) with Gemini-3.8-Flash, GPT-5.6-Luna and DeepSeek-V4-Flash, and Exec, the executability score from SkillLearnBench's LLM judge, computed as the average of its four original sub-metrics (completeness, determinism, consistency, usability; rescaled to 0–100; [5], Sec. 3.2.1):

MethodGemini (All/held-out/Exec)Luna (All/held-out/Exec)DeepSeek (All/held-out/Exec)
No Skill39.8 / 41.0 / –56.0 / 53.0 / –29.7 / 28.5 / –
One-Shot [5]55.7 / 58.5 / 50.743.2 / 44.5 / 59.130.5 / 27.5 / 54.0
Self-Feedback [5]59.5 / 57.5 / 49.243.5 / 42.0 / 48.730.8 / 33.0 / 47.9
SkillOpt [1]47.0 / 46.5 / 50.746.5 / 42.5 / 69.245.8 / 46.2 / 62.7
SkillGrad [2]46.0 / 45.0 / 49.649.2 / 46.8 / 56.535.0 / 36.0 / 52.8
EvoSkill [3]42.8 / 43.5 / 47.650.8 / 49.2 / 64.640.2 / 46.0 / 64.3
TF-GRPO [4]36.0 / 36.0 / 37.852.7 / 54.0 / 37.843.0 / 39.8 / 40.2
Skill$^2$75.7 / 74.8 / 53.060.5 / 62.5 / 61.547.2 / 46.5 / 74.5
Human-Authored (ref.)81.3 / 80.0 / 54.179.2 / 80.0 / 54.158.5 / 56.8 / 54.1

[1] Y. Yang et al. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904, 2026.
[2] H. Wang et al. SkillGrad: Optimizing Agent Skills Like Gradient Descent. arXiv:2605.27760, 2026.
[3] S. Alzubi et al. EvoSkill: Automated Skill Discovery for Multi-Agent Systems. arXiv:2603.02766, 2026.
[4] Y. Cai et al. Training-Free Group Relative Policy Optimization. arXiv:2510.08191, 2025.
[5] S. Zhong et al. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. arXiv:2604.20087, 2026.

AuthorsOpenReview ↗

Response to Reviewer fomx (2)

Response to W2 & Q2

Could you report held-out task accuracy for the existing ablations to clarify whether the components improve actual task performance?

We now report task accuracy for all ablations on all 20 tasks with GPT-5.6-Luna, with the same rollout budget as full Skill$^2$:

VariantAll (%)Held-out (%)
Skill$^2$ (full)60.562.5
w/o Controlled Comparison53.5 (−7.0)54.3 (−8.2)
w/o Control Reuse51.2 (−9.3)50.3 (−12.2)
w/o Synthesis52.8 (−7.7)50.0 (−12.5)

Removing any component lowers task accuracy by 7–10 points on all instances and by 8–13 points on held-out instances, so each component contributes to actual task performance.

Response to W3

During skill search, each new candidate is evaluated once on the seed instance. Although execution trajectories also inform the comparisons, the stability of the resulting dimension-level conclusions across repeated executions is not evaluated.

We do not systematically repeat each candidate's execution due to cost (each round involves containerized multi-turn agent rollouts). However, dimension-level conclusions are not fixed after a single execution: later rounds carry earlier-decided values into new skills, so those values are tested again and a contradicted conclusion can be revised. See following case study:

Case study (backbone: Gemini, task: python-scala-translation): the companion_module_object entry point failed in round 2 (0.0), was adopted again on a newer baseline in round 6 and passed (1.0), and was adopted by the final skill. We view this as a partial, built-in re-checking mechanism rather than a guarantee of stability.

Thanks for this clarification. I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.