acceptodds
Under review as a conference paper at ICLR 2027

Progressive Agent Skill Generation via Reinforcement Learning

Abstract

Recent large language model agents often use external skills as modular procedural units that condition inference and improve complex task solving. Thus, automatically generating high-quality skills from documents or experience has become an important problem. Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the agent on downstream tasks. To address this challenge, we propose Skill-, a reinforcement learning method that learns a unified policy for progressive skill generation. Specifically, we construct each skill by repeatedly applying the learned policy to successive source evidence and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill- generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill- improves average downstream success rates over the strongest skill-generation baseline by 3.1 points on CL-Bench and 6.7 points on tau2-bench. Further ablations and analysis validate the importance of rollback reward and progressive generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent Skill Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NnuHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Skill-α learns to turn documents and execution traces into reusable textual skills. A Qwen3-8B editor applies CREATE, UPDATE, MERGE, PRUNE, or NOOP to an evolving skill. It is first trained on synthetic demonstrations, then optimized with GRPO on fixed local editing states. The rollback reward compares the worker's performance with the original and edited skills on the same training query. The query is used for reward evaluation and is hidden from the editor. After training, skill generation requires neither anchor queries nor parameter updates. Experiments cover CL-Bench, SpreadsheetBench, and tau2-bench, with transfer to BFCL and other worker models.

Critical strengths
  1. The tau2-bench results are the strongest evidence for the method. With GPT-4o, Table 3 reports an average of 55.83, compared with 49.17 for SkillPro and 33.33 without a skill. Across three trials with different optimization seeds and independently sampled anchors, the corresponding means are 56.67 ± 0.83 for Skill-α and 48.61 ± 2.55 for SkillPro (Table A8). The same generated skills also improve Claude-Sonnet-4.5 from 63.33 to 70.83. This gives me confidence that the experience-derived skills are useful beyond the worker used in training.

  2. Using one editing policy for documents and trajectories is a useful contribution. The SFT-only and action-space ablations in Table 5 support the roles of RL and skill maintenance. Table A10 also shows that rollback outperforms direct reward after NOOP is removed, so the result cannot be explained entirely by the abstention rule. Applying the trained editor to BFCL without further parameter updates extends the evaluation beyond the training domains.

Critical weaknesses
  1. I would like a clearer explanation of what rollback changes during training. On SpreadsheetBench and tau2-bench, feedback is binary, so Eq. 5 gives R_i = (1 - r_ctrl) * r_edit_i for valid content-changing edits. If the control fails, this is direct binary reward; if it succeeds, every such edit receives zero. Without NOOP, successful-control groups therefore contribute no policy-gradient signal, while failed-control groups retain the same reward ordering as direct reward. Table A10 establishes a performance difference, but leaves open how much comes from changing which groups contribute updates. Appendix A's expected-reward analysis is consistent with this observation. Control success rates, counts of informative update groups, and worker execution costs would help explain the gain. This equivalence applies to the binary benchmarks; CL-Bench uses graded training feedback.

  2. The smaller generalization gains need uncertainty estimates. On CL-Bench with GPT-4o, Skill-α scores 10.38 versus 8.57 without a skill. The 3.1-point gain quoted in the paper compares against the strongest skill-generation baseline, which is below No Skill. On BFCL, gains over No Skill are 1.01 points with GPT-4o and 4.02 with Claude-Sonnet-4.5. Table A8 covers neither benchmark. Tasks sharing a CL-Bench context, and BFCL variants sharing a base task, should be treated as dependent when estimating uncertainty. These results are promising, but I place more weight on tau2-bench in my recommendation. I lean toward weak acceptance because of its repeated gains and the transfer across workers.

Questions for authors
  1. For each binary-verifier benchmark, what fractions of training groups have a successful control or zero reward variance? How many groups contribute policy-gradient updates under rollback and direct reward, and what are their worker execution costs? These statistics would help assess whether rollback mainly improves training by concentrating updates on states where the current skill fails.

  2. Can you provide uncertainty estimates for gains over No Skill on CL-Bench and BFCL under both workers? A paired bootstrap over source contexts for CL-Bench and base tasks for BFCL would help, preserving the reported macro averages and overlapping Sub-API assignments. Repeated runs would also help quantify generation and execution variability.

  3. Appendix C gives 50, 8, 19, and 19 editing steps for the four experience source groups, averaging 24, while Table A13 reports 40 generator calls per skill. What accounts for the additional calls? Were the cost and accuracy measurements obtained with the same generation settings?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We thank Reviewer NnuH for the supportive comments. Below, we address all raised questions and will incorporate the feedback into the revised paper.

Review (W1&Q1 - Part 1): I would like a clearer explanation of what rollback changes during training. For each binary-verifier benchmark, what fractions of training groups have a successful control or zero reward variance? How many groups contribute policy-gradient updates under rollback and direct reward?

Response: Thanks for this insightful observation. Your analysis of the binary-verifier case is exact. When the observed control fails with rctrl=0r^{\text{ctrl}}=0, rollback reward preserves the same candidate reward ordering as direct binary reward among valid content-changing candidates, while the NOOP fallback rule can provide an additional signal when no valid candidate edit improves over the failed control. Thus, on binary benchmarks, rollback reward acts by filtering which states and actions receive positive advantages based on the observed control outcome. Specifically, on successful-control groups where rctrl=1r^{\text{ctrl}}=1, removing NOOP causes rollback to assign 00 to all content edits, yielding zero reward variance std(R)=0\text{std}(R)=0 and no reward-driven update. When valid NOOP is present on rctrl=1r^{\text{ctrl}}=1 groups, rollback assigns 11 to NOOP and 00 to content edits, giving a positive advantage to NOOP and negative advantages to unnecessary edits.

To quantify this effect across the 512 SpreadsheetBench and 768 τ2\tau^2-bench editing states with G=8G=8, valid NOOP included, and informative update groups defined by std(R)>0\text{std}(R)>0, we report the training-group statistics logged from the rollback reward RrbR^{\text{rb}} and direct reward RdirR^{\text{dir}} training runs below.

BenchmarkTotal GroupsControl Success (rctrl=1r^{\text{ctrl}}=1)Zero-Variance Groups (RrbR^{\text{rb}} / RdirR^{\text{dir}})Informative Update Groups (RrbR^{\text{rb}} / RdirR^{\text{dir}})
SpreadsheetBench512149 (29.1%)64.8% / 52.3%180 (35.2%) / 244 (47.7%)
τ2\tau^2-bench768411 (53.5%)71.0% / 61.7%223 (29.0%) / 294 (38.3%)

When rctrl=1r^{\text{ctrl}}=1, direct reward assigns 11 to any candidate that achieves riedit=1r^{\text{edit}}_i=1, giving it a positive advantage whenever at least one candidate in the group fails. Across the rctrl=1r^{\text{ctrl}}=1 groups in the logged training runs, direct reward assigns positive advantages to 918 CREATE and UPDATE candidates that do not improve over the observed control score, whereas rollback reward assigns zero positive advantage to content edits on rctrl=1r^{\text{ctrl}}=1 states, as shown below.

BenchmarkPositive-Advantage CREATE on rctrl=1r^{\text{ctrl}}=1 (RdirR^{\text{dir}} / RrbR^{\text{rb}})Positive-Advantage UPDATE on rctrl=1r^{\text{ctrl}}=1 (RdirR^{\text{dir}} / RrbR^{\text{rb}})Total Positive-Advantage Edits on rctrl=1r^{\text{ctrl}}=1 (RdirR^{\text{dir}} / RrbR^{\text{rb}})
SpreadsheetBench229 / 0100 / 0329 / 0
τ2\tau^2-bench432 / 0157 / 0589 / 0
Total661 / 0257 / 0918 / 0

These logged training statistics confirm that direct reward assigns positive advantages to content edits that do not improve over the observed control score on rctrl=1r^{\text{ctrl}}=1 states, whereas rollback reward restricts positive content-editing advantages to states with an observed rctrl=0→riedit=1r^{\text{ctrl}}=0 \to r^{\text{edit}}_i=1 transition. We have added these statistics to Section 5.3 of the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

Review (W1&Q1 - Part 2): What are their worker execution costs?

Response: With G=8G=8 valid candidates per group and isolating the effect of skipping candidate evaluations when rctrl=1r^{\text{ctrl}}=1, the average worker executions per group under short-circuit rollback is 1+8(1−pctrl)1 + 8(1 - p_{\text{ctrl}}), as shown in the table below.

BenchmarkControl Success (rctrl=1r^{\text{ctrl}}=1)Direct Reward (RdirR^{\text{dir}})Standard Rollback (RrbR^{\text{rb}})Short-Circuit Rollback (RrbR^{\text{rb}})Savings vs. Direct Reward
SpreadsheetBench149 / 512 (29.1%)8.009.006.6716.60%
τ2\tau^2-bench411 / 768 (53.5%)8.009.004.7241.02%

While standard synchronous rollback evaluates 1 control rollout and 8 candidate rollouts per group, binary rollback allows skipping all 8 candidate evaluations whenever rctrl=1r^{\text{ctrl}}=1, since Rirb=0R^{\text{rb}}_i = 0 for all non-NOOP edits and Rrb(NOOP)=1R^{\text{rb}}(\text{NOOP})=1 deterministically. This reduces average worker executions per group below direct reward by 16.60% on SpreadsheetBench and 41.02% on τ2\tau^2-bench. When within-group NOOP evaluation reuse is additionally enabled for all methods, the average calls per group for Direct, Standard Rollback, and Short-Circuit Rollback are 7.21, 7.86, and 5.59 on SpreadsheetBench and 7.51, 8.23, and 4.05 on τ2\tau^2-bench, corresponding to savings of 22.47% and 46.03% over direct reward.

Review (W2&Q2): The smaller generalization gains need uncertainty estimates. Can you provide uncertainty estimates for gains over No Skill on CL-Bench and BFCL under both workers? A paired bootstrap over source contexts for CL-Bench and base tasks for BFCL would help, preserving the reported macro averages and overlapping Sub-API assignments. Repeated runs would also help quantify generation and execution variability.

Response: Following your suggestion, we have computed 95% confidence intervals using a 10,000-replicate clustered paired bootstrap, resampling over source contexts for CL-Bench and unique base tasks for BFCL while preserving macro averages and overlapping Sub-API assignments. The results against both No-Skill and the strongest skill-generation baseline are shown in the table below.

BenchmarkBackboneSkill-α\alpha Macro Avg.Δ\Delta vs. No-Skill [Clustered 95% CI]Best Skill-Gen BaselineΔ\Delta vs. Best Skill-Gen Baseline [Clustered 95% CI]
CL-BenchGPT-4o10.38%+1.81 [-1.04, +4.75]Ctx2Skill (7.30%)+3.07 [+0.21, +5.99]
CL-BenchClaude-Sonnet-4.59.55%+1.85 [-0.70, +4.36]AutoSkill (8.40%)+1.15 [-1.53, +3.86]
BFCL Multi-TurnGPT-4o57.13%+1.01 [-5.24, +7.43]SkillPro (51.50%)+5.63 [+0.20, +10.85]
BFCL Multi-TurnClaude-Sonnet-4.562.94%+4.02 [-2.01, +10.34]SkillPro (57.54%)+5.41 [-0.82, +11.69]

On CL-Bench and BFCL, while the single-run 95% intervals over No-Skill include zero due to the limited number of evaluation entries, Skill-α\alpha achieves strictly positive 95% confidence intervals over the strongest skill-generation baselines on both benchmarks under GPT-4o, and remains the only method that consistently improves over No-Skill across both benchmarks and workers. We have added these bootstrap intervals to Appendix D.4 of the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Review (Q3): Appendix C gives 50, 8, 19, and 19 editing steps for the four experience source groups, averaging 24, while Table A13 reports 40 generator calls per skill. What accounts for the additional calls? Were the cost and accuracy measurements obtained with the same generation settings?

Response: Thanks for pointing out this detail. We confirm that the cost in Table A16 and accuracy in Table 3 are measured from the exact same skill-generation runs under identical settings. In our setup, a batch size of 4 specifies an upper bound of up to 4 trajectories per editing step under the fixed 8,192-token prompt budget. Appendix C reports the nominal step count of 24 obtained by dividing the total source trajectories by the maximum batch size of 4, whereas Table A16 logs the actual LLM calls required during progressive generation.

Specifically, each generator call in Skill-α\alpha must fit the system prompt, the evolving skill zt−1z_{t-1}, and the trajectories within the 8,192-token prompt cap. As zt−1z_{t-1} grows to an average of 2,968 tokens, packing 4 full trajectories alongside zt−1z_{t-1} often exceeds the 8,192-token limit. Whenever this happens, the loader packs fewer trajectories into the current call and processes the remaining trajectories in subsequent steps so that no evidence is truncated, increasing the average number of progressive editing calls from 24 to 40.00. The same prompt-budget sub-batching also explains why Progressive Prompt Skill logs 30.75 calls instead of 24 without separate merge calls, while other baselines incur additional calls from recursive chunk merges. We have clarified this in Appendix C and Appendix D.7 of the revised paper.

AuthorsOpenReview ↗

Kind Request for Your Feedback on Our Rebuttal

Thank you very much for your thoughtful review, as well as the valuable questions and constructive suggestions you provided. In our rebuttal and revised paper, we have done our best to address all of your concerns and clarifications. As the author-reviewer discussion period will end in about 1 day, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. If you have any remaining questions or would like further clarification on any point, please feel free to let us know and we will respond promptly. Thank you very much for your time and consideration!

Reviewer_b5mnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors proposed a RL based method to learn the generation of skills. A Qwen3-8B skill generator is trained through supervised fine-tuning followed by GRPO to progressively create, update, merge, prune, or retain skill content, while keeping the downstream worker fixed. Its central contribution is a rollback reward that compares worker performance under the original and edited skills on the same training query, providing feedback on whether an individual edit improves task execution.

Critical strengths

The paper proposes an elegant approach to training LLMs to generate skills. The rollback reward evaluates edits through their effect on downstream task execution, comparing the original and edited skills on the same query with a fixed worker. The experiments in both the main text and the appendix are clearly presented and complemented by insightful ablation studies.

Critical weaknesses

None

(Not critical) line 483 "uture" the "F" is missing

Questions for authors

Given that the method has access to two successive skill variants, zt−1z_{t-1} and ztz_t, along with their corresponding rewards, have the authors explored using these pairs for Direct Preference Optimization (DPO) [1]?

Have the authors evaluated whether skills learned on one task transfer to new tasks, either directly in a zero-shot setting or through few-shot adaptation?

[1] Rafailov et al. (2024). Direct Preference Optimization: Your Language Model Is Secretly a Reward Model.

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank Reviewer b5mn for the encouraging evaluation. We have addressed all raised questions below and carried out additional experiments to validate our findings.

Review (W1): Line 483 "uture" the "F" is missing

Response: Thanks for your careful reading. We have corrected this typo to "Future" and proofread the entire paper to eliminate any remaining typos.

Review (Q1): Given that the method has access to two successive skill variants, zt−1z_{t-1} and ztz_t, along with their corresponding rewards, have the authors explored using these pairs for Direct Preference Optimization (DPO)?

Response: Thanks for this suggestion. Using the same SFT checkpoint and the 2,048 fixed editing states, we feed the exact same prompt (zt−1,xt)(z_{t-1}, x_t) to the policy, sample 8 candidates per state, and evaluate zt−1z_{t-1} and ztz_t with the GPT-4o worker and benchmark verifier on the anchored query qtancq_t^{\text{anc}}. Under a strict improvement threshold redit>rctrlr^{\text{edit}} > r^{\text{ctrl}}, valid improving edits are paired as chosen against NOOP and non-improving edits with redit≤rctrlr^{\text{edit}} \le r^{\text{ctrl}} as rejected. Conversely, when no valid candidate edit satisfies redit>rctrlr^{\text{edit}} > r^{\text{ctrl}} on a non-empty zt−1z_{t-1}, NOOP is paired as chosen against the candidate edits as rejected, while NOOP on an empty skill zt−1z_{t-1} is rejected and excluded from preference construction. We train the policy using the standard sigmoid DPO objective with β=0.1\beta = 0.1, without label smoothing or an auxiliary SFT loss. The results are shown in the table below.

MethodCL-Bench Avg.SpreadsheetBenchτ2\tau^2-bench Avg.
Skill-α\alpha w/o Rollback Reward3.68%17.00%46.67%
DPO4.06%18.50%49.17%
Skill-α\alpha (Ours)10.38%27.50%55.83%

As shown in the table, DPO outperforms Skill-α\alpha w/o Rollback Reward across all three benchmarks, confirming that redit>rctrlr^{\text{edit}} > r^{\text{ctrl}} comparisons help the model distinguish edits that genuinely improve verifier scores from redundant additions. However, DPO remains below Skill-α\alpha because it trains offline on static candidates sampled once from the pre-RL policy. It cannot explore valid edits absent from the initial samples and becomes overly conservative when all initial candidates satisfy redit≤rctrlr^{\text{edit}} \le r^{\text{ctrl}}. Therefore, GRPO in Skill-α\alpha performs better by repeatedly sampling and verifying new candidate edits from the evolving policy on these editing states. We have added this comparison to Section 5.3 of the revised paper.

Review (Q2): Have the authors evaluated whether skills learned on one task transfer to new tasks, either directly in a zero-shot setting or through few-shot adaptation?

Response: Thanks for this suggestion. We evaluate zero-shot cross-task transfer on WikiTQ [1] by directly reusing the exact same frozen SpreadsheetBench skills from Table 3 for Skill-α\alpha and all baselines, without using any WikiTQ data to select skills, tune prompts, or modify skill content. Due to rebuttal time constraints, we evaluate on the first 2,539 test examples in the official order of the 4,344-example pristine-unseen-tables test split under the same GPT-4o worker. The results are shown below.

MethodWikiTQ AccuracyAbsolute Gain (pp)
No-Skill (Base)82.87%-
Trace2Skill82.91%+0.04
Anthropic Skill-Creator83.65%+0.78
Progressive Prompt Skill83.85%+0.98
Skill-α\alpha (Ours)84.01%+1.14

Although the base worker already achieves high accuracy on WikiTQ, the frozen SpreadsheetBench skill generated by Skill-α\alpha transfers effectively in a zero-shot setting and achieves the largest gain over all baselines. Combined with the cross-domain and cross-worker analyses in our paper, this result provides further evidence that Skill-α\alpha distills transferable procedural guidance rather than task-specific shortcuts. We have added this experiment to Appendix D.1 of the revised paper.


References

[1] Panupong Pasupat, Percy Liang. Compositional Semantic Parsing on Semi-Structured Tables. In ACL 2015.

I thank the authors for their additional experiments, especially the one from question 1. I am satisfied with the answers and maintain my score.

AuthorsOpenReview ↗

Thanks to Reviewer b5mn

Thank you very much for your timely feedback and for taking the time to carefully consider our rebuttal and revised paper. We sincerely appreciate your recognition of our additional experiments, as well as your constructive suggestions throughout the review process. Your feedback has been invaluable in helping us strengthen the paper.

Reviewer_F1yTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

LLM based agents use skills as their procedural memory while solving complex tasks. Limitations of current skill generation systems include: 1) heuristics or pipeline systems extract skills from documents/experiences. Such approaches need to be specialized per evidence source. 2) Learning systems need a supervision signal i.e. whether the generated skill is useful for future tasks, such supervision is absent or expensive to gather.

Authors propose Skill-alpha, a reinforcement learning (RL) framework for automatically generating/editing external agent skills (SKILL.md-style procedural guidance) from either static documents or execution trajectories. Learned skill-curation policy looks at the evidence and selects an action amongst {create, update, merge, prune, noop} to be applied to a skill. Since supervision is still a challenge, this paper proposes rollback reward (similar to A/B testing in SWE), fixed worked agent is run on relevant training query with and without the specific edit to skill, and binary reward of 1 only if the verifier score strictly improves (r_edit > r_ctrl).

Critical strengths
  • The paper rightly identifies that skill generation lacks a direct supervision signal and that one-shot generation is challenging because there are multiple interacting aspects: abstraction, denoising, conflict resolution, and compression. Authors propose rollback reward for execution-grounded, control-relative local credit assignment over well-defined action space for progressive skill generation.

  • The paper does extensive experimental evaluation over established benchmarks (CL-Bench, SpreadsheetBench, tau2-bench,BFCL multi-turn) comparing against strong baselines including Ctx2Skill, AutoSkill, Trace2Skill, SkillX, and SkillPro, 2 backbones (GPT-4o, Claude-Sonnet-4.5) Skill-alpha achieves empirical gains over heuristic and pipeline baselines on CL-Bench, SpreadsheetBench, and tau2-bench, alongside zero-shot transfer to an unseen benchmark (BFCL multi-turn) and cross-worker transfer to Claude-Sonnet-4.5 and DeepSeek-V4-Flash.

  • Hyperparameters, data counts, exact prompt templates (Appendix E), and computational costs (Appendix D.6) are thoroughly documented.

Critical weaknesses
  • While the input evidence is identical while comparing with baselines, Skill-alpha was trained using 2,048 states x 8 candidate rollouts judged by the benchmark environment or by GPT-5.5 under the official task rubrics. None of the baselines are allowed to use benchmark verifiers or execution feedback.

  • Reported gains across tau-bench, CL-Bench, and BFCL come from just a few tasks or small test splits. E.g. tau-bench domains only have 20–40 tasks, so 1 task equals 2.5% to 5.0% performance, the main tables (Tables 2–4) do not report confidence intervals or statistical significance tests. Also, on CL-Bench with GPT-4o, simply giving the agent the raw document (NO SKILL, 8.57%) beats every skill-generation baseline (6.37%–7.30%), is close to Skill-alpha overall (10.38%), and outperforms Skill-alpha on “Rule System Application” (21.82% vs. 20.91%). This weakens skill-alpha’s claims for superior performance.

Questions for authors
  • Why were the prompt/pipeline baselines capped at 2,600 output tokens? Skill-alpha on the other hand generates 2,968-token skills with no retrieval cap?

  • Skill-alpha during its SFT stage was warmed up using DeepSeek-V4-Pro generated data. How does a baseline like Progressive Prompt Skill perform when driven by DeepSeek-V4-Pro without restriction on output tokens?

  • From my understanding, Progressive Prompt Skill is added to isolate the effect of RL training from multi-step prompting alone. However, Progressive Prompt Skill is forced to rewrite the entire skill from scratch within a 2,600 token limit. To make this a fair ablation, can you evaluate a prompt-only baseline e.g. Progressive Prompt Skill when given the same {CREATE, UPDATE, MERGE, PRUNE, NOOP} actions space and a >3,000-token skill budget?

  • Because several held-out splits are small (e.g., 20–40 tasks per domain on tau-bench, where a 2.5–5.0% margin is only 1 task), please report task-level 95% confidence intervals or paired significance tests (e.g., paired bootstrap or McNemar's test against NO SKILL and the strongest baseline) computed directly over your existing per-task evaluation predictions for Tables 2, 3, 4 alongside the task-weighted overall average for CL-Bench.

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/4)

We thank Reviewer F1yT for your positive feedback. We are pleased to address your concerns below and will incorporate all feedback to improve the paper.

Review (W1): While the input evidence is identical while comparing with baselines, Skill-alpha was trained using 2,048 states x 8 candidate rollouts judged by the benchmark environment or by GPT-5.5 under the official task rubrics. None of the baselines are allowed to use benchmark verifiers or execution feedback.

Response: Thanks for this observation. First, we agree that the main tables compare a trained skill editor against training-free pipelines, so the supervision conditions are not identical. Specifically, while all experience-based baselines receive the same source trajectories, tool observations, and pre-collected execution feedback, and spend substantially more inference time per skill with a GPT-4o writer through multi-round self-play, internal LLM judging, gating on recorded trajectory rewards, and recursive LLM merging, they do not run new anchored-query verifier executions during skill construction. In contrast, Skill-α\alpha shifts this computation to a one-time offline RL stage to train an 8B policy πϕ\pi_\phi, and at skill-generation time constructs each skill in a single progressive pass without querying any anchored tasks, benchmark verifiers, or test feedback.

Second, our controlled ablations show that additional verifier supervision on anchored queries cannot by itself explain the full gains of Skill-α\alpha. Specifically, w/o rollback reward in Table 5, Direct Reward - NOOP in Table A13, and our new DPO baseline reported for Reviewer b5mn (Q1) all use the same 2,048 fixed editing states, the same SFT initialization, and a candidate group size of eight, where actual verifier execution counts additionally depend on control evaluations, invalid candidates, and NOOP reuse. However, all of them underperform Skill-α\alpha, with w/o rollback reward falling below the strongest training-free baseline and remaining close to SFT only. Therefore, these experiments confirm that training-time verifier supervision alone does not yield the full performance advantage without rollback credit assignment and online candidate sampling from the evolving policy.

Third, Table 4 confirms that the learned policy generalizes zero-shot to domains where no training verifier is ever used. On the unseen BFCL multi-turn benchmark, no BFCL trajectories or verifiers are used during training. At test time, Skill-α\alpha processes the same source BFCL trajectories as all baselines to construct skills without any verifier access or parameter updates, and still outperforms all baselines under both backbones.

AuthorsOpenReview ↗

Rebuttal by Authors (2/4)

Review (W2&Q4 - Part 1): Reported gains across tau-bench, CL-Bench, and BFCL come from just a few tasks or small test splits. Because several held-out splits are small, please report task-level 95% confidence intervals or paired significance tests computed directly over your existing per-task evaluation predictions for Tables 2, 3, 4.

Response: Thanks for this suggestion. We have computed paired 95% bootstrap confidence intervals with 10,000 resamples over source contexts for CL-Bench and tasks for SpreadsheetBench, τ2\tau^2-bench, and BFCL. To avoid overlap when No-Skill exceeds competing generators, we report comparisons against both No-Skill and the strongest skill-generation baseline in each setting below.

BenchmarkBackboneSkill-α\alpha Avg.Δ\Delta vs. No-Skill [95% CI]Best Skill-Gen BaselineΔ\Delta vs. Best Skill-Gen Baseline [95% CI]
SpreadsheetBenchGPT-4o27.50%+9.50 [+3.00, +16.00]Anthropic Skill-Creator (26.00%)+1.50 [-5.00, +8.00]
SpreadsheetBenchClaude-Sonnet-4.531.50%+5.50 [-2.00, +13.00]SkillX (31.50%)0.00 [-7.50, +7.50]
τ2\tau^2-benchGPT-4o55.83%+22.50 [+11.67, +33.33]SkillPro (49.17%)+6.67 [-5.00, +18.33]
τ2\tau^2-benchClaude-Sonnet-4.570.83%+7.50 [-2.50, +17.50]Anthropic Skill-Creator (65.83%)+5.00 [-5.00, +15.00]
CL-Bench (Macro)GPT-4o10.38%+1.81 [-1.04, +4.75]Ctx2Skill (7.30%)+3.07 [+0.21, +5.99]
CL-Bench (Macro)Claude-Sonnet-4.59.55%+1.85 [-0.70, +4.36]AutoSkill (8.40%)+1.15 [-1.53, +3.86]
BFCL Multi-TurnGPT-4o57.13%+1.01 [-5.24, +7.43]SkillPro (51.50%)+5.63 [+0.20, +10.85]
BFCL Multi-TurnClaude-Sonnet-4.562.94%+4.02 [-2.01, +10.34]SkillPro (57.54%)+5.41 [-0.82, +11.69]

Under GPT-4o, Skill-α\alpha achieves strictly positive 95% confidence intervals over No-Skill on τ2\tau^2-bench and SpreadsheetBench, as well as against 6 of the 7 skill-generation baselines on SpreadsheetBench, with multi-seed runs in Table A10 further confirming low variance at 56.67±0.83%56.67 \pm 0.83\% and 26.67±0.76%26.67 \pm 0.76\%. On CL-Bench and BFCL, while the single-run 95% intervals over No-Skill include zero due to the limited number of evaluation entries, Skill-α\alpha achieves strictly positive 95% confidence intervals over the strongest skill-generation baselines under GPT-4o and remains the only method that consistently improves over No-Skill across all benchmarks and backbones in Tables 2, 3, 4, and A5. We have added these results to Appendix D.4 of the revised paper.

AuthorsOpenReview ↗

Rebuttal by Authors (3/4)

Review (W2&Q4 - Part 2): Please report the task-weighted overall average for CL-Bench. Also, on CL-Bench with GPT-4o, simply giving the agent the raw document (NO SKILL, 8.57%) beats every skill-generation baseline (6.37%–7.30%) and is close to Skill-alpha overall (10.38%).

Response: We report the task-weighted overall average across all 1,065 held-out CL-Bench tasks under both workers in the table below.

BenchmarkBackboneSkill-α\alphaNo-SkillΔ\Delta vs. No-SkillBest Skill-Gen BaselineΔ\Delta vs. Best Skill-Gen Baseline
CL-Bench (Task-Wtd)GPT-4o7.61% (81/1,065)6.38% (68/1,065)+1.22%Anthropic (5.26%, 56/1,065)+2.35%
CL-Bench (Task-Wtd)Claude-Sonnet-4.58.64% (92/1,065)6.20% (66/1,065)+2.44%Ctx2Skill (7.61%, 81/1,065)+1.03%

As shown in the table, Skill-α\alpha outperforms both No-Skill and all skill-generation baselines on the task-weighted CL-Bench average under both workers. On CL-Bench, No-Skill already receives the raw source context in the benchmark messages, and generated skills are appended to these original messages at evaluation time. Consequently, if a generated skill contains imprecise abstractions, redundant rules, or conflicting sections, appending it introduces misleading guidance or attention distraction that degrades performance below No-Skill, as seen for all competing baselines on GPT-4o. In contrast, the editor in Skill-α\alpha is trained with rollback reward to favor edits that improve over the control and consolidate overlapping rules, making it the only method that consistently improves over the raw context under both macro and task-weighted averages.

Review (W2&Q4 - Part 3): On CL-Bench with GPT-4o, NO SKILL outperforms Skill-alpha on “Rule System Application” (21.82% vs. 20.91%). This weakens skill-alpha’s claims for superior performance.

Response: We acknowledge that on Rule System Application under GPT-4o, Skill-α\alpha does not improve over No-Skill, scoring 20.91% versus 21.82%, which corresponds to 23 versus 24 solved tasks out of 110. However, all competing skill-generation baselines suffer large drops on this split down to 14.55%–16.36% in Table 2 because their appended skills introduce negative interference, whereas Skill-α\alpha largely avoids this degradation on GPT-4o and achieves a positive gain over No-Skill under Claude-Sonnet-4.5 from 17.27% to 19.09%, tying for best across all methods.

Meanwhile, on the remaining three CL-Bench categories under GPT-4o, especially Procedural Task Execution where accuracy more than doubles from 4.30% to 9.68%, and across all four categories under Claude-Sonnet-4.5, Skill-α\alpha consistently improves over No-Skill. Thus, Skill-α\alpha avoids the severe negative interference seen in other baselines while delivering clear overall gains across both backbones.

Review (Q1): Why were the prompt/pipeline baselines capped at 2,600 output tokens? Skill-alpha on the other hand generates 2,968-token skills with no retrieval cap?

Response: Thanks for this question. First, our generation logs confirm that the 2,600-token per-call output cap does not truncate the baselines. As shown in Table A8, the final generated skills of Anthropic Skill-Creator, Progressive Prompt Skill, and Trace2Skill average 1,427, 2,174, and 2,077 tokens, remaining well below 2,600 tokens because LLMs tend to compress content when rewriting or consolidating text [1].

Second, the 2,600-token limit is a per-call output cap paired with the 16,384-character chunk input budget for Anthropic Skill-Creator, Progressive Prompt Skill, Trace2Skill, ExpeL, and AWM, where chunked baselines recursively merge chunk-level outputs and Skill-α\alpha reaches 2,968 tokens by accumulating local edits across steps (Appendix B.2). Meanwhile, baselines whose native schemas require larger single-call outputs retain their original limits, including AutoSkill at 8,192 tokens, SkillX at 5,000 tokens, and Ctx2Skill with endpoint defaults.

Third, while character budgets for input chunks or retrieval views and token counts are not identical units, we strictly preserve each baseline's native artifact format and inference protocol without arbitrary restriction. Only retrieval-based methods, namely ExpeL, AWM, and SkillX, apply a 10,000–12,000-character cap when selecting top entries from a larger pool at test time, whereas all non-retrieval baselines inject their entire generated SKILL.md into the worker without any retrieval cap, following the exact same injection protocol as Skill-α\alpha.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their constructive feedback and recognition of our work. Based on your suggestions, we have updated the revised paper as follows.

  1. Proofreading. We have corrected all remaining typos.
  2. Protocol Clarifications (Section 5, Appendix B.2, and Appendix D.7) We have clarified the baseline output and retrieval budget conventions and prompt-budget sub-batching during progressive skill generation.
  3. Statistical Confidence Intervals (Appendix D.4). We have added bootstrap 95% confidence intervals across Tables 2, 3, and 4.
  4. Controlled Ablations on Writer Backbone, Action Space, and DPO (Appendix D.3 and Section 5.3). We have added new ablations evaluating Progressive Prompt Skill with an uncapped DeepSeek-V4-Pro writer, prompt-only Action Prompt Skill baselines using the same {CREATE, UPDATE, MERGE, PRUNE, NOOP} action space, and the DPO baseline.
  5. Zero-Shot Cross-Task Skill Transfer on WikiTQ (Appendix D.1) We have added a zero-shot transfer evaluation on 2,539 unseen WikiTQ examples by directly reusing the frozen SpreadsheetBench skills from Table 3 without any task-specific selection or tuning.
  6. Rollback Training-Group Dynamics (Section 5.3) We have added logged training-group variance and advantage statistics comparing rollback reward against direct reward on binary-verifier benchmarks.
AuthorsOpenReview ↗

Paper Clarification (1/3)

We would like to thank all reviewers for their thorough reading of our paper and insightful comments. We at first address the most common and central questions raised across reviews to avoid any misunderstandings.

1. Statistical Uncertainty, Task-Weighted CL-Bench Evaluation, and Avoidance of Negative Transfer

To quantify evaluation uncertainty across benchmarks with varying split sizes, we compute 10,000-replicate paired bootstrap 95% confidence intervals over source contexts for CL-Bench, unique base tasks for BFCL, and tasks for SpreadsheetBench and τ2\tau^2-bench, together with the task-weighted overall average across all 1,065 held-out CL-Bench tasks.

BenchmarkBackboneSkill-α\alpha Avg.Δ\Delta vs. No-Skill [95% CI]Best Skill-Gen BaselineΔ\Delta vs. Best Skill-Gen Baseline [95% CI]
SpreadsheetBenchGPT-4o27.50%+9.50 [+3.00, +16.00]Anthropic Skill-Creator (26.00%)+1.50 [-5.00, +8.00]
SpreadsheetBenchClaude-Sonnet-4.531.50%+5.50 [-2.00, +13.00]SkillX (31.50%)0.00 [-7.50, +7.50]
τ2\tau^2-benchGPT-4o55.83%+22.50 [+11.67, +33.33]SkillPro (49.17%)+6.67 [-5.00, +18.33]
τ2\tau^2-benchClaude-Sonnet-4.570.83%+7.50 [-2.50, +17.50]Anthropic Skill-Creator (65.83%)+5.00 [-5.00, +15.00]
CL-Bench (Macro)GPT-4o10.38%+1.81 [-1.04, +4.75]Ctx2Skill (7.30%)+3.07 [+0.21, +5.99]
CL-Bench (Macro)Claude-Sonnet-4.59.55%+1.85 [-0.70, +4.36]AutoSkill (8.40%)+1.15 [-1.53, +3.86]
CL-Bench (Task-Wtd)GPT-4o7.61% (81/1,065)+1.22% (vs. 6.38%, 68/1,065)Anthropic (5.26%, 56/1,065)+2.35%
CL-Bench (Task-Wtd)Claude-Sonnet-4.58.64% (92/1,065)+2.44% (vs. 6.20%, 66/1,065)Ctx2Skill (7.61%, 81/1,065)+1.03%
BFCL Multi-TurnGPT-4o57.13%+1.01 [-5.24, +7.43]SkillPro (51.50%)+5.63 [+0.20, +10.85]
BFCL Multi-TurnClaude-Sonnet-4.562.94%+4.02 [-2.01, +10.34]SkillPro (57.54%)+5.41 [-0.82, +11.69]

Under GPT-4o, Skill-α\alpha achieves strictly positive 95% confidence intervals over No-Skill on τ2\tau^2-bench and SpreadsheetBench, as well as against 6 of the 7 skill-generation baselines on SpreadsheetBench, with multi-seed runs in Table A10 further confirming low variance at 56.67±0.83%56.67 \pm 0.83\% and 26.67±0.76%26.67 \pm 0.76\%. On CL-Bench and BFCL, while the single-run 95% intervals over No-Skill include zero due to the limited number of evaluation clusters, Skill-α\alpha achieves strictly positive 95% confidence intervals over the strongest skill-generation baselines under GPT-4o and outperforms all baselines on the 1,065-task CL-Bench task-weighted average under both workers. On CL-Bench, No-Skill already receives the raw source context in the benchmark messages, so appending skills with imprecise or redundant rules introduces attention distraction and causes competing generators to fall below No-Skill on GPT-4o. In contrast, Skill-α\alpha is the only method that consistently improves over No-Skill across all benchmarks and backbones in Tables 2, 3, 4, and A5.

AuthorsOpenReview ↗

Paper Clarification (2/3)

2. Supervision Conditions and Controlled Ablations on the Source of Gains

While all experience-based baselines receive the same source trajectories, tool observations, and pre-collected execution feedback, and spend substantially more inference time per skill with a GPT-4o writer through multi-round self-play, internal LLM judging, gating on recorded trajectory rewards, and recursive LLM merging, they do not run new anchored-query verifier executions during construction. In contrast, Skill-α\alpha shifts this computation to a one-time offline RL stage to train an 8B policy πϕ\pi_\phi, and at test time generates each skill in a single progressive pass without querying anchored tasks, benchmark verifiers, or test feedback.

To verify that the gains of Skill-α\alpha cannot be explained simply by training-time verifier access, teacher distillation, output token budgets, or action-space prompting, we summarize our controlled ablations below.

MethodSupervision & Generation SetupCL-Bench Avg.SpreadsheetBenchτ2\tau^2-bench Avg.
Progressive Prompt Skill (GPT-4o)Prompt-only full rewrite (2,600-token cap)7.11%18.00%38.33%
Progressive Prompt Skill (DeepSeek-V4-Pro, Uncapped)Prompt-only full rewrite with SFT teacher (Uncapped)4.47%18.50%36.67%
Action Prompt Skill (GPT-4o, Retry Max 3)Prompt-only {CREATE, UPDATE, MERGE, PRUNE, NOOP} + quality gates6.06%21.50%44.17%
Skill-α\alpha (SFT Only)DeepSeek-V4-Pro SFT initialization without RL5.21%15.00%44.17%
Skill-α\alpha w/o Rollback RewardSame 2,048 fixed states + direct verifier reward (RdirR^{\text{dir}})3.68%17.00%46.67%
DPO (Control-Relative Pairs)Same 2,048 fixed states + offline DPO (redit>rctrl,β=0.1r^{\text{edit}} > r^{\text{ctrl}}, \beta=0.1)4.06%18.50%49.17%
Skill-α\alpha (Ours)Same 2,048 fixed states + GRPO with Rollback Reward (RrbR^{\text{rb}})10.38%27.50%55.83%

As shown in the table, replacing GPT-4o with an uncapped DeepSeek-V4-Pro writer or equipping GPT-4o with our structured editing action space and quality-gate retries still leaves a large gap below Skill-α\alpha. More importantly, w/o rollback reward and DPO both use the exact same SFT initialization, the same 2,048 fixed editing states, and benchmark verifier evaluations, yet both underperform Skill-α\alpha by a wide margin. Therefore, these controlled comparisons confirm that the performance advantage of Skill-α\alpha stems from rollback credit assignment combined with online candidate sampling from the evolving policy during GRPO.

3. Mechanism and Execution Efficiency of Rollback Reward on Binary Benchmarks

On binary-verifier benchmarks, when the observed control fails with rctrl=0r^{\text{ctrl}}=0, rollback reward preserves the same candidate ordering as direct binary reward among valid content-changing edits. However, when the observed control already succeeds with rctrl=1r^{\text{ctrl}}=1, direct reward assigns 11 to any candidate that achieves riedit=1r^{\text{edit}}_i=1, giving positive advantages to non-improving content edits whenever at least one candidate in the group fails. Across the 512 SpreadsheetBench and 768 τ2\tau^2-bench groups in our logged training runs, rollback reward reduces informative update groups from 244 to 180 on SpreadsheetBench and from 294 to 223 on τ2\tau^2-bench, eliminating all 918 positive-advantage CREATE and UPDATE edits that occur on rctrl=1r^{\text{ctrl}}=1 states under direct reward, including 329 on SpreadsheetBench and 589 on τ2\tau^2-bench.

Furthermore, because binary rollback deterministically assigns Rirb=0R^{\text{rb}}_i = 0 to all content edits and Rrb(NOOP)=1R^{\text{rb}}(\text{NOOP})=1 whenever rctrl=1r^{\text{ctrl}}=1, short-circuit rollback skips all 8 candidate evaluations on successful-control groups. At 1+8(1−pctrl)1 + 8(1 - p_{\text{ctrl}}) executions per group, short-circuit rollback reduces average worker executions per group from 8.00 under direct reward to 6.67 on SpreadsheetBench and 4.72 on τ2\tau^2-bench, saving 16.60% and 41.02% of worker executions, or 22.47% and 46.03% when within-group NOOP evaluation reuse is enabled across all methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.