acceptodds
Withdrawn from ICLR 2027

CODESKILL: Learning Self-Evolving Skills for Coding Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding agents produce rich trajectories while solving software-engineering tasks. To enable agent self-evolution, these trajectories can be distilled into reusable procedural skills that compactly encode experience to guide future behavior. However, existing skill construction and maintenance methods often rely on fixed prompts and heuristic update rules, leaving it unclear how knowledge should be selected, abstracted, and maintained to best serve downstream agents. We propose CODESKILL, an LLM-based framework that reformulates skill extraction and skill-bank maintenance as a learnable management policy. CODESKILL extracts multi-granularity procedural skills from coding-agent trajectories, evolves skills with new experience, and maintains a compact skill bank for future task solving. We train CODESKILL with reinforcement learning, using a hybrid reward that combines dense rubric-based skill-quality feedback with sparse verifiable execution feedback from the frozen downstream agent. Experiments on EnvBench, SWE-Bench Verified, and Terminal-Bench 2 show that CODESKILL improves average pass rate by 11.03 over the no-skill baseline and by 5.10 over the strongest prompt-based or memory baseline, while maintaining a compact skill bank.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Skill, Self-evolving agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8aeFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CODESKILL learns a policy for extracting, revising, and maintaining natural-language procedural skills from coding-agent trajectories while keeping the downstream solver frozen. A Qwen3.5-4B manager receives supervised warmup and GRPO training with a hybrid reward combining rubric-based quality, improvement over cached no-skill execution, and a behavior-alignment factor. Task-level and event-driven skills are retrieved and injected into the solver context. On four task groups from three benchmarks, Table 1 reports a macro-average pass rate of 40.35% with the Qwen solver, versus 29.32% without skills and 35.25% for GPT-based prompt management. The lifecycle ablation reduces the bank from 1,252 to 676 skills while improving aggregate accuracy. These are promising results, but they rely on two preparatory passes over the evaluation set, which limits the generalization claim.

Critical strengths
  1. The learned management policy is a useful contribution with measurable utility. Under the Qwen solver, the full model exceeds its SFT-only version by 5.81 percentage points in macro-average accuracy and the GPT prompt-management baseline by 5.10 points (Table 1). With GPT as the downstream solver, it also improves the average from 21.80% to 31.62%. This supports the value of optimizing an experience-management module without retraining the solver.
  2. The paper evaluates the skill lifecycle rather than just skill generation. Table 2 reports averages of 37.88%, 39.53%, and 40.35% for extraction, extraction plus evolution, and the full lifecycle. Maintenance removes about 46% of entries while increasing aggregate accuracy. That is consequential because an accumulating library must control redundancy. The benefit is aggregate: Java accuracy falls from 41.12% with extraction alone to 38.32% with the full lifecycle, so the result should not be presented as uniformly positive.
Critical weaknesses
  1. The test-time protocol leaves the evidence boundary and acquisition cost insufficiently specified. Section 4.2 constructs the bank through two passes over the evaluation set, and Appendix C restricts retrieval to the same benchmark. This is a legitimate batch-adaptation setting and can test whether the learned manager handles a new distribution. The current-instance exclusion is also a useful safeguard. However, the second pass uses skill-conditioned trajectories: provenance must account for skills that influenced those trajectories, not only the instances directly used to extract or merge entries. The manuscript does not explain this dependency tracking, so the exclusion cannot be fully audited. Moreover, the reported step savings omit the two preparatory passes and skill-management calls, leaving the practical cost of the reported improvement unresolved.
  2. The distinctive RL reward design is insufficiently isolated by the experiments. The final objective gates execution improvement by an LLM judgment of whether behavior reflects the skill, but matching behavior can arise from the solver’s existing competence. The paper does not compare the full reward with quality-only, execution-only, or an otherwise matched objective without alignment. Its SFT-versus-RL comparison supports the overall RL stage, but cannot identify the value of the proposed attribution mechanism. Skip, add, and drop receive only rubric feedback, so the method also does not directly validate every maintenance decision through execution. Without these controls and uncertainty estimates, the empirical case for the specific reward design remains incomplete.
Questions for authors

The following requests concern the current method and evidence; their answers could change my recommendation.

  1. Please clarify the evaluation boundary. Are both preparatory passes performed over the full evaluation set for every method, and when is the bank frozen? Does provenance include skills that influenced second-pass trajectories as well as the instances directly used in extraction, evolution, and merging? Can you report a complementary evaluation in which bank construction and evaluation use disjoint instance pools, or explicitly restrict the claims to adaptation using other target-set instances?
  2. Please specify the runtime skill-loading mechanism: event-retrieval timing, top-k and thresholds, deduplication, skill token limits, exact injection position, and solver decoding settings. For reverse retrieval during RL, what is K, and can the selected verification task be the source of the extraction trajectory? The distinction affects whether feedback measures reuse beyond the source task.
  3. What evidence shows that the alignment gate improves downstream performance? Please provide a controlled reward ablation, particularly full reward versus the same objective without alignment, with training data and budget matched. Also state the exact quality-reward weights for skip, add, and drop, and explain how reward scales remain comparable when different operation types occur in a group.
  4. Are Tables 1 and 2 single evaluation runs or averages? Please report evaluation repeats or paired uncertainty estimates, the Terminal-Bench evaluation denominator, and the sizes of the common solved subsets used for Steps. Confirm that baselines share the preparatory data access, provenance filtering, and prompt budgets. Please also quantify preparation and online retrieval costs; fewer steps on the common solved subset alone do not establish lower total cost.
Reviewer_TCHtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors train a skill manager — a Qwen3.5-4B policy that writes, revises, adds, merges, and drops reusable markdown "skills" extracted from coding-agent trajectories — while the downstream coding policy stays frozen. The manager is fine-tuned on teacher-generated data and then optimized with GRPO. The reward combines a rubric-based LLM judge score with an execution term: a task instance is found by matching it to the generated skill ("reverse retrieval"), the frozen solver is run on it, and the improvement over a cached no-skill baseline is gated by a judge-estimated "did the agent actually follow this skill" factor. Results are reported on EnvBench, SWE-Bench Verified, and Terminal-Bench 2 with two different frozen solvers.

Critical strengths
  1. Clean problem decomposition. Isolating the manager from the solver is a good experimental design: it makes credit assignment testable and lets skills transfer to solvers that cannot be trained.
  2. The two reward mechanisms are sensible. Reverse retrieval solves a real problem (an arbitrary generated skill has no matching task), and the alignment gate targets the genuine confounder that the solver may succeed on its own.
  3. The maintenance result is interesting. Compressing the bank from 1252 to 676 skills while improving average pass rate (Table 2) is a genuinely useful finding, provided it survives the caveats below.
Critical weaknesses

W1. The evaluation protocol is transductive, so the generalization claims do not hold. The skill bank is built by two passes over the test set before final evaluation (§4.1). Therefore (a) "held-out" instances are held out only with respect to training, not with respect to bank construction; (b) Terminal-Bench 2 is presented as out-of-distribution, but its bank is grown from TB2's own instances, so the 25.88 → 36.47 gain mixes "the manager transfers" with "we ran two passes on the target distribution"; (c) the no-skill baseline receives no comparable exploration budget, so the headline +11.03 is inflated relative to a clean setting. The paper acknowledges in the Conclusion that streaming is future work, but no non-transductive number is reported anywhere.

W2. The comparison that decides the thesis is missing. The central claim is that learning beats fixed prompts at a small model size. But the only "stronger model" baseline is GPT-5.4-mini with a single-shot fixed prompt — no feedback loop, no best-of-n, no self-critique. The authors already run Qwen3.5-35B-A3B as the frozen solver and as the reward rollout engine, so using it as the manager is nearly free and would directly answer whether RL buys capability or only saves cost. Relatedly, "lightweight" is a stated motivation but there is no cost table (tokens, latency, $, or amortization of the reported ~210 GPU-hours), and "state-of-the-art among memory-based methods" is unsupported because every baseline is a self-reimplementation. Note also the internal signal: SFT-CODESKILL (34.54) is below the GPT-5.4-mini prompt manager (35.25), i.e. distilling the teacher into 4B loses; only RL wins. The contribution should be framed around execution feedback, not around lightweightness.

W3. Positioning is too thin for a crowded area, and the closest neighbor is not cited. "Defining the quality of a generated artifact by its downstream utility for a frozen consumer" is already the core of CapRL (arXiv 2509.22647), which additionally constructs verifiable instances and obtains attribution structurally (the consumer is vision-free, so success must come from the artifact). The paper does not cite it, does not explain why structural attribution is unavailable here, and does not argue why its soft judge-based gate is preferable. Separately, the dual-granularity bank closely overlaps with Tu et al. (2026), which appears only inside a citation list, and §1 claims SkillRL "relies on fixed prompts and heuristic criteria" while §2.2 correctly lists it as an RL method — an internal inconsistency a reviewer will read as under-reading the prior work. The phrase "reformulates skill extraction and maintenance as a learnable policy" makes a novelty claim the literature already owns.

W4. The only remaining novel components are never ablated. After the above, the defensible contributions are the alignment gate and reverse retrieval. Neither is ablated: no comparison against λ·R_Q + R_E without gating, no comparison against random-task or source-task reward assignment. In addition, the "higher weight" applied to skip/add/drop is never given numerically, and these actions share a single GRPO group normalization with operations whose rewards live on a different scale — and add/drop receive no execution feedback at all, so a harmful drop is never penalized.

W5. Reverse retrieval is an unjustified and biased reward estimator. The choice is never motivated and K is never reported. Formally, if ΔV is nonzero only where a skill applies, the paper's objective is J = P(applicable) × E[ΔV | applicable], whereas reverse retrieval estimates approximately E[ΔV | applicable] — it divides out the applicability probability. Consequences: the estimator prefers narrow, high-hit-rate skills; this is in direct tension with the reusability dimension of the quality rubric, which rewards generality; and it may explain the reported "event-driven skills are stronger than task-level skills" finding, which could be an artifact of the estimator rather than a property of the knowledge. The estimator is also not the objective stated in Eq. (1), and at RL time nothing prevents the retrieved instance from being the source instance the skill was distilled from.

W6. Statistical support is thin and some numbers are inconsistent. Back-computing sample sizes from the percentages gives n = 43 (EnvBench-Python), 107 (Java), 150 (SWE-Bench Verified), 85 (TB2). Most reported gaps are 5–16 task flips, i.e. roughly 1.5–2σ, with no seeds, no error bars, and no paired tests. Reporting also looks selective: EnvBench's Issues metric is never discussed although CODESKILL (62.74) is far worse than the GPT-5.4-mini prompt baseline (31.58) and worse than its own "extraction + evolution" ablation (43.00), which contradicts "without sacrificing aggregate performance"; the full pipeline is below extraction-only on EnvBench-Java (38.32 vs 41.12); and 46.67% pass rate in 7.56 steps with GPT-5.4-mini is implausible for SWE-Bench and needs explanation before the cross-solver claim can be trusted.

Questions for authors
  1. Can you report a non-transductive result: bank built only from training data, evaluated zero-shot on all three benchmarks? And a strictly streaming variant?
  2. Can you add Qwen3.5-35B-A3B as the manager under the identical pipeline, and a GPT-5.4-mini manager with iterative refinement?
  3. What is the cost per skill operation?
  4. What is the ablation of R_A gating and of reverse retrieval (vs. random task, vs. source task)?
  5. What are K, the fraction of samples with R_E ≠ 0, the fraction with R_A > 0, and the fraction of retrievals returning the source instance?
  6. What is the numerical weight for skip/add/drop, and how do you justify one group normalization over different reward ranges?
  7. Why does GPT-5.4-mini solve 46.67% of SWE-Bench Verified in 7.56 steps?
  8. Why does full lifecycle increase EnvBench-Python Issues from 43.00 to 62.74?
  9. What are the bank/memory sizes for the prompt and subtask-memory baselines, for the compactness comparison?
Reviewer_si8KOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose CODESKILL that learns to extract, evolve, and maintain reusable procedural skills from coding-agent trajectories. It trains a skill-management policy using supervised warmup followed by GRPO, with a hybrid reward combining rubric-based skill quality, downstream execution improvement, and behavioral alignment. Experiments on EnvBench, SWE-Bench Verified, etc show improvements over no-skill, prompt-based skill-management, and memory baselines.

Critical strengths

The proposed framework covers comprehensive skill lifecycle: integrates task-level and event-driven skills with extraction, evolution, and bank maintenance. Ablations suggest that these components capture complementary knowledge and that maintenance reduces the skill bank by approximately 46% while improving overall performance. It also shows encouraging results across diverse benchmarks and downstream policies.

Critical weaknesses

The test-time evaluation protocol complicates the generalization claims. The skill bank is constructed through two preparatory passes over the test set using no-skill and skill-conditioned rollouts. Although skills originating from the current instance are excluded during its final evaluation, skills derived from other test instances remain available. Why is the skill bank constructed using rollouts from the entire test set before final evaluation? Do all skill-based baselines receive equivalent access to cross-instance test trajectories?

How sensitive are the results to the quality-reward weight, the behavior-alignment reward, and the number of no-skill baseline rollouts? Can the authors provide evidence that the LLM-as-judge rewards correlate with human assessments of skill utility or actual downstream improvements?

Besides, code is not provided, raising concerns on reproducibility.

Questions for authors

See the weakness section above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.