- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CODESKILL learns a policy for extracting, revising, and maintaining natural-language procedural skills from coding-agent trajectories while keeping the downstream solver frozen. A Qwen3.5-4B manager receives supervised warmup and GRPO training with a hybrid reward combining rubric-based quality, improvement over cached no-skill execution, and a behavior-alignment factor. Task-level and event-driven skills are retrieved and injected into the solver context. On four task groups from three benchmarks, Table 1 reports a macro-average pass rate of 40.35% with the Qwen solver, versus 29.32% without skills and 35.25% for GPT-based prompt management. The lifecycle ablation reduces the bank from 1,252 to 676 skills while improving aggregate accuracy. These are promising results, but they rely on two preparatory passes over the evaluation set, which limits the generalization claim.
Critical strengths
- The learned management policy is a useful contribution with measurable utility. Under the Qwen solver, the full model exceeds its SFT-only version by 5.81 percentage points in macro-average accuracy and the GPT prompt-management baseline by 5.10 points (Table 1). With GPT as the downstream solver, it also improves the average from 21.80% to 31.62%. This supports the value of optimizing an experience-management module without retraining the solver.
- The paper evaluates the skill lifecycle rather than just skill generation. Table 2 reports averages of 37.88%, 39.53%, and 40.35% for extraction, extraction plus evolution, and the full lifecycle. Maintenance removes about 46% of entries while increasing aggregate accuracy. That is consequential because an accumulating library must control redundancy. The benefit is aggregate: Java accuracy falls from 41.12% with extraction alone to 38.32% with the full lifecycle, so the result should not be presented as uniformly positive.
Critical weaknesses
- The test-time protocol leaves the evidence boundary and acquisition cost insufficiently specified. Section 4.2 constructs the bank through two passes over the evaluation set, and Appendix C restricts retrieval to the same benchmark. This is a legitimate batch-adaptation setting and can test whether the learned manager handles a new distribution. The current-instance exclusion is also a useful safeguard. However, the second pass uses skill-conditioned trajectories: provenance must account for skills that influenced those trajectories, not only the instances directly used to extract or merge entries. The manuscript does not explain this dependency tracking, so the exclusion cannot be fully audited. Moreover, the reported step savings omit the two preparatory passes and skill-management calls, leaving the practical cost of the reported improvement unresolved.
- The distinctive RL reward design is insufficiently isolated by the experiments. The final objective gates execution improvement by an LLM judgment of whether behavior reflects the skill, but matching behavior can arise from the solver’s existing competence. The paper does not compare the full reward with quality-only, execution-only, or an otherwise matched objective without alignment. Its SFT-versus-RL comparison supports the overall RL stage, but cannot identify the value of the proposed attribution mechanism. Skip, add, and drop receive only rubric feedback, so the method also does not directly validate every maintenance decision through execution. Without these controls and uncertainty estimates, the empirical case for the specific reward design remains incomplete.
Questions for authors
The following requests concern the current method and evidence; their answers could change my recommendation.
- Please clarify the evaluation boundary. Are both preparatory passes performed over the full evaluation set for every method, and when is the bank frozen? Does provenance include skills that influenced second-pass trajectories as well as the instances directly used in extraction, evolution, and merging? Can you report a complementary evaluation in which bank construction and evaluation use disjoint instance pools, or explicitly restrict the claims to adaptation using other target-set instances?
- Please specify the runtime skill-loading mechanism: event-retrieval timing, top-k and thresholds, deduplication, skill token limits, exact injection position, and solver decoding settings. For reverse retrieval during RL, what is K, and can the selected verification task be the source of the extraction trajectory? The distinction affects whether feedback measures reuse beyond the source task.
- What evidence shows that the alignment gate improves downstream performance? Please provide a controlled reward ablation, particularly full reward versus the same objective without alignment, with training data and budget matched. Also state the exact quality-reward weights for skip, add, and drop, and explain how reward scales remain comparable when different operation types occur in a group.
- Are Tables 1 and 2 single evaluation runs or averages? Please report evaluation repeats or paired uncertainty estimates, the Terminal-Bench evaluation denominator, and the sizes of the common solved subsets used for Steps. Confirm that baselines share the preparatory data access, provenance filtering, and prompt budgets. Please also quantify preparation and online retrieval costs; fewer steps on the common solved subset alone do not establish lower total cost.