- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Hierarchical Skill Evolution (HSE), a framework for extracting reusable procedural knowledge from coding-agent trajectories without updating the underlying coding model. The main idea is to aggregate experience progressively: individual trajectories are first converted into task-level evidence, evidence from the same repository is then summarized into repository-conditioned candidates, and recurring patterns across repositories are finally consolidated into transferable procedural or recovery skills. A writer–evaluator loop controls candidate construction and promotion.
The paper evaluates HSE on SWE-bench Verified and DeepSWE. The static setting is the most convincing result: ten skills evolved from SWE-Gym are frozen and transferred to the target benchmarks, where they improve GLM-5.2 under both Pi and Claude Code. The same skill pack also improves GPT-5.5 and Opus 4.8 under Pi. The paper further reports target-benchmark adaptation, where the skill library is repeatedly updated using trajectories collected from the target benchmark.
Overall, I think the paper studies an important problem and the static-transfer results are promising. My main concern is that the experiments do not yet cleanly establish that the proposed hierarchical evolution mechanism, rather than more general trajectory aggregation and repeated target-set experience, is responsible for the strongest gains.
Critical strengths
The strongest aspect of the paper is that the same compact skill library improves several coding-agent configurations without modifying the solver. This is practically attractive and avoids conflating skill learning with model fine-tuning.
I also appreciate that the authors evaluate more than aggregate performance. The procedural/recovery decomposition, cross-harness transfer, and controller/hierarchy ablations are useful attempts to understand why the method works rather than only reporting a higher SWE-bench number.
Critical weaknesses
- The target-adaptation experiment should be interpreted as adaptation rather than generalization. The largest improvements in the paper come from the eight-round target-benchmark adaptation experiment. However, each round performs a complete rollout over the target benchmark and then updates the skill library from those trajectories. The resulting library is subsequently evaluated on the same complete target task set. It is useful that verifier outcomes are not exposed to the writer or evaluator, but the trajectories themselves still contain substantial information about the target tasks: issue descriptions, inspected code, attempted patches, tests, and failure modes. After several rounds, the system has therefore seen a considerable amount of target-task experience. I think this is a valid test-time or continual-adaptation experiment, but it is qualitatively different from the static transfer result. In its current form, it is difficult to tell how much of the very large Gate-8 improvement comes from reusable skill evolution versus repeated exposure to the same target task population. A cleaner experiment would construct skills from one subset of target instances and evaluate on a disjoint subset. A leave-one-task-out version would also be informative.
- The hierarchy ablation does not cleanly isolate the value of hierarchical aggregation.
The main methodological claim is that repository-level aggregation provides an important intermediate abstraction between individual trajectories and cross-repository skills. However, the “w/o Hierarchical Aggregation” experiment randomly shuffles task trajectories across repositories.
I am not convinced this is the right counterfactual. Random shuffling destroys meaningful repository structure and creates artificial groups, so a performance drop is not particularly surprising. What I would really like to know is whether the proposed three-stage hierarchy is better than a flat but otherwise matched aggregation strategy. For example, directly aggregate L1 task evidence into transferable skills using the same trajectories, writer, evaluator, and final skill budget. Without such a comparison, Table 4 shows that incorrect grouping hurts, but does not fully establish that repository-conditioned aggregation is necessary. - The controller contribution is still difficult to attribute. The evaluator uses manually defined evidence features and fixed thresholds, with relatively simple weight updates over time. The writer also evolves through constrained rule changes. Yet the paper ablates “controller updates” only as a single block. This leaves an important question unanswered: is the gain coming from adaptive writer behavior, adaptive evaluator weights, or simply from a reasonably designed fixed set of aggregation rules? I would like to see the two components separated. A fixed-writer/adaptive-evaluator variant and an adaptive-writer/fixed-evaluator variant would make the methodological contribution much clearer. Sensitivity to the evaluator thresholds would also help, since these thresholds directly determine which skills enter the library. A smaller issue is statistical reporting. The main GLM experiments use three runs, which is helpful, but some differences against the strongest baseline are modest—for example, less than one percentage point on SWE-bench Verified in some settings. A paired per-instance analysis or bootstrap confidence interval over the improvement would make these comparisons easier to interpret. The GPT-5.5 and Opus 4.8 results would also benefit from uncertainty reporting.
Questions for authors
1.Can the authors evaluate target adaptation using disjoint adaptation and evaluation task pools, or otherwise ensure that a task cannot benefit from skills influenced by its own earlier trajectories?
2.Can the authors add a matched flat-aggregation baseline that removes the repository level without randomly corrupting repository membership?
3.Can the writer update and evaluator update be ablated separately? How sensitive are the results to the evaluator weights and the fixed accept/revise thresholds?
4.Are the GPT-5.5 and Opus 4.8 results based on one run? If so, can paired uncertainty or additional runs be reported?