- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Skill-Forest targets capability-matched Agent Skill optimization, where the same relatively weak backbone both executes tasks and optimizes Skills using only outcome-level supervision, without relying on a stronger optimizer. The method brings the randomize-and-aggregate principle of Random Forests into textual Skill optimization: starting from the same initial Skill, it runs K K independent Skill trees, each with its own sampled training-validation split; within each tree, patch-based branching preserves alternative update paths. At the forest level, a shared expansion budget, validation-gated pruning, gain-prioritized node expansion, early rejection, and Root validation reuse control construction cost. Tree-local winners are then aggregated into one deployable Skill via Select or Merge. Experiments cover five benchmarks spanning single-round QA, multi-round tool use, and persistent environment interaction. Skill-Forest outperforms baselines such as SkillOpt, GEPA, and Trace2Skill overall and reports substantial rollout savings.
Critical strengths
1.Reasonable and well-motivated direction: It extends Skill optimization from a single textual-gradient path to multi-tree search with independent data views, directly targeting the high variance of single-path optimization. The idea is natural and has practical engineering value.
2.Relatively complete efficiency design: The shared expansion budget, validation-gated pruning, gain-prioritized expansion, upper-bound early rejection, and Root evaluation reuse jointly reduce forest-construction rollouts. The paper also provides a fairly detailed cost breakdown.
3.Good experimental coverage: The five benchmarks span single-round direct chat, multi-round tool use, and persistent environment interaction. Under the capability-matched setting, Skill-Forest overall outperforms several representative baselines without relying on a stronger optimizer.
Critical weaknesses
1.Internal tension in the method: The paper acknowledges that tree-local validation gains are not comparable across trees, yet uses them to rank nodes in a global priority queue for allocating expansion budget. This may divert budget to trees with easier validation subsets or lower initial scores rather than to genuinely promising search directions.
2.Concerns about data usage and selection fairness: Each tree uses only about 50% of the training data, while baselines mostly use the full training set. Final Select/Merge also scores winners on the complete training split, without an independent selection set separate from the tree splits, raising the risk of selection overfitting.
3.Insufficient evidence strength: Main results lack cross-run standard deviations and significance tests. The efficiency comparison only counts forest construction and does not fully include aggregation/Merge rollouts and LLM call costs. It remains unclear whether the 2–3 pp gains are stable and how much is attributable to the forest structure itself.
Questions for authors
see Critical Weaknesses