- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the problem of budget allocation during skill self-evolution. It proposes VeriTree, which maintains a posterior state and dynamically decides at each iteration whether to spend additional budget on validating an existing skill candidate or on generating a new skill version. Through comparisons with several representative baselines, the authors report that VeriTree can obtain high-accuracy skills at substantially lower cost.
Critical strengths
- Treating verification as a budget is an important research direction, and I strongly agree with this perspective.
- VeriTree dynamically chooses between further validating existing candidates and generating new skill versions based on posterior estimates and the remaining budget. I find this adaptive allocation mechanism conceptually interesting.
Critical weaknesses
- The current experimental evaluation is insufficient to fully establish the effectiveness and generality of VeriTree. The paper jointly compares accuracy and cost across methods, but VeriTree does not appear to consistently achieve Pareto dominance over the baselines. A more informative evaluation would therefore compare accuracy under a fixed cost budget, or cost required to reach a fixed accuracy level, and ideally report accuracy–cost curves for different methods to characterize their marginal returns as the budget increases. In addition, four of the five evaluated tasks are knowledge-oriented QA tasks, while only one involves a more operational or agentic setting, leaving it unclear whether the method generalizes to broader agent scenarios. The current ablation studies also do not sufficiently explain where the accuracy improvements come from.
- The methodological contribution appears relatively incremental and primarily focused on improving efficiency in a specific optimization setting. VeriTree shows a clear relative reduction in optimization cost, but it does not demonstrate a consistent advantage in final task accuracy, and the absolute cost savings in some settings appear modest. As a result, I am not yet convinced that the empirical gains are strong enough to establish a sufficiently substantial contribution. Stronger evidence of generality, larger absolute savings, or a clearer qualitative advantage over existing skill-optimization pipelines would help strengthen the case.
Questions for authors
- Given the rapid improvement of foundation-model capabilities, how do the authors view the long-term applicability of skill self-evolution? In particular, do skills optimized by VeriTree transfer effectively to other models, or are they highly specific to the model used during optimization?
- Could the authors plot the prompt-token length of the selected skill version over the course of optimization and provide the full text of at least one final optimized skill? This would help clarify whether the gains mainly come from more precise and structured rules, or from accumulating increasingly detailed descriptions of corner cases. This question is also related to my concern about understanding the source of the accuracy gains.
- VeriTree appears to provide a substantial cost advantage, but in Configuration 1 on the SSB task, it reduces cost by roughly 90% while achieving an accuracy about 10 percentage points lower than SkillOpt. If the budget allocated to VeriTree is increased, can it continue improving and eventually approach or match the accuracy achieved by SkillOpt?
3 comments
Q2: SSB in Config. 1 — can VeriTree approach SkillOpt with a larger budget?
Yes. SSB has a pool of only 120 questions and rounds, so the default budget is very small. We increased the number of epochs ( is unchanged, since is already capped at ):
| VeriTree budget | Rounds | Test acc. | Cost (RMB) |
|---|---|---|---|
| 1× (, paper) | 12 | 0.4821 | 2.72 |
| 2× () | 24 | 0.5357 | 5.31 |
| 4× () | 48 | 0.5714 | 10.90 |
| 8× () | 96 | 0.5893 | 21.62 |
| SkillOpt (paper) | – | 0.5857 | 23.71 |
Accuracy increases monotonically with the budget and matches SkillOpt at a slightly lower cost. We also note that SkillOpt's +23.93 pp gain in this cell has a high seed variance (0.5643 ± 0.0255, Table R2) and does not recur in Config. 2 (+1.79 pp).
We thank the reviewer again for the thoughtful suggestions, which have substantially strengthened our evaluation. We hope the new matched-budget analyses, agentic tasks and transfer results address the soundness concerns, and we would be grateful if the reviewer would reconsider the score.
Thank you for the additional experiments and for sharing the results. As a follow-up to Q2 in my original review, could you plot the prompt token count of the selected skill at each optimization iteration and share the full text of one final optimized skill?
We thank the reviewer for all concrete suggestions.
W1-a: No consistent Pareto dominance; accuracy at fixed cost, cost at fixed accuracy, and accuracy–cost curves.
First, from the existing Table 1: since VeriTree has the lowest cost in all 15 cells, it is Pareto non-dominated in 15 of 15 cells. It strictly dominates (higher or equal accuracy at lower cost) SkillOpt in 9 of 15 cells and GEPA in 8 of 15 cells, and no baseline dominates VeriTree in any cell.
Second, following the reviewer's suggestion, we added:
W1-b: Four of the five tasks are knowledge QA; generality to agentic settings.
We added two multi-step agentic tasks under Config. 2 (target model Qwen3.6-35B-A3B): ALFWorld (embodied text navigation and manipulation) and τ2-bench retail (multi-turn tool use with a simulated user). The task-success signal is binary, so VeriTree's Beta posterior applies unchanged.
Agentic rollouts are long and expensive, so the evaluation budget matters even more. VeriTree's relative cost (31% and 34% of SkillOpt's) is in line with the main results.
W1-c: The ablations do not explain where the accuracy comes from.
Table R1 in the General Response decomposes the gain on the same tree and budget: retaining versions contributes +1.1 to +3.8 pp over the chain, the posterior-separation switching rule adds a further +0.5 to +1.8 pp over the best hand-tuned fixed ratio, and both arms are necessary.
W2: Incremental contribution; no consistent accuracy advantage; limited absolute savings.
Q1: Long-term value of skill self-evolution; do skills transfer across models?
As base models improve, generic gains on high-baseline tasks shrink, but skills increasingly encode what models cannot know a priori: tool conventions, organization-specific procedures and recurring failure modes. Moreover, stronger target models make each rollout more expensive, which makes budget-aware verification more relevant, not less. We applied skills optimized in Config. 2 (target Qwen3.6-35B-A3B) zero-shot to the two other targets; Δ is relative to each target's own S0.
Transferred VeriTree skills retain roughly 60–80% of the in-model gain and transfer at least as well as SkillOpt's skills.