- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces EvoBranch, an outer optimization framework that wraps existing skill optimizers with branch-specific history, multiple local revisions, global evolutionary proposals, and validation-based beam selection. It evaluates the framework with three optimizers, three backbone models, and four benchmarks, aiming to show that organizing optimization across competing refinement paths improves skill optimization.
Critical strengths
-
Separating native proposal mechanisms from outer search has practical value. The shared adapter design supports SkillOpt, SSO, and SkillGen, while branch-specific issue and revision histories provide a structured alternative to independent restarts.
-
The appendix provides unusually informative diagnostic material. Detailed costs, repeated runs, final skills, transfer results, and benchmark-specific shortcut analyses allow readers to examine the sources and limitations of the reported gains. These observations could support a useful contribution on the behavior of skill optimization.
Critical weaknesses
-
The complete results do not establish the claimed advantage of the search strategy under comparable conditions. The abstract and main text state that longer trajectories and restarts do not reproduce the gains. However, Table 10 reports that a native SkillOpt trajectory truncated to the budget achieves 69.2%, compared with EvoBranch’s 67.3%, using fewer calls (963 versus 1,047) and tokens (5.32M versus 5.36M). Appendix B.3 explicitly acknowledges that equal-compute superiority has not been demonstrated. Table 11 also shows substantial variation: full EvoBranch obtains 50.5% and 67.3% in identical configurations, while removing global evolution yields 70.1% and 63.6%. Furthermore, the SSO comparison changes the selection supervision from a label-free pairwise judge to labeled validation accuracy. Together, these findings prevent attributing the reported gains specifically to better search organization.
-
The strongest improvements largely reflect benchmark shortcuts rather than stronger task-solving skills. Appendix B.5 shows that the SkillOpt–Luna LiveMath improvement from 41.1% to 67.3% comes entirely from exploiting a meta-option that is always correct when present: correct answers increase from 10 to 41 out of 48 such questions, while performance on the other 59 questions decreases from 34 to 31. On ChemLLMBench, always answering “No” achieves 80.4%, and the authors report that every 80.4% entry exhibits this constant behavior; no configuration exceeds that baseline. The highlighted maximum gain of 51.4 percentage points reaches this majority-class result. These findings do not show that the optimization algorithm is unusable, but substantially weaken the evidence for meaningful, reusable skill improvement.
-
Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems is relevant prior work on refining a persistent skill using multiple rollouts and contrastive feedback. Discussing it would help distinguish multiple execution trajectories under one incumbent skill from EvoBranch’s simultaneous refinement paths with branch-specific histories and cross-branch evolution.
Questions for authors
See Weakness