- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SkillVine replaces single-path skill optimization with a search over a graph of skill versions. A trunk follows the standard linear update, and at each step branches grow from historical versions selected by validation accuracy, expansion count, and complementarity with the trunk. When the trunk update is rejected, branches accept their edits one at a time. Across five benchmarks and two models, SkillVine achieves the best test score in nine of ten settings.
Critical strengths
S1. Framing skill evolution as search over a version graph is a natural and general extension of linear evolution. Keeping earlier versions and allowing new branches from them addresses a clear failure mode of linear methods, which stall when successive updates are rejected. The framework can also be built directly on top of existing linear optimizers.
S2. The evaluation covers five benchmarks and two models, and SkillVine achieves the best test score in nine of the ten settings. The ablations and the evolution-path analysis further show that the choice of where to branch affects the result.
Critical weaknesses
W1. The comparisons do not fully control for compute. Best-of-3 and Random try the same number of updates per step as SkillVine. However, when the trunk update is rejected, SkillVine tests each edit separately on the validation set, so it runs more validation evaluations in total. The paper acknowledges this extra cost but reports cost only for its own variants, not for any baseline. The paper itself also notes that producing more candidates increases the chance of finding a better version. It is therefore unclear how much of the gain comes from branching rather than from these extra evaluations.
W2. AGBU relies on an assumption that is not well supported. It assumes that if a training batch leads to harmful edits for the trunk, it will also lead to harmful edits for the branches. But each branch proposes its own edits from a different skill version, so this need not hold. The paper supports the assumption with only two selected examples. The rationale for using the trunk outcome to decide how branches are updated is therefore weak.
Questions for authors
Q1. Please report the training rollouts and validation evaluations used by every method, and compare SkillVine with methods such as Best-of-N under the same number of validation evaluations. This would show whether the gains come from branching or from the additional evaluations.
Q2. AGBU assumes that a batch harmful to the trunk is also harmful to the branches. How was this conclusion reached, and what evidence supports it?
Q3. What motivates the form of the selector score, and how were λexp = 0.08, λcomp = 0.5, and β = 1.6 chosen? The paper reports no sensitivity analysis for these coefficients. It also states that mixing discovery and refinement selectors yields more consistent gains, but reports no comparison with either selector alone.
8 comments
Response to Weakness-2 and Question-2
Thank you for this helpful comment!
We analyzed the w/o AGBU runs to measure the correlation between the trunk’s acceptance or rejection and performance drops in both branches at the same training step. The results are shown below. “Both ↓” means that both branch candidates have lower validation scores than their own parents.
| Model | Trunk | Both ↓ | Not both ↓ |
|---|---|---|---|
| DeepSeek-v4-Flash | Accepted | 36.4% | 63.6% |
| DeepSeek-v4-Flash | Rejected | 79.1% | 20.9% |
| Qwen3.8-Flash | Accepted | 50.0% | 50.0% |
| Qwen3.8-Flash | Rejected | 83.9% | 16.1% |
The table shows that both branches are much more likely to lose performance when the trunk update is rejected. This shows a strong association and supports using trunk acceptance or rejection as a signal for AGBU.
The reason is that updates are largely guided by the training trajectories. Even when the trunk and branches have different parents, they can learn the same skill from the same training batch. For example, one parent may lack that skill, so the update adds it. Another may have an older version, so the update replaces it with the same new skill. Both updates therefore lead to the same skill content.
Response to Question-3
Thank you for this helpful suggestion.
Equation (3) considers three factors:
- Validation accuracy favors strong parent nodes. Improving an already strong version can help raise the best score found during evolution.
- Complementarity encourages branches to explore capabilities that the trunk lacks. This helps branches follow different paths rather than spend the budget on similar directions.
- The exploration bonus gives less-expanded nodes more opportunities. When validation scores and complementarity repeatedly favor the same nodes, this bonus reduces concentration on a few paths and keeps other versions available for exploration.
Together, these terms guide the search toward promising but different evolution paths. Compared with Best-of-3, SkillVine avoids concentrating all updates on one current version. Compared with Random, it uses both node performance and complementarity with the trunk to guide exploration.
We set the current coefficients based on observations from actual runs to keep each term at an appropriate scale and ensure that variations in its value meaningfully distinguish candidate nodes.
We tested how and in Eq. (3) affect performance on SpreadsheetBench using DeepSeek-v4-Flash. We changed one coefficient at a time while keeping the others fixed. The results are shown below:
| Test Score | ||
|---|---|---|
| 0.001 | 0.5 | 69.75 |
| 0.08 | 0.5 | 71.53 |
| 1 | 0.5 | 66.90 |
| 5 | 0.5 | 63.35 |
| 0.5 | 0.05 | 65.12 |
| 0.5 | 0.25 | 65.84 |
| 0.5 | 0.5 | 69.75 |
| 0.5 | 1 | 67.97 |
The table shows that performance tends to drop when increases or decreases. These changes make the selector rely more on node scores and exploration rewards, and less on complementarity. This highlights the importance of complementarity in node selection.
We also varied the gain reward and regression penalty coefficients in Eq. (4), while keeping . We used the existing SkillVine run on SpreadsheetBench with DeepSeek-v4-Flash and recalculated branch node selection at each training step. Each calculation used only the nodes available at that step. We then checked how many steps kept the original parent choices before the first difference. The results are shown below:
| Varied coefficient | Values | Matching prefix | First divergence |
|---|---|---|---|
| Gain reward | 0, 0.001, 0.1, 0.25 | 3/8 | Step 4 |
| Gain reward | 0.5, 0.8, 1, 2 | 7/8 | Step 8 |
| Gain reward | 5, 10, 100, 1000 | 8/8 | None |
| Regression penalty | 0, 0.001, 0.1, 0.5 | 3/8 | Step 4 |
| Regression penalty | 0.8, 1, 2, 5, 10 | 7/8 | Step 8 |
The results show that increasing either coefficient puts more weight on complementarity. The selector then makes the same parent choices as the original run for more steps.
To address your second concern, we tested two alternatives to the mixed selectors: two refinement selectors and two discovery selectors. We used the same compute budget for comparison.
Using two refinement selectors corresponds to our budget-matched Best-of-3 experiment. The results are shown below:
| Model | Method | Spreadsheet | LiveMath | OfficeQA | SearchQA | TABVERSE |
|---|---|---|---|---|---|---|
| DeepSeek-v4-Flash | Best-of-3 | 63.70 | 50.00 | 45.27 | 71.50 | 63.88 |
| SkillVine | 71.53 | 53.77 | 55.41 | 74.43 | 67.35 | |
| Qwen3.8-Flash | Best-of-3 | 66.90 | 34.91 | 44.59 | 70.93 | 59.80 |
| SkillVine | 67.97 | 39.62 | 54.05 | 72.40 | 63.67 |
For two discovery selectors, we tested the five model–benchmark combinations where parent choices differ from those in the original mixed-selector runs. The results are shown below:
| Model | Selector | OfficeQA | SearchQA | TABVERSE |
|---|---|---|---|---|
| DeepSeek-v4-Flash | Discovery only | — | 70.29 | 56.33 |
| Mixed (SkillVine) | — | 74.43 | 67.35 | |
| Qwen3.8-Flash | Discovery only | 50.68 | 70.71 | 60.20 |
| Mixed (SkillVine) | 54.05 | 72.40 | 63.67 |
The results show that mixed selectors outperform refinement-only selection in all ten combinations and discovery-only selection in all five tested combinations. This supports using discovery and refinement together rather than either alone.
Response to Weakness-1 and Question-1
We thank the reviewer for recognizing the value of our version-graph search, its compatibility with existing linear optimizers, and the breadth and strength of our experimental results, and for the helpful suggestions.
The table below shows each method’s ratio of total validation rollouts to total training rollouts in the original Table 1:
These ratios vary across benchmarks because we follow SkillOpt-Lite’s dataset splits, which use very different training and validation set sizes. For example, SearchQA and SpreadsheetBench both use 40 training examples per update, but each validation pass uses 200 examples on SearchQA and only 39 on SpreadsheetBench. Validation therefore costs more relative to training on SearchQA.
The original experiments did not enforce equal validation budgets for the following reasons:
To make the source of our gains clearer, we added budget-matched experiments for all baselines except Linear, which remains a naive reference. Its budget-matched counterpart is effectively budget-matched Best-of-3. At each step, these baselines match SkillVine’s validation evaluation count while keeping their own update acceptance rules. All methods use exactly the same training-batch construction procedure. For Hermes-SE, we retain its node-selection and path-evolution logic within SkillOpt-Lite. Implementation details unrelated to path evolution, such as the prompts used to generate skill updates, remain the same as SkillVine’s. This allows us to compare search strategies under the same setup.
The results are as follows:
SkillVine still performs best with the same validation budget. This suggests that its gains come from using the budget more effectively, rather than simply running more evaluations.