- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes EVOLUTION LEARNING for turning failures in execution into reusable agent skills and internalize the knowledge:
1.SKILL EVOLUTION After a failed rollout, an editor finds the failure evidence and proposes new rule change. Then a deterministic Harness checks. The revised skill gets a new rollout. Then the first successful version is certified. And the failed parent is also kept. It gives a failed-to-success transition.
-
SKILL TRANSFER put certified skills into a SkillBank. A retriever selects skills. If it fails, a repair loop may revise a copy without changing the shared bank.
-
SKILL INTERNALIZATION / EvoOPD trains a skill-free student model with the on-policy distillation. It generate actions without a skill. Two views of the base model score the same response under the certified skill and the failed parent. The positive view is the distillation target. Teacher disagreement is measured by JS distance. The disagreement assigns higher weight to action whose scores change after repair.
Critical strengths
- The proposed failed-to-success transition is useful and well-motivated. Keeping the failed parent and using it as the positive view to weight the distillation target. It is a clear way to exploit verified repairs without imitating the failure. It connects environment-verified skill repair with the knowledge internalization. It does not stop at retrieval or prompt-level skill reuse.
- The improvement on ALFWorld skill-free internalization is good. The frozen base model reaches 7.86% and 4.48% on Seen and Unseen. The trained student reaches 54.52% and 64.18%. This suggests that certified skill supervision can transfer into model behavior when skills are removed at test time.
Critical weaknesses
1.the transfer story is inconsistent across benchmarks. SKILL EVOLUTION and SKILL TRANSFER are single runs, while the main internalization comparison against Vanilla AgentOPD has confidence intervals that include zero. At the same time, WebShop frozen-bank transfer barely improves success and lowers partial score, so the “certified skills transfer” claim is much weaker than the ALFWorld results suggest. This is consequential because the paper’s main contribution is a lifecycle that is supposed to generalize across source repair, transfer, and internalization; the evidence is strongest only for ALFWorld and for the final distillation stage.
2.The novelty and attribution concern. EvoOPD is one on-policy distillation with a positive teacher plus a disagreement-based action weighting from a failed skill view. The paper does not convincingly separate this from additional teacher compute, from positive-only AgentOPD, or from simpler weighting schemes. The theoretical analysis only bounds top-K truncation error and does not explain why the JS-distance weighting should improve downstream success. Since the central methodological contribution is the weighting rule, and the current evidence does not establish that it is the source of the gain rather than extra teacher scoring or other protocol differences.
Questions for authors
- Can the authors provide at least three seeds for SKILL EVOLUTION and SKILL TRANSFER, with paired confidence intervals? Would the main claims survive if the single-run results were treated as point estimates only?
- Can EvoOPD be compared to Vanilla AgentOPD under matched teacher-scored tokens or matched teacher compute? How much of the gain remains when the positiveonly baseline receives the same teacher budget?
- What is the false-positive rate of single-rollout certification? Can certification require multiple successful rollouts, and how does that change the bank size and downstream results?
- Can the authors isolate the JS distance weighting from simpler alternatives, e.g., uniform weights, binary disagreement weights, KL instead of JS, no square root, no clipping, or random weights with the same mean? Which design choices are necessary for the reported gain?
- WebShop internalization. Why is skill-free internalization not evaluated on WebShop? Is the lifecycle expected to hold there, or is WebShop a negative result for transfer?
- Stage-specific pools. Internalization uses 18-revision certificates while transfer uses a 5-revision bank. How much of the internalization advantage depends on the deeper revision budget or on the different training subset?
- Baseline fairness. How close are the adapted baselines implementations to their original mechanisms? Would stronger original implementations change the source-task ranking?
6 comments
Response to Reviewer SKYz (2/4) — W2 continued; Q1–Q3: Replication, budgets, and certification
the budget-matched uniform weights reach 58.46±1.14%, a difference of +5.72 percentage points; smoothed KL and no-square-root come in at 62.19% and 62.44%. Budgets are topped up through extra updates, so the effect of the changed update count still needs separate control.Under default parameters, removing clipping is equivalent to the current design and is one single run.
Q1 — Multiple Seeds and Confidence Intervals
In the original submission, source evolution and frozen transfer were evaluated with a single run, which could not establish stability. The three internalization training seeds and task bootstrap from that run did not measure variation across independent evolution and transfer runs. We have now completed three independent evolution and frozen-transfer runs, as reported below.
We fix 3 independent sets of editing and model-sampling conditions to rebuild the ALFWorld skill library; we compare evolution results on the 3,553 source tasks, plus frozen-skills versus no-skills performance on the 134 unseen tasks:
| Scenario | Method successes | Control successes | Mean diff and 95% interval / points |
|---|---|---|---|
| ALF source: skill evolution vs Reflexion | 1650/1640/1660; each n=3553 | 1554/1548/1560; each n=3553 | +2.70 [+2.42, +2.98] |
| ALF unseen env: frozen skills vs no skills | 47/45/50; each n=134 | 7/8/7; each n=134 | +29.85 [+24.29, +35.41] |
Source evolution improves over Reflexion by 2.70 percentage points with a 95% interval of [+2.42, +2.98]; the frozen-skill gain on unseen tasks is +29.85 points with a 95% interval of [+24.29, +35.41]. Intervals are computed from paired differences across the three runs.
Q2 — Gains Under Matched Teacher Budget
In the original submission, the comparison did not match teacher compute. EvoOPD used 617,781–635,809 teacher-scored token positions, while Vanilla used 432,634–442,302, a gap of roughly 1.43×. That comparison could not isolate the influence of additional teacher compute. We have now completed the budget-matched comparison reported below.
With the same training source, initialization, and optimization settings, we give the positive-only baseline extra updates so that each seed matches the teacher budget of the current design; seeds 42, 43, 44 are paired one-to-one and evaluated on the 134 unseen tasks:
| Condition | Unseen-env successes | Success rate mean±sample SD /% | Teacher-scored token positions |
|---|---|---|---|
| JS distance weighting / current design (the EvoOPD setting of paper Table 4) | 83/91/84; each n=134 | 64.18 ± 3.25 | 617,781/623,995/635,809 |
| No clipping / α=1 (mathematically equivalent to the current design, same run) | 83/91/84; each n=134 | 64.18 ± 3.25 | 617,781/623,995/635,809 |
| Uniform weights / budget-matched (positive-only distillation spends the full budget) | 77/80/78; each n=134 | 58.46 ± 1.14 | ≈617,781/623,995/635,809 (topped up via extra updates) |
The current design reaches 64.18±3.25%; after budget matching, uniform weights reach 58.46±1.14%, a difference of +5.72 percentage points, with a 95% interval paired over 3 seeds of [+0.37, +11.07].
Q3 — Single-Rollout Certification False Positives
A single certification only shows that the one run succeeded; it does not guarantee the skill still succeeds on repeated use. We froze 100 originally certified candidates in each of ALFWorld and WebShop, fixing skills, model, and environment seed=42; each candidate then received 2 additional screenings plus 5 sampled audits:
| Environment | Frozen original candidates | Retained after 2 screenings | All candidates audit success | Retained candidates audit success | Screened out but passed all 5 audits |
|---|---|---|---|---|---|
| ALFWorld | 100 | 88/100 | 454/500 (90.8%) | 418/440 (95.0%) | 2 |
| WebShop | 100 | 86/100 | 455/500 (91.0%) | 411/430 (95.6%) | 4 |
| Environment | All: failed ≥1 of 5 audits | Retained: failed ≥1 of 5 audits | Candidate-level 95% interval: overall success rate |
|---|---|---|---|
| alfworld | 24/100 | 14/88 | 86.4%–94.6% |
| webshop | 21/100 | 11/86 | 86.4%–95.0% |
After two screenings we retain 88 and 86 candidates respectively; audit success rates among retained candidates are 95.0% and 95.6%, versus 90.8% and 91.0% across all candidates. On ALFWorld 24/100 candidates and on WebShop 21/100 candidates fail at least once in the 5 audits.
This is retest instability under temperature 0.6, not the theoretical false-positive rate of the original temperature-0 certification. Post-screening differences embed a selection effect. The campaign completed 1,400 runs, 10,374 model calls, and 18,304,462 tokens; in addition, 4 candidates were replayed successfully at temperature 0 solely for pipeline checking.
Using the full-bank original candidate counts as the base, adding two screenings per candidate yields these retention counts and extra interactions:
Response to Reviewer SKYz (3/4) — Q3 continued; Q4–Q5: Screening, ablations, and WebShop
| Environment | Total candidates | Retained | Retention rate | Extra screening runs |
|---|---|---|---|---|
| ALFWorld | 1160 | 1021 | 88.00% | 2320 |
| WebShop | 3066 | 2637 | 86.00% | 6132 |
The upstream pool for the ALFWorld transfer library is the 1,160 candidates from five revisions, not the 1,644 candidates used in the internalization stage.
The success rates and WebShop score of the original rule library arevalues as reported in the paper.
We apply additional screening to all 653 skills, form the retained skills into a strictly screened library, and compare it against the original rule library with the same model and tasks:
| Env | Tests | Original rule library (653 skills) | Strictly screened library | Change / points | Improved / regressed tasks | WS score /100 |
|---|---|---|---|---|---|---|
| ALFWorld | 274 | 100/274 (36.50%) | 92/274 (33.58%) | -2.92 | 7/15 | N/A |
| WebShop | 500 | 114/500 (22.80%) | 105/500 (21.00%) | -1.80 | 6/15 | 46.04→43.20 |
The strictly screened library achieves an ALFWorld success rate of 33.58% and a WebShop rate of 21.00% with score 43.20; the two screenings trade coverage against retest reliability.
Q4 — Comparison With Simpler Weighting Schemes
We first conducted a local diagnostic on a single fixed batch, then completed the three-seed training ablations reported below. The single-batch diagnostic covers 8 tasks, 171 rounds, and 1,028 action tokens:
| Weight rule | Mean weight | Weighted loss | Logits gradient norm | Gradient cosine vs. original rule |
|---|---|---|---|---|
| Uniform weights set to 1 | 1.000000 | 0.524793044 | 0.009927163 | 0.995870870 |
| Original √JS weights | 1.043331 | 0.597175452 | 0.011107898 | 1.000000000 |
| Fixed at the full-batch mean | 1.043331 | 0.547532750 | 0.010357315 | 0.995870870 |
| Shuffled within same-length action segment groups | 1.043331 | 0.554271215 | 0.010431625 | 0.991978522 |
Same-mean shuffling exchanges weights within groups of action segments of the same length; gradients are computed analytically only in the full-vocabulary logits space, with no parameter updates. With the default α=1 and cap 2, clipping and not clipping are fully identical; the wording in the paper appendix that clipping takes effect in 6–11% of updates needs clarification.
We fix the training source, initialization, optimizer, and schedule, changing only the weight rule; each condition uses 3 seeds and is evaluated on the 134 unseen tasks:
| Condition | Unseen-env successes | Success rate mean±sample SD /% | Versus uniform weights / points | Teacher-scored token positions |
|---|---|---|---|---|
| Span JS distance / current design (paper Table 4) | 83/91/84; each n=134 | 64.18 ± 3.25 | +3.23 | 617781/623995/635809 |
| Same scoring pipeline / uniform weights (successes equal Vanilla in paper Table 4) | 85/84/76; each n=134 | 60.95 ± 3.68 | 0.00 | 432,634–442,302 |
| Same-mean randomly shuffled weights | 81/88/79; each n=134 | 61.69 ± 3.53 | +0.75 | 621600/626400/618200 |
| Same-mean constant scaling | 86/83/78; each n=134 | 61.44 ± 3.02 | +0.50 | 623500/622100/629400 |
| Binary disagreement weights | 83/89/80; each n=134 | 62.69 ± 3.42 | +1.74 | 618900/621300/635000 |
| Smoothed KL weights | 86/90/81; each n=134 | 63.93 ± 3.37 | +2.99 | 628500/619800/630700 |
| JS without square root | 82/90/82; each n=134 | 63.18 ± 3.45 | +2.24 | 626200/620500/633100 |
| Per-token distance weights | 85/90/80; each n=134 | 63.43 ± 3.73 | +2.49 | 625500/617300/631200 |
| No clipping / α=1 (mathematically equivalent to the current design, same run) | 83/91/84; each n=134 | 64.18 ± 3.25 | +3.23 | 617781/623995/635809 |
| Cap 1.25 / α=1 | 82/88/81; each n=134 | 62.44 ± 2.83 | +1.49 | 622300/620700/628100 |
| α=0.5 / cap 2 | 82/90/78; each n=134 | 62.19 ± 4.56 | +1.24 | 620900/617600/632500 |
| α=2 / cap 2 | 84/87/78; each n=134 | 61.94 ± 3.42 | +1.00 | 627400/624900/630600 |
In the table, the current-design and uniform-weight rows correspond to the existing results in paper Table 4; removing clipping and the current design are mathematically equivalent and constitute the same run. This table tops up no extra teacher budget, so it must not be conflated with the budget-matched conditions of W2 / Q2.
Smoothed KL reaches 63.93%, no square root 63.18%, and per-token 63.43%, all close to the current design. These comparisons show that simple alternatives can also come close to the current design, so no single specific design choice can be taken as necessary.
Q5 — WebShop Internalization and Negative Transfer
The paper reports that frozen-transfer success on WebShop rises from 22.60% to 22.80% while partial score falls from 53.83 to 46.04. Limited direct transfer does not prove that internalization training would fail, and the extra test-time compute used by local repair should also be reported separately.
Response to Reviewer SKYz (1/4) — W1 and W2: Transfer, internalization, and attribution
Thank you for the detailed and thoughtful review. We appreciate your close examination of transfer, gain attribution, and the experimental controls. We address each concern below.
W1 — Transfer Story Inconsistent Across Benchmarks
We accept this criticism: the current evidence does not establish that the full lifecycle holds stably across different environments. After re-analyzing 12 evaluation receipts, both cross-seed paired intervals of EvoOPD over Vanilla include zero:
The paper reports WebShop frozen-transfer success rising from 22.60% to 22.80%, while partial score falls from 53.83 to 46.04. Local repair consumes extra test-time compute and must not be counted as direct transfer gains. We will present source evolution, frozen transfer, and internalization separately and narrow the cross-environment claim.
We independently rebuild the ALFWorld and WebShop skill libraries, comparing source evolution, frozen transfer, and local repair respectively; every condition uses 3 independent runs:
ALFWorld source evolution improves over Reflexion by 2.70 percentage points; the WebShop frozen-transfer difference is +0.13 points with interval [-1.61, +1.88] including zero, while local repair adds +7.60 points. Intervals are computed from paired differences across the three runs.
On WebShop we train positive-only distillation and EvoOPD separately, 3 seeds each, against the frozen base model; at test time skills, teacher, and retrieval are removed, and each training seed uses 822 unique tasks, 824 slots, and 103 updates:
The EvoOPD success rate is 25.00±0.53%, versus 24.53±0.61% for positive-only distillation, a difference of +0.47 points; official scores are 47.50 and 47.17 respectively.
W2 — EvoOPD Novelty and Gain Attribution
The existing top-K truncation error bound characterizes a mathematical property of truncation; it cannot prove that JS-distance weighting improves downstream success rates. Nor has any training-level ablation simultaneously ruled out extra teacher compute and alternative weightings, so the gain cannot be attributed solely to JS weighting.
We fix the training source, initialization, and optimization settings, and add controls whose per-seed teacher budgets match those of the current design; matched controls top up their budgets through extra updates. Seeds 42, 43, and 44 are paired one-to-one, compared on the 134 unseen tasks:
JS weighting reaches 64.18±3.25%;