- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates a potential weakness of skill–policy co-evolution: while an agent's policy and skill bank improve together during training, the resulting policy may become overly dependent on the exact skill bank it was trained with. As skills are modified, removed, or replaced after deployment, this dependence can lead to performance degradation. The authors first provide a theoretical analysis of this problem through an idealized cooperative acceptance rule, then empirically examine how sensitivity to skill modifications changes throughout training. They introduce CODA, an evaluation protocol that measures robustness under different forms of skill drift, including meaning-preserving rewrites, incomplete instructions, and misleading skills. Experiments cover four co-evolution methods across five benchmarks. The results show that performance on modified skill banks can be substantially lower than performance on the original bank, suggesting that conventional evaluations may overestimate how robust co-evolved agents are after deployment.
Critical strengths
S1. Important and relevant problem. The paper raises an interesting concern about whether agents trained with evolving skill banks can still use those skills reliably after deployment. This is particularly relevant for systems where skills may change more frequently than the underlying policy.
S2. Evaluation across different co-evolution frameworks. The experiments cover four methods and five benchmarks, including interactive and reasoning tasks. The consistent performance degradation across these settings provides useful evidence that sensitivity to skill drift is not limited to one particular implementation.
S3. Interesting analysis of how fragility changes during training. Rather than evaluating only the final policy, the authors examine intermediate checkpoints and show that sensitivity to skill modifications often increases as training progresses. This provides a useful perspective on the relationship between improving task performance and robustness.
S4. Practical evaluation protocol. CODA introduces several complementary measurements of skill robustness, including consistency under meaning-preserving edits and resistance to corrupted skills. The use of paired evaluations and an unperturbed noise baseline makes the proposed protocol more informative than simply reporting performance drops.
Critical weaknesses
W1. The theoretical analysis relies on strong assumptions. Proposition 1 assumes that the selected skill bank maximizes the current policy's performance over a candidate set covering all considered perturbations. This is unlikely to hold in practical co-evolution systems, where only a limited number of skill candidates are explored. The result therefore seems closer to a consequence of the assumptions than a general explanation of why co-evolution causes fragility.
W2. Limited evidence that co-evolution itself causes the observed fragility. The experiments show that co-evolved policies are sensitive to changes in their skill banks, but it remains unclear whether this sensitivity is specific to co-evolution. A comparison with policies trained using fixed skill banks or other non-co-evolving training strategies would help isolate the proposed mechanism.
W3. Limited connection between the proposed evaluation and realistic deployment changes. CODA focuses its scoring on three content-level perturbations, while changes to the skill-bank composition are excluded from the final verdict. Since adding, removing, and replacing skills are among the deployment scenarios motivating the paper, this leaves an important part of the problem outside the main evaluation.
Questions for authors
See Weaknesses
5 comments
Response to Reviewer vhER (part 2)
Co-evolution and fixed-bank training across four settings.
| Setting | Training | Original SR (%) | T1 SR (%) | T2 SR (%) | T3 SR (%) | Worst-case SR (%) | Peak-to-worst drop (pp) | T1 CR (%) | T2 RR (%) | T3 RR (%) |
|---|---|---|---|---|---|---|---|---|---|---|
| SkillRL–ALFWorld | Co-evolution | 88.8 | 78.6 | 79.3 | 79.5 | 78.6 | 10.2 | 87.0 | 70.2 | 71.2 |
| Fixed bank | 83.3 | 77.9 | 76.9 | 76.4 | 76.4 | 6.9 | 90.7 | 70.7 | 70.2 | |
| ReSkill–ALFWorld | Co-evolution | 76.9 | 64.3 | 66.6 | 65.4 | 64.3 | 12.6 | 84.2 | 60.6 | 58.9 |
| Fixed bank | 72.1 | 65.5 | 66.0 | 64.8 | 64.8 | 7.4 | 88.6 | 61.2 | 60.7 | |
| Evolving-RL–Mind2Web | Co-evolution | 25.2 | 23.2 | 20.3 | 20.4 | 20.3 | 4.9 | 77.5 | 12.2 | 11.4 |
| Fixed bank | 23.7 | 22.1 | 20.2 | 19.9 | 19.9 | 3.8 | 79.0 | 12.4 | 11.8 | |
| ARISE–AMC23 | Co-evolution | 87.4 | 82.5 | 81.5 | 80.9 | 80.9 | 6.5 | 82.1 | 63.5 | 65.4 |
| Fixed bank | 84.8 | 80.7 | 79.6 | 79.3 | 79.3 | 5.5 | 83.6 | 64.0 | 65.2 |
Performance drop and retention under T1–T3. Each perturbation column reports drop (pp; ) / SR retention (%; ).
| Setting | Training | T1 drop / retention | T2 drop / retention | T3 drop / retention | No-skill SR (%) |
|---|---|---|---|---|---|
| SkillRL–ALFWorld | Co-evolution | 10.2 / 88.5 | 9.5 / 89.3 | 9.3 / 89.5 | 74.5 |
| Fixed bank | 5.5 / 93.4 | 6.4 / 92.3 | 6.9 / 91.7 | 74.0 | |
| ReSkill–ALFWorld | Co-evolution | 12.6 / 83.6 | 10.3 / 86.6 | 11.5 / 85.0 | 63.3 |
| Fixed bank | 6.7 / 90.8 | 6.2 / 91.4 | 7.4 / 89.8 | 63.8 | |
| Evolving-RL–Mind2Web | Co-evolution | 2.0 / 92.1 | 4.9 / 80.6 | 4.8 / 81.0 | 15.8 |
| Fixed bank | 1.6 / 93.2 | 3.5 / 85.2 | 3.8 / 84.0 | 15.9 | |
| ARISE–AMC23 | Co-evolution | 4.9 / 94.4 | 5.9 / 93.2 | 6.5 / 92.6 | 75.3 |
| Fixed bank | 4.1 / 95.2 | 5.2 / 93.9 | 5.5 / 93.5 | 75.5 |
(Note: For each axis, drop = Original SR − perturbed SR, and retention = perturbed SR / Original SR × 100. The ALFWorld fixed-bank values are calculated from unrounded evaluation outcomes; the remaining rows use the reported SRs.)
The comparison yields three observations:
-
Fixed-bank training induces baseline sensitivity, which serves as a reference point. Fixed-bank controls lose 1.6–6.7 pp under T1, showing that moderate sensitivity to wording can arise even when the bank is held fixed during policy training, likely due to repeated exposure to fixed skill expressions.
-
Co-evolution consistently amplifies losses under all three perturbation axes. Across all four settings, its peak-to-worst drop is 1.0–5.2 pp larger, and it retains a smaller fraction of original success under each axis. Fixed-bank controls also have 1.5–4.4 pp higher T1 consistency. Resistance rates show a less uniform pattern: fixed-minus-co-evolution differences in T2/T3 RR range from −1.0 to +1.8 pp.
-
The extra performance gain from co-evolution is disproportionately eroded under drift. For SkillRL–ALFWorld, the co-evolution advantage over fixed-bank training shrinks by 87%, from +5.5 pp on the original bank to just +0.7 pp under T1; for ReSkill–ALFWorld, this advantage is completely erased and inverts from +4.8 pp (76.9% vs. 72.1%) to −1.2 pp under T1 (64.3% vs. 65.5%). Nevertheless, co-evolution retains higher worst-case SR in three of four settings, while no-skill SR differs by at most 0.5 pp between paired groups. Thus, while co-evolution improves peak performance, much of the additional gain achieved by co-evolving the bank over keeping it fixed is fragile to routine rewording.
Together, these observations are consistent with co-evolution amplifying an existing sensitivity through mutual adaptation between the policy and its bank. Because both branches share identical policy-update rules, retrieval pipelines, and rollout budgets—differing only in whether the skill bank continues to be updated and filtered via cooperative acceptance—this controlled comparison directly isolates the amplifying effect of cooperative skill evolution.
Response to Reviewer vhER (part 3)
W3. How Does CODA Evaluate Changes to Skill-Bank Composition?
CODA focuses its scoring on three content-level perturbations, while changes to the skill-bank composition are excluded from the final verdict. Since adding, removing, and replacing skills are among the deployment scenarios motivating the paper, this leaves an important part of the problem outside the main evaluation.
Thank you for highlighting this connection to deployment. Skill-bank composition changes are already evaluated in CODA under axis T4, which adds, removes, or replaces skills and re-runs retrieval on the modified bank. These results are reported in Section 3 and the Appendix, although T4 is excluded from the binary pass/fail verdict.
T1–T3 modify skill content while holding the selected skills and retrieval fixed; T4 changes bank composition and re-runs retrieval. As discussed in the Appendix, T4 operations also differ across methods with substantially different bank sizes. Their effects therefore depend on the composition edit and the resulting changes in skill coverage and retrieval. We report T4 as a complementary structural diagnostic, keeping these deployment effects visible without combining heterogeneous composition changes into the content-drift verdict.
For completeness, the table below summarizes the existing T4 results at the last reported policy checkpoint in each setting, using the corresponding unperturbed-bank evaluation from the Appendix.
| Setting | Checkpoint | Original SR (%) | T4 SR (%) | Drop (pp) |
|---|---|---|---|---|
| SkillRL–ALFWorld | s90 | 88.8 | 77.6 | 11.2 |
| SkillRL–WebShop | s70 | 52.4 | 52.4 | 0.0 |
| ReSkill–ALFWorld | s360 | 76.9 | 59.4 | 17.5 |
| ReSkill–ScienceWorld | s300 | 50.3 | 48.0 | 2.3 |
| Evolving-RL–Mind2Web | s54 | 25.2 | 20.5 | 4.7 |
| ARISE–AMC23 | s1000 | 87.4 | 76.5 | 10.9 |
The tested composition edits reduce success by 2.3–17.5 pp across five of the six settings (while SkillRL–WebShop remains unchanged at the reported precision). These results show that CODA already captures deployment risks from bank composition drift alongside content drift. Following your helpful suggestion, we will elevate the T4 results directly into the main CODA report card and explicitly label the binary verdict as a "Content-Drift Verdict (T1–T3)" alongside the "Composition-Drift Diagnostic (T4)", making both dimensions equally visible.
Once again, we sincerely appreciate your careful review and valuable feedback, which have substantially improved the quality and clarity of our work. We hope our response and revisions will further strengthen your confidence in our submission.
Response to Reviewer vhER (part 1)
We greatly appreciate your recognition of our contributions and insightful feedback. Following your suggestions, we have added controlled fixed-bank training comparisons across all four methods (W2) and elevated the T4 composition-change evaluation in the CODA report card (W3). We hope our response below will further strengthen your confidence in our work.
W1. Strong Assumptions in the Theoretical Analysis
Thank you for the question. Proposition 1 formalizes a conditional selection effect. Selecting skills for their utility to the current policy favors the retained bank within the explored candidate set. Exact maximization captures an idealized version of this preference; practical implementations, including SkillRL's rollout-based screening and ReSkill's Thompson sampling, approximate it in different ways (Table 1 and Appendix A).
We recognize that finite candidate sets may not cover the perturbation neighborhood and that practical selection need not achieve exact optimality. The proposition's guarantee depends on both assumptions. Its purpose is to explain the implication of policy-dependent skill selection under stated conditions, rather than claiming that every practical update achieves exact optimality. However, exploring only a limited number of skill candidates does not prevent fragility in practice: when a policy is repeatedly updated on a small, sparse set of accepted skill phrasings without exposure to meaning-preserving alternatives, this can encourage specialization to the particular wording of the retained bank.
The evaluation concern extends beyond this idealization. Proposition 2 separates return on the delivered bank from the performance loss under drift through an exact algebraic identity that does not require optimality or candidate coverage. It makes clear why optimizing performance on the current bank alone does not explicitly constrain robustness to subsequent skill changes.
Our experiments then examine the response pattern under actual training procedures. Across four methods and six settings, none of the 18 tested setting–axis combinations (T1–T3) improves success beyond evaluation noise at delivery (a pattern that also holds across 8 additional settings with larger 27B–31B models in our response to Reviewer n69A, W2). This provides consistent empirical support for the predicted pattern under approximate selection rules, rather than verification of the assumptions themselves. The fixed-bank controls below further examine whether continuing co-evolution increases sensitivity relative to policy training with a fixed bank.
W2. Is the Observed Fragility Specific to Co-Evolution?
Thank you for this valuable suggestion. We added fixed-bank training controls for all four methods to distinguish sensitivity arising with a fixed bank from the additional effect of evolving the bank alongside the policy.
The comparisons cover SkillRL and ReSkill on ALFWorld, Evolving-RL on Mind2Web, and ARISE on AMC23, with their original backbones. Each pair shares the starting policy and bank, training tasks, policy-update settings, retrieval and injection procedures, and subsequent rollout budget. Both branches continue policy training, but only the co-evolution branch updates the bank. For ReSkill, the comparison starts from the shared policy and nonempty bank at s60, with matched remaining training budgets through s360.
We test each resulting policy on the same held-out tasks using its own unperturbed bank, the T1–T3 variants, and no skills. Worst-case SR is the lowest SR across T1–T3; the peak-to-worst drop measures its decrease from original-bank success.