acceptodds
Under review as a conference paper at ICLR 2027

Evolving Together, Failing Apart: The Skill-Policy Co-evolution Trap in Self-Improving Agents

Abstract

Skill–policy co-evolution—in which an agent’s policy model and skill bank improve each other during training—is a promising paradigm for self-improving agents. Current evaluations assess the co-evolved policy on the final bank with which it was trained (the delivered bank). After deployment, however, skills may be rewritten, added, or removed more frequently than the policy model is retrained, leaving the policy to operate on a drifted skill bank addressed by neither the training objective nor existing evaluation protocols. Through theoretical and empirical analysis, we establish that the fragility masked by this blind spot is structural rather than incidental. The acceptance rule shared by these methods, which we term cooperative acceptance, ties a policy’s competence to the bank with which it co-evolved, inducing fragility under skill drift regardless of implementation; empirically, sensitivity to such drift grows as co-evolution training proceeds. We therefore introduce the CO-evolution Drift Audit (CODA), a deployment-oriented evaluation protocol that simulates skill drift and measures worst-case success rate, consistency rate under meaning-preserving edits, and resistance rate to corrupted skills. Across four co-evolution methods and five benchmarks (six settings), worst-case success rates fall 5–13 points below the peaks attained by the corresponding delivered pairs, erasing 52–93% of the delivered banks’ gains over the no-skill baseline. CODA thus reframes the target of co-evolution—from peak performance on the co-evolved bank to the ability to harness skills however they evolve—and provides a shared standard for auditing future methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, co-evolution, self-improving agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vhEROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates a potential weakness of skill–policy co-evolution: while an agent's policy and skill bank improve together during training, the resulting policy may become overly dependent on the exact skill bank it was trained with. As skills are modified, removed, or replaced after deployment, this dependence can lead to performance degradation. The authors first provide a theoretical analysis of this problem through an idealized cooperative acceptance rule, then empirically examine how sensitivity to skill modifications changes throughout training. They introduce CODA, an evaluation protocol that measures robustness under different forms of skill drift, including meaning-preserving rewrites, incomplete instructions, and misleading skills. Experiments cover four co-evolution methods across five benchmarks. The results show that performance on modified skill banks can be substantially lower than performance on the original bank, suggesting that conventional evaluations may overestimate how robust co-evolved agents are after deployment.

Critical strengths

S1. Important and relevant problem. The paper raises an interesting concern about whether agents trained with evolving skill banks can still use those skills reliably after deployment. This is particularly relevant for systems where skills may change more frequently than the underlying policy.

S2. Evaluation across different co-evolution frameworks. The experiments cover four methods and five benchmarks, including interactive and reasoning tasks. The consistent performance degradation across these settings provides useful evidence that sensitivity to skill drift is not limited to one particular implementation.

S3. Interesting analysis of how fragility changes during training. Rather than evaluating only the final policy, the authors examine intermediate checkpoints and show that sensitivity to skill modifications often increases as training progresses. This provides a useful perspective on the relationship between improving task performance and robustness.

S4. Practical evaluation protocol. CODA introduces several complementary measurements of skill robustness, including consistency under meaning-preserving edits and resistance to corrupted skills. The use of paired evaluations and an unperturbed noise baseline makes the proposed protocol more informative than simply reporting performance drops.

Critical weaknesses

W1. The theoretical analysis relies on strong assumptions. Proposition 1 assumes that the selected skill bank maximizes the current policy's performance over a candidate set covering all considered perturbations. This is unlikely to hold in practical co-evolution systems, where only a limited number of skill candidates are explored. The result therefore seems closer to a consequence of the assumptions than a general explanation of why co-evolution causes fragility.

W2. Limited evidence that co-evolution itself causes the observed fragility. The experiments show that co-evolved policies are sensitive to changes in their skill banks, but it remains unclear whether this sensitivity is specific to co-evolution. A comparison with policies trained using fixed skill banks or other non-co-evolving training strategies would help isolate the proposed mechanism.

W3. Limited connection between the proposed evaluation and realistic deployment changes. CODA focuses its scoring on three content-level perturbations, while changes to the skill-bank composition are excluded from the final verdict. Since adding, removing, and replacing skills are among the deployment scenarios motivating the paper, this leaves an important part of the problem outside the main evaluation.

Questions for authors

See Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer vhER (part 1)

We greatly appreciate your recognition of our contributions and insightful feedback. Following your suggestions, we have added controlled fixed-bank training comparisons across all four methods (W2) and elevated the T4 composition-change evaluation in the CODA report card (W3). We hope our response below will further strengthen your confidence in our work.

W1. Strong Assumptions in the Theoretical Analysis

The theoretical analysis relies on strong assumptions. Proposition 1 assumes that the selected skill bank maximizes the current policy's performance over a candidate set covering all considered perturbations. This is unlikely to hold in practical co-evolution systems, where only a limited number of skill candidates are explored. The result therefore seems closer to a consequence of the assumptions than a general explanation of why co-evolution causes fragility.

Thank you for the question. Proposition 1 formalizes a conditional selection effect. Selecting skills for their utility to the current policy favors the retained bank within the explored candidate set. Exact maximization captures an idealized version of this preference; practical implementations, including SkillRL's rollout-based screening and ReSkill's Thompson sampling, approximate it in different ways (Table 1 and Appendix A).

We recognize that finite candidate sets may not cover the perturbation neighborhood and that practical selection need not achieve exact optimality. The proposition's guarantee depends on both assumptions. Its purpose is to explain the implication of policy-dependent skill selection under stated conditions, rather than claiming that every practical update achieves exact optimality. However, exploring only a limited number of skill candidates does not prevent fragility in practice: when a policy is repeatedly updated on a small, sparse set of accepted skill phrasings without exposure to meaning-preserving alternatives, this can encourage specialization to the particular wording of the retained bank.

The evaluation concern extends beyond this idealization. Proposition 2 separates return on the delivered bank from the performance loss under drift through an exact algebraic identity that does not require optimality or candidate coverage. It makes clear why optimizing performance on the current bank alone does not explicitly constrain robustness to subsequent skill changes.

Our experiments then examine the response pattern under actual training procedures. Across four methods and six settings, none of the 18 tested setting–axis combinations (T1–T3) improves success beyond evaluation noise at delivery (a pattern that also holds across 8 additional settings with larger 27B–31B models in our response to Reviewer n69A, W2). This provides consistent empirical support for the predicted pattern under approximate selection rules, rather than verification of the assumptions themselves. The fixed-bank controls below further examine whether continuing co-evolution increases sensitivity relative to policy training with a fixed bank.

W2. Is the Observed Fragility Specific to Co-Evolution?

Limited evidence that co-evolution itself causes the observed fragility. The experiments show that co-evolved policies are sensitive to changes in their skill banks, but it remains unclear whether this sensitivity is specific to co-evolution. A comparison with policies trained using fixed skill banks or other non-co-evolving training strategies would help isolate the proposed mechanism.

Thank you for this valuable suggestion. We added fixed-bank training controls for all four methods to distinguish sensitivity arising with a fixed bank from the additional effect of evolving the bank alongside the policy.

The comparisons cover SkillRL and ReSkill on ALFWorld, Evolving-RL on Mind2Web, and ARISE on AMC23, with their original backbones. Each pair shares the starting policy and bank, training tasks, policy-update settings, retrieval and injection procedures, and subsequent rollout budget. Both branches continue policy training, but only the co-evolution branch updates the bank. For ReSkill, the comparison starts from the shared policy and nonempty bank at s60, with matched remaining training budgets through s360.

We test each resulting policy on the same held-out tasks using its own unperturbed bank, the T1–T3 variants, and no skills. Worst-case SR is the lowest SR across T1–T3; the peak-to-worst drop measures its decrease from original-bank success.

AuthorsOpenReview ↗

Response to Reviewer vhER (part 2)

Co-evolution and fixed-bank training across four settings.

SettingTrainingOriginal SR (%)T1 SR (%)T2 SR (%)T3 SR (%)Worst-case SR (%)Peak-to-worst drop (pp) ↓\downarrowT1 CR (%) ↑\uparrowT2 RR (%) ↑\uparrowT3 RR (%) ↑\uparrow
SkillRL–ALFWorldCo-evolution88.878.679.379.578.610.287.070.271.2
Fixed bank83.377.976.976.476.46.990.770.770.2
ReSkill–ALFWorldCo-evolution76.964.366.665.464.312.684.260.658.9
Fixed bank72.165.566.064.864.87.488.661.260.7
Evolving-RL–Mind2WebCo-evolution25.223.220.320.420.34.977.512.211.4
Fixed bank23.722.120.219.919.93.879.012.411.8
ARISE–AMC23Co-evolution87.482.581.580.980.96.582.163.565.4
Fixed bank84.880.779.679.379.35.583.664.065.2

Performance drop and retention under T1–T3. Each perturbation column reports drop (pp; ↓\downarrow) / SR retention (%; ↑\uparrow).

SettingTrainingT1 drop / retentionT2 drop / retentionT3 drop / retentionNo-skill SR (%)
SkillRL–ALFWorldCo-evolution10.2 / 88.59.5 / 89.39.3 / 89.574.5
Fixed bank5.5 / 93.46.4 / 92.36.9 / 91.774.0
ReSkill–ALFWorldCo-evolution12.6 / 83.610.3 / 86.611.5 / 85.063.3
Fixed bank6.7 / 90.86.2 / 91.47.4 / 89.863.8
Evolving-RL–Mind2WebCo-evolution2.0 / 92.14.9 / 80.64.8 / 81.015.8
Fixed bank1.6 / 93.23.5 / 85.23.8 / 84.015.9
ARISE–AMC23Co-evolution4.9 / 94.45.9 / 93.26.5 / 92.675.3
Fixed bank4.1 / 95.25.2 / 93.95.5 / 93.575.5

(Note: For each axis, drop = Original SR − perturbed SR, and retention = perturbed SR / Original SR × 100. The ALFWorld fixed-bank values are calculated from unrounded evaluation outcomes; the remaining rows use the reported SRs.)

The comparison yields three observations:

  1. Fixed-bank training induces baseline sensitivity, which serves as a reference point. Fixed-bank controls lose 1.6–6.7 pp under T1, showing that moderate sensitivity to wording can arise even when the bank is held fixed during policy training, likely due to repeated exposure to fixed skill expressions.

  2. Co-evolution consistently amplifies losses under all three perturbation axes. Across all four settings, its peak-to-worst drop is 1.0–5.2 pp larger, and it retains a smaller fraction of original success under each axis. Fixed-bank controls also have 1.5–4.4 pp higher T1 consistency. Resistance rates show a less uniform pattern: fixed-minus-co-evolution differences in T2/T3 RR range from −1.0 to +1.8 pp.

  3. The extra performance gain from co-evolution is disproportionately eroded under drift. For SkillRL–ALFWorld, the co-evolution advantage over fixed-bank training shrinks by 87%, from +5.5 pp on the original bank to just +0.7 pp under T1; for ReSkill–ALFWorld, this advantage is completely erased and inverts from +4.8 pp (76.9% vs. 72.1%) to −1.2 pp under T1 (64.3% vs. 65.5%). Nevertheless, co-evolution retains higher worst-case SR in three of four settings, while no-skill SR differs by at most 0.5 pp between paired groups. Thus, while co-evolution improves peak performance, much of the additional gain achieved by co-evolving the bank over keeping it fixed is fragile to routine rewording.

Together, these observations are consistent with co-evolution amplifying an existing sensitivity through mutual adaptation between the policy and its bank. Because both branches share identical policy-update rules, retrieval pipelines, and rollout budgets—differing only in whether the skill bank continues to be updated and filtered via cooperative acceptance—this controlled comparison directly isolates the amplifying effect of cooperative skill evolution.

AuthorsOpenReview ↗

Response to Reviewer vhER (part 3)

W3. How Does CODA Evaluate Changes to Skill-Bank Composition?

CODA focuses its scoring on three content-level perturbations, while changes to the skill-bank composition are excluded from the final verdict. Since adding, removing, and replacing skills are among the deployment scenarios motivating the paper, this leaves an important part of the problem outside the main evaluation.

Thank you for highlighting this connection to deployment. Skill-bank composition changes are already evaluated in CODA under axis T4, which adds, removes, or replaces skills and re-runs retrieval on the modified bank. These results are reported in Section 3 and the Appendix, although T4 is excluded from the binary pass/fail verdict.

T1–T3 modify skill content while holding the selected skills and retrieval fixed; T4 changes bank composition and re-runs retrieval. As discussed in the Appendix, T4 operations also differ across methods with substantially different bank sizes. Their effects therefore depend on the composition edit and the resulting changes in skill coverage and retrieval. We report T4 as a complementary structural diagnostic, keeping these deployment effects visible without combining heterogeneous composition changes into the content-drift verdict.

For completeness, the table below summarizes the existing T4 results at the last reported policy checkpoint in each setting, using the corresponding unperturbed-bank evaluation from the Appendix.

SettingCheckpointOriginal SR (%)T4 SR (%)Drop (pp)
SkillRL–ALFWorlds9088.877.611.2
SkillRL–WebShops7052.452.40.0
ReSkill–ALFWorlds36076.959.417.5
ReSkill–ScienceWorlds30050.348.02.3
Evolving-RL–Mind2Webs5425.220.54.7
ARISE–AMC23s100087.476.510.9

The tested composition edits reduce success by 2.3–17.5 pp across five of the six settings (while SkillRL–WebShop remains unchanged at the reported precision). These results show that CODA already captures deployment risks from bank composition drift alongside content drift. Following your helpful suggestion, we will elevate the T4 results directly into the main CODA report card and explicitly label the binary verdict as a "Content-Drift Verdict (T1–T3)" alongside the "Composition-Drift Diagnostic (T4)", making both dimensions equally visible.

Once again, we sincerely appreciate your careful review and valuable feedback, which have substantially improved the quality and clarity of our work. We hope our response and revisions will further strengthen your confidence in our submission.

Reviewer_5LNLOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper identifies a concrete problem in skill-policy coevolution: the policy model is sensitive to the drift in the co-evolved skill bank that may occur after deployment, exposing a limitation of existing evaluations that only measure fit to a fixed delivered bank. The paper demonstrates its mechanism and signature both theoretically and empirically, and proposes CODA, an evaluation protocol making the deployment robustness of the delivered policy–bank pair measurable.

Critical strengths
  1. The paper identifies an important evaluation blind spot in skill–policy co-evolution. Existing methods typically evaluate the co-evolved policy on its final skill bank, overlooking potential performance degradation when the bank changes after deployment. The paper provides empirical evidence across multiple methods and benchmarks that such degradation can be substantial, even under meaning-preserving skill rewrites. This finding challenges the reliability of conventional evaluations and highlights an important problem for self-evolving agents.

  2. CODA provides a practical evaluation protocol for measuring coevolution methods' robustness to skill drift. It introduces complementary metrics that assess behavioral consistency under semantic rewrites, resistance to corrupted skills, and worst-case task success. By requiring only the final delivered policy–bank pair and standardizing the perturbation procedure, CODA enables convenient robustness evaluation across different co-evolution methods.

Critical weaknesses
  1. The theoretical contribution is relatively limited, and its connection to the observed fragility is not fully established. Proposition 1 largely follows from the idealized optimality and candidate coverage assumptions, which may not hold in practical co-evolution methods.
Questions for authors
  1. How well do the optimality and candidate coverage assumptions underlying Proposition 1 hold in the evaluated co-evolution methods? What evidence supports the applicability of the theoretical conclusions when actual acceptance rules are approximate and candidate coverage is limited?

  2. To what extent do the empirical results support cooperative acceptance as the mechanism underlying the observed fragility? Could the authors clarify whether similar fragility might also arise from skill-conditioned policy training without cooperative skill selection?

  3. The paper measures fragility at different policy checkpoints against the same final skill bank, rather than the corresponding skill bank at each checkpoint. Could the authors clarify whether the increasing fragility trend also holds when each policy is evaluated against its skill bank at the same step? I think this would help distinguish increasing specialization to the final bank from increasing fragility of the co-evolved policy–bank pairs themselves.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5LNL (part 1)

Thank you for your time and effort in reviewing our work. Following your constructive suggestions, we have added two controlled ablations—Fixed-Bank Training (Q2) and Checkpoint-Matched Skill Bank Evaluation (Q3)—to more cleanly isolate the mechanism and trajectory of the co-evolution trap. We hope our responses below address your concerns and further increase your confidence in our work.

W1 & Q1. Applicability of the Theoretical Analysis

The theoretical contribution is relatively limited, and its connection to the observed fragility is not fully established. Proposition 1 largely follows from the idealized optimality and candidate coverage assumptions, which may not hold in practical co-evolution methods.

How well do the optimality and candidate coverage assumptions underlying Proposition 1 hold in the evaluated co-evolution methods? What evidence supports the applicability of the theoretical conclusions when actual acceptance rules are approximate and candidate coverage is limited?

Thank you for the question. Proposition 1 formalizes a shared selection preference: skills are retained according to their utility to the current policy. As summarized in Table 1 and Appendix A, this preference appears in different implementations, including rollout-based screening in SkillRL and Thompson sampling in ReSkill. Exact maximization idealizes this common preference to isolate the direction in which cooperative selection pushes the policy–bank pair.

In practice, these procedures need not find the best bank within the candidate set, and their finite candidate sets need not cover the full perturbation neighborhood. The guarantee is therefore conditional on optimality and coverage, rather than a universal claim about every practical update or deployment-time edit. However, limited candidate coverage does not prevent fragility in practice: repeatedly updating a policy on a sparse subset of accepted skill expressions—without exposure to meaning-preserving alternatives—still reinforces specialization to the surface form of the retained bank.

The broader implication is clarified by Proposition 2: optimizing return on the current bank does not explicitly constrain performance under skill drift. The return decomposition in Proposition 2 is an exact algebraic identity that does not require exact optimality or complete candidate coverage. Thus, high performance on the delivered bank alone does not certify robustness to changes in that bank.

To examine whether this concern manifests in practical implementations, Section 3 evaluates four methods across six settings. Across all 18 combinations of setting and content-perturbation axis (T1–T3), no tested perturbation improves performance beyond evaluation noise at delivery (a pattern that also holds across 8 additional settings with larger 27B–31B models on long-horizon tasks in our response to Reviewer n69A, W2). These observations support the predicted response pattern under the methods' actual, approximate selection procedures. We address the separate question of whether cooperative skill selection adds to sensitivity from policy training alone in Q2.

Q2. Can Training with a Fixed Skill Bank Produce Similar Fragility?

To what extent do the empirical results support cooperative acceptance as the mechanism underlying the observed fragility? Could the authors clarify whether similar fragility might also arise from skill-conditioned policy training without cooperative skill selection?

Thank you for suggesting this valuable control. It helps distinguish sensitivity observed with a fixed skill bank from the additional effect of evolving the bank alongside the policy.

We compare standard co-evolution with fixed-bank training for SkillRL and ReSkill on ALFWorld, Evolving-RL on Mind2Web, and ARISE on AMC23, using their original backbones. Within each comparison, we match the starting policy and bank, training tasks, policy-update settings, retrieval and injection procedures, and subsequent rollout budget. For ReSkill, both branches start from the same policy checkpoint and skill bank at s60 and use the same remaining training budget through s360. After branching, both groups update the policy; the fixed-bank group freezes the bank, while the co-evolution group continues updating it.

On the same held-out tasks, we evaluate each trained policy with its own unperturbed bank and the CODA T1–T3 variants. Worst-case SR is the minimum across T1–T3, and the peak-to-worst drop is Original SR minus Worst-case SR.

AuthorsOpenReview ↗

Response to Reviewer 5LNL (part 2)

Fixed-bank training versus co-evolution.

SettingTrainingOriginal SR (%)T1 SR (%)T2 SR (%)T3 SR (%)Worst-case SR (%)Peak-to-worst drop (pp) ↓\downarrowT1 CR (%) ↑\uparrowT2 RR (%) ↑\uparrowT3 RR (%) ↑\uparrow
SkillRL–ALFWorldCo-evolution88.878.679.379.578.610.287.070.271.2
Fixed bank83.377.976.976.476.46.990.770.770.2
ReSkill–ALFWorldCo-evolution76.964.366.665.464.312.684.260.658.9
Fixed bank72.165.566.064.864.87.488.661.260.7
Evolving-RL–Mind2WebCo-evolution25.223.220.320.420.34.977.512.211.4
Fixed bank23.722.120.219.919.93.879.012.411.8
ARISE–AMC23Co-evolution87.482.581.580.980.96.582.163.565.4
Fixed bank84.880.779.679.379.35.583.664.065.2

Performance drop and retention under T1–T3. Each perturbation column reports drop (pp; ↓\downarrow) / SR retention (%; ↑\uparrow).

SettingTrainingT1 drop / retentionT2 drop / retentionT3 drop / retentionNo-skill SR (%)
SkillRL–ALFWorldCo-evolution10.2 / 88.59.5 / 89.39.3 / 89.574.5
Fixed bank5.5 / 93.46.4 / 92.36.9 / 91.774.0
ReSkill–ALFWorldCo-evolution12.6 / 83.610.3 / 86.611.5 / 85.063.3
Fixed bank6.7 / 90.86.2 / 91.47.4 / 89.863.8
Evolving-RL–Mind2WebCo-evolution2.0 / 92.14.9 / 80.64.8 / 81.015.8
Fixed bank1.6 / 93.23.5 / 85.23.8 / 84.015.9
ARISE–AMC23Co-evolution4.9 / 94.45.9 / 93.26.5 / 92.675.3
Fixed bank4.1 / 95.25.2 / 93.95.5 / 93.575.5

(Note: For each axis, drop = Original SR − perturbed SR; SR retention = perturbed SR / Original SR × 100. For the ALFWorld fixed-bank controls, these quantities are computed from unrounded evaluation outcomes. Values for the other rows are calculated from the reported SRs.)

These results clarify the concern in three respects:

  1. Fixed-bank policies exhibit a baseline sensitivity to skill wording. T1 rewrites reduce success by 1.6–6.7 pp in the fixed-bank groups. Thus, moderate sensitivity is present even when the bank is held fixed during the controlled training phase, as repeated exposure to a static skill expression can already foster dependence.
  2. Co-evolution consistently produces larger SR drops across T1–T3 in all four settings. Its peak-to-worst drop is 1.0–5.2 pp larger (nearly doubling the T1 drop on ALFWorld: 10.2 vs. 5.5 pp for SkillRL and 12.6 vs. 6.7 pp for ReSkill), and the fixed-bank controls retain a larger fraction of original success under all three axes. Fixed-bank controls also achieve 1.5–4.4 pp higher T1 CR, while T2/T3 RR differences are mixed (−1.0 to +1.8 pp, fixed minus co-evolution).
  3. Higher original-bank performance is disproportionately lost after rewriting. For SkillRL–ALFWorld, the co-evolution advantage over fixed-bank training shrinks by 87% from +5.5 pp on the original bank (88.8% vs. 83.3%) to +0.7 pp under T1 (78.6% vs. 77.9%); for ReSkill–ALFWorld, the +4.8 pp clean advantage (76.9% vs. 72.1%) is completely erased and inverts to −1.2 pp under T1 (64.3% vs. 65.5%). Co-evolution nevertheless retains higher worst-case SR in three of four settings, while no-skill SR differs by at most 0.5 pp between groups. Thus, while co-evolution improves peak performance, the additional gain it achieves over a fixed bank is highly vulnerable to routine rewording.

These results are consistent with mutual adaptation amplifying sensitivity to skill wording. With a fixed bank, the policy adapts to the retained skill text. Co-evolution adds a reciprocal process: skill candidates are selected for their utility to the current policy, which then further adapts to the selected bank. Without explicitly checking whether gains survive equivalent rewrites, this process can reinforce dependence on particular skill expressions. This helps explain why co-evolution achieves higher original-bank performance but larger losses after rewriting: part of its additional gain may depend on the specific policy–bank pairing.

AuthorsOpenReview ↗

Response to Reviewer 5LNL (part 3)

Q3. Does Fragility Also Increase When Each Checkpoint Uses Its Own Skill Bank?

The paper measures fragility at different policy checkpoints against the same final skill bank, rather than the corresponding skill bank at each checkpoint. Could the authors clarify whether the increasing fragility trend also holds when each policy is evaluated against its skill bank at the same step? I think this would help distinguish increasing specialization to the final bank from increasing fragility of the co-evolved policy–bank pairs themselves.

Our original design fixes the skill bank to control the evaluation conditions. In Section 3, every policy checkpoint is evaluated against the same final bank and the same perturbed variants, so the policy is the only changing component. This measures how its sensitivity to a fixed set of skill edits changes during training. We appreciate your suggestion to complement this analysis with checkpoint-matched skill banks, which directly assesses the robustness of the evolving policy–bank pairs themselves.

Following your suggestion, we evaluate ReSkill on ALFWorld and ARISE on AMC23 at five intermediate checkpoints each, pairing every policy with the bank saved at the same training step. At each checkpoint, we compare the unperturbed bank with a verified meaning-preserving T1 rewrite on the same held-out tasks, using the same rewrite protocol and fixing the retrieved skills within each comparison. The tables place these new results alongside the original fixed-final-bank evaluations.

Original SR refers to the unperturbed bank in each row. T1 drop = Original SR − T1 SR; SR retention = T1 SR / Original SR × 100. The new matched-bank ALFWorld results aggregate 420 task evaluations per checkpoint, with differences and ratios computed before rounding.

ReSkill–ALFWorld.

CheckpointEvaluation bankOriginal SR (%)T1 SR (%)T1 drop (pp) ↓\downarrowSR retention (%) ↑\uparrow
s60Fixed final bank33.836.4−2.6107.7
Matched bank37.436.70.798.1
s120Fixed final bank41.440.50.997.8
Matched bank44.342.61.796.2
s180Fixed final bank61.060.70.399.5
Matched bank62.458.83.694.3
s240Fixed final bank26.928.1−1.2104.5
Matched bank29.528.31.296.0
s300Fixed final bank56.150.75.490.4
Matched bank57.452.64.891.7

ARISE–AMC23.

CheckpointEvaluation bankOriginal SR (%)T1 SR (%)T1 drop (pp) ↓\downarrowSR retention (%) ↑\uparrow
s100Fixed final bank15.516.0−0.5103.2
Matched bank19.218.90.398.4
s300Fixed final bank20.020.00.0100.0
Matched bank22.121.60.597.7
s500Fixed final bank26.723.33.487.3
Matched bank29.528.31.295.9
s700Fixed final bank55.053.31.796.9
Matched bank55.853.72.196.2
s900Fixed final bank78.475.03.495.7
Matched bank78.074.53.595.5

The increase in later-checkpoint sensitivity becomes even cleaner and more pronounced with contemporaneous skill banks:

  1. Elimination of Early Cross-Checkpoint Mismatch Noise. Under the Fixed final bank evaluation, early checkpoints (s60 and s240 in ReSkill; s100 in ARISE) showed slightly negative drops (−2.6, −1.2, and −0.5 pp) because early policies had not yet adapted to the final bank. Under your suggested Matched bank evaluation, this cross-step mismatch disappears completely—every checkpoint exhibits a positive T1 drop.
  2. ReSkill–ALFWorld: T1 drop rises from 0.7 pp at s60 to 4.8 pp at s300, while retention falls from 98.1% to 91.7%. The trajectory is not monotonic: the drop temporarily decreases at s240, alongside a dip in unperturbed performance under both bank conditions.
  3. ARISE–AMC23: T1 drop rises from 0.3 pp at s100 to 3.5 pp at s900, while retention declines overall from 98.4% to 95.5%. The later policy–bank pairs therefore lose a larger fraction of their original performance under meaning-preserving rewrites.

These results confirm increasing fragility of the contemporaneous policy–bank pairs in both settings, beyond specialization to the fixed final bank alone. The decline in retention also shows the pattern on a relative scale, complementing the absolute drops.

Once again, we sincerely thank you for your careful review and constructive feedback. We have addressed the concerns with these new controlled experiments, which will be incorporated into the revised manuscript to further strengthen its clarity and rigor. If any aspect remains unclear, we would be glad to further elaborate.

Reviewer_n69AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a blind spot in the evaluation of skill–policy co-evolution: the fact that a policy performs well on the skill bank it was co-trained with does not guarantee its robustness against "skill drift" during deployment. Through theoretical and empirical analysis, the authors demonstrate that "cooperative acceptance"—a mechanism based on current policy rewards—drives the policy and its specific skill bank to become mutually specialized. Consequently, the policy may become increasingly sensitive to alterations in these skills as training progresses. To address this, the paper introduces CODA, a framework that evaluates success rates, consistency, and resilience against compromised skills by applying perturbations such as semantic-preserving rewrites, information pruning, and the injection of misleading content. Across six experimental settings—encompassing four methods and five benchmarks—the worst-case success rate drops by approximately 5–13 percentage points compared to the original policy-skill pairings, representing a loss of 52–93% of the performance gains achieved over a "no-skill" baseline. The core contribution of this work is the revelation that high performance on a fixed, co-trained pair can mask vulnerabilities that emerge during deployment, alongside the provision of a systematic auditing protocol designed to assess robustness against skill drift.

Critical strengths
  1. The paper captures an important blind spot in current self-improving agent research. While existing literature focuses almost exclusively on peak success rates achieved on fixed, co-evolved skill banks, this work highlights the practical reality that skill libraries in real-world deployment are inherently dynamic and subject to drift.
  2. The theoretical and empirical framing of the "co-evolution trap" is rigorous and well-supported.
Critical weaknesses
  1. The paper identifies the "Co-Evolution Trap" but fails to propose any straightforward improvements or fixes. It lacks a complete loop—from problem identification to resolution—making its methodological contribution somewhat thin.

  2. While the paper claims that the identified pain points stem from real-world application scenarios, the tasks used in the experiments are overly simplistic. Furthermore, the study is limited to Qwen2.5-7B and Qwen3-4B models, creating a significant gap between this work and the requirements of state-of-the-art agent applications. It is unclear whether the paper’s conclusions hold true in contemporary, cutting-edge, real-world scenarios, or if the issues described remain relevant to current foundation models for agents. A key value of "skills" lies in their ability to provide agents with the domain-specific "know-how" of human experts, better aligning the agent’s reasoning with that of a human professional. In my view, simple agent tasks—such as those found in ALFWorld or WebShop—do not require such skills. Attempting to maintain these "skills" using outdated models for such tasks is highly likely to lead to overfitting.

  3. The CODA protocol mandates intensive rollout evaluations for every deployed system, involving multiple dimensions, various perturbation axes (T1–T3), and numerous repetitions. In real-world agent applications, however, rollouts are extremely costly—incurring significant expenses in terms of time, API usage, and computational resources. This limitation constrains the practical applicability and overall value of the approach for real-world industrial deployment.

Questions for authors

See weaknesses. My primary concern relates to the adaptability of the proposed approach to current, more realistic, and complex "long-horizon agent tasks"—which necessitate the involvement of expert knowledge—as well as whether the conclusions presented in the paper hold true for larger-scale models, including some of the current state-of-the-art (SOTA) models. Further analysis from the authors on these points would help me decide whether to adjust my score.

7 comments
AuthorsOpenReview ↗

Response to Reviewer n69A (part 1)

Thank you for your constructive feedback. To fully address your concerns, we have conducted extensive additional experiments during the rebuttal period:

  1. Mitigation Loop (W1): Implemented and evaluated a model-agnostic skill-bank augmentation intervention across 4 methods and 3 benchmarks.
  2. Larger Models & Complex Tasks (W2/Q1): Scaled co-evolution experiments to Qwen3.8-27B and Gemma-4-31B-IT on long-horizon, know-how-intensive benchmarks (τ²-Bench Telecom and SkillGym).
  3. Audit Efficiency (W3): Quantified exact wall-clock overhead across all 14 settings and evaluated budget-constrained auditing strategies.

We sincerely hope this clarification encourages you to reconsider and reevaluate our contributions.

W1. Lack of a Straightforward Mitigation for the Co-Evolution Trap

The paper identifies the "Co-Evolution Trap" but fails to propose any straightforward improvements or fixes. It lacks a complete loop—from problem identification to resolution—making its methodological contribution somewhat thin.

We appreciate the valuable suggestion. Our work combines (1) a theoretical and empirical investigation that identifies and characterizes the previously overlooked co-evolution trap with (2) the development of a new evaluation protocol called CODA.

To close the loop from discovery and evaluation to mitigation, we propose a straightforward, model-agnostic training intervention. During training, we vary the skill text across rollouts, using current versions, verified paraphrases that preserve their meaning, and unmodified versions from earlier skill bank snapshots. This exposes the policy to both changes in wording and revisions made during co-evolution.

The table below reports CODA results with and without augmentation for SkillRL and ReSkill on ALFWorld, Evolving-RL on Mind2Web, and ARISE on AMC23.

SettingPeak SRWorst-case SRPeak-to-worst dropT1 SRT2 SRT3 SRNo-skill SRT1 CRT2 RRT3 RR
SkillRL–ALFWorld88.878.610.278.679.379.574.587.070.271.2
+ augmentation88.579.98.681.479.980.074.389.070.571.2
ReSkill–ALFWorld76.964.312.664.366.665.463.384.260.658.9
+ augmentation76.666.110.567.167.066.163.586.360.959.4
Evolving-RL–Mind2Web25.220.34.923.220.320.415.877.512.211.4
+ augmentation25.120.54.624.020.720.515.779.012.411.4
ARISE–AMC2387.480.96.582.581.580.975.382.163.565.4
+ augmentation87.281.06.284.081.881.075.284.063.765.3

The results reveal two key insights that both provide an actionable baseline and reinforce our theoretical findings:

  • Effective Mitigation of Lexical and Historical Drift (T1): Across all four settings, this straightforward training intervention consistently improves T1 SR by 0.8–2.8 pp and T1 Consistency Rate (CR) by 1.5–2.1 pp, recovering up to 40.0% of the T1 performance drop (e.g., on Evolving-RL–Mind2Web) with negligible impact on clean peak performance (<= 0.3 pp). This provides practitioners with an immediate, drop-in mitigation against routine skill rewording.
  • Empirical Validation of Structural Fragility (T2 & T3): While exposure to paraphrases and past snapshots improves worst-case SR by 1.3–1.8 pp on ALFWorld, gains on T2/T3 Resistance Rates (RR) remain modest (-0.1 to +0.5 pp). Crucially, this empirically validates our central theoretical claim: the Co-evolution Trap is structural rather than merely lexical. Because cooperative acceptance rewards unconditional reliance on the co-evolved bank, surface-level augmentation alone cannot teach the policy to reject corrupted skills (T2/T3). Fully resolving T2/T3 requires modifying the acceptance rule itself—precisely the research direction that CODA is designed to benchmark and guide.
AuthorsOpenReview ↗

Response to Reviewer n69A (part 2)

W2 & Q1. Do the Findings Extend to Larger Models and Realistic Long-Horizon Tasks?

While the paper claims that the identified pain points stem from real-world application scenarios, the tasks used in the experiments are overly simplistic. Furthermore, the study is limited to Qwen2.5-7B and Qwen3-4B models, creating a significant gap between this work and the requirements of state-of-the-art agent applications. It is unclear whether the paper’s conclusions hold true in contemporary, cutting-edge, real-world scenarios, or if the issues described remain relevant to current foundation models for agents. A key value of "skills" lies in their ability to provide agents with the domain-specific "know-how" of human experts, better aligning the agent’s reasoning with that of a human professional. In my view, simple agent tasks—such as those found in ALFWorld or WebShop—do not require such skills. Attempting to maintain these "skills" using outdated models for such tasks is highly likely to lead to overfitting.

We strongly agree with your insight that the core value of skills lies in injecting domain-specific expert "know-how" into complex workflows. While our initial benchmarks followed the standard evaluation suites of prior co-evolution papers (SkillRL, ReSkill, Evolving-RL, ARISE), testing whether the Co-evolution Trap persists when frontier-scale dense models genuinely rely on complex expert know-how is critical.

To test this directly, we scaled co-evolution training to two much larger backbones from distinct families—Qwen3.8-27B and Gemma-4-31B-IT (4$\times$ – 8$\times$ larger than our initial models)—on two realistic, long-horizon benchmarks requiring intricate procedural knowledge: τ²-Bench Telecom (complex troubleshooting workflows) and SkillGym (expert human-written skill libraries).

(Experimental setup: On Telecom, we train on 512 sampled tasks and evaluate on all 114 official base tasks; on SkillGym, we train on 512 tasks and evaluate on 128 held-out tasks, initializing both SkillRL and ReSkill from the same expert skill banks.)

MethodBenchmarkBackbonePeak SR (%)Worst-case SR (%)Peak-to-worst drop (pp)T1 SR (%)T2 SR (%)T3 SR (%)No-skill SR (%)T1 CR (%)T2 RR (%)T3 RR (%)
SkillRLτ²-Bench TelecomQwen3.8-27B79.274.84.476.576.074.860.791.457.556.3
Gemma-4-31B-IT68.163.15.064.964.063.148.989.845.744.6
SkillGymQwen3.8-27B28.124.04.125.825.024.016.490.114.413.6
Gemma-4-31B-IT22.718.83.920.619.518.811.289.39.48.8
ReSkillτ²-Bench TelecomQwen3.8-27B78.173.44.774.074.973.459.190.856.254.8
Gemma-4-31B-IT67.361.85.561.863.862.647.589.144.543.6
SkillGymQwen3.8-27B27.322.74.623.724.222.715.988.513.912.9
Gemma-4-31B-IT23.419.04.419.020.119.811.888.09.99.3

These results directly address both parts of your concern

  1. Genuine Reliance on Expert Know-How (Not Toy-Task Overfitting): Across all 8 settings, the delivered skill banks boost success over the no-skill baseline by +11.4 to +19.8 percentage points (e.g., lifting Qwen3.8-27B on Telecom from 60.7% to 79.2%, and nearly doubling SkillGym success from 11.2% to 22.7% on Gemma-4-31B-IT). These tasks genuinely require domain know-how that even 27B–31B models lack zero-shot.
  2. The Co-evolution Trap Persists at Scale: Even on 27B and 31B models, post-deployment skill drift drops worst-case success by 3.9–5.5 pp, erasing 24%–40% of the entire gain provided by the expert skill bank.
  3. High Vulnerability to Purely Lexical Edits (T1): Strikingly, degradation is not limited to corrupted skills. Meaning-preserving rewrites alone (T1) reduce success by 2.1–5.5 pp. In fact, for ReSkill with Gemma-4-31B-IT on both Telecom and SkillGym, T1 (meaning-preserving edits) is already the worst-case perturbation (dropping 5.5 pp and 4.4 pp, respectively). Thus, scaling model capacity and task complexity does not eliminate the structural coupling induced by co-evolution.
AuthorsOpenReview ↗

Response to Reviewer n69A (part 3)

W3. What Is the Evaluation Cost of CODA?

The CODA protocol mandates intensive rollout evaluations for every deployed system, involving multiple dimensions, various perturbation axes (T1–T3), and numerous repetitions. In real-world agent applications, however, rollouts are extremely costly—incurring significant expenses in terms of time, API usage, and computational resources. This limitation constrains the practical applicability and overall value of the approach for real-world industrial deployment.

We appreciate this practical perspective and would like to clarify two key design properties that keep CODA's cost strictly bounded and practical for industrial workflows:

  1. One-Time Pre-Deployment Gate (Zero Online Overhead): CODA is an offline pre-release audit (analogous to pre-deployment safety or stress testing), not an online runtime requirement. It adds zero latency or API cost after deployment.
  2. Log Reuse Across Metrics (4 ×\times Additional Rollouts, Not Combinatorial): Although CODA reports multiple metrics (Worst-case SR, CR, and RR), all metrics are computed post-hoc from the exact same rollout logs. Furthermore, perturbed skill banks (T1–T3) are generated once offline and reused across repeats. Thus, adding the four audit conditions (No-skill and T1–T3) alongside the standard original-bank evaluation requires a strictly bounded 5 ×\times total rollout cost (4 ×\times additional overhead).

1. Exact Runtime Overhead Across All 14 Settings

As shown below, a full CODA audit takes 1.6 to 6.5 hours on most benchmarks, and under 20 hours even on our heaviest 31B long-horizon setting—a highly manageable one-time offline cost prior to model release.

Original Benchmarks

MethodBenchmarkBackboneOriginal-bank eval. (min)Full CODA rollout time (min, ratio)
SkillRLALFWorldQwen2.5-7B41204 (4.98×)
WebShopQwen2.5-7B2041,001 (4.91×)
ReSkillALFWorldQwen3-4B58312 (5.38×)
ScienceWorldQwen3-4B133709 (5.33×)
Evolving-RLMind2WebQwen2.5-7B78392 (5.03×)
ARISEAMC23Qwen3-4B2096 (4.80×)

Additional Long-Horizon Benchmarks

MethodBenchmarkBackboneOriginal-bank eval. (min)Full CODA rollout time (min, ratio)
SkillRLτ²-Bench TelecomQwen3.8-27B54277 (5.13×)
Gemma-4-31B-IT73382 (5.23×)
SkillGymQwen3.8-27B163848 (5.20×)
Gemma-4-31B-IT2051,085 (5.29×)
ReSkillτ²-Bench TelecomQwen3.8-27B57299 (5.25×)
Gemma-4-31B-IT76407 (5.36×)
SkillGymQwen3.8-27B172915 (5.32×)
Gemma-4-31B-IT2131,157 (5.43×)

2. Practical Strategies to Further Reduce Audit Cost

When evaluation budgets are tightly constrained during iterative development, CODA supports two practical cost-reduction strategies:

  • Strategy A: Targeted Single-Axis Auditing (+1 ×\times overhead). In many production pipelines, the primary risk during routine maintenance is benign skill rephrasing (T1). Auditing T1 alone requires only one additional rollout pass (2 ×\times total cost instead of 5 ×\times) while directly measuring both T1 SR drop and Consistency Rate (CR). As shown in W2, T1 alone already captures a major fraction (and sometimes 100%) of the worst-case drop.
  • Strategy B: Stratified Task Subsampling (40%–61% time reduction). Alternatively, keeping the standard 100% evaluation on the original bank while running the four additional audit conditions on a shared 50% or 25% task subset substantially reduces cumulative rollout time:
Coverage for added conditionsMean original-bank eval. (min)Mean sampled CODA (min)Mean full CODA (min)Time saved / cumulative-time reduction factorMean absolute relative error in worst-case drop (%) ↓\downarrow
100% (full CODA)110.5577.4577.40% / 1.00×0
50%110.5344.2577.440.4% / 1.68×12.0
25%110.5227.5577.460.6% / 2.54×22.3

Subsampling offers a clear cost–precision trade-off: 50% and 25% coverage cut total audit time by 40.4% (1.68 ×\times speedup) and 60.6% (2.54 ×\times speedup), respectively, with mean absolute relative errors of 12% and 22.3% in the worst-case drop. Consequently, practitioners can use T1-only auditing or 50% subsampling for rapid screening during development, reserving the full 100% CODA audit for final pre-release sign-off.

Once again, we sincerely appreciate your time and thoughtful feedback, which have significantly strengthened the quality and completeness of our work. We hope our response will prompt you to reassess our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.