acceptodds
Under review as a conference paper at ICLR 2027

Agentic Knowledge-Model Co-Evolution via Consistency Regularization

Abstract

LLM agents can evolve by learning from accumulated experience. Existing approaches extract reusable external knowledge, such as skills and memories, from experience to guide subsequent task solving, with some further internalizing this knowledge through model training. However, different ways of analyzing and transforming the same experience can yield knowledge with substantially different utility, even when the resulting guidance appears reasonable. Inspired by consistency regularization, we identify consistent utility gains across different ways of analyzing the same experience, rather than relying on any single way of analyzing it. We introduce SPLICE, a framework for consistency-regularized knowledge-model co-evolution. It constructs evolution views by perturbing how the same experience is analyzed, producing different knowledge candidates. Knowledge updates are then guided by common gain, which accounts for utility gains and their consistency across views. During model evolution, self-distillation internalizes the evolved knowledge alongside reinforcement learning, using knowledge associated with consistent utility degradation as a contrastive reference. The two processes interleave, with model updates driving further knowledge evolution. Experiments across diverse domains show that SPLICE outperforms knowledge evolution and model training baselines by up to 7.1% and 4.6% in average accuracy, while its evolved knowledge reduces output tokens per correct solution by up to 44% and transfers across model sizes and families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Agentic Evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V8N7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPLICE, a framework for knowledge–model co-evolution in LLM agents. The key idea is that different ways of analyzing the same experience may produce knowledge with very different utility, so the authors introduce multiple evolution views and update knowledge only when the resulting utility gains are consistent across views. The proposed common gain combines average gain with a cross-view consistency penalty, and the resulting knowledge is further internalized into the model through GRPO and gated contrastive self-distillation. The framework is evaluated on four domains, including mathematics, coding, spreadsheet manipulation, and web shopping. SPLICE improves both training-free knowledge evolution and knowledge–model co-evolution over several recent baselines, and the evolved knowledge also shows cross-model-size and cross-family transferability.

Critical strengths
  1. The problem is practically meaningful. Existing agent knowledge-evolution methods often assume that extracted knowledge is useful, while seemingly reasonable knowledge may not actually improve downstream performance. Using cross-view utility consistency to determine whether a knowledge update is reliable is intuitive and relatively novel.

  2. The proposed common gain is simple and directly aligned with the main hypothesis. It considers not only average utility gain but also disagreement across evolution views, and the paper provides a theoretical interpretation for this design.

  3. The empirical results are strong and cover four diverse task domains. SPLICE consistently improves over prior methods in both training-free knowledge evolution and knowledge–model co-evolution.

  4. The ablation studies directly test the central mechanism. Replacing common gain with mean gain causes a clear performance drop and makes harmful knowledge updates much more frequent, which supports the importance of the consistency term.

  5. The cross-model transfer results are convincing. Knowledge evolved with Qwen3.5-9B transfers to other model sizes and different model families, suggesting that the learned knowledge is not entirely model-specific.

Critical weaknesses
  1. The theoretical assumptions are not fully aligned with the practical implementation. Proposition 1 assumes that different evolution views are independent and identically distributed, while in practice the views are generated by the same knowledge optimizer from the same experience and may therefore be strongly correlated.

  2. The computational overhead is substantial. Each round of knowledge evolution generates multiple candidates and performs task replay, while model evolution additionally requires multiple rollout groups and self-distillation.

  3. Knowledge updates are repeatedly accepted or rejected based on a held-out validation set. Such repeated adaptive selection may gradually overfit the validation set.

  4. The framework relies on an external LLM as the knowledge optimizer. The quality and diversity of the evolution views therefore depend on the capability of this external model, and robustness across different knowledge optimizers is not sufficiently explored.

Questions for authors
  1. Proposition 1 assumes that evolution views are i.i.d., while in practice they are generated by the same knowledge optimizer from the same experience. Could the authors clarify the validity of this assumption and analyze how correlation among views affects common gain?

  2. The method relies on an external LLM to generate evolution views and update knowledge. How robust is SPLICE when using weaker knowledge optimizers or optimizers from different model families?

  3. The default setting uses three evolution views. How sensitive is performance to the number of views? Do additional views consistently help, or does performance quickly saturate?

  4. Knowledge updates are repeatedly accepted or rejected using a held-out validation set. Have the authors observed validation overfitting, and have they tested dynamic validation splits or an independent holdout set?

4 comments

Thank you for the thoughtful and constructive review and for recognizing the idea of common gain and our empirical results. We address your concerns below.


W1, Q1 & Q3: The i.i.d. assumption, dependence among views, and the number of views.


On the i.i.d. assumption. The i.i.d. assumption in Proposition 1 is an idealization, and we agree that it does not strictly hold in practice. Note that the proposition already conditions on the task, the experience, the model, and the current knowledge, so sharing the same experience is part of the conditioning rather than a violation. The remaining dependence comes from the views being generated by the same optimizer, which may give them similar tendencies. The assumption is only needed for the probability bound, which motivates selecting evidence supported by several views. Common gain itself is well defined without it. When the views are dependent, the bound can be optimistic, and we analyze below how such dependence affects common gain in practice.

Moreover, generating the views with the optimizer is only one way to instantiate SPLICE. The views can also be predefined, for example to incorporate human expertise in analyzing experience, and the consistency criterion and common gain apply without change. We have revised the discussion in the paper to clarify these points.

How dependence affects common gain. The more correlated the views are, the smaller the differences among their gains and the smaller the inconsistency penalty, so common gain approaches mean gain. In the extreme case where all views are identical, common gain equals mean gain and the consistency criterion no longer has any effect. Dependence therefore weakens the selection rather than causing the method to select worse evidence. To quantify how much the views actually differ, we measured their gains on the same task. For each round we fix the model and the three candidate knowledge documents, replay each candidate 15 times on every task, average the 15 replays to remove rollout noise, and subtract the reward of the original rollout to obtain the gain Δj\Delta_j of each view on that task. We then report three metrics:

  • Sign disagreement: the fraction of tasks on which some views improve the task while others do not.
  • Relative inconsistency: the inconsistency penalty 12 meanj<ℓ∣Δj−Δℓ∣\frac{1}{2}\,\mathrm{mean}_{j<\ell}|\Delta_j-\Delta_\ell| divided by meanj∣Δj∣\mathrm{mean}_j|\Delta_j|, ranging from 0 to 1, where 0 means that all views give identical gains.
  • Filtered by common gain: the fraction of tasks with a positive mean gain whose common gain is not positive.
BenchmarkModelSign disagreementRelative inconsistencyFiltered by common gain
WebShop9B15.0%0.3625.6%
WebShop4B21.2%0.4230.2%
SpreadSheet9B25.0%0.3318.2%
SpreadSheet4B52.5%0.5144.8%

The views disagree substantially on the same task, and a considerable portion of the tasks with a positive mean gain are filtered out as a result. The consistency term therefore remains active rather than degenerating into mean gain, which is consistent with Table 2.

Number of views. We ran complete knowledge evolution on WebShop and LiveMath with N=2,3,4N=2,3,4, replaying each candidate once and keeping all other settings unchanged. The final Test accuracy (%) is:

NNWebShop 9BWebShop 4BLiveMath 9BLiveMath 4B
265.8059.0055.3051.99
366.4060.4053.9752.32
465.4064.4054.6449.34

Increasing NN from 2 to 3 improves most settings, while increasing it to 4 does not bring consistent further gains, indicating that performance saturates quickly. Since each additional view increases both replay and optimizer calls linearly, the default N=3N=3 is a reasonable trade-off between effectiveness and cost.


W2: Computational overhead.


We first clarify that multiple rollout groups and self-distillation are not additional costs introduced by SPLICE, but are shared by many existing methods that update the model with RL. SkillRL and ReSkill likewise sample multiple rollouts per task and update their skills during training, and methods that internalize knowledge into model parameters generally also require self-distillation or even heavier training procedures [1,2]. Our measurements agree with this: the training-step cost of SPLICE per parameter update is 0.573 GPU-hours for 9B and 0.272 for 4B, comparable to 0.520 and 0.243 for ReSkill and lower than 0.571 for SkillRL 4B.

The additional cost of SPLICE lies in knowledge evolution. The table below reports the GPU-hours and optimizer tokens per round of the knowledge evolution methods on WebShop. All methods were run on a single 8×H200 node.

MethodModelRolloutKnowledge optimizationInput tokensOutput tokens
ReasoningBank9B0.2390.171414K8.8K
ReasoningBank4B0.1690.167375K8.6K
ACE9B0.7070.677745K65.9K
ACE4B0.3830.493536K37.0K
EvoSkill9B0.2220.168226K5.9K
EvoSkill4B0.1550.250178K7.6K
SkillOpt9B0.2160.117493K6.3K
SkillOpt4B0.1770.131598K6.9K
SPLICE9B0.2500.248764K11.3K
SPLICE4B0.1700.272804K12.0K

SPLICE costs more per round than ReasoningBank, EvoSkill, and SkillOpt, while remaining well below ACE. The extra cost comes from the multiple views: each view requires its own optimizer calls, which form the largest component (0.248 GPU-hours for 9B and 0.272 for 4B), and replaying all candidates adds 0.102 and 0.065 GPU-hours. In co-evolution, knowledge evolution runs in parallel with training and therefore does not noticeably slow it down.

This additional cost is worthwhile. SPLICE achieves the highest average accuracy in both the training-free and the training-based groups, and the gains do not come from extra computation alone: the Best branch variant in Figure 4 also generates multiple views and replays their candidates, yet performs clearly worse, indicating that the benefit comes from how common gain exploits utility consistency across views.

[1] Shi Y, Chen Y, Lu Z, et al. Skill1: Unified evolution of skill-augmented agents via reinforcement learning, 2026.

[2] Wu R, Wang X, Mei J, et al. Evolver: Self-evolving llm agents through an experience-driven lifecycle, 2025.


W3 & Q4: Validation overfitting.


In our setup, the validation and test splits are strictly separated: the test split never enters knowledge evolution process. The test results in Table 1 therefore already reflect the generalization of the evolved knowledge. Nevertheless, we agree with the reviewer that repeatedly selecting updates on a fixed validation set could lead to overfitting it. We therefore conducted three additional experiments.

Denser validation and Test curves. On three WebShop 9B knowledge evolution runs, we re-evaluated Test500 with the knowledge retained at certain rounds and compared it with the validation accuracy of the same round. The table reports the average over the three runs (%).

RoundVal100Test500
07.010.20
2051.056.33
4052.356.73
6056.359.13
8056.359.13

The two curves rise together, and we do not observe a divergence where validation keeps improving while Test stalls or drops.

An independent holdout. We held out 400 tasks from the official WebShop dev split that overlap neither the 100 validation tasks nor Test500. They never enter knowledge evolution process, and are used only for post hoc evaluation (accuracy, %).

Round9B Val / Holdout4B Val / Holdout
08.0 / 11.252.0 / 2.50
2049.0 / 52.5053.0 / 55.25
4049.0 / 52.5053.0 / 55.25
6053.0 / 60.0061.0 / 59.00
8053.0 / 60.0061.0 / 59.00

The improvements obtained on the validation set also appear on this holdout, which never took part in any selection.

Dynamic validation splits. We also tested a rotating validation set: the 600 tasks from the dev split are divided into six non-overlapping subsets of 100 tasks, and the active subset rotates every certain rounds. At each rotation the current knowledge is re-evaluated. After the knowledge evolution, the final accuracy is 75.00 on Test500 and 76.25 on holdout400 for 9B, and 65.60 and 60.00 for 4B. Together, these three experiments show no sign that SPLICE overfits a fixed validation set and remains effective when the validation set rotates during evolution.

Overall, the improvements from knowledge evolution also hold on data that never participated in selection, and we do not observe signs of validation overfitting. A more systematic comparison of validation and holdout strategies is left for future work.


W4 & Q2: Knowledge optimizer.


We would like to first clarify that using an LLM to analyze experience and generate knowledge is a common practice in knowledge evolution [1,2,3] rather than a design choice specific to SPLICE. SPLICE also makes no particular assumption about the optimizer. Its contribution lies in selecting evidence through utility consistency across evolution views, and the single-view knowledge generation can be performed by any optimizer.

We nevertheless agree with the reviewer that it is important to examine the robustness of SPLICE to the choice of optimizer. To this end, we replaced the optimizer on WebShop with three weaker models, including models from other families: Claude Haiku 4.5, Qwen3-Next-80B-A3B, and DeepSeek-V3. Under each optimizer, SPLICE, SkillOpt, and EvoSkill each run complete knowledge evolution from empty knowledge with the task model unchanged. The final Test accuracy (%) is:

Task modelMethodHaiku 4.5Qwen3-NextDeepSeek-V3
9BSkillOpt7.6012.8011.80
9BEvoSkill7.808.6011.20
9BSPLICE49.6033.0028.00
4BSkillOpt12.402.802.00
4BEvoSkill1.405.002.20
4BSPLICE49.8039.4021.40

SPLICE clearly outperforms the other two methods under every optimizer and for both model sizes, whereas the two baselines nearly fail with the weaker optimizers. The main reason is that weaker optimizers often produce lengthy and overly procedural knowledge. For example, an EvoSkill skill generated by Haiku 4.5 requires multi-step validation before every search, causing many actions to be truncated. SPLICE also produces candidates of mixed quality, but such candidates show consistently negative gains across views during replay and are filtered out by common gain, so only evidence that brings gains under multiple views is retained.

These results suggest that SPLICE remains effective across different optimizers. This is consistent with our motivation: by retaining only evidence whose utility is consistent across views, SPLICE can still extract generalizable knowledge.

[1] Zhao A, Huang D, Xu Q, et al. Expel: Llm agents are experiential learners, 2024

[2] Zhang Q, Hu C, Upasani S, et al. Agentic context engineering: Evolving contexts for self-improving language models, 2026

[3] Yang Y, Gong Z, Huang W, et al. Skillopt: Executive strategy for self-evolving agent skills, 2026

Dear Reviewer,

Thank you again for your thoughtful and constructive review. We have carefully considered your feedback, conducted additional experiments and analyses to address your concerns, and revised the manuscript accordingly.

As the discussion period draws to a close, we would sincerely appreciate your feedback on whether our responses have addressed your main concerns. We would be happy to discuss and clarify any remaining questions.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_7xncOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SPLICE, a framework for co-evolving external knowledge and LLM agents. It generates knowledge candidates by perturbing experience-analysis instructions and selects update evidence using common gain, which combines mean replay gains with a cross-view disagreement penalty. For model evolution, it combines GRPO with gated contrastive self-distillation using knowledge associated with consistent utility degradation as a contrastive reference. Experiments across four benchmarks and two model sizes report improvements in average accuracy, token efficiency, and cross-model knowledge transfer.

Critical strengths
  1. Cross-view utility consistency provides a principled criterion for reducing reliance on isolated successes. The common-gain ablation (Table 2) and multi-view strategy comparisons (Figure 4) support this design.
  2. The method achieves strong average performance across four task domains and two model sizes. Cross-model knowledge transfer further suggests practical value.
Critical weaknesses
  1. The paper lacks controlled ablations comparing standard self-distillation with variants incorporating degradation-derived reference knowledge and token-level gating. Their individual contributions to the reported improvements therefore remain unclear.
  2. Each knowledge candidate is evaluated with only one replay, although the paper includes comparisons of multi-view selection strategies, end-to-end computational costs relative to co-evolution baselines remain insufficiently characterized.
Questions for authors
  1. Can the authors provide controlled ablations comparing standard self-distillation with CSD variants that remove the negative reference or token-level gating?
  2. How stable are the gains across independent training runs and increased replay counts? Can the authors report uncertainty estimates and end-to-end computational budgets for the training-based comparisons?
3 comments

Thank you for the thoughtful and constructive review and for recognizing the utility-consistency criterion and our empirical results. We address your concerns below.


W1 & Q1: Ablation of CSD.


Following the reviewer's suggestion, we conducted this controlled ablation. The knowledge evolution procedure, the number of rollouts with and without knowledge, and the training budget are kept unchanged, and only the training objective varies:

  • No self-distillation (λ=0\lambda=0)
  • Standard self-distillation, i.e., CSD without K−K^- (γt=0\gamma_t=0)
  • CSD without the token-level gate (γt=1\gamma_t=1)
  • Full CSD

The last three use λ=0.3\lambda=0.3. Experiments are conducted on WebShop and LiveMath. We report the Test accuracy (%) averaged over three evaluation seeds.

ObjectiveWebShop 9BWebShop 4BLiveMath 9BLiveMath 4B
No self-distillation (λ=0\lambda=0)66.1375.8064.2465.67
Standard SD (γt=0\gamma_t=0)73.4074.9366.2365.89
CSD w/o token-level gate (γt=1\gamma_t=1)70.8061.5363.8066.23
Full CSD76.1378.6768.8767.44

Full CSD achieves the best accuracy in all settings, outperforming both standard SD and the ungated variant. This shows that CSD is beneficial, and that both the negative reference and the token-level gate contribute. To exclude the influence of variation in the evolved knowledge, we further conducted a fixed-knowledge experiment on WebShop 9B, where all four objectives share the same final KK from Know. Evo and its corresponding K−K^-. Full CSD again performs best (79.87%), while removing either the contrastive reference (73.20%) or the gate (69.53%) falls below no self-distillation (76.87%). This is consistent with Section 3.2: K−K^- complements the guidance from KK, and the gate preserves the useful guidance they share.


W2 & Q2: Stability across training runs and replay counts, and computational budgets.


Stability across independent training runs. We repeated the complete SPLICE (Co-Evo) training on WebShop with five independent training seeds, keeping all other settings unchanged. The result of each training run is the Test accuracy averaged over three evaluation seeds. Across the five runs, the mean ± sample standard deviation is 76.73 ± 8.15 for 9B and 71.32 ± 4.91 for 4B. Although results vary across training seeds, the average accuracy of SPLICE remains higher than that of all training-based baselines in Table 1.

Number of replays. We first conducted a repeated-replay experiment with the model and candidates fixed. On WebShop and LiveMath, we take six evenly spaced rounds from SPLICE (Know. Evo) run per model size. With the model and the three candidates of each round fixed, each candidate knowledge is replayed 15 times on all tasks of the round. We split the 15 replays into five non-overlapping groups. Each group computes common gain from the mean reward of its first RR replays (R=1,2,3R=1,2,3) and selects tasks accordingly. We then measure the consistency across the five groups with three metrics:

  • Common-gain variance: the variance of a task's common gain across the five groups.
  • Rank correlation: the Spearman correlation between groups when all tasks are ranked by common gain.
  • Selection disagreement: the fraction of all tasks in a round whose selection status differs between two groups.

Consistency of common gain and task selection (RR = 1 / 2 / 3)

BenchmarkModelCommon-gain varianceRank correlationSelection disagreement
WebShop9B0.040 / 0.022 / 0.0150.75 / 0.85 / 0.862.5% / 2.2% / 3.3%
WebShop4B0.048 / 0.021 / 0.0140.71 / 0.83 / 0.866.1% / 7.2% / 6.7%
LiveMath9B0.072 / 0.036 / 0.0260.75 / 0.86 / 0.8911.1% / 9.4% / 8.3%
LiveMath4B0.061 / 0.031 / 0.0200.78 / 0.84 / 0.898.1% / 8.1% / 5.8%

As RR increases, the variance decreases and the rank correlation increases, but even a single replay already yields fairly stable scores and selections. Selection disagreement mainly occurs for tasks whose common gain is close to zero and just crosses the threshold. To examine whether such changes admit ineffective evidence, we select tasks using a single replay and recompute their common gain on the remaining 14 replays. The selected tasks keep a positive mean held-out common gain in all settings (+0.21 to +0.56), showing that evidence selected with a single replay remains beneficial.

To further examine whether more replays change the final outcome, we ran complete knowledge evolution on WebShop and LiveMath with R=1,2,3R=1,2,3. Each candidate is replayed RR times, and rewards are averaged before computing common gain. The number of views, the task model, the training tasks, and the number of evolution rounds are kept unchanged.

Effect of the number of replays on final performance (Test accuracy, %)

BenchmarkModelR=1R=1R=2R=2R=3R=3
WebShop9B66.4068.8067.40
WebShop4B60.4062.0064.60
LiveMath9B53.9753.3156.29
LiveMath4B52.3255.3055.63

Increasing the number of replays generally brings some improvement, but the gains are limited, while the replay cost grows linearly with RR. The default R=1R=1 already achieves competitive results and offers a reasonable trade-off between effectiveness and cost. We have added these results to the revised appendix.

End-to-end computational budgets of training-based methods. We measured the GPU-hours per parameter update of each training-based method on WebShop. All methods were run on a single 8×H200 node.

GPU-hours per parameter update

MethodModelParameter updateRolloutTraining stepKnowledge evolutionTotal
SkillRL9B0.1980.2200.4180.0500.468
SkillRL4B0.1310.4400.5710.0790.650
ReSkill9B0.1780.3420.5200.0380.558
ReSkill4B0.0760.1670.2430.0110.254
SPLICE9B0.1620.4110.5730.1620.735
SPLICE4B0.0880.1840.2720.0880.360

SPLICE's additional cost comes from multi-view knowledge evolution, namely candidate replay and the optimizer calls of multiple views. Since knowledge evolution runs in parallel with training, this overlap reduces the time overhead. Overall, the additional computation is worthwhile, as SPLICE achieves the highest average accuracy among the training-based methods. This advantage does not come merely from spending more computation. The Best branch variant in Figure 4 also generates multiple views and replays their candidates, yet performs clearly worse, indicating that the gains come from consistency regularization.

Dear Reviewer,

Thank you again for your thoughtful and constructive review. We have carefully considered your feedback, conducted additional experiments and analyses to address your concerns, and revised the manuscript accordingly.

As the discussion period draws to a close, we would sincerely appreciate your feedback on whether our responses have addressed your main concerns. We would be happy to discuss and clarify any remaining questions.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_8dNVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPLICE jointly evolves an agent's external knowledge and policy. It generates knowledge candidates from different analyses of the same experience, then replays the source tasks to measure their utility. A common-gain score favors task evidence whose reward improvements are consistent across candidates. The selected evidence is consolidated into a knowledge update and checked on held-out tasks before adoption. Policy training combines GRPO with task-weighted, gated contrastive self-distillation, using knowledge associated with consistent degradation as a negative reference. Knowledge and policy updates proceed asynchronously. The experiments cover four benchmarks and two model sizes, with additional transfer and output-token analyses.

Critical strengths
  1. I think the knowledge-selection idea is well motivated. Different lessons from the same experience can sound reasonable yet have different effects on behavior. Common gain gives a simple way to favor evidence that remains useful across these alternatives, and the theoretical analysis explains its conservative behavior under the stated assumptions.
  2. SPLICE matches or exceeds the listed training-free baselines in all eight model/benchmark settings in Table 1, with average gains of 6.56 and 7.05 percentage points over SkillOpt.
Critical weaknesses
  1. The knowledge optimizer is central to SPLICE, yet its model/version and full prompts are not specified. Table 5 lists possible values of (n) and (\lambda) without reporting the settings used in each experiment. Matching training epochs also does not account for differences in replay, validation, and external-LLM costs. These omissions make the method difficult to reproduce and the comparison budgets difficult to assess. The authors should provide the exact configurations and a cost breakdown for SPLICE and its baselines.
  2. The main comparisons change both knowledge evolution and model training, leaving CSD’s added value unclear. A controlled ablation should compare no self-distillation ((\lambda=0)), self-distillation without the contrastive term ((\gamma_t=0) for all tokens), and the full objective. The knowledge-evolution procedure and the number of rollouts with and without knowledge should remain unchanged. This would show whether CSD adds value beyond knowledge evolution and GRPO.
  3. Table 5 uses only three views and one replay per candidate, so the score may reflect execution noise as well as differences in knowledge utility. Proposition 1 concerns expected reward gains and does not establish the reliability of this finite-replay estimate. Repeating replay evaluations with the model and candidates fixed, then reporting score variation and changes in the selected tasks, would directly test this concer.
Questions for authors

These are additional clarifications and presentation suggestions.

  1. Please explain the resampling procedure used to compute the paired confidence intervals in Figure 6, including the resampling unit and how pairing is preserved. Also clarify which steps of knowledge evolution and evaluation are repeated across the three seeds.
  2. The “up to 44%” claim should refer specifically to the reduction in output tokens per correct attempt, since Table 3 excludes input knowledge tokens and the costs of knowledge evolution and training.
4 comments

Thank you for the thoughtful and constructive review and for recognizing the paper’s motivation and empirical results. We address each concern below.


W1: More details of the experiments.


1. Knowledge optimizer and prompts. All knowledge evolution methods use Claude Sonnet 4.6 as the knowledge optimizer. In SPLICE, the optimizer is called to (i) generate the analysis-instruction revisions that define the NN evolution views for a batch of experience; (ii) propose and edit a knowledge candidate under each view; (iii) consolidate the knowledge update from the evidence of the selected tasks; and (iv) construct the noisy knowledge K−K^- in co-evolution.

The paper focuses on how SPLICE selects evidence through utility consistency across evolution views, rather than on the specific pipeline that generates each single-view candidate. We thank the reviewer for this question and have revised the appendix to include these details and prompts for better reproducibility.

In new experiments on WebShop, we further replaced the optimizer with three weaker models, and compared SPLICE, SkillOpt, and EvoSkill. SPLICE achieves the highest accuracy among the three methods with all three alternative optimizers and both model sizes (see our response to Reviewer V8N7).

2. Experimental configurations. All experiments in the paper use N=3N=3 evolution views and one replay per candidate (R=1R=1). In co-evolution, the CSD loss weight λ\lambda (Eq. 7) is 0.3 for all reported results. The most frequently used evolution batch size BB and number of selected tasks nn for each benchmark are:

BenchmarkEvolution batch BBSelected tasks nn
LiveMath122
LiveCode204
SpreadSheet101
WebShop122 (4B) / 3 (9B)

We have corrected the range of nn in Table 5 to {1,2,3,4}\{1,2,3,4\}. For the baselines, we primarily follow their original implementations and the configurations reported in their papers.

3. Computational budgets. A strictly compute-matched comparison is difficult. Training-free methods spend computation mainly on knowledge evolution and API calls, whose procedures vary widely: on WebShop, ACE's repeated reflection and re-execution cost 2.6–3.4× the GPU-hours of ReasoningBank per round. Training-based methods spend it mainly on rollouts and parameter updates. Therefore, we ensure that every training task is used by the model the same number of times, i.e., all methods are trained for the same number of epochs (Table 4). On top of this, we measured the GPU-hours of each method on WebShop and report their composition below. All methods were run on a single 8×H200 node.

Training-free methods: GPU-hours and optimizer tokens per round

MethodModelRolloutKnowledge optimizationInput tokensOutput tokens
ReasoningBank9B0.2390.171414K8.8K
ReasoningBank4B0.1690.167375K8.6K
ACE9B0.7070.677745K65.9K
ACE4B0.3830.493536K37.0K
EvoSkill9B0.2220.168226K5.9K
EvoSkill4B0.1550.250178K7.6K
SkillOpt9B0.2160.117493K6.3K
SkillOpt4B0.1770.131598K6.9K
SPLICE9B0.2500.248764K11.3K
SPLICE4B0.1700.272804K12.0K

The extra cost of SPLICE comes mainly from knowledge optimization with multiple views, while candidate replay accounts for only 0.07–0.10 GPU-hours per round. ACE is the most expensive because its growing playbook is included in every rollout (about 59K input tokens per call with 9B, versus fewer than 5K for the other methods), in addition to its repeated re-execution. This extra computation is put to effective use: the Best branch variant in Figure 4 spends the same multi-view generation and replay but performs clearly worse, showing that the gains come from the proposed consistency regularization rather than from additional computation alone.

Training-based methods: GPU-hours per parameter update

MethodModelParameter updateRolloutTraining stepKnowledge evolutionTotal
SkillRL9B0.1980.2200.4180.0500.468
SkillRL4B0.1310.4400.5710.0790.650
ReSkill9B0.1780.3420.5200.0380.558
ReSkill4B0.0760.1670.2430.0110.254
SPLICE9B0.1620.4110.5730.1620.735
SPLICE4B0.0880.1840.2720.0880.360

The training-step cost of SPLICE is close to that of ReSkill. The additional cost comes from multi-view knowledge evolution, namely candidate replay and the optimizer calls of multiple views. In the asynchronous design, knowledge evolution runs on the inference GPUs concurrently with parameter updates, and this overlap reduces the time overhead. Overall, the additional cost of SPLICE is worthwhile, as it brings clear accuracy gains: SPLICE achieves the highest average accuracy in both the training-free and the training-based groups.


W2: Ablation of CSD.


Following the reviewer's suggestion, we conducted this controlled ablation. The knowledge evolution procedure, the number of rollouts with and without knowledge, and the training budget are kept unchanged, and only the training objective varies:

  • No self-distillation (λ=0\lambda=0)
  • Self-distillation without the contrastive reference (γt=0\gamma_t=0)
  • CSD without the token-level gate (γt=1\gamma_t=1)
  • Full CSD

The last three use λ=0.3\lambda=0.3. Experiments are conducted on WebShop and LiveMath. We report the Test accuracy (%) averaged over three evaluation seeds.

ObjectiveWebShop 9BWebShop 4BLiveMath 9BLiveMath 4B
No self-distillation (λ=0\lambda=0)66.1375.8064.2465.67
SD w/o contrastive reference (γt=0\gamma_t=0)73.4074.9366.2365.89
CSD w/o token-level gate (γt=1\gamma_t=1)70.8061.5363.8066.23
Full CSD76.1378.6768.8767.44

Full CSD achieves the best accuracy in all settings, showing that CSD is beneficial. To exclude the influence of variation in the evolved knowledge, we further conducted a fixed-knowledge experiment on WebShop 9B, where all four objectives share the same final KK from Know. Evo and its corresponding K−K^-. Full CSD again performs best (79.87%), while removing either the contrastive reference (73.20%) or the gate (69.53%) falls below no self-distillation (76.87%). This is consistent with Section 3.2: K−K^- complements the guidance from KK, and the gate preserves the useful guidance they share.


W3: Reliability of common gain.


We conducted the suggested experiment as follows. On WebShop and LiveMath, we take six evenly spaced rounds from SPLICE (Know. Evo) run per model size. With the model and the three candidates of each round fixed, each candidate knowledge is replayed 15 times on all tasks of the round. We split the 15 replays into five non-overlapping groups. Each group computes common gain from the mean reward of its first RR replays (R=1,2,3R=1,2,3) and selects tasks accordingly. We then measure the consistency across the five groups with three metrics:

  • Common-gain variance: the variance of a task's common gain across the five groups.
  • Rank correlation: the Spearman correlation between groups when all tasks are ranked by common gain.
  • Selection disagreement: the fraction of all tasks in a round whose selection status differs between two groups.

Consistency of common gain and task selection (RR = 1 / 2 / 3)

BenchmarkModelCommon-gain varianceRank correlationSelection disagreement
WebShop9B0.040 / 0.022 / 0.0150.75 / 0.85 / 0.862.5% / 2.2% / 3.3%
WebShop4B0.048 / 0.021 / 0.0140.71 / 0.83 / 0.866.1% / 7.2% / 6.7%
LiveMath9B0.072 / 0.036 / 0.0260.75 / 0.86 / 0.8911.1% / 9.4% / 8.3%
LiveMath4B0.061 / 0.031 / 0.0200.78 / 0.84 / 0.898.1% / 8.1% / 5.8%

As RR increases, the variance decreases and the rank correlation increases, but even a single replay already yields fairly stable scores and selections. Selection disagreement mainly occurs for tasks whose common gain is close to zero and just crosses the threshold.

To examine whether such changes admit ineffective evidence, we select tasks using a single replay and recompute their common gain on the remaining 14 replays. The selected tasks keep a positive mean held-out common gain in all settings (+0.21 to +0.56), showing that evidence selected with a single replay remains beneficial.

To further examine whether more replays change the final outcome, we ran complete knowledge evolution on WebShop and LiveMath with R=1,2,3R=1,2,3. Each candidate is replayed RR times, and rewards are averaged before computing common gain. The number of views, the task model, the training tasks, and the number of evolution rounds are kept unchanged.

Effect of the number of replays on final performance (Test accuracy, %)

BenchmarkModelR=1R=1R=2R=2R=3R=3
WebShop9B66.4068.8067.40
WebShop4B60.4062.0064.60
LiveMath9B53.9753.3156.29
LiveMath4B52.3255.3055.63

Increasing the number of replays generally brings some improvement, but the gains are limited, while the replay cost grows linearly with RR. The default R=1R=1 already achieves competitive results and offers a reasonable trade-off between effectiveness and cost. We have added these results to the revised appendix.


Q1: Clarification on evaluation details.


We would like to clarify both parts below.

Paired confidence intervals in Figure 6. The knowledge of each method is fixed to the final knowledge it evolved with Qwen3.5-9B and is applied to the target model without further evolution. For each target model, benchmark, and method, every test task is executed 8 times, and the gain is the difference in avg@8 accuracy between using and not using the knowledge on the same target model. We compute the intervals with a paired bootstrap over test tasks. We first average the 8 executions of each task, and then resample the test tasks with replacement 10,000 times. In each resample, both conditions use the same set of tasks, and we compute the mean gain. The 95% confidence interval is given by the 2.5th and 97.5th percentiles of the resampled gains.

What is repeated across the three seeds. For the training-free methods in Table 1, each seed is an independent, complete knowledge evolution run: it starts from empty knowledge, reruns the entire evolution procedure with a different sampling seed, and evaluates the final knowledge on the Test set. The reported value is the average over these three runs.


Q2: Presentation.


We thank the reviewer for pointing this out and have revised the paper accordingly.

Dear Reviewer,

Thank you again for your thoughtful and constructive review. We have carefully considered your feedback, conducted additional experiments and analyses to address your concerns, and revised the manuscript accordingly.

As the discussion period draws to a close, we would sincerely appreciate your feedback on whether our responses have addressed your main concerns. We would be happy to discuss and clarify any remaining questions.

Thank you for your time and consideration.

Best regards, The Authors

Comments on the paper

2 comments

Dear Area Chairs and Reviewers,

We sincerely thank the AC for overseeing the review process and the reviewers for their thoughtful and constructive feedback. During the rebuttal, we conducted targeted additional experiments and analyses to address the central concerns. We summarize the paper and the new evidence below.

1. Paper in one paragraph

  • Problem: Different ways of analyzing the same agent experience can yield knowledge with substantially different utility, even when the resulting guidance appears reasonable. Relying on any single analysis risks adopting knowledge whose gains reflect an occasional success or view-specific guidance, which co-evolution may then internalize into the model.
  • Method: We propose SPLICE, which extends consistency regularization from predictions to knowledge utility. It perturbs how the same experience is analyzed to form multiple evolution views, and updates knowledge using common gain, which favors utility gains shared across views. Model evolution combines GRPO with gated contrastive self-distillation, using knowledge associated with consistent utility degradation as a contrastive reference.
  • Results: On math, coding, spreadsheet, and web-shopping benchmarks with Qwen3.5-4B and 9B, SPLICE improves average accuracy over the strongest knowledge evolution baseline by 6.6 and 7.1 points, and over the strongest co-evolution baseline by 3.0 and 4.6 points. Its evolved knowledge reduces output tokens per correct attempt by up to 44% and transfers across model sizes and families.

2. Main issues raised and what we did

Contribution of CSD

  • Controlled ablation (8dNV, 7xnc): Under a fixed training budget, full CSD is best in all four settings, outperforming standard self-distillation by 1.55 to 3.74 points.
  • Fixed-knowledge check: With identical knowledge, removing the contrastive reference or the gate drops accuracy below no self-distillation (73.20 / 69.53 vs. 76.87), while full CSD reaches 79.87.

Reliability of common gain

  • Replay noise (8dNV, 7xnc): A single replay already gives stable rankings (Spearman 0.71 to 0.78), and the selected tasks keep positive gains on 14 held-out replays (+0.21 to +0.56).
  • The i.i.d. assumption (V8N7): It is needed only for the bound in Proposition 1. Views still disagree in sign on 15.0% to 52.5% of tasks, so the consistency term remains active.
  • Number of views (V8N7): Performance saturates at 3 views, supporting our default.

Robustness and generalization

  • Weaker optimizers (V8N7): With Claude Haiku 4.5, Qwen3-Next, or DeepSeek-V3, SPLICE reaches 21.40 to 49.80 on WebShop, while SkillOpt and EvoSkill reach at most 12.80.
  • Validation overfitting (V8N7): Gains carry over to an independent 400-task holdout never used for selection (e.g., 11.25 to 60.00 for 9B).
  • Training seeds (7xnc): Over five independent runs, SPLICE's mean accuracy exceeds every training-based baseline on WebShop.

Cost and reproducibility

  • Cost (8dNV, 7xnc, V8N7): SPLICE's training-step cost is comparable to ReSkill, and its per-round cost is far below ACE. The Best-branch variant uses the same multi-view compute yet performs clearly worse, so the gains do not come from computation alone.
  • Reproducibility (8dNV): We added full prompts and exact configurations to the appendix.

3. Current status

  • Methodological contribution: All three reviewers found the core idea well motivated. 8dNV highlighted the knowledge-selection idea and its theoretical analysis, 7xnc viewed cross-view utility consistency as a principled criterion for reducing reliance on isolated successes, and V8N7 described it as intuitive and relatively novel.
  • Empirical support: Reviewers recognized that SPLICE matches or exceeds all training-free baselines in every setting (8dNV), performs strongly across four domains and two model scales, and is supported by ablations that directly test the central mechanism and by convincing cross-model transfer (7xnc, V8N7). The new CSD ablation, multi-seed runs, and weaker-optimizer experiments further strengthen this evidence.
  • Concerns addressed: We believe our new experiments and analyses fully address the main concerns. As the reviewers have not yet responded to our rebuttal, we hope this summary helps in assessing whether these concerns have been resolved.

Overall, we sincerely appreciate the reviewers' constructive feedback, which has helped us strengthen the paper. We hope this summary provides a useful overview.

Dear Reviewers,

As the discussion period is coming to an end tomorrow, I would be grateful if you could review the authors' rebuttals and share your thoughts on whether your concerns have been adequately addressed, including any remaining questions.

Thank you very much for your time and thoughtful efforts throughout the review process!

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.