acceptodds
Under review as a conference paper at ICLR 2027

Discover and Internalize: A Continual Co-Evolving Training Loop for Self-Improvement Agents

Abstract

Agent self-improvement requires not only learning better behaviors, but also discovering what better behaviors could be. Existing approaches typically couple behavioral discovery with policy optimization, limiting the exploration of alternative behaviors and making it difficult to fully exploit potentially useful behavioral strategies. We reframe agent self-improvement as a behavioral hypothesis search problem and view skills as a behavioral hypothesis space: a skill is a candidate hypothesis to be tested, rather than an already validated capability. Based on this view, we propose Discover-and-Internalize Co-Evolution (DICE), a co-evolutionary framework between behavioral discovery and policy internalization. Behavioral discovery evolves and composes skills to explore new behavioral hypotheses, while policy internalization consolidates promising behaviors into the model weights. The updated policy then provides a stronger foundation for the next round of discovery, forming a mutually reinforcing evolution loop. Our results demonstrate that explicitly separating behavioral discovery from policy internalization, while combining them into a co-evolutionary loop, provides an effective paradigm for continual agent self-improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, agentic RL, on policy distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9LVKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a self-improvement method for LLM agents. The method establishes a teacher–student distillation framework and employs an LLM-based skill curator to evolve and compose skills. Based on task outcomes, it dynamically determines the direction of distillation between a skill-free Student and skill-augmented Teachers. Finally, the policy is jointly trained using GRPO and correctness-gated distillation.

Critical strengths

Major Strengths

The proposed method is very interesting.

Minor Strengths

The paper cites recent literature and builds upon recent research.

Critical weaknesses

The paper contains significant errors in its references. The cited work "AlphaEvolve Team. Alphaevolve: A coding agent for scientific and mathematical discovery" cannot be found, while a closely related work exists [1]. Additionally, the authors of "Training Language Models to Self-Correct via Reinforcement Learning" are incorrectly listed. These errors make me suspect that the authors used AI generative tools to produce the references without verifying them. Furthermore, the Introduction contains no references, making it difficult to verify the accuracy of some statements.

[1] Novikov, Alexander, et al. "Alphaevolve: A coding agent for scientific and algorithmic discovery." arXiv preprint arXiv:2506.13131 (2025).

The novelty of the paper's core contributions is insufficiently established. Previous works such as SkillRL and UCOB have already explored combining external behavioral discovery with policy internalization, while UCOB has also employed bidirectional distillation based on behavioral differences between Skill and No-Skill configurations. Moreover, the experimental comparisons lack closely related methods such as UCOB.

Questions for authors
  1. Can the authors clarify the differences between the proposed method and existing methods such as SkillRL and UCOB? If the main difference is simply introducing Teachers with different Skill configurations, how does this constitute a substantial methodological contribution?

  2. Can the authors provide additional experimental comparisons with closely related methods, particularly UCOB, to better demonstrate the effectiveness of the proposed method?

  3. Can the authors correct the reference errors and provide appropriate citations in the Introduction, particularly for statements such as "Existing on-policy distillation and self-distillation approaches largely emphasize the second process"?

  4. Can the authors further analyze the effect of using multiple Teachers, for example, by comparing with a strategy that performs multiple runs and selects the best one?

  5. Can the authors provide additional ablation studies or discussion to clarify the contribution of correctness gating and the roles of different gating mechanisms?

Reviewer_EEwLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Discover-and-Internalize Co-Evolution (DICE), a framework that alternates between external skill discovery and policy internalization. A skill-free student and multiple skill-conditioned teachers share policy weights and generate trajectories under different contexts. Verified outcomes determine the direction of distillation, allowing teacher-to-student, student-to-teacher, and cross-teacher transfer. An independent LLM curator updates the skill repository, while the policy is optimized jointly with GRPO and correctness-gated distillation. Experiments on ALFWorld and WebShop using Qwen3-1.7B and Qwen3-4B report substantial improvements, primarily evaluated on the skill-free student. Validation includes component ablations, a comparison of global and turn-level distillation, and analyses of discovery–internalization dynamics.

Critical strengths

Evaluating the skill-free Student directly addresses whether training-time skill assistance produces capabilities that persist without repository access at deployment. The reported Qwen3-4B improvements over the strongest other methods in Table 1 are substantial: 14.0 percentage points on ALFWorld and 3.6 percentage points in WebShop success rate. These results make the framework worth investigating and suggest potential practical value. DICE operationalizes the observation that skill-conditioned configurations can either help or mislead the policy. Its supervision graph supports reverse correction and transfer across different skill compositions. Cross-Teacher transfer is a potentially useful extension beyond two-view formulations, provided that its independent benefit can be established under controlled conditions.

Critical weaknesses

The framework is coherent, but its methodological contribution appears primarily incremental. Several central components have already been explored in closely related work: skill validation and skill-free internalization in SIRI, evolving hindsight-based supervision in SEED, and performance-dependent bidirectional distillation with skill evolution in UCOB. DICE’s most identifiable extension is the use of multiple skill-conditioned configurations and cross-teacher transfer within a continuing skill-evolution loop. While this is a reasonable extension, the conceptual overlap with prior work limits the paper’s originality. Given the incremental nature of the method, direct comparisons with the closest approaches are particularly important for assessing its significance. However, the main results do not include comparisons with UCOB, SIRI, or SEED under comparable conditions, leaving DICE’s empirical advantage over these alternatives unclear. Consistent improvements over these methods would strengthen the paper’s contribution, even with limited conceptual novelty. The current comparisons therefore do not fully establish the empirical value of the proposed extension.

Questions for authors
  1. What additional benefit does student-to-teacher distillation provide? When the skill-free student succeeds but a skill-conditioned teacher fails, why not omit the reverse distillation signal and revise or prune the misleading skills, while retaining the standard RL updates? Since the student and teachers share policy weights, student-to-teacher distillation does not train an independent teacher model. What additional benefit does this reverse distillation provide for the final skill-free policy? Could the authors quantify its marginal contribution through an ablation that enables or disables student-to-teacher distillation while keeping the other training components and budgets comparable?

  2. How does DICE compare with the closest approaches? Could the authors provide direct comparisons with closely related methods, particularly UCOB, SIRI, and SEED, using comparable backbone models, evaluation protocols, and training budgets, while explicitly accounting for differences in skill generation and curator usage? These comparisons would help establish whether DICE’s multi-configuration design and cross-teacher transfer yield consistent empirical improvements over existing approaches, and could materially affect my assessment of the paper’s contribution.

Reviewer_zTV1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents DICE, a method for jointly optimizing language model weights and input context represented as textual skills. It alternates between skill optimization through an independent LLM and model weight updates using reinforcement learning and distillation. Evaluated on natural language tasks against methods in reinforcement learning and distillation, DICE reports improved agent performance. Overall, the experiments support the benefit of combining skill optimization with policy training.

Critical strengths

The article makes a compelling argument for optimizing skills and weights in tandem. Policy-based updates depend on the inputs given to an LLM agent, so it makes sense to optimize both jointly. Skills are of increasing use in LLM agents, and make a natural target for context optimization. The experimental study, especially the ablations, provides evidence that combining skill optimization with weight updates improves performance over either optimizing skills without distillation or distilling from a fixed skill repository.

Critical weaknesses

The first weakness is positioning of the article relative both to the quickly growing state of the art in this field, as well as broader context. For the former, the article does not explain how DICE relates to other methods, including those used in the experimental comparison. Even a short description of how DICE differs from baselines RLSD, SDAR, and OCSD would help position. However, the greater issue is comparison with methods that do very similar things to DICE. Some of these, such as DreamCoder and Co-Harness, are cited, however many are absent. These articles propose methods that are very similar to DICE and should at least be mentioned:

  • EMPO²: Liu, Zeyuan, et al. “Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization.” International Conference on Learning Representations, 2026.
  • BetterTogether: Soylu, Dilara, et al. “Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, edited by Yaser Al-Onaizan et al., Association for Computational Linguistics, 2024, pp. 10696–10710.
  • MetaTuner: Bo, Xiaohe, et al. “Prompt and Parameter Co-Optimization for Large Language Models.” International Conference on Learning Representations, 2026.
  • MEM1: Zhou, Zijian, et al. “MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents.” International Conference on Learning Representations, 2026.

Of these, at least one should be included in the experimental study. The experiments provide evidence for DICE’s effectiveness relative to the selected baselines and for the contribution of its components. However, they do not establish its advantage over existing methods that jointly adapt model parameters and external context. A comparison with a closely related approach, particularly EMPO², under the same training and evaluation conditions would bring the experimental study up the the standards of ICLR. For the second positioning issue around the broader context, this paper is positioned in the domain of evolution, or more specifically co-evolution, without any grounding in those fields. For example, these two references would better position the contribution, especially if the algorithm of DICE was described using the terms of evolutionary algorithms:

  • Rosin, Christopher D., and Richard K. Belew. “New Methods for Competitive Coevolution.” Evolutionary Computation, vol. 5, no. 1, 1997, pp. 1–29.
  • Potter, Mitchell A., and Kenneth A. De Jong. “Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents.” Evolutionary Computation, vol. 8, no. 1, 2000, pp. 1–29.

The second, and related, weakness is the presentation of the article. The text is very difficult to follow. Terms are used before being clearly defined, complicating a linear reading of the article. Some examples of non-standard or new terms that are used before their definition, or without them completely: "policy internalization", "promising strategies", "behavioral hypothesis", "LLM curator", "reusable artifacts." These appear as technical terms, and some do map directly to things for which there are established terms, like "distillation" for "policy internalization." I think "skills" is also a new and nebulous enough term that it should be clearly defined. The central distillation mechanism depends on whether a configuration “succeeds,” but the paper does not clearly specify the benchmark-specific success criteria or when gating uses binary success versus comparisons of graded rewards. The related work and results section suffers from similar presentation issues. The related works present one or multiple references per sentence, without relating them to the current work or providing any analysis. The results section does an accounting of the experimental results without providing any clear insight or analysis on what the results mean. Finally, while the proposed method is framed as "co-evolution", nothing is explained using existing terms from evolutionary computation. The LLM "curator" acts as both a variation and selection operator, in standard terminology, and does more than curation on the skill repository, though it is not clearly described. How skills are generated and revised, how composition works, how candidates are validated, and how “least useful” entries are identified for removal are unclear. Given the combination of these presentation issues, I believe it would be very difficult for a researcher unfamiliar with this literature to parse and understand the article, much less gain insight from it.

Questions for authors

How does the repository update step work? In detail, how are skills generated and revised? How are skill candidates validated? How are entries ranked for removal?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.