acceptodds
Under review as a conference paper at ICLR 2027

CoPE: Co-Evolving Policy and Experience via Context Distillation for Open-Ended Generation

Abstract

Reinforcement learning with rubric-based rewards is widely used for open-ended generation tasks. However, scalar rewards discard the rich information contained in language feedback. Context distillation aims to internalize knowledge from external context, which we refer to as experience, into model parameters. Fixed, pre-defined context can become suboptimal as the policy improves during training. We propose CoPE, an on-policy context-distillation framework that jointly improves a single policy model's downstream-task performance and rubric-generation capability. The evolving rubrics are then used by a frozen feedback model to extract adaptive experience for subsequent policy training. CoPE includes two phases of context distillation: a) training the policy model with experience as additional context and b) training the policy model as the rubric generator to produce better rubrics for experience extraction. The two phases are performed iteratively and enhance each other during training. We evaluate CoPE on open-ended generation tasks and evaluation ability, where the policy needs to generate rubrics to choose the preferred response. Our results show that CoPE outperforms scalar reward RL and previous experience learning approaches and co-evolves the policy model on downstream tasks and rubric generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
language model, open-ended generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jBHhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Rubric-based RL compresses rubric evaluations into a scalar reward, discarding actionable feedback, while fixed rubrics can lead to suboptimal policies. The paper uses an external feedback model to extract natural-language experiences from rubrics and responses, improves the policy via on-policy context distillation, then repurposes the policy as a rubric generator for further rubric training, enabling rubric–policy co-evolution.

Critical strengths
  1. Rubrics are no longer fixed but evolve alongside policy training, while the capability of the rubric generator is also continuously improved.
  2. Rubric-based natural-language experiences provide more specific guidance than scalar rewards.
  3. The training pipeline is relatively complex, and demonstrating stable training would itself require substantial effort.
Critical weaknesses
  1. An ablation with a fixed rubric generation policy is missing to demonstrate the effectiveness of rubric training.
  2. What strategies are used to select high-quality rubrics, and why are these strategies reliable indicators of rubric quality?
  3. Why does rubric training use the highest-scoring and lowest-scoring responses as teacher context, rather than directly applying SFT to learn rubrics deemed high-quality by the authors? Moreover, since these scores are determined by the current rubrics, could using these responses as context introduce self-bias?
  4. How do the rubrics and experiences evolve throughout training? I suggest highlighting the key changes identified by the authors in the case studies, rather than presenting entire passages. Additional comparisons of experiences generated by different feedback models and rubrics generated by different policies would help clarify the improvements brought by rubric training.
  5. The paper emphasizes "co-evolution with policy and experience." When does this co-evolution reach a plateau, and what factors limit further improvement?
Questions for authors
  1. The training FLOPs x-axis in Figure 2 lacks numerical tick labels.
Reviewer_izdSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoPE, an on-policy context-distillation framework for post-training LLMs on open-ended generation tasks where ground-truth answers are unavailable. A single policy model is trained in two alternating phases: (a) a policy-training phase that distills language experience extracted by a frozen feedback model, using the current rubrics as context into the policy via on-policy context distillation; and (b) a rubric-training phase that trains the same policy to generate better rubrics using contrastive positive/negative response pairs as context. Because the rubrics evolve with the policy, the extracted experience is intended to remain aligned with the policy's current failure modes, in contrast to the fixed-rubric Experience Learning baseline. On five open-ended benchmarks with Qwen3-8B and OLMo-3-7B as policy backbones, CoPE improves over both scalar-reward RL (GRPO with rubric-as-reward) and EL, and the same policy, when used as a rubric generator, also improves scores on RewardBench 2 and JudgeBench.

Critical strengths
  • Clean algorithmic design. Algorithm 1 and Figure 1 make the α/β alternation easy to understand and re-implement. Training the policy as a rubric generator (rather than directly as an experience generator) is a smart choice: rubrics are more structured and easier to filter via deduplication and discriminativeness pruning.
  • Consistent empirical gains across configurations. Table 1 shows improvements over RL and EL for policy backbones (Qwen3-8B, OLMo-3-7B) and two feedback models (self-feedback, GPT-4o) on all five open-ended benchmarks. Table 2 shows the policy, repurposed as a rubric generator, also gains on two of the tasks.
  • Insightful ablations. Authors ablate on context design for the policy loss and the rubric loss in Table 3; evolving experience vs. evolving rubrics Table 3; initial rubric generator, rubric management, and K_max in Table ). Taken together, these isolate the role of each design choice reasonably well.
  • Encouraging out-of-domain generalization. §3.4 shows a model trained on WildChat-IF + Tulu3 improves on medical and research tasks.
Critical weaknesses
  • Lack of Novelty: The idea that fixed rubrics become stale as the policy evolves, and the idea that language feedback is richer than scalar rewards are stated are not first proposed by authors. The algorithmic machinery (KL-based context distillation, GRPO-style grouping) is standard.
  • Delta over very recent concurrent work is modest. The method builds directly on EL, EvoLM, EvoRubrics, and Rubric-ARM. Table 7 attempts to differentiate CoPE from each, but the core novelty — alternating between training a single model to generate responses and to generate rubrics — is a natural combination of existing components. A more direct experimental comparison to EvoRubrics (closest conceptually) in Table 1 would help.

Minor issues:

  • "Eq. (3) and Eq. (4) define different objectives", the key design point is that both objectives update the same parameters; worth stating explicitly.
  • §4 could cite Critique-GRPO and related RL-with-critique approaches more prominently; but currently they appear in a single sentence.
  • The term "experience" is used in both the informal and technical senses; a short formal definition in §2.1 would help readers new to the EL line of work.
Questions for authors
  • Compute parity is claimed but not fully isolated. §3.1 notes that EL and CoPE compute are kept "similar" without exact matching, and Figure 3 shows CoPE at ≥ 1.0× relative training time vs. EL at ~0.7×. A wall-matched EL baseline would strengthen the central claim that co-evolution not simply more compute — drives the gain.
  • Hyperparameter selection protocol is underspecified. (α, β) = (2, 1) is used throughout the main experiments; Figure 3 shows that (1, 1) is comparable at some compute budgets and (2, 1) is better at others. It would help to clarify whether (2, 1) was selected on a held-out validation split or on the test benchmarks.
  • Rubric evolution is illustrated but not quantified. Tables 8 and 9 give one qualitative example of rubric and experience evolution. Aggregate measures, rubric diversity across epochs, drift from the initial rubric distribution, discriminativeness over time, would make the "rubrics evolve meaningfully" claim more concrete.
  • Reward-benchmark comparisons are qualitatively presented. Table 2 notes that public baselines use different training sets, so the comparison is "for reference." The improvement in rubric-generation ability is nonetheless the headline result of Table 2; the authors should at minimum compare to one baseline (e.g., EvoLM) retrained on CoPE's training set, or clearly flag that these numbers are suggestive rather than head-to-head.
  • Variance is not reported in the main table. Table 1 is described as an average of three runs, but no standard deviations are shown. For some comparisons (e.g., CoPE vs. EL on WildChat) the gap is large enough that noise is unlikely to explain it, but the error bars matter for interpreting narrower gaps.
Reviewer_29HcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CoPE, a framework that alternates between training a single model to generate responses and to develop evaluation rubrics. A frozen feedback model extracts textual experience based on these dynamic rubrics, which is then used to refine the model’s strategy via in-context distillation. Experiments conducted across two model families, five generation benchmarks, and evaluation capability tests demonstrate that CoPE consistently outperforms baselines relying on fixed rubrics—both in terms of experience-based learning and reinforcement learning (RL) with scalar rewards.

Critical strengths

CoPE consistently outperforms baselines that use fixed rubrics across various model families and feedback model configurations, yielding robust empirical gains. Furthermore, the framework enhances the model’s proficiency in both response generation and rubric creation, underscoring its practical utility.

Critical weaknesses

The study lacks a control condition—specifically, one where the rubric generator remains frozen while rubrics are continuously refreshed and applied using the same selection strategy—making it difficult to confirm whether the observed gains truly stem from a process of co-evolution. Additionally, the reliance on GPT-4o for primary generation evaluation, combined with the absence of reported uncertainty, complicates the assessment of smaller performance improvements.

Questions for authors
  1. Under an equivalent computational budget, how would the model perform if the rubric generator were kept frozen, with only the rubrics themselves being continuously refreshed and filtered?
  2. Do the results across three random seeds represent independent training runs, or merely different sampling during evaluation? Could you provide confidence intervals, as well as results obtained using independent evaluators?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.