acceptodds
Under review as a conference paper at ICLR 2027

EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics

Abstract

Language models encode substantial evaluative knowledge from pretraining, yet current post-training methods rely on external supervision (human annotations, proprietary models, or scalar reward models) to produce reward signals. Each source brings a different constraint: human judgment is costly and hard to scale, proprietary APIs create dependencies, and verifiable rewards cover only domains with ground-truth answers. A model’s own evolution offers a natural source of supervision for self-improvement, yet remains largely untapped by current methods. We introduce EVOLM, a post-training method that turns this supervision into rewards for further learning. EVOLM trains two capabilities within a single language model in alternation: (1) a rubric generator producing instance-specific evaluation criteria optimized for discriminative utility, which maximizes a small frozen judge’s ability to distinguish preferred from dispreferred responses; and (2) a policy trained using those rubric-conditioned scores as reward. All preference signals are constructed from the policy’s own outputs via temporal contrast with earlier checkpoints, requiring no human annotation or external supervision. The co-trained policy achieves 69.2% average on the OLMo3-Adapt suite, exceeding policies trained with GPT-4.1-prompted rubrics by 2.5 points and those trained with the state-of-the-art 8B reward model Skywork-RM-V2 by 9.5 points. The learned rubrics transfer to judges they were never trained against, raising a Qwen3-8B judge’s RewardBench-2 accuracy from 39.7% to 62.4%. Rubrics that co-evolve with the policy therefore supply training signal that external reward sources do not.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-improving models, LM co-evolution, RL with non-verifiable rewards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TmwAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author proposes EvoLM, a self-evolving framework that alternates between policy optimization and rubric generation, using temporal contrast between current and earlier policy responses to construct preference signals without external labels. Experiments across 12 benchmarks demonstrate improvements over existing rubric-based RL methods, with additional evidence of transferability across models and judges.

Critical strengths
  1. It offers an interesting approach to improving both rubric generation and policy optimization without requiring externally annotated preference labels. Using temporal contrast to construct preference pairs provides a potentially scalable self-improvement mechanism.

  2. The paper presents extensive experiments across 12 benchmarks, including comparisons with recent rubric-based RL methods. The cross-model and cross-judge evaluations, together with comprehensive ablations, provide useful evidence of the method's effectiveness.

Critical weaknesses
  1. Although the paper investigates co-evolving rubric generation and policy optimization, the improvements over sequential training are relatively modest. The main comparison also uses different preference construction strategies, making it difficult to determine how much of the gain comes specifically from co-evolution rather than other design choices.

  2. Despite emphasizing open-ended tasks, EvoLM achieves only 42.2 on AlpacaEval v3, below both Qwen3-prompted (45.9) and GPT-4.1-prompted rubrics (45.5). Much of the improvement appears to come from more verifiable tasks such as code generation. Further (human) evaluation on diverse open-ended tasks is needed to establish the effectiveness of the learned rubrics.

Questions for authors
  1. Could the authors compare EvoLM with GPT-4.1 using the same rubric-generation prompt and provide a sequential-training baseline with matched preference supervision? This would help establish how much of the improvement comes specifically from co-evolution.

  2. How often does the assumption that later responses are better than earlier ones hold at the individual-response level? Could the authors validate the constructed preference pairs using an independent evaluator?

  3. Why does co-evolution underperform the prompted baseline on Llama-3.1-8B, particularly with the severe AlpacaEval degradation reported in Table 24? What explains this instability?

1 comment

We thank the reviewer for finding the approach interesting and the experiments, cross-model and cross-judge evaluations and ablations useful, and for the specific requests, which we answer below.

What is new in EvoLM. EvoLM trains the reward itself, without labels. One Qwen3-8B model writes a weighted checklist (rubric) for each question and also answers questions; a frozen Qwen3-1.7B judge scores answers against the checklist, and the rubric generator is trained so that the judge's scores separate a later answer from an earlier one. Co-evolving training (EvoLM) alternates the two roles in one run; sequential training trains the rubric generator first and then freezes it.

"EvoLM achieves only 42.2 on AlpacaEval v3, below both Qwen3-prompted (45.9) and GPT-4.1-prompted rubrics (45.5)."

With the reasoning block removed before scoring, EvoLM is within a few points of every rubric method. The submitted version scored IFEval and AlpacaEval on responses that still contained the model's reasoning text. Re-scored, EvoLM gets 60.2 on AlpacaEval, against 61.5 and 62.5 for the two prompted baselines and 56.3 to 63.8 for the other rubric methods (Table 1). EvoLM's average goes from 69.2 to 74.8, the highest in Table 1.

"Much of the improvement appears to come from more verifiable tasks such as code generation."

The gains are largest on code, and the rubrics also match experts on open-ended questions. On HumanEval+, EvoLM's policy reaches 86.2, against 80.5 for the next-best method and 73.7 and 58.7 for the prompted baselines (Table 1), using no unit tests, external reference answers or human preference labels. On open-ended deep-research questions, EvoLM's rubrics rank responses in agreement with expert-written rubrics more often than GPT-4.1's (HealthBench 58.4 against 52.5, ResearchQA 59.3 against 51.0; Table 2). We have not run a human evaluation.

"… provide a sequential-training baseline with matched preference supervision?"

Tables 21 and 1 report this comparison. Two pair types run under both schedules. In inferred-question pairs, the weaker answer responds to a question the model reconstructed from the stronger answer; in rubric-conditioned pairs, the stronger answer was written with the checklist in view and the weaker without it. Matched by pair type, co-evolving and sequential training give 74.8 and 73.7 (inferred question) and 74.4 and 74.7 (rubric-conditioned).

"… the improvements over sequential training are relatively modest."

The two schedules perform comparably. On matched pairs they are within 1.1 points, as above. EvoLM's main run, which uses temporal-contrast pairs, and the best sequential run differ by +0.16 (interval [−1.05, 1.36], Table 25). Both schedules score above the prompted baselines (73.0 and 72.4) and Skywork-RM-V2 (64.9) in Table 1, and the five co-evolving pair-building variants give 73.6 to 74.8 (Table 21).

"Could the authors compare EvoLM with GPT-4.1 …?"

What training adds is visible in the rubrics themselves. Over the main run, label-only criteria fall from 23.5% to 0.3% and criteria that embed a specific expected value rise from 6.9% to 19.3% (Appendix B.3, Table 14). Training a policy on GPT-4.1 rubrics written in EvoLM's rubric format needs an additional run, which was not possible within the rebuttal period.

"How often does the assumption that later responses are better than earlier ones hold at the individual-response level?"

Appendix A.10 reports this pair by pair (Table 10). Over 22,776 pairs, scored by the training judge, the later answer scores higher in 29.8% of pairs, lower in 29.2%, and ties in 40.9%; among non-tied pairs the later answer wins 50.5% overall and 55.1% on safety questions. We did not use an independent evaluator.

"Why does co-evolution underperform the prompted baseline on Llama-3.1-8B, particularly with the severe AlpacaEval degradation reported in Table 24? What explains this instability?"

The co-evolved Llama policy stops ending its responses. On AlpacaEval, 786 of its 805 responses run past 20,000 characters (median 83,500, against 2,900 for the prompted Llama policy); they begin coherently and degrade into unrelated tokens, which gives the 0.2 score (Table 24). We have not identified what in training causes it. The rubric generator itself transfers to Llama: used frozen, the generator trained in the Qwen3-8B run trains a Llama-3.1-8B policy to 46.9 on average, against 45.7 with GPT-4.1 rubrics, with AlpacaEval at 18.4 against 22.6 (Table 3).

We hope these measurements answer the reviewer's questions. If they do, we would be grateful if the reviewer would consider raising the score.

Reviewer_H5ZnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a past-future preference pair construction method. The response from a past checkpoint is used as the dispreferred response. Such past-future pairs are used to train the rubric generator. The whole process does not include external annotations, so it is self-evolving.

Critical strengths
  • The paper is easy to follow.
  • The idea makes sense. The reviewer would trust that it will work.
Critical weaknesses
  • Well, the biggest question is that the idea is not new. [1] basically has the same idea. And, some other self-play works [2-4].

  • I think if we just do preference optimization like the paper shows in the experiment section, these works can also do that. There will be no big difference. This is also shown in Table 1, 2 points gap between Qwen3-8B (prompted) and EVOLM.

  • This is not a bad paper. BUT, because of these similar works, I will vote for rejection.

[1] Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future. https://arxiv.org/pdf/2508.06026

[2] Self-Rewarding Language Models. https://arxiv.org/pdf/2401.10020

[3] LEARNING TO REASON WITHOUT EXTERNAL REWARDS. https://arxiv.org/pdf/2505.19590

[4] Self-Consistency Preference Optimization. https://arxiv.org/pdf/2411.04109

Questions for authors

Please check the Critical Weaknesses.

1 comment

We thank the reviewer for finding the paper easy to follow and the idea sound, and for the four pointers, all of which the revised paper cites and discusses in Section 4 ([3] appears there as Zhao et al., 2026).

What is new in EvoLM. EvoLM trains the reward itself, without labels. One Qwen3-8B model writes a weighted checklist (rubric) for each question and also answers questions; a separate frozen Qwen3-1.7B judge scores answers against the checklist, and the policy is trained with reinforcement learning on those scores. The rubric generator is trained so that the judge's scores separate a later answer from an earlier one. Each pair consists of a current answer and an earlier answer from the same model; no later model is involved (Section 2.4).

"[1] basically has the same idea."

EvoLM differs from [1] in who scores responses. In [1] and [2] the model judges its own responses. In EvoLM the model writes the criteria, and a separate frozen model applies them. [3] and [4] use no judge.

How responses are scoredTraining signalUses a next-generation model?
TSRLM [1]the model itself, prompted as a judgepreference optimization on its own judgmentsyes: chosen responses are curated using a next-generation model's predictions
Self-Rewarding LMs [2]the model itself, prompted as a judge; its judging improves in trainingpreference optimization on its own judgmentsno
Intuitor [3]no judgereinforcement learning on the model's self-certaintyno
ScPO [4]no judgepreference optimization on pairs ranked by agreement among the model's sampled answersno
EvoLMa separate frozen model applying a checklist the model wrotepolicy: reinforcement learning on the judge's score; rubric generator: the judge's score margin between a later and an earlier answer, plus a format rewardno

The ingredient EvoLM shares with [1] is an earlier answer as the weaker side of a pair.

"I think if we just do preference optimization like the paper shows in the experiment section, these works can also do that."

EvoLM's policy is trained with reinforcement learning, not preference optimization. The policy is trained on the judge's scores of its own answers, not on pairs. Training additional policies to compare against [1]–[4] was not possible within the rebuttal period.

"2 points gap between Qwen3-8B (prompted) and EVOLM" — "no big difference."

The gap is distinguishable from zero. The revised paper scores IFEval and AlpacaEval on the response with the reasoning block removed (Table 1), which turns the submitted 69.2 against 67.5 into 74.8 against 73.0. Over 12 benchmarks, EvoLM leads the policy trained with prompted Qwen3-8B rubrics by 1.86 points and the policy trained with prompted GPT-4.1 rubrics by 2.40, with evaluation-level 95% intervals [0.6, 3.1] and [1.1, 3.7], and leads the policy trained with the Skywork-RM-V2 reward model by 9.87 [8.6, 11.2] (Appendix E.1, Table 25). The largest differences are on code: HumanEval+ is 86.2 against 73.7 and 58.7, and against 80.5 for the next-best method (Table 1), obtained without unit tests, external reference answers or human preference labels.

We hope this clarifies how EvoLM differs from [1]–[4]. If it addresses the reviewer's concerns, we would be grateful if the reviewer would consider raising the score.

Reviewer_PWf9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EvoLM, a post-training method where a single model learns both to generate responses and to write the rubrics used to score them, with no human labels, verifiers, or external teacher models. The rubric generator is trained to maximize discriminative utility, rewarding rubrics that help a small frozen judge score the preferred response above the dispreferred one. These preference pairs come from the policy itself through temporal contrast, where the current response to a prompt is treated as better than one from an earlier checkpoint. Then, the training alternates between updating the policy with rubric-based rewards and updating the rubric generator on these pairs, so as the policy improves, the rubrics need to become more specific to stay discriminative. The authors motivate this objective by treating rubrics as latent variables and deriving it through variational inference. The result shows that the policy reaches 69.2% average on the OLMo3-Adapt suite, which is higher than the policy trained using GPT-4.1-prompted rubrics (66.7%), four recent rubric-based RL methods (66.7 to 67.6%), and the Skywork-RM-V2 scalar reward model (59.7%), even though the reward model has the best static preference accuracy. The learned rubrics also transfer to unseen judges and policies, basically agree better with expert rubrics on HealthBench and ResearchQA than GPT-4.1, and shift over training from abstract labels to concrete, checkable criteria. The main setup uses Qwen3-8B as both policy and rubric generator, Qwen3-1.7B as the frozen judge, and Tulu 3 prompts.

Critical strengths

The method doesn’t need any preference annotations, verifiers or proprietary teacher model to train the rubric generator, since the preference pairs come from the policy’s own earlier checkpoints.

Critical weaknesses

First, I don’t think the evaluation tests the setting where rubric-based rewards should actually help. Almost all of the OLMo3-Adapt benchmarks are verifiable, and AlpacaEval is the only open-ended one. RLVR would probably do as well or better in these domains, and the paper’s own analysis shows the math rubrics mostly turn into answer checking with the expected answer embedded. HealthBench and ResearchQA are where rubrics should matter, but they are only used to check rubric agreement, not to train or evaluate a policy. So it is somewhat unclear what EvoLM gives over a simpler self-improvement setup with RLVR, e.g., using the model’s own majority-vote answers as reward.

Also, it is unclear where the improvement actually comes from. First, the adjusted rubric prompt is only given to EvoLM, while all prompted baselines use a generic prompt. In the paper’s own ablation, co-evolving with the generic prompt gets 67.6, basically the same as the untrained prompted baseline (67.5). Also, on OLMo-3-7B a frozen generator using the main prompt matches co-evolution. So the gain over GPT-4.1 and the other baselines may mostly come from the prompt rather than from training the rubric generator. Furthermore, no one checks whether the rubrics or the temporal pairs are correct. The “better” response is just the later one, and the expected answers inside the rubrics come from the same model, so the training could just as well be pushing the policy toward its own answers and habits as making the rubrics better.

Questions for authors

See weaknesses.

1 comment

We thank the reviewer for the careful review, and for recognizing that EvoLM needs no preference annotations, verifiers or proprietary teacher model.

What is new in EvoLM. EvoLM trains the reward itself, without labels. One Qwen3-8B model writes a weighted checklist (rubric) for each question and also answers questions; a frozen Qwen3-1.7B judge scores answers against the checklist, and the rubric generator is trained so that the judge's scores separate a later answer from an earlier one. The revised paper scores IFEval and AlpacaEval on the response with the reasoning block removed (EvoLM 69.2 → 74.8, prompted Qwen3-8B 67.5 → 73.0), and reruns the RewardBench-2 evaluation for prompted Qwen3-8B, whose original run had failed (Ties 9.7%, below the 25% of random choice); the rerun gives 46.0.

Results. Over 12 benchmarks, EvoLM's policy averages 74.8, against 73.0 and 72.4 for policies trained with prompted rubrics from Qwen3-8B and GPT-4.1 (evaluation-level 95% intervals on the differences [0.6, 3.1] and [1.1, 3.7], Table 25). It leads the policy trained with Skywork-RM-V2 by 9.9 points, although Skywork is far more accurate on RewardBench-2 (86.4 against 46.0): accuracy on a fixed preference benchmark did not predict how well a reward trains a policy.

"… the training could just as well be pushing the policy toward its own answers and habits as making the rubrics better."

The gains appear on benchmarks scored independently of the model. On HumanEval+, scored by executing unit tests that training never uses, EvoLM's policy reaches 86.2, against 80.5 for the next-best method and 73.7 and 58.7 for the prompted baselines (Table 1); training uses no unit tests, external reference answers or human preference labels. The rubrics also carry beyond the policy they were trained with: used frozen, the generator trains a Qwen3-4B policy to 70.2 and a Llama-3.1-8B policy to 46.9, against 69.6 and 45.7 with GPT-4.1 rubrics (Table 3).

"So the gain over GPT-4.1 and the other baselines may mostly come from the prompt rather than from training the rubric generator."

Training changes what the rubrics contain. Over the main run, label-only criteria fall from 23.5% to 0.3%, criteria grow from 59 to 112 characters, criteria that embed a specific expected value rise from 6.9% to 19.3%, and criteria that check explicit constraints rise from 7.5% to 20.3% (Appendix B.3, Table 14). The policy trained with these rubrics leads rubric methods from prior work, each with its own rubric pipeline: RRD by 1.6 points (interval [0.4, 2.9]), RaR by 0.9 ([−0.3, 2.1]) and Rubric-ARM by 1.2 ([−0.1, 2.4]) (Table 25).

"… no one checks whether the rubrics or the temporal pairs are correct."

The revision checks the temporal pairs pair by pair (Appendix A.10, Table 10). Scored by the training judge, the later answer scores higher in 29.8% of pairs, lower in 29.2%, and ties in 40.9%; among non-tied pairs the later answer wins 50.5%. The answers embedded in rubrics are not separately verified; the held-out evaluations above check the trained policy's answers independently.

"Almost all of the OLMo3-Adapt benchmarks are verifiable … majority-vote answers as reward."

EvoLM needs no reference answers, and its rubrics agree with experts where none exist. EvoLM uses no unit tests or external reference answers in training, so it applies where neither exists. HealthBench and ResearchQA test the rubrics against expert-written rubrics in such domains: EvoLM's rubrics rank responses in agreement with the expert rubrics more often than GPT-4.1's (58.4 against 52.5, and 59.3 against 51.0; Table 2). A comparison with majority-vote training needs a new policy run, and a policy trained in a domain without reference answers is future work.

We hope these results address the reviewer's concerns. If so, we would be grateful if the reviewer would consider raising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.