- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The author proposes EvoLM, a self-evolving framework that alternates between policy optimization and rubric generation, using temporal contrast between current and earlier policy responses to construct preference signals without external labels. Experiments across 12 benchmarks demonstrate improvements over existing rubric-based RL methods, with additional evidence of transferability across models and judges.
Critical strengths
-
It offers an interesting approach to improving both rubric generation and policy optimization without requiring externally annotated preference labels. Using temporal contrast to construct preference pairs provides a potentially scalable self-improvement mechanism.
-
The paper presents extensive experiments across 12 benchmarks, including comparisons with recent rubric-based RL methods. The cross-model and cross-judge evaluations, together with comprehensive ablations, provide useful evidence of the method's effectiveness.
Critical weaknesses
-
Although the paper investigates co-evolving rubric generation and policy optimization, the improvements over sequential training are relatively modest. The main comparison also uses different preference construction strategies, making it difficult to determine how much of the gain comes specifically from co-evolution rather than other design choices.
-
Despite emphasizing open-ended tasks, EvoLM achieves only 42.2 on AlpacaEval v3, below both Qwen3-prompted (45.9) and GPT-4.1-prompted rubrics (45.5). Much of the improvement appears to come from more verifiable tasks such as code generation. Further (human) evaluation on diverse open-ended tasks is needed to establish the effectiveness of the learned rubrics.
Questions for authors
-
Could the authors compare EvoLM with GPT-4.1 using the same rubric-generation prompt and provide a sequential-training baseline with matched preference supervision? This would help establish how much of the improvement comes specifically from co-evolution.
-
How often does the assumption that later responses are better than earlier ones hold at the individual-response level? Could the authors validate the constructed preference pairs using an independent evaluator?
-
Why does co-evolution underperform the prompted baseline on Llama-3.1-8B, particularly with the severe AlpacaEval degradation reported in Table 24? What explains this instability?
We thank the reviewer for finding the approach interesting and the experiments, cross-model and cross-judge evaluations and ablations useful, and for the specific requests, which we answer below.
What is new in EvoLM. EvoLM trains the reward itself, without labels. One Qwen3-8B model writes a weighted checklist (rubric) for each question and also answers questions; a frozen Qwen3-1.7B judge scores answers against the checklist, and the rubric generator is trained so that the judge's scores separate a later answer from an earlier one. Co-evolving training (EvoLM) alternates the two roles in one run; sequential training trains the rubric generator first and then freezes it.
With the reasoning block removed before scoring, EvoLM is within a few points of every rubric method. The submitted version scored IFEval and AlpacaEval on responses that still contained the model's reasoning text. Re-scored, EvoLM gets 60.2 on AlpacaEval, against 61.5 and 62.5 for the two prompted baselines and 56.3 to 63.8 for the other rubric methods (Table 1). EvoLM's average goes from 69.2 to 74.8, the highest in Table 1.
The gains are largest on code, and the rubrics also match experts on open-ended questions. On HumanEval+, EvoLM's policy reaches 86.2, against 80.5 for the next-best method and 73.7 and 58.7 for the prompted baselines (Table 1), using no unit tests, external reference answers or human preference labels. On open-ended deep-research questions, EvoLM's rubrics rank responses in agreement with expert-written rubrics more often than GPT-4.1's (HealthBench 58.4 against 52.5, ResearchQA 59.3 against 51.0; Table 2). We have not run a human evaluation.
Tables 21 and 1 report this comparison. Two pair types run under both schedules. In inferred-question pairs, the weaker answer responds to a question the model reconstructed from the stronger answer; in rubric-conditioned pairs, the stronger answer was written with the checklist in view and the weaker without it. Matched by pair type, co-evolving and sequential training give 74.8 and 73.7 (inferred question) and 74.4 and 74.7 (rubric-conditioned).
The two schedules perform comparably. On matched pairs they are within 1.1 points, as above. EvoLM's main run, which uses temporal-contrast pairs, and the best sequential run differ by +0.16 (interval [−1.05, 1.36], Table 25). Both schedules score above the prompted baselines (73.0 and 72.4) and Skywork-RM-V2 (64.9) in Table 1, and the five co-evolving pair-building variants give 73.6 to 74.8 (Table 21).
What training adds is visible in the rubrics themselves. Over the main run, label-only criteria fall from 23.5% to 0.3% and criteria that embed a specific expected value rise from 6.9% to 19.3% (Appendix B.3, Table 14). Training a policy on GPT-4.1 rubrics written in EvoLM's rubric format needs an additional run, which was not possible within the rebuttal period.
Appendix A.10 reports this pair by pair (Table 10). Over 22,776 pairs, scored by the training judge, the later answer scores higher in 29.8% of pairs, lower in 29.2%, and ties in 40.9%; among non-tied pairs the later answer wins 50.5% overall and 55.1% on safety questions. We did not use an independent evaluator.
The co-evolved Llama policy stops ending its responses. On AlpacaEval, 786 of its 805 responses run past 20,000 characters (median 83,500, against 2,900 for the prompted Llama policy); they begin coherently and degrade into unrelated tokens, which gives the 0.2 score (Table 24). We have not identified what in training causes it. The rubric generator itself transfers to Llama: used frozen, the generator trained in the Qwen3-8B run trains a Llama-3.1-8B policy to 46.9 on average, against 45.7 with GPT-4.1 rubrics, with AlpacaEval at 18.4 against 22.6 (Table 3).
We hope these measurements answer the reviewer's questions. If they do, we would be grateful if the reviewer would consider raising the score.