- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoRubric, a self-evolving rubric RL framework for open-ended tasks. A shared trainable policy acts as both a Reasoner and a Rubric Generator: it generates candidate responses, proposes new evaluation criteria from those responses, and jointly optimizes both roles. The generated rubrics are filtered by a frozen Meta-Verifier and scored by an external Grader, with discrimination, peer consensus, and memory used to maintain informative criteria. Experiments across Medical, Writing, and Science tasks show consistent improvements over Base, Static Rubric-RL, and External Evolving-RL baselines.
Critical strengths
-
EvoRubric provides a principled way to adapt the reward specification as the policy improves. By jointly training the Reasoner and Rubric Generator, it addresses a key limitation of static or frozen-rubric approaches, where the evaluation criteria may become outdated as the policy distribution shifts.
-
The empirical analysis provides strong support for the effectiveness of the proposed framework. The paper includes comprehensive ablation studies that clearly isolate the contribution of each major component, as well as human preference evaluations that provide additional evidence beyond automatic benchmark scores. Together, these results make the source of the performance gains relatively transparent and strengthen the empirical case for the proposed co-evolution design.
Critical weaknesses
-
The reward construction is potentially self-referential and prone to adaptive reward overfitting. EvoRubric generates rubrics from the current policy responses and then uses those rubrics to reward the same responses, while the Reasoner and Rubric Generator also share parameters. This creates a co-adaptation loop in which the policy may optimize criteria that are easy to satisfy or discriminate rather than criteria that reflect true task quality. The current verifier and consensus mechanisms do not provide an independent correctness signal, making this a central concern for the validity of the proposed learning objective. [1,4]
-
The empirical comparison is not sufficient to isolate the benefit of the proposed co-evolution mechanism. The paper omits several closely related evolving-rubric baselines, including OnlineRubrics, RLCER, and a fully matched RLER/DR Tulu setup, while EvoRubric also uses additional trainable rubric rollouts and extra verifier/grader computation. Without stronger baselines and token/FLOP-matched controls, it is unclear whether the reported gains come from the proposed mechanism or simply from training data with same distribution, stronger supervision and additional compute. [1,2,3]
[1] MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Yunzhong He, and Afra Feyza Akyürek. Online Rubrics Elicitation from Pairwise Comparisons. arXiv:2510.07284, 2025. [2] Leheng Sheng, Wenchang Ma, Ruixin Hong, Xiang Wang, An Zhang, and Tat-Seng Chua. Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics. arXiv:2602.10885, 2026. [3] Rulin Shao et al. DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. ICML 2026; arXiv:2511.19399. [4] Ran Xu, Tianci Liu, Zihan Dong, Tony You, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang. Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training. arXiv:2602.01511, 2026.
Questions for authors
See Critical Weaknesses
5 comments
Kind Reminder of Our Response to Reviewer 1Sqb
Dear Reviewer 1Sqb,
Thank you for taking the time to review our paper.
In our response, we addressed your concern about adaptive reward overfitting with evidence from human audits and independent evaluations already reported in our paper. We also clarified our baseline comparisons and provided estimated training costs.
As the discussion period is nearing its end, could you please take a look at our response when you have a chance? We would appreciate hearing whether it addresses your concerns. Please let us know if anything needs further explanation, and we would be happy to clarify.
Thank you for your time and attention to our work!
Manuscript Revision Addressing Your Concerns
Dear Reviewer 1Sqb,
Thank you for your review. We have revised Appendix A to discuss OnlineRubrics and explain why DR-Tulu is closer to our setting: both DR-Tulu and EvoRubric use current responses and existing rubrics to propose new criteria. With the same Qwen3-8B response model, EvoRubric outperformed our DR-Tulu-style GPT-4.1 generator baseline on both medical benchmarks (Table 3). Appendix H now reports that EvoRubric used more local GPU time but had a 17.5% lower estimated cost per run when counting GPU rental and rubric-generation API fees.
On reward overfitting, our original submission reports a human audit of rubric correctness. With the frozen Meta-Verifier, the share of retained items judged invalid in the final 8B memory pool fell from 39.06% to 5.10% (Table 5). Independent model and human evaluations also showed gains in response quality (Tables 1 and 6). Finally, Table 2 isolates the contribution of training the Rubric Generator: keeping rubric sampling, verification, and grading in place but removing the Generator Loss lowered HealthBench and LLMEval-Med scores by 3.55 and 3.82 points, respectively.
Thank you again for raising these concerns. We hope the revision makes both the baseline comparison and its computational cost clearer.
Response to Reviewer 1Sqb
Dear Reviewer 1Sqb,
Thank you for your feedback on our design and experiments. We address each concern in detail below.
We already took this risk into account when designing EvoRubric. Peer consensus measures scoring agreement, and a frozen Meta-Verifier checks each new rubric item for correctness before it can be used to reward the same responses. The verifier stays fixed as the shared policy changes.
Independent audit of rubric correctness. In the final 8B memory pool, enabling the Meta-Verifier reduced the share of items judged invalid by human annotators from 39.06% to 5.10% and the share with factual conflicts from 15.63% to 2.55% (Table 5). This reduces the chance that invalid rubric items are reused as rewards.
Independent evaluation of response quality. EvoRubric outscored matched baselines on all five benchmarks when Claude Sonnet 4.6 applied each benchmark’s own rubric (Table 1). In blind human evaluation against the GPT-4.1 rubric-generator baseline, EvoRubric recorded 48% wins, 14% ties, and 38% losses on overall quality (Table 6).
The human audit and independent evaluations show that EvoRubric learns correct and useful rubrics whose benefits extend beyond the training reward.
Our paper already discusses RLCER and DR Tulu (Appendix A) and reports a comparison with a DR Tulu-style external rubric-generation baseline (Section 4.3, Table 3). OnlineRubrics and DR Tulu both use external models to update rubrics. Since DR Tulu proposes new items from current responses and existing rubrics, its strategy is closer to our setting. We use this strategy with GPT-4.1 in our medical experiments. With the same Qwen3-8B response backbone, EvoRubric leads this baseline by 2.88 and 0.83 points on HealthBench and LLMEval-Med, respectively. We will also discuss OnlineRubrics in the revision.
RLCER also trains the Reasoner and Rubric Generator jointly. It selects rubric items based on how strongly item satisfaction correlates with final-answer correctness on math problems. A direct comparison with RLCER as designed is therefore impossible in our open-ended setting, which lacks this verifiable correctness signal.
Table 2 compares full EvoRubric with a variant that removes the Generator Loss. Both variants retain rubric sampling, verification, grading, and memory. Removing the loss lowers HealthBench and LLMEval-Med scores by 3.55 and 3.82 points, respectively. Directly training the Rubric Generator improves performance within the same evolving-rubric pipeline.
We also estimated per-run costs for EvoRubric and the GPT-4.1 rubric-generator baseline from Table 3. Each used eight H20 GPUs. EvoRubric took 36 hours (288 GPU-hours), and the GPT-4.1 baseline took 28 hours (224 GPU-hours). The estimated GPU rental costs were \427.88and332.80, respectively. The GPT-4.1 baseline also incurred an average of $185.70 per run in rubric-generation API fees (84.11M tokens, Appendix H).
EvoRubric scored higher on both benchmarks (Table 3) at a 17.5% lower estimated cost per run (\427.88vs.518.50), counting GPU rental and rubric-generation API fees.