acceptodds
Under review as a conference paper at ICLR 2027

EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation

10 views

Abstract

Reinforcement Learning (RL) has advanced Large Language Models (LLMs) in verifiable domains, while open-ended generation remains challenging due to the absence of definitive rewards. Rubric-based RL provides explicit evaluation criteria, but learning to construct these criteria remains challenging when final-answer correctness is not verifiable. We propose EvoRubric, a co-evolutionary RL framework that combines criterion-validity feedback, response discrimination, and peer agreement to learn rubrics for open-ended generation. A shared policy acts as both a Reasoner and a Rubric Generator, using its current responses and historical rubrics to discover new evaluation dimensions. To combine adaptive rubric discovery with a stable validity check, a frozen copy of the initial policy serves as the Meta-Verifier, while a frozen Grader scores responses against the retained criteria. Discriminative feedback, Leave-One-Out peer consensus, and a persistent memory pool transform this feedback into complementary rewards that jointly optimize both policy roles, closing the loop between response improvement and rubric discovery. EvoRubric improves over matched static and external evolving-rubric baselines across five benchmarks in Medical, Writing, and Science. Across three training seeds, it achieves five-benchmark averages of at 8B and at 14B, exceeding the strongest matched baselines by 3.09 and 2.06 points, respectively. Human audits assess criterion validity and response quality, and experiments with expert-initialized rubrics demonstrate compatibility with human priors.

Primary area:
generative models
Keywords:
Large Language Models, Reinforcement Learning, Open-Ended Generation, Rubric-Based Evaluation, Self-Evolving Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1SqbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoRubric, a self-evolving rubric RL framework for open-ended tasks. A shared trainable policy acts as both a Reasoner and a Rubric Generator: it generates candidate responses, proposes new evaluation criteria from those responses, and jointly optimizes both roles. The generated rubrics are filtered by a frozen Meta-Verifier and scored by an external Grader, with discrimination, peer consensus, and memory used to maintain informative criteria. Experiments across Medical, Writing, and Science tasks show consistent improvements over Base, Static Rubric-RL, and External Evolving-RL baselines.

Critical strengths
  1. EvoRubric provides a principled way to adapt the reward specification as the policy improves. By jointly training the Reasoner and Rubric Generator, it addresses a key limitation of static or frozen-rubric approaches, where the evaluation criteria may become outdated as the policy distribution shifts.

  2. The empirical analysis provides strong support for the effectiveness of the proposed framework. The paper includes comprehensive ablation studies that clearly isolate the contribution of each major component, as well as human preference evaluations that provide additional evidence beyond automatic benchmark scores. Together, these results make the source of the performance gains relatively transparent and strengthen the empirical case for the proposed co-evolution design.

Critical weaknesses
  1. The reward construction is potentially self-referential and prone to adaptive reward overfitting. EvoRubric generates rubrics from the current policy responses and then uses those rubrics to reward the same responses, while the Reasoner and Rubric Generator also share parameters. This creates a co-adaptation loop in which the policy may optimize criteria that are easy to satisfy or discriminate rather than criteria that reflect true task quality. The current verifier and consensus mechanisms do not provide an independent correctness signal, making this a central concern for the validity of the proposed learning objective. [1,4]

  2. The empirical comparison is not sufficient to isolate the benefit of the proposed co-evolution mechanism. The paper omits several closely related evolving-rubric baselines, including OnlineRubrics, RLCER, and a fully matched RLER/DR Tulu setup, while EvoRubric also uses additional trainable rubric rollouts and extra verifier/grader computation. Without stronger baselines and token/FLOP-matched controls, it is unclear whether the reported gains come from the proposed mechanism or simply from training data with same distribution, stronger supervision and additional compute. [1,2,3]

[1] MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Yunzhong He, and Afra Feyza Akyürek. Online Rubrics Elicitation from Pairwise Comparisons. arXiv:2510.07284, 2025. [2] Leheng Sheng, Wenchang Ma, Ruixin Hong, Xiang Wang, An Zhang, and Tat-Seng Chua. Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics. arXiv:2602.10885, 2026. [3] Rulin Shao et al. DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. ICML 2026; arXiv:2511.19399. [4] Ran Xu, Tianci Liu, Zihan Dong, Tony You, Ilgee Hong, Carl Yang, Linjun Zhang, Tao Zhao, and Haoyu Wang. Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training. arXiv:2602.01511, 2026.

Questions for authors

See Critical Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1Sqb

Dear Reviewer 1Sqb,

Thank you for your feedback on our design and experiments. We address each concern in detail below.

W1: The reward construction may be self-referential and prone to adaptive reward overfitting.

We already took this risk into account when designing EvoRubric. Peer consensus measures scoring agreement, and a frozen Meta-Verifier checks each new rubric item for correctness before it can be used to reward the same responses. The verifier stays fixed as the shared policy changes.

  1. Independent audit of rubric correctness. In the final 8B memory pool, enabling the Meta-Verifier reduced the share of items judged invalid by human annotators from 39.06% to 5.10% and the share with factual conflicts from 15.63% to 2.55% (Table 5). This reduces the chance that invalid rubric items are reused as rewards.

  2. Independent evaluation of response quality. EvoRubric outscored matched baselines on all five benchmarks when Claude Sonnet 4.6 applied each benchmark’s own rubric (Table 1). In blind human evaluation against the GPT-4.1 rubric-generator baseline, EvoRubric recorded 48% wins, 14% ties, and 38% losses on overall quality (Table 6).

The human audit and independent evaluations show that EvoRubric learns correct and useful rubrics whose benefits extend beyond the training reward.

W2-1: The comparison lacks stronger, fully matched evolving-rubric baselines (DR Tulu, RLCER, OnlineRubrics).

Our paper already discusses RLCER and DR Tulu (Appendix A) and reports a comparison with a DR Tulu-style external rubric-generation baseline (Section 4.3, Table 3). OnlineRubrics and DR Tulu both use external models to update rubrics. Since DR Tulu proposes new items from current responses and existing rubrics, its strategy is closer to our setting. We use this strategy with GPT-4.1 in our medical experiments. With the same Qwen3-8B response backbone, EvoRubric leads this baseline by 2.88 and 0.83 points on HealthBench and LLMEval-Med, respectively. We will also discuss OnlineRubrics in the revision.

RLCER also trains the Reasoner and Rubric Generator jointly. It selects rubric items based on how strongly item satisfaction correlates with final-answer correctness on math problems. A direct comparison with RLCER as designed is therefore impossible in our open-ended setting, which lacks this verifiable correctness signal.

W2-2: Extra rubric rollouts and computation may explain the performance gains.

Table 2 compares full EvoRubric with a variant that removes the Generator Loss. Both variants retain rubric sampling, verification, grading, and memory. Removing the loss lowers HealthBench and LLMEval-Med scores by 3.55 and 3.82 points, respectively. Directly training the Rubric Generator improves performance within the same evolving-rubric pipeline.

We also estimated per-run costs for EvoRubric and the GPT-4.1 rubric-generator baseline from Table 3. Each used eight H20 GPUs. EvoRubric took 36 hours (288 GPU-hours), and the GPT-4.1 baseline took 28 hours (224 GPU-hours). The estimated GPU rental costs were \427.88and427.88 and \\332.80, respectively. The GPT-4.1 baseline also incurred an average of $185.70 per run in rubric-generation API fees (84.11M tokens, Appendix H).

EvoRubric scored higher on both benchmarks (Table 3) at a 17.5% lower estimated cost per run (\427.88vs.427.88 vs. \\518.50), counting GPU rental and rubric-generation API fees.

AuthorsOpenReview ↗

Kind Reminder of Our Response to Reviewer 1Sqb

Dear Reviewer 1Sqb,

Thank you for taking the time to review our paper.

In our response, we addressed your concern about adaptive reward overfitting with evidence from human audits and independent evaluations already reported in our paper. We also clarified our baseline comparisons and provided estimated training costs.

As the discussion period is nearing its end, could you please take a look at our response when you have a chance? We would appreciate hearing whether it addresses your concerns. Please let us know if anything needs further explanation, and we would be happy to clarify.

Thank you for your time and attention to our work!

AuthorsOpenReview ↗

Manuscript Revision Addressing Your Concerns

Dear Reviewer 1Sqb,

Thank you for your review. We have revised Appendix A to discuss OnlineRubrics and explain why DR-Tulu is closer to our setting: both DR-Tulu and EvoRubric use current responses and existing rubrics to propose new criteria. With the same Qwen3-8B response model, EvoRubric outperformed our DR-Tulu-style GPT-4.1 generator baseline on both medical benchmarks (Table 3). Appendix H now reports that EvoRubric used more local GPU time but had a 17.5% lower estimated cost per run when counting GPU rental and rubric-generation API fees.

On reward overfitting, our original submission reports a human audit of rubric correctness. With the frozen Meta-Verifier, the share of retained items judged invalid in the final 8B memory pool fell from 39.06% to 5.10% (Table 5). Independent model and human evaluations also showed gains in response quality (Tables 1 and 6). Finally, Table 2 isolates the contribution of training the Rubric Generator: keeping rubric sampling, verification, and grading in place but removing the Generator Loss lowered HealthBench and LLMEval-Med scores by 3.55 and 3.82 points, respectively.

Thank you again for raising these concerns. We hope the revision makes both the baseline comparison and its computational cost clearer.

Reviewer_swb8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Modern AI models can be post-trained to show exceptional performance on verifiable tasks. It would be great if we could extend this to less verifiable tasks. One solution being explored is to try and turn non-verifiable tasks into verifiable tasks by writing a rubric. But the question is then, who writes the rubric and how do you know whether it is any good?

This paper offers one solution, which is to learn a rubric generator (from a frozen model) whilst evolving answer generation. The answers are scored according to the rubric. There are steps that allow the rubric to be independently verified and the eventual scoring is performed by a third-party model. The final evaluation is a subjective rating by both humans and a more powerful model (Sonnet 4.6).

This seems like a clever idea but there's also something slightly circular about the approach. Ultimately, if you knew exactly what the ground truth rating was going to be (e.g. it was going to be given by a particular Claude model) then presuambly you could just ask that model to provide scores and train against those? The round-the-houses rubric generation stuff might be interesting if you cared about properties of the rubric (here they do audit the rubrics, but mainly to check they pass basic checks like factuality). In the end the approach is likely to be more or less sucessful to the extent that the rubric matches the policy of the rater that you eventually care about. Perhaps in many cases this would be a human (the authors do use human ratings here too, to their credit) but then a natural thing might be to try and learn a rubric from humans via cloning? But this is not what they do.

One nice thing about this project is that the authors have some sensible baselines (fixed and externally-generated rubrics) and the authors' approach clearly beats these when assessed with Claude. Human raters seem a bit more ambivalent though. They authors' approach seems to win only on concisenss and relevance.

I also think that the generalisation conditions are pretty weak here. I'd be more impressed if (say) you could learn a rubric for scoring tasks that could generalise to new tasks. But there's no attempt to do this here - just held out questions from the benchmarks they focus on.

So overall, this is not a bad paper but I didn't find the approach terribly compelling, the data are not massively impressive, and there are a lot of choices that I found a bit arbitrary. I also found the paper a bit indigestible in places.

Critical strengths

Important problem. Creative approach.

Critical weaknesses

Relatively weak results. Limited generealisability.

Questions for authors

What's ultimately the ground truth here?

3 comments
AuthorsOpenReview ↗

Response to Reviewer swb8

Dear Reviewer swb8,

Thank you for your thoughtful review and questions. Below, we clarify our evaluation setup and address your concerns.

W1 & Q1: What is the ground truth, and why not train directly on Claude’s scores?

Thank you for raising this question. We would like to clarify why we chose this setup.

Here, the ground truth for evaluation comes from the rubrics defined by each benchmark (some benchmarks provide expert-written rubrics). A judge (typically an LLM) then scores each response against these rubrics.

However, in RL, optimizing for an LLM judge’s scores can lead to reward hacking, as prior work has shown [1, 2]. If the same LLM serves as both the training judge and the final evaluator (Claude, for example), the policy may learn to exploit that judge’s scoring preferences. As a result, Claude may give the responses high scores even when human raters or a different LLM judge rate them lower.

To reduce this bias, we deliberately use different judges for training and final evaluation (GPT-OSS-120B and Claude Sonnet 4.6, respectively) and supplement automatic evaluation with blind human ratings. This gives us an independent assessment that better reflects actual response quality. Prior work also uses separate judges for training and evaluation on several benchmarks [3].

[1] Reward Hacking in Rubric-Based Reinforcement Learning. NeurIPS 2026.

[2] Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training. EMNLP 2026.

[3] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. ICML 2026.

W2: Why not learn rubrics from humans via imitation?

Learning rubrics from humans via imitation is a reasonable option. However, as discussed in our introduction, rubrics written before RL can miss new strengths and errors in the policy’s responses. RL keeps producing new responses, and at scale it is almost impossible for experts (humans) to review them and write new rubric items. This is a practical limit of relying on imitation alone when rubrics need to adapt during RL.

EvoRubric addresses this by automatically discovering correct and useful rubric items from the policy’s current responses and updating the rubrics during training. Further, EvoRubric can start from expert-written rubrics and build on them to improve response quality (Section 4.4, Table 4).

W3: Human evaluation seems to favor EvoRubric only on conciseness and relevance.

Conciseness and relevance showed the clearest gains. Human raters assessed overall quality separately, and EvoRubric won more often than it lost on overall quality against both baselines. In blind evaluation on 50 medical questions, the win/tie/loss rates were 42%/30%/28% against the 8B rubric generator and 48%/14%/38% against the GPT-4.1 rubric generator (Table 6).

The latter is a strong baseline that uses GPT-4.1 to generate training rubrics. Both methods use Qwen3-8B to generate responses. EvoRubric also outscored this baseline by 2.88 and 0.83 points on HealthBench and LLMEval-Med, respectively (Table 3).

W4: The experiments only use held-out questions from the same benchmarks.

We would like to clarify that three of our five evaluation benchmarks are separate datasets from those used for RL training (Section 4.1). We train a separate model in each domain on 3,000 examples.

DomainRL training dataEvaluation data
MedicalHealthBenchHeld-out HealthBench questions and LLMEval-Med (independent benchmark)
WritingLongWriter-Zero-RLDataWritingBench and Creative Writing (both independent benchmarks)
ScienceResearchQAHeld-out ResearchQA questions

EvoRubric outperformed matched baselines on all three independent benchmarks (Table 1). These results demonstrate that the gains in response quality generalise across datasets within the same domain.

I'm not quite sure I understand the rebuttal - when I said train directly on Claude's scores, it doesn't have to be the same Claude (to avoid reward hacking). Perhaps I'm missing something.

I appreciate the clarification on the hold-out set. This does make a difference to my evaluation and I'll change my score to a 3.

Thank you for your positive feedback on our evaluation setup!

Sorry we may not have fully understood your point earlier. Do you mean asking a strong LLM judge to give each response an overall score without rubrics? This can work. However, for open-ended tasks, rubrics give the judge clear checks and help make scores more consistent (direct LLM scores can be unstable) [1]. Prior work also shows that rubric-based RL can achieve better results than RL using direct overall scores [2]. This is why we use rubrics.

Plus, our final evaluation is independent. It uses each benchmark’s own fixed, high-quality rubrics and a fixed Claude judge (our model does not generate extra rubrics to score the test set). All methods use the same evaluation setup.

We only update rubrics during training to add missing checks (for new strengths or errors in the training rollouts that the initial rubrics do not cover). And even when starting from expert-written rubrics, these updates improve results under independent evaluation.

[1] CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists. EMNLP 2025

[2] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. ICLR 2026

Thank you again for reconsidering our work and raising your score! We would be happy to clarify further if anything remains unclear.

Reviewer_tiJwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoRubric, a self-evolving rubric-driven reinforcement learning framework for open-ended generation. The method jointly trains a Reasoner and a Rubric Generator within a shared policy, optimizing rubric generation without verifiable final-answer correctness.

Experiments across multiple benchmarks spanning Medical, Writing, and Science domains show improvements over both static and externally evolving rubric baselines. The paper also includes component ablations and human evaluations of response quality.

Critical strengths

Learning useful evaluation criteria without ground-truth answer correctness is an important challenge for applying reinforcement learning to open-ended generation. The paper provides a practical framework that achieves consistent empirical improvements across multiple benchmarks.

Critical weaknesses

Insufficient Justification for the Rubric-Learning Objective

The paper does not convincingly establish that the proposed combination of discriminative feedback and peer consensus encourages the Rubric Generator to discover genuinely useful evaluation criteria. While these signals provide a practical way to optimize rubrics without ground-truth answer correctness, it remains unclear whether optimizing them leads to higher-quality supervision or merely reinforces features that are easy to discriminate and widely agreed upon among the generated rubric sets.

First, the discriminative reward only encourages criteria that distinguish the currently sampled responses. If all responses share the same critical error, a criterion identifying that error would receive no discriminative reward, even though it may capture an important evaluation dimension that is otherwise overlooked.

Second, the use of peer consensus is not sufficiently justified. Agreement among generated rubric sets does not necessarily imply that the evaluation criteria are correct or useful, especially since all candidate rubric sets are generated by the same policy and evaluated on the same group of responses.

If a rubric batch contains an important but uncommon evaluation criterion, it may receive a lower reward due to its disagreement with the majority of other rubrics, or even be excluded from subsequent training. Since consensus is computed at the batch level, the presence of such a criterion may also reduce the reward assigned to the entire batch.

Furthermore, since the Reasoner and Rubric Generator share the same policy, this may create a self-reinforcing feedback loop in which both roles gradually favor evaluation dimensions that are easier to satisfy and consistently rewarded, rather than those that genuinely matter for response quality.

Questions for authors

When all sampled responses share the same critical error, how does your proposed method discover important evaluation criteria?

Can the learned rubrics generalize beyond the responses used to construct them?

Does sharing the same policy between the Reasoner and the Rubric Generator introduce self-reinforcing reward biases?

5 comments
AuthorsOpenReview ↗

Response to Reviewer tiJw

Dear Reviewer tiJw,

Thank you for the helpful feedback. We address each concern in detail below.

W1 & W3: Can peer consensus establish that the learned rubrics are correct or useful?

We would like to clarify that EvoRubric uses peer consensus to measure scoring agreement and a frozen Meta-Verifier to check each new rubric item for correctness. The experiments below, already reported in our paper, show that EvoRubric learns correct and useful rubrics.

  1. Rubric validity and correctness. Three experienced medical rubric annotators audited rubric items for 20 questions across four training epochs (Section 4.5). With the Meta-Verifier, the share of items judged invalid in the final 8B memory pool fell from 39.06% to 5.10%. New, valid rubric dimensions accounted for 75.16% and 76.26% of all entries in the final 8B and 14B pools, respectively (Table 5). The human audit shows that the vast majority of retained rubric items are correct and valid.
  2. Effectiveness of training supervision. Removing peer consensus lowered HealthBench and LLMEval-Med scores by 4.33 and 3.59 points, respectively. Removing discriminative feedback lowered them by 6.21 and 2.64 points (Section 4.3, Table 2). EvoRubric outscored matched baselines on all five benchmarks in third-party evaluation (Table 1). Blind human evaluation also favored EvoRubric over both external baselines on overall quality (Table 6). The gains in response quality show that the learned rubrics provide useful training supervision.

W2 & Q1: Rubric items for errors shared by all sampled responses receive no discriminative reward.

We would like to clarify that excluding these rubric items from the discriminative reward is a deliberate design choice, and we think it is not a weakness. This is because GRPO subtracts the group mean from each reward, so a penalty shared by all responses cancels out and provides no relative optimization signal.

Prior work follows the same principle. DAPO [1] filters out groups where all responses are correct or all are incorrect. DR Tulu [2] drops rubric items that give every response in a group the same score. Our ablation also supports this design (Table 2).

Shared errors matter, but our focus is on turning differences between responses into training signals. When some responses make an error and others avoid it, the corresponding rubric item can guide the policy toward better responses.

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS 2025.

[2] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. ICML 2026.

W4: Peer consensus may suppress important but uncommon evaluation criteria.

As discussed above, our experiments already show that EvoRubric discovers and retains many correct and valid new rubric items (Table 5) and that peer consensus improves downstream performance (Table 2).

Peer consensus compares how entire rubric batches score the same responses. An item can contribute to the current update even if it is proposed only once, provided it passes verification and its batch is eligible. For later updates, only the eligible batch with the highest consensus supplies new items to the memory pool.

An uncommon item can affect batch selection through its effect on the batch’s scores. The overall effect of peer consensus measured in our ablation is positive.

W5 & Q3: Does sharing the same policy introduce self-reinforcing reward biases?

We already took this risk into account when designing EvoRubric and introduced a frozen Meta-Verifier to limit it.

  1. Verification during training. The Meta-Verifier stays fixed at the initial checkpoint and checks each new rubric item proposed by the shared policy. With the Meta-Verifier, the share of items judged invalid by human annotators in the final 8B memory pool fell from 39.06% to 5.10% (Table 5).

  2. Independent evaluation after training. EvoRubric’s gains also hold under independent evaluation by Claude Sonnet 4.6 and in blind human evaluation (Tables 1 and 6).

Our experiments show that the Meta-Verifier reduces the risk of self-reinforcing reward bias and helps EvoRubric learn correct and useful rubrics.

Q2: Can the learned rubrics generalize beyond the responses used to construct them?

Thank you for the suggestion. We tested this on 100 HealthBench questions. For each question, we generated rubrics from one set of responses and used them unchanged to score two unseen responses.

We excluded seven questions for which the physician-written rubrics gave both responses the same score. On the remaining 93 questions, pairwise ranking agreement with the physician-written rubrics was 64.5% (60/93) with base rubrics alone, 68.8% (64/93) with external rubric items added, and 76.3% (71/93) with EvoRubric items added. EvoRubric’s new items improved agreement on unseen responses by 11.8 and 7.5 percentage points over the base and external baselines, respectively.

AuthorsOpenReview ↗

Kind Reminder of Our Response to Reviewer tiJw

Dear Reviewer tiJw,

Thank you again for your review.

Our earlier response addresses possible misunderstandings of our method in your review. We highlighted the human audits and independent evaluation results showing that the learned rubrics are correct and useful, and explained the basic reasoning behind our reward design.

As the discussion period comes to an end, we would greatly appreciate it if you could take a look at our response and let us know whether it addresses your concerns. Your feedback would mean a lot to us, and we would be happy to clarify any remaining questions.

Thank you for your time and consideration!

Thank you for the detailed response. My concerns are: If all sampled responses make the same mistake, can the model still learn a rubric that identifies this error? If a particular evaluation criterion is very important, but only a few models consider it important, could the consensus mechanism mistakenly exclude it?

I still believe that, although the additional experiments demonstrate improvements in overall performance, you do not sufficiently explain why the current rubric-learning objective is well justified.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Summary

Dear PCs, SACs, ACs and Reviewers,

Thank you for the thoughtful discussion. EvoRubric learns not only how to answer an open-ended question, but also what to check in an answer. A single policy learns both roles: as its responses change during RL, it can discover useful criteria that the initial rubrics missed, without requiring a verifiable final answer. We summarize below how the original evidence and the new analyses address the main questions raised during review.

  1. Rubric correctness, usefulness, and reward hacking (tiJw, 1Sqb). Both reviewers asked whether the shared policy might learn criteria that are easy to satisfy rather than criteria that reflect answer quality. Peer consensus measures scoring agreement; a separate, frozen Meta-Verifier checks each new item for correctness. In the original human audit, enabling the verifier reduced the share of invalid items in the final 8B pool from 39.06% to 5.10% (Table 5). We also tested whether the gains extended beyond the training reward: EvoRubric improved scores under independent benchmark rubrics, and blind human raters favored its answers on overall quality against both external-generator baselines (Tables 1 and 6). The revision adds a direct test of the rubrics themselves: on unseen responses to the same questions, adding EvoRubric items improved ranking agreement with physician-written rubrics from 64.5% to 76.3% (Appendix G.1).

  2. Shared errors and uncommon criteria (tiJw). In response to tiJw’s follow-up, we examined the training trajectories directly. For 35.0% of the ResearchQA questions processed in the first epoch, EvoRubric generated a criterion that identified a requirement missed by every sampled response. We also found that an important criterion proposed by only one candidate set can survive selection and remain in memory: consensus compares how whole sets score responses, not how often an individual criterion is proposed (Appendix I.3).

  3. Generalization across datasets (swb8). Three of the five evaluation benchmarks are separate from the RL training datasets, and EvoRubric outperformed the matched baselines on all three (Table 1). Reviewer swb8 explicitly noted that this clarification changed their assessment and raised their rating.

  4. Performance under a matched local GPU-time budget (1Sqb). Reviewer 1Sqb acknowledged that some concerns had been addressed and identified computational confounding as the remaining issue. We therefore compared training trajectories on the same eight H20 GPUs. Within the GPT-4.1-generator baseline’s local training-time budget, EvoRubric scored 55.51 versus 54.59 on HealthBench. It also reached 55.51 before the same-scale external and static baselines finished at 52.45 and 50.69, respectively (Appendix H.1).

We appreciated the two rounds of discussion with Reviewers tiJw and 1Sqb. We shared both follow-up analyses before the deadline and posted the GPU-time comparison shortly before the discussion closed. While we have not yet heard back from either reviewer, we believe these results can address their remaining concerns, and we hope the reviewers and the Area Chair find them helpful.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.