- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
As the paper mentions, rubrics need to be stable and discriminative. If not discriminative, they won't provide any RL signal, and if gradings are not stable, they will provide noisy signals. They combine these into a reliability scores and drop rubrics that are not stable or discriminative. They train two Qwen models and show improvements over three benchmarks.
Critical strengths
The argument around stability and discriminative make sense and are valid. They train two different size models and show improvements on three different benchmarks. Also, they compared with three different baselines and show their method consistently outperforms the others. They also show that expert agreements increase and judges take lower effort to grade the filtered rubrics.
Critical weaknesses
I think there's a very important factor that is not discussed in the paper: consistency is not a function of the rubric alone, it also heavily depends on the judge used for evaluation. I think they should be viewed as a pair, a stronger judge will be more consistent on rubrics, and a weaker one will be less consistent. So I think it's not easy to disentangle whether the source of inconsistency is the rubric or the judge. I think more analysis on the effect of different judges can make this paper more robust.
There should also be more discussion on why it is acceptable to drop rubrics entirely. To me, it's unintuitive to remove some important objectives from reward design and it could even introduce reward hacking. The best rubric is the most comprehensive one. I think another option would be to rewrite and augment less consistent rubrics to be easier to grade (for a weaker judge like oss-120b) as opposed to dropping them entirely. i'm curious what authors think about this.
Questions for authors
see weaknesses.
4 comments
Response to Reviewer aBZZ (2/2)
Judge choice in on-policy RL must account for grading latency because rewards are computed repeatedly for newly sampled responses. This makes very large judges costly to use throughout training. For example, the original Rubrics as Rewards study uses gpt-4o-mini to assign training rewards [2]. We chose gpt-oss-120b because it combines strong performance with support for local deployment, making it practical for our RL training setup [3].
To examine whether stronger judges eliminate grading instability, we extended the cross-judge analysis in Figure 6 with an additional repeated-grading experiment. We selected 100 outputs and graded each output four times using GPT-6 Astra, Claude Opus 5.5, and DeepSeek 4.1 Flash.
| Judge | Mean score | Mean SD | Same score (4 runs) |
|---|---|---|---|
| GPT-6 Astra | 26.91 | 1.45 | 76% |
| Claude Opus 5.5 | 29.24 | 3.06 | 62% |
| DeepSeek 4.1 Flash | 30.36 | 5.42 | 40% |
“Same score (4 runs)” reports the percentage of outputs receiving identical total scores across all four grading runs.
These results confirm that grading stability varies across judges. However, all three judges still exhibit variability when grading the same outputs. Even GPT-6 Astra, which has the lowest variability in this experiment, produces different total scores across runs for 24% of outputs. Thus, using a stronger judge alone does not eliminate instability in rubric-based grading. Such instability can also arise when a rubric is inapplicable to evaluating the current sampled response.
Regarding rewriting and augmenting less consistent rubrics, doing so during RL training is impractical in our setting for two reasons. First, we cannot readily verify the correctness of automatically rewritten or augmented criteria. An edit may unintentionally change the intended requirement, omit an important qualification, or introduce an incorrect requirement, thereby adding substantial noise to the reward. Second, rewriting and augmenting criteria would require additional generation and validation, increasing latency within the training loop. This additional delay is difficult to accommodate in on-policy RL.
RAR² assesses the reliability of rubric judgments on the currently sampled responses and filters unreliable reward contributions during training. It therefore provides a practical way to reduce unstable supervision without requiring rubric generation and validation at each update.
[2] Gunjal et al. (2025). Section 4.2 specifies gpt-4o-mini as the training judge. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains.
[3] OpenAI (2025). Introducing gpt-oss.
Gentle Reminder
Dear Reviewer aBZZ,
As the discussion period ends in about a day, we would greatly appreciate your feedback on whether our responses and additional results address your concerns. Specifically, we have
(1) Clarified that grading stability depends jointly on the rubric, judge model, and sampled response.
(2) Added an experiment in which three judges each evaluated the same 100 outputs four times, showing that grading variability persists even with stronger judges.
(3) Explained why some rubric criteria may be inapplicable to evaluating a particular sampled response, and discussed the correctness and latency challenges of rewriting or augmenting rubrics during RL training.
If these clarifications and results resolve your concerns, we would be grateful if you would consider updating your rating. We would also be happy to answer any remaining questions before the deadline.
Thank you again for your time and thoughtful review.
Dear Reviewer,
With less than two hours remaining in the discussion period, we would like to kindly follow up on our responses to your comments. If your schedule permits, we would greatly appreciate hearing whether our responses have addressed your concerns or whether any points still require clarification.
Thank you again for your time and thoughtful feedback.
Best regards, The Authors
Response to Reviewer aBZZ (1/2)
Thank you for your feedback. We agree that grading stability depends on the judge. More broadly, it depends jointly on three factors (rubric, judge model, sampled response).
Rubrics can be imperfect, even when they aim to cover the key points of a comprehensive expert solution. HealthBench, for example, uses physician-written, conversation-specific criteria [1]. During RL training, these rubrics are applied to responses sampled from the current policy. Some criteria may be inapplicable to the sampled response because they were derived from expert-written outputs. For that sampled response, including such criteria in the reward can instead lead to reward hacking.
The following example illustrates this issue.
Question
Model response (selected excerpts)
Rubric (original wording, +6 points)
The sampled response focuses on actions after arrival, whereas the rubric assesses preparations before departure, including changing sleep schedules before traveling and arriving several days early. These actions concern a decision point that has already passed. Although such advice could be useful for a future trip, its absence does not necessarily constitute a failure of this particular sampled response. Including this rubric in the reward for the current response can instead encourage reward hacking.
[1] Arora et al. (2025). Section 4.3 describes the construction of physician-written rubrics. HealthBench: Evaluating Large Language Models Towards Improved Human Health.