acceptodds
Under review as a conference paper at ICLR 2027

RAR²: Reliability-Aware Rubrics as Rewards for Reinforcement Learning

Abstract

LLMs are increasingly deployed across open-ended tasks, where high-quality outputs must satisfy multiple, task-specific criteria. Rubric-based evaluation makes these criteria explicit and is used to assess complex outputs in specialized, high-stakes domains. The Rubrics as Rewards (RaR) framework extends this approach to reinforcement learning by aggregating rubric-level verdicts into structured reward signals. We characterize rubric signals along two dimensions: discrimination across policy-generated responses and stability across repeated evaluations. The two dimensions divide rubric signals into four quadrants, with stable and discriminative signals providing the most reliable guidance for policy learning. However, standard RaR implicitly assumes noise-free rubric verdicts and therefore cannot distinguish genuine differences between responses from those induced by unstable judgments. To address this limitation, we propose Reliability-Aware Rubrics as Rewards (RAR²). RAR² measures stability through within-response variance and discrimination through between-response variance. A Beta–Binomial random-effects model then assesses whether each rubric's apparent discrimination is supported by stable judgments, and a reliability gate filters out rubric signals that lack such support. Experiments show that RAR² achieves the best performance across models of different sizes and diverse tasks. Additional analyses demonstrate that RAR² delivers consistent performance gains across judge reasoning-effort settings and improves agreement with expert assessments across judge models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, rubric-based rewards, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aBZZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

As the paper mentions, rubrics need to be stable and discriminative. If not discriminative, they won't provide any RL signal, and if gradings are not stable, they will provide noisy signals. They combine these into a reliability scores and drop rubrics that are not stable or discriminative. They train two Qwen models and show improvements over three benchmarks.

Critical strengths

The argument around stability and discriminative make sense and are valid. They train two different size models and show improvements on three different benchmarks. Also, they compared with three different baselines and show their method consistently outperforms the others. They also show that expert agreements increase and judges take lower effort to grade the filtered rubrics.

Critical weaknesses

I think there's a very important factor that is not discussed in the paper: consistency is not a function of the rubric alone, it also heavily depends on the judge used for evaluation. I think they should be viewed as a pair, a stronger judge will be more consistent on rubrics, and a weaker one will be less consistent. So I think it's not easy to disentangle whether the source of inconsistency is the rubric or the judge. I think more analysis on the effect of different judges can make this paper more robust.

There should also be more discussion on why it is acceptable to drop rubrics entirely. To me, it's unintuitive to remove some important objectives from reward design and it could even introduce reward hacking. The best rubric is the most comprehensive one. I think another option would be to rewrite and augment less consistent rubrics to be easier to grade (for a weaker judge like oss-120b) as opposed to dropping them entirely. i'm curious what authors think about this.

Questions for authors

see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aBZZ (1/2)

Thank you for your feedback. We agree that grading stability depends on the judge. More broadly, it depends jointly on three factors (rubric, judge model, sampled response).

Rubrics can be imperfect, even when they aim to cover the key points of a comprehensive expert solution. HealthBench, for example, uses physician-written, conversation-specific criteria [1]. During RL training, these rubrics are applied to responses sampled from the current policy. Some criteria may be inapplicable to the sampled response because they were derived from expert-written outputs. For that sampled response, including such criteria in the reward can instead lead to reward hacking.

The following example illustrates this issue.

Question

After my flight from Boston to Madrid, i'm so drained i can barely stay awake. should i drink coffee or use melatonin? i hate feeling groggy.

Model response (selected excerpts)

“Try to go to sleep at the same time as locals in Madrid (e.g., 10 PM–11 PM).”

“Get sunlight exposure during the day to help reset your internal clock.”

“Drink coffee if you're trying to stay awake.”

“Use melatonin only if you're struggling to fall asleep — and only at night.”

Rubric (original wording, +6 points)

Recommends strategies to adjust the body's natural clock before traveling to your desired destination, including going to bed an hour or two later than usual if traveling west and an hour or two earlier if traveling east, and suggests traveling a few days earlier before important occasions to allow the body's natural clock to adjust to the new destination.

The sampled response focuses on actions after arrival, whereas the rubric assesses preparations before departure, including changing sleep schedules before traveling and arriving several days early. These actions concern a decision point that has already passed. Although such advice could be useful for a future trip, its absence does not necessarily constitute a failure of this particular sampled response. Including this rubric in the reward for the current response can instead encourage reward hacking.

[1] Arora et al. (2025). Section 4.3 describes the construction of physician-written rubrics. HealthBench: Evaluating Large Language Models Towards Improved Human Health.

AuthorsOpenReview ↗

Response to Reviewer aBZZ (2/2)

Judge choice in on-policy RL must account for grading latency because rewards are computed repeatedly for newly sampled responses. This makes very large judges costly to use throughout training. For example, the original Rubrics as Rewards study uses gpt-4o-mini to assign training rewards [2]. We chose gpt-oss-120b because it combines strong performance with support for local deployment, making it practical for our RL training setup [3].

To examine whether stronger judges eliminate grading instability, we extended the cross-judge analysis in Figure 6 with an additional repeated-grading experiment. We selected 100 outputs and graded each output four times using GPT-6 Astra, Claude Opus 5.5, and DeepSeek 4.1 Flash.

JudgeMean scoreMean SDSame score (4 runs)
GPT-6 Astra26.911.4576%
Claude Opus 5.529.243.0662%
DeepSeek 4.1 Flash30.365.4240%

“Same score (4 runs)” reports the percentage of outputs receiving identical total scores across all four grading runs.

These results confirm that grading stability varies across judges. However, all three judges still exhibit variability when grading the same outputs. Even GPT-6 Astra, which has the lowest variability in this experiment, produces different total scores across runs for 24% of outputs. Thus, using a stronger judge alone does not eliminate instability in rubric-based grading. Such instability can also arise when a rubric is inapplicable to evaluating the current sampled response.

Regarding rewriting and augmenting less consistent rubrics, doing so during RL training is impractical in our setting for two reasons. First, we cannot readily verify the correctness of automatically rewritten or augmented criteria. An edit may unintentionally change the intended requirement, omit an important qualification, or introduce an incorrect requirement, thereby adding substantial noise to the reward. Second, rewriting and augmenting criteria would require additional generation and validation, increasing latency within the training loop. This additional delay is difficult to accommodate in on-policy RL.

RAR² assesses the reliability of rubric judgments on the currently sampled responses and filters unreliable reward contributions during training. It therefore provides a practical way to reduce unstable supervision without requiring rubric generation and validation at each update.

[2] Gunjal et al. (2025). Section 4.2 specifies gpt-4o-mini as the training judge. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains.

[3] OpenAI (2025). Introducing gpt-oss.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer aBZZ,

As the discussion period ends in about a day, we would greatly appreciate your feedback on whether our responses and additional results address your concerns. Specifically, we have

(1) Clarified that grading stability depends jointly on the rubric, judge model, and sampled response.

(2) Added an experiment in which three judges each evaluated the same 100 outputs four times, showing that grading variability persists even with stronger judges.

(3) Explained why some rubric criteria may be inapplicable to evaluating a particular sampled response, and discussed the correctness and latency challenges of rewriting or augmenting rubrics during RL training.

If these clarifications and results resolve your concerns, we would be grateful if you would consider updating your rating. We would also be happy to answer any remaining questions before the deadline.

Thank you again for your time and thoughtful review.

Dear Reviewer,

With less than two hours remaining in the discussion period, we would like to kindly follow up on our responses to your comments. If your schedule permits, we would greatly appreciate hearing whether our responses have addressed your concerns or whether any points still require clarification.

Thank you again for your time and thoughtful feedback.

Best regards, The Authors

Reviewer_jkNCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies unreliable rubric-based rewards for LLM reinforcement learning, distinguishing reproducible differences between responses from variation caused by inconsistent judge verdicts. RAR² collects repeated binary judgments, fits a Beta–Binomial random-effects model for each rubric, and applies a posterior reliability gate before constructing rewards for group-relative policy optimization. Experiments cover two Qwen3 policy sizes and three professional-reasoning benchmarks, with additional analyses of judgment count, judge reasoning effort, runtime, and expert agreement. RAR² achieves the highest mean score in all ten model–evaluation combinations in Table 1.

Critical strengths
  1. The distinction between discrimination and reliability addresses a consequential training problem. A rubric can separate sampled responses because of judge randomness rather than reproducible response differences, potentially compromising group-relative credit assignment. The variance decomposition and the observation that a single judgment cannot identify the reliability parameter provide a coherent foundation for addressing this issue. This makes the problem formulation useful beyond the particular implementation.
  2. The main comparison provides evidence of benefits beyond repeated judging alone. Table 1 explicitly states that the RaR baseline also averages multiple judgments, while evaluation restores the full rubric set. RAR² nevertheless outperforms RaR, ARL, and POW3R across both policy sizes and all reported aggregate evaluation slices. For example, the larger policy’s HealthBench score increases from 69.74 with RaR to 74.34 with RAR². These controls and consistent improvements make the approach practically promising.
Critical weaknesses
  1. The added value of the Bayesian gate over simpler filtering remains insufficiently established. Appendix A.2 derives a moment-based reliability/ICC estimator, but the paper does not compare training with this estimator against posterior gating. A random gate with a matched removal rate is also absent. Consequently, the experiments support the complete procedure without establishing whether its benefits require Bayesian reliability estimation or arise largely from simpler filtering or rubric removal as regularization. Appendix C.1 reports removal rates rather than final performance as the threshold varies. These omissions are consequential because they leave the specific methodological contribution insufficiently isolated.
  2. The evidence connecting reproducibility to expert-correct supervision is underspecified. Equation (6) measures reliability under a fixed judge protocol, which can remain high when the judge makes systematically incorrect distinctions. Figure 6 is therefore important, but the paper does not clearly define the unit of expert agreement, how the three experts are combined, or whether filtering changes the denominator. Higher agreement on retained rubrics would not by itself establish improved correctness over the full objective. Furthermore, the main scores in Appendix B.2 are automated rubric evaluations, and an independent assessment of the final policies’ improvements is not clearly established. This limits the evidence that RAR² improves supervision quality beyond consistency with the judging protocol.
Questions for authors
  1. For Figure 6, what exactly are the numerator and denominator of expert agreement? Is agreement measured on individual rubric verdicts, response rankings, or another learning signal? Are dropped rubrics excluded, what fraction of judgments remains, and how are the three experts’ annotations aggregated? Are RaR and RAR² compared on a common set of evaluated decisions?
  2. Which judge model and reasoning-effort configuration are used for held-out scoring in Table 1 and Figure 4? Is the evaluation protocol fixed when the training judge’s reasoning effort changes? Do the cross-judge results assess the final trained policies or only their training signals?
  3. Under the same repeated judgments and training budget, how does posterior gating compare with gating using the Appendix A.2 estimator and with random removal at a matched rate? This comparison would directly determine whether the proposed reliability model offers an advantage over simpler alternatives.
3 comments
AuthorsOpenReview ↗

Response to Reviewer jkNC (1/3)

Thank you for your review and constructive suggestions. Below, we address each of your concerns.

Weakness1

To examine the added value of Bayesian reliability gating, we conducted additional Qwen3-4B-Instruct experiments using the moment-based reliability estimator in Appendix A.2 and a random removal baseline. For random removal, we fixed a random rubric mask before training, matching the dataset-level removal rates of RAR² in Figure 7.

Rebuttal Table 1. Gate ablations. All scores are percentages.

BenchmarkRaRRandom removalMoment-based gateRAR²
HealthBench46.2343.1548.8649.71
PRBench Non-Hard39.5638.7142.7143.01
PRBench Hard18.6712.5420.2420.76
OneMillion-Bench Global23.5719.4225.8226.11
OneMillion-Bench CN25.0622.8328.5328.96

Random removal scores below RaR on all five evaluation slices. The moment-based gate improves over RaR, and RAR² further improves on the moment-based gate across all five slices. These comparisons provide evidence for an additional benefit of posterior gating over direct empirical reliability filtering.

The moment-based gate uses a reliability point estimate calculated from the empirical variance decomposition. Under a finite judgment budget, this estimate can be sensitive to sampling variation. RAR² uses the count likelihood and priors to obtain a posterior distribution and makes its selection based on the probability that reliability exceeds the required level.

To assess how the gate threshold affects final performance, we evaluated τ=0.40\tau=0.40 and τ=0.70\tau=0.70, in addition to the default τ=0.55\tau=0.55.

Rebuttal Table 2. Threshold sensitivity. All scores are percentages.

BenchmarkRaRτ=0.40\tau=0.40τ=0.55\tau=0.55, defaultτ=0.70\tau=0.70
HealthBench46.2346.7149.7144.62
PRBench Non-Hard39.5640.1243.0139.54
PRBench Hard18.6718.8520.7618.49
OneMillion-Bench Global23.5723.9826.1123.11
OneMillion-Bench CN25.0625.3428.9624.79

At τ=0.40\tau=0.40, the removal rates on HealthBench, PRBench, and OneMillion-Bench are 1.57%, 1.87%, and 1.33%, respectively, and performance remains close to RaR. At τ=0.70\tau=0.70, the rates rise to 25.76%, 11.91%, and 12.25%, respectively, and performance falls below the default setting on every slice. The default threshold achieves the highest score on each slice among the three settings tested.

We will add the moment-based and random removal controls to the revised experiments and include the threshold-dependent final scores in Appendix C.1.

AuthorsOpenReview ↗

Response to Reviewer jkNC (2/3)

Weakness2

Thank you for pointing out the need to clarify expert agreement more precisely. Equation (6) measures reproducibility under a fixed judge protocol. Figure 6 assesses agreement with expert judgments about training-time rubric decisions. The annotation protocol and treatment of filtering are detailed below.

Q1: For Figure 6, what exactly are the numerator and denominator of expert agreement?

The numerator is the number of rubric items for which the method's decision matches the final expert label. The denominator is the total number of rubric items associated with the 100 sampled examples for each benchmark. Rubrics filtered by RAR² remain in the denominator.

For RaR, a match is Pass/Pass or Fail/Fail. For RAR², Drop/Drop also counts as a match. Every off-diagonal entry counts as a disagreement.

Q2: Is agreement measured on individual rubric verdicts, response rankings, or another learning signal? Are dropped rubrics excluded, what fraction of judgments remains, and how are the three experts' annotations aggregated?

Agreement is measured on one rubric criterion applied to one model-generated response. The analysis includes all sampled rubric items, including those labeled Drop. The matrices below show the complete distribution of retained and dropped decisions.

Each response and its corresponding rubrics are provided to two experts for separate assessment. Each expert assigns a Pass/Fail label for RaR and a Pass/Fail/Drop label for RAR² to each rubric. If their labels agree, the agreed label is adopted as the final expert judgment. Disagreements are referred to a third expert, whose adjudication determines the final label. These final labels serve as the reference for Figure 6.

Q3: Are RaR and RAR² compared on a common set of evaluated decisions?

Yes. For each benchmark, we uniformly sampled 100 training examples during the first epoch. The same examples and associated rubric-response items are used for both methods and all judge models. Expert labels use the decision options available to the corresponding method.

The following raw confusion matrices give the PRBench results underlying Figure 6(b). Each entry is a percentage of all sampled rubric items. Each judge's matrix sums to 100%.

Rebuttal Table 3. RaR. Column labels indicate expert/model decisions.

JudgePass/PassPass/FailFail/PassFail/Fail
gpt-oss-120b23.89%14.13%13.65%48.33%
GPT-6 Astra26.82%9.42%8.96%54.80%
Claude Opus 527.85%8.11%8.99%55.05%
Hy4 Preview24.99%12.86%11.75%50.40%

Rebuttal Table 4. RAR². Rows indicate expert decisions and columns indicate method decisions.

JudgeExpert decisionPassFailDrop
gpt-oss-120bPass26.84%1.12%0.87%
Fail0.64%55.43%0.55%
Drop0.52%0.39%13.64%
GPT-6 AstraPass28.93%0.97%0.12%
Fail0.28%56.77%0.36%
Drop0.09%1.16%11.32%
Claude Opus 5Pass27.63%1.86%0.27%
Fail0.33%56.04%0.54%
Drop0.05%0.31%12.97%
Hy4 PreviewPass27.27%0.82%0.18%
Fail0.13%57.47%0.24%
Drop0.11%0.31%13.47%

The RAR² comparison includes expert agreement about whether a rubric should be removed. We will clarify the decision labels, full denominator, sampling procedure, and expert adjudication protocol in the revised manuscript and Figure 6 description.

AuthorsOpenReview ↗

Response to Reviewer jkNC (3/3)

Weakness2 (continued)

To assess the final policies beyond automated rubric scoring, we conducted an additional human pairwise evaluation. We randomly sampled questions from the HealthBench and PRBench test sets and provided experts with the RaR and RAR² responses to the same question. The preference results are reported below from RAR²'s perspective.

Rebuttal Table 5. Human preference results.

BenchmarkWinTieLoss
HealthBench72%19%9%
PRBench79%3%18%

Experts preferred RAR² outputs more often than RaR outputs on both benchmarks. We will add this assessment of the final policy outputs to the revised evaluation section.

Q4: Which judge model and reasoning-effort configuration are used for held-out scoring in Table 1 and Figure 4?

As described in Section 4, Table 1 uses gpt-oss-120b with high reasoning effort. Figure 4 uses the same judge model, with the reasoning-effort setting indicated on the horizontal axis.

Q5: Is the evaluation protocol fixed when the training judge's reasoning effort changes?

The held-out test examples and judge model are fixed across the Figure 4 settings. Every policy is scored using the complete evaluation rubric set. The reasoning-effort settings are indicated on the horizontal axis. We will clarify these evaluation details in the revised experimental setup.

Q6: Do the cross-judge results assess the final trained policies or only their training signals?

Figure 6 assesses agreement between the training-time rubric decisions and expert judgments across judge models. The additional human pairwise evaluation in Rebuttal Table 5 assesses the final trained policies.

Q7: Under the same repeated judgments and training budget, how does posterior gating compare with gating using the Appendix A.2 estimator and with random removal at a matched rate?

The comparisons in Rebuttal Table 1 use the same repeated-judgment settings and training budget. The moment-based control gates rubrics using the Appendix A.2 reliability point estimate. The random control fixes a random rubric mask before training at the dataset-level removal rates reported in Figure 7.

Posterior gating outperforms the moment-based gate on all five evaluation slices, while random removal scores below RaR on every slice. These results support an additional benefit from posterior gating over the empirical gate and show that random removal at matched rates does not reproduce the gains of RAR². We will add both controls and their implementation details to the revised manuscript.

Reviewer_mnhSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors introduce Reliability-aware rubrics as rewards, a framework designed to improve rubric-based reinforcement learning by explicitly accounting for the reliability of LLM-generated rubric judgements. The authors argue that existing rubric-based reward methods primarily focus on discriminating among policy-generated responses while overlooking judgment instability, which may introduce spurious reward differences and negatively affect policy optimization. To address this issue, the proposed framework characterizes rubric signals along two dimensions: within-response judgment stability and between-response discrimination. The proposed method repeatedly evaluates each response under a fixed judge protocol and employs a Beta–Binomial random-effects model to estimate the reliability of rubric-level reward signals. A Bayesian reliability gate then filters rubrics whose observed response differences lack sufficient statistical support, while preserving the original weights of retained rubrics.

Critical strengths
  1. The paper is well-written and easy to follow;

  2. Explicitly distinguishing judgement stability from response discrimination is a reasonable research question.

  3. The paper investigates how the number of repeated evaluations and judge reasoning effort affect training performance, it provides useful analysis of judgement repetition and reasoning effort.

  4. The decomposition of total judgment variance into between-response variation and within-response noise is well motivated, and the Beta–Binomial random-effects model provides a natural way to represent repeated binary judgments.

Critical weaknesses
  1. A central claim of the paper is that the authors give rubrics that are both stable and discriminative. However, the reliability gate is defined through ρK=Kϕ1+(K−1)ϕ\rho_K=\frac{K\phi}{1+(K-1)\phi}. Importantly, this quantity does not directly constrain the absolute magnitude of between-response discrimination, given by σB2=μ(1−μ)ϕ\sigma_B^2=\mu(1-\mu)\phi. Consequently, a rubric may have high relative reliability while providing little meaningful discrimination among responses, particularly when μ\mu is close to zero or one. This will lead to a gap between the two-dimensional motivvation and the actual selection criterion.

  2. The current experiments do not give sufficient ablation studies to guarantee the contribution of the bayesian reliability gate. In particular, it remains unclear whether the improvements arise from accurate reliability estimation or from more generic effects of rubric filtering, reward renormalization, or the reduction of noisy training signals.

  3. Figure 3 reports that a substantial proportion of rubrics exhibit unstable judgments, whereas Figure 7 suggests that the proportion of rubrics removed by the reliability gate at the default threshold is considerably smaller. These observations are not necessarily inconsistent.

  4. Figure 6 reports substantial improvements in agreement with human expert assessments, while the authors did not provide how the judgements of the three human experts are aggregated.

  5. The proposed reliability gate primarily thresholds a posterior probability involving the intraclass correlation coefficient, rather than independently ensuring both judgment stability and meaningful response discrimination, as suggested by the paper's central framing.

Questions for authors

Please see Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer mnhS (1/2)

We thank the reviewer for the thoughtful comments. We address your concerns point by point below.

Weakness 1

We appreciate the reviewer’s observation and will clarify the connection between our two-dimensional motivation and the selection criterion. Our gate jointly evaluates response discrimination and judgment stability through

ρK=σB2σB2+σW2/K=Kϕ1+(K−1)ϕ.\rho_K = \frac{\sigma_B^2}{\sigma_B^2+\sigma_W^2/K} = \frac{K\phi}{1+(K-1)\phi}.

Accordingly, the posterior-based selection condition can be written as

P(ρK≥12∣nr)=P(σB2≥σW2K∣nr)≥τ.P\left(\rho_K \ge \frac{1}{2} \mid \mathbf{n}_r\right) = P\left(\sigma_B^2 \ge \frac{\sigma_W^2}{K} \mid \mathbf{n}_r\right) \ge \tau.

This defines an acceptable relationship between response contrast and judgment noise: smaller contrasts can be retained when judgments are sufficiently reproducible, whereas greater judgment noise requires stronger response contrast.

When μ\mu approaches zero or one, both between-response variance and within-response noise decrease through the same factor μ(1−μ)\mu(1-\mu):

σB2=μ(1−μ)ϕ,σW2=μ(1−μ)(1−ϕ).\sigma_B^2 = \mu(1-\mu)\phi, \qquad \sigma_W^2 = \mu(1-\mu)(1-\phi).

A small absolute between-response variance therefore does not by itself establish that the signal is uninformative. For example, infrequent but consistently identified failures may still provide useful feedback for relative credit assignment. The absence of a minimum absolute discrimination threshold is a deliberate design choice.

Weakness 2

The Bayesian reliability gate addresses uncertainty in estimating rubric reliability from a limited number of responses and repeated judgments. In small response groups, empirical variance estimates can fluctuate substantially. Bayesian modeling incorporates prior regularization and allows uncertainty in these estimates to enter the selection decision.

To examine the additional contribution of posterior-based selection, we conducted an ablation that replaces the Bayesian gate with direct empirical reliability estimation. This variant retains the reliability-based filtering framework and uses the noise-corrected empirical variance ratio described in Appendix A.2.

The results for Qwen3-4B-Instruct are:

Evaluation settingEmpirical reliability gateRAR²Improvement
HealthBench48.8649.71+0.85
PRBench Non-Hard42.7143.01+0.30
PRBench Hard20.2420.76+0.52
OneMillion-Bench Global25.8226.11+0.29
OneMillion-Bench CN28.5328.96+0.43

Improvements are reported in percentage points. RAR² achieves higher scores across all five evaluation settings, with gains ranging from 0.29 to 0.85 points and an average gain of 0.48 points. The consistent improvements support the additional benefit of the Bayesian gate. We will include this ablation in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer mnhS (2/2)

Weakness 3

We agree that these two observations are not inconsistent. To provide a precise empirical characterization of rubric categories, Figure 3 uses the empirical reliability estimate. Rubrics are assigned to categories through explicit threshold-based rules using the empirical diagnostics. Figure 7 reports the fraction of rubrics actually removed by the posterior-based reliability gate in RAR², including its retention safeguards.

A rubric classified as unstable by an empirical estimate is not necessarily discarded by the posterior-based gate, which accounts for estimation uncertainty when making the selection decision. Retention safeguards can also preserve rubrics that would otherwise be removed. We will clarify these definitions and specify the empirical classification thresholds in the figure descriptions.

Weakness 4

Thank you for pointing out this omission. Each model-generated response, together with its corresponding rubrics, is provided to two human experts for separate assessment. Each expert assigns a Pass/Fail label for RaR and Pass/Fail/Drop label for RAR² to each rubric.

If their labels agree, the agreed label is adopted as the final expert judgment. Disagreements are referred to a third expert, whose adjudication determines the final label. These final expert labels serve as the reference for the agreement reported in Figure 6. We will clarify this annotation and adjudication protocol in the revised manuscript.

Weakness 5

The ICC-based formulation explicitly incorporates both between-response variation and within-response judgment noise:

ϕ=σB2σB2+σW2.\phi = \frac{\sigma_B^2}{\sigma_B^2+\sigma_W^2}.

For the average of KK judgments, the corresponding reliability is

ρK=σB2σB2+σW2/K.\rho_K = \frac{\sigma_B^2}{\sigma_B^2+\sigma_W^2/K}.

Thresholding a posterior probability involving this quantity therefore implements a joint assessment of response discrimination and judgment stability. Both dimensions enter the gate explicitly through the variance decomposition.

This joint criterion offers two practical advantages. First, selection uses a single tunable posterior-confidence threshold τ\tau. This reduces the complexity of specifying and tuning separate thresholds for stability and discrimination. Second, the criterion accommodates judgment variability when it is supported by sufficiently strong response discrimination. Rubrics whose judgments are relatively unstable can still be retained if their between-response variation is sufficiently large. Such rubrics may provide useful learning signals. Jointly evaluating the two variance components allows us to preserve these signals while controlling their relative noise.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer mnhS,

As the discussion period ends in about a day, we would greatly appreciate your feedback on whether our responses and additional ablation address your concerns. Specifically, we have:

(1) Clarified how the gate jointly considers stability and discrimination.

(2) Added an ablation comparing posterior-based gating with direct empirical reliability estimation.

(3) Explained the relationship between Figures 3 and 7 and detailed the expert annotation and adjudication protocol for Figure 6.

If these clarifications and results resolve your concerns, we would be grateful if you would consider updating your rating. We would also be happy to answer any remaining questions before the deadline.

Thank you again for your time and thoughtful review!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion and Revisions

Summary of the Discussion and Revisions

We thank the AC and the reviewers for a constructive discussion.

Strengths noted. Reviewers found that "the distinction between discrimination and reliability addresses a consequential training problem" and makes "the problem formulation useful beyond the particular implementation" (jkNC), that the variance decomposition is "well motivated" and the Beta-Binomial model "a natural way to represent repeated binary judgments" (mnhS), and that the method "consistently outperforms the others" across two policy sizes, three benchmarks and three baselines while "expert agreements increase" (aBZZ).

New analyses added during the discussion

  • Joint stability and discrimination in the gate (mnhS W1, W5). Section 3.2 and Appendix A.3 show that ρK≥12\rho_K\ge\tfrac12 is equivalent to σB2≥σW2/K\sigma_B^2\ge\sigma_W^2/K and to ϕ≥1/(K+1)\phi\ge 1/(K+1). The gate therefore weighs a rubric's contrast against the judge noise left after averaging KK verdicts, and the gated event does not depend on the pass rate μ\mu.
  • Contribution of the Bayesian gate (mnhS W2, jkNC W1 and Q7). In Appendix C.2, over three seeds, random removal at matched rates falls below RaR on all five slices, while a moment-based gate at the same reliability boundary gains 1.69 to 3.39 points and RAR² exceeds that gate on every slice by 0.20 to 1.06 points.
  • Expert agreement protocol (mnhS W4, jkNC W2 and Q1 to Q3). Appendix C.4 defines an item as one criterion applied to one response, over 100 sampled examples per benchmark, with identical items for both methods and all four judges. Two experts label each item and a third adjudicates disagreements. Dropped items stay in the denominator, and Pass/Fail confusions fall from 27.78% to 1.76% for gpt-oss-120b.
  • Held-out evaluation protocol (jkNC Q4 to Q6). Appendix B.2 fixes one protocol for every method, using gpt-oss-120b at high effort with K=3K=3 and the full rubric set with the gate disabled.
  • Judge-dependent grading stability (aBZZ W1). In Appendix C.6 we grade 100 outputs four times with each of three judges. The mean standard deviation ranges from 1.45 to 5.42 points and the share of identical scores from 40% to 76%, yet even the most stable judge changes 24% of outputs.

Status. Reviewer jkNC raised the rating from 2 to 3, noting that keeping dropped items in the denominator "resolves my original concern about Figure 6", that the three-seed ablation is "more informative than the initial single-run ablation", and that the human evaluation is "the strongest new evidence", so that "these results sufficiently address my main concerns". Reviewer mnhS acknowledged that our responses solved the concerns raised in the review, and the new gate ablations, the clarified relation between Figures 3 and 7, and the expert annotation protocol resolved W2 through W5, which drew no further questions. For the remaining W1, Appendix A.3 proves that the gated event is independent of the pass rate, and our follow-up shows that adding absolute floors does not improve over RaR. For Reviewer aBZZ we added the cross-judge repeated-grading experiment in Appendix C.6 and explained why we gate rather than rewrite rubrics, and we remain glad to answer any further questions.

We thank the reviewers for their time and the AC for considering this summary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.