acceptodds
Under review as a conference paper at ICLR 2027

Rubric Rewards from Item Response Theory

Abstract

Many language tasks have no single answer that can be checked automatically. Rubrics provide criteria for judging responses to these tasks. For reinforcement learning, the resulting verdicts must be combined into a scalar reward. A common approach sums the points assigned to satisfied criteria. Distinct verdict patterns can thus receive the same reward, and the fixed points encode how much each criterion should count, not how strongly its verdict distinguishes the current rollouts. Beyond this aggregation problem, judging the full rubric needs more judge requests as the criterion count grows. To address these limitations, Rubric Response Theory (RRT) measures quality and selects criteria when rubric criteria are monotone indicators of a shared target. Rather than adding assigned points, RRT uses a two parameter item response model that treats the verdict pattern as evidence about scalar quality specific to the rubric. Under this model, its likelihood score maximizes the local signal-to-noise ratio for quality. Its Response Parameter Network (RPN) reads the prompt and criterion text to predict criterion difficulty and discrimination. As the policy distribution changes during training, RRT uses online expectation maximization to update the RPN from current rollout verdicts. With Qwen3.5-4B as the policy, RRT's macro criterion score across Medical, Science, Rubrics as Rewards Science, and RubricBench is 1.7 points above that of group relative policy optimization (GRPO). On hard and very hard criteria in Medical and Science, RRT gains 2.8 to 5.6 points over GRPO. At half the criterion budget, adaptive Fisher selection with a frozen RPN keeps the macro criterion score across four datasets within 0.1 points of GRPO with full judging. These results show RRT can reduce judge requests while remaining competitive with GRPO.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
large language models, reinforcement learning, rubric-based rewards, item response theory, reward aggregation, LLM-as-a-judge, adaptive criterion selection, Fisher information

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ve73OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Rubric Rewards from Item Response Theory (RRT) that treats multiple rubric criteria as noisy measurements of a shared latent response quality, rather than directly aggregating their binary verdicts through weighted summation. It introduces a Response Parameter Network (RPN) to estimate criterion difficulty and discrimination, and infers latent quality scores from the observed verdict patterns to serve as rewards for GRPO training. Additionally, the framework uses Fisher information to adaptively select informative criteria, achieving competitive policy performance while substantially reducing judging costs. Experiments on RLHF demonstrate the effectiveness of RRT.

Critical strengths
  1. Simply summing or averaging multiple criterion rewards in rubric reward + GRPO training settings can overlook differences in criterion difficulty and informativeness. The paper offers an interesting perspective on how these signals could be aggregated more meaningfully.

  2. The IRT-based formulation is intuitive and well motivated. Treating rubric criteria as noisy measurements of an underlying response quality provides a principled alternative to conventional weighted aggregation. Modeling criterion difficulty and discrimination also allows the framework to make better use of different verdict patterns.

Critical weaknesses
  1. The assumption of a shared latent quality may be too strong for general rubric-based RL. RRT assumes that all criteria are monotonic indicators of a single underlying response quality. However, real-world rubrics often involve different objectives, which may have genuine trade-offs.

  2. It is unclear whether the inferred latent quality better reflects actual response quality or human preferences. The paper demonstrates that RRT can distinguish different verdict patterns and reduce reward ties, but greater reward discrimination does not necessarily mean better reward quality. In particular, criterion difficulty and statistical informativeness are not necessarily equivalent to criterion importance. Some independent evaluation against human preferences would make this claim more convincing. For example, I suggest that the authors could conduct additional experiments on human preference datasets (like helpsteer3) to examine whether the inferred latent-quality scores better align with human preference judgments than conventional additive reward aggregation.

  3. I am also somewhat concerned about the additional complexity introduced by the Response Parameter Network (RPN). Although the authors report relatively low computational overhead for online RPN updates, the framework still requires a separate warm-start procedure and an additional parameterized model that is continuously updated alongside the policy. Since the RPN directly determines the inferred reward, its initialization quality and calibration stability could potentially affect the overall RL training dynamics. Moreover, the improvement from online RPN updates over a frozen RPN appears relatively modest (Table 2). It would be helpful to further investigate sensitivity to RPN initialization, update frequency, and training hyperparameters, as well as whether the additional online calibration consistently improves policy performance across different settings.

  4. The downstream RL improvements are relatively modest and not fully consistent across models. (That's fine, but experiments on LLMs with more parameters would be better, like 32B.)

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Quality and human preferences (Part 1 of 2)

Thank you for your feedback and positive assessment of the motivation and the formulation of RRT. The thoughtful concerns also helped improve the paper.

Changes to the manuscript are shown in blue. Section, appendix, and table numbers below refer to the revised manuscript unless marked as the submission.

W1: Shared latent quality and tradeoffs

The reviewer raises a useful concern about whether one quality variable can represent criteria with different or conflicting objectives. In RRT, ziz_i represents quality specific to a prompt and its rubric. Sec. 3.1 assumes that each criterion's pass probability increases with this quality, and the abstract and Introduction state this scope. A rollout can still satisfy some criteria and fail others.

GRPO takes one scalar reward per rollout, so any rubric reward places all criteria on one scale. A weighted sum of verdicts orders rollouts exactly as a logistic model with one quality variable does (Theorem 4), so RRT keeps the single scale of rubric points and makes it explicit and testable.

Correctness, safety, completeness, and conciseness can contribute to this shared target. As a case study, consider the Medical prompt with source index 70651. It asks for the two most widely accepted treatments for insomnia and when to prefer each. Its rubric includes criteria of all four kinds.

KindCriterion
CorrectnessThe response states that major clinical guidelines designate CBT-I as the first-line (gold-standard) treatment for chronic insomnia.
SafetyThe answer includes a brief caution that treatment decisions should be individualized and guided by a healthcare professional, especially given medication risks.
CompletenessThe answer describes core CBT-I components such as sleep restriction, stimulus control, cognitive restructuring, and relaxation training.
ConcisenessThe response provides sufficient detail to capture key distinctions and indications without unnecessary length or excessive technical digression.

In this case study, each criterion describes a feature of the same target answer. The conciseness criterion requires sufficient detail without unnecessary length, so it is defined relative to the content that the other criteria require. In addition, the rubric sources avoid conflicting criteria by design. RubricHub, the source of Medical and Science, constrains rubric generation so that each criterion does not strongly depend on, contradict, or overlap with others (Li et al., 2026), and RubricBench checks each rubric for internal conflicts (Zhou et al., 2026). For rubrics with explicit tradeoffs, Sec. 5 discusses modeling multiple quality targets.

To assess the shared quality assumption empirically, App. D.6 examines dependence between criterion verdicts (Table 22). Fitted quality explains 72.8% to 85.3% of pairwise mutual information. Furthermore, quality inferred from the other criteria predicts omitted verdicts 15.8 points above chance in macro ROC-AUC within criteria (Table 1). Together, these results support a shared quality variable for the evaluated rubrics.

W2: Criterion importance and human preferences

Thank you for suggesting a comparison with human preferences. Rubric points specify how much each criterion should count, while RRT measures how much its verdict reveals about shared quality (Sec. 1). Sec. 4.2 and App. C.8 now compare rubric points and RRT on all 150 RubricBench test pairs (Table 12). The comparison uses existing human preference labels from RubricBench, which includes response pairs from HelpSteer3, the dataset the reviewer suggested (Zhou et al., 2026).

For this comparison, both rewards use the same released criterion verdicts, with the RPN fitted on training prompts and held fixed for evaluation. Rubric points assign one point per criterion. Human preferences are used only for evaluation. Preference accuracy gives half credit for ties. Values are percentages.

RewardPreference accuracy
Rubric points56.3
RRT58.3

RRT improves human preference accuracy by 2.0 percentage points over rubric points. In the primary run on Qwen3.5-4B, RRT + online RPN also exceeds Vanilla GRPO by 1.7 points in macro normalized points score, which weights criteria by their assigned points and is the reward Vanilla GRPO trains on (Table 2).

W3: RPN cost, sensitivity, and online calibration

The reviewer raises useful questions about the cost of the RPN, its initialization and sensitivity, and what online updates add beyond a frozen RPN. Online updates raise Pearson correlation with criterion pass rates at the next policy step by 1.7 points in the macro mean over Medical and Science, using existing verdicts without additional judge requests (App. C.6, Table 10).

To be continued in Part 2.

AuthorsOpenReview ↗

RPN and policy models (Part 2 of 2)

Continued from Part 1.

W3: RPN cost, sensitivity, and online calibration (continued)

App. E.6 now reports the mean and standard deviation of criterion score over three training seeds on Qwen3.5-4B (Table 27).

ConditionMedicalScienceRaR ScienceRubricBenchMacro meanpp value vs. online RPN
Vanilla GRPO67.6 ±\pm 0.578.2 ±\pm 0.470.4 ±\pm 0.477.0 ±\pm 0.273.3 ±\pm 0.20.020
POW3R70.3 ±\pm 0.678.9 ±\pm 0.270.0 ±\pm 0.477.2 ±\pm 0.274.1 ±\pm 0.20.082
RRT + frozen RPN69.9 ±\pm 0.579.4 ±\pm 0.470.5 ±\pm 0.577.8 ±\pm 0.374.4 ±\pm 0.30.361
RRT + online RPN70.4 ±\pm 0.679.4 ±\pm 0.470.4 ±\pm 0.678.2 ±\pm 0.474.6 ±\pm 0.4N/A

The tt tests use macro score differences between each method and RRT + online RPN for runs with the same training seed.

With the RPN warm start held fixed, RRT + frozen RPN gains 1.1 macro criterion points over Vanilla GRPO across three training seeds, while RRT + online RPN, with one RPN update per policy step, gains 1.3 points over Vanilla GRPO (p=0.020p=0.020). Relative to the frozen RPN, the online RPN gains on Medical and RubricBench. Sec. 4.3 now reports the mean and standard deviation over three training seeds.

In addition, the paper's ablations show the RPN's contribution to policy scores. With the same reward, RPN parameters raise the macro criterion score over parameters from empirical pass rates (Table 2). RPN verdict prediction stays between 85.6% and 88.7% macro ROC-AUC across all tested RPN configurations and rollout policies (Table 5).

App. G.3 now reports calculated judge requests for the RPN warm start and policy training (Table 37). Accordingly, the abstract, Introduction, Sec. 4.4, and Sec. 5 now report savings including initialization.

# Judge requestsMedicalScienceRaR ScienceRubricBenchMacro mean
RPN initialization only481,032428,768119,63239,176267,152
Vanilla GRPO, full judging, total2,404,6722,148,624600,72078,3521,308,092
RRT with adaptive Fisher (0.50), training only1,222,7361,094,880328,86442,432672,228
RRT with adaptive Fisher (0.50), including initialization1,703,7681,523,648448,49681,608939,380
Reduction vs. full Vanilla GRPO, training only49.2%49.0%45.3%45.8%47.3%
Reduction vs. full Vanilla GRPO, including initialization29.1%29.1%25.3%−4.2%19.9%

Including initialization, judge requests fall by 19.9% on average. The requests saved in one epoch cover initialization on Medical, Science, and RaR Science.

Furthermore, App. G.3 reports the timing comparison in hours (Table 38).

DatasetVanilla GRPO, full judging, training runRRT (0.50), initializationRRT (0.50), training runRRT (0.50), totalReduction vs. Vanilla GRPO
Medical205.314.8157.8172.615.9%
Science214.512.0176.1188.112.3%
Macro mean209.913.4167.0180.414.1%

Including initialization, RRT reduces total elapsed time by 14.1% on average across Medical and Science. At criterion budget 0.50, adaptive Fisher selection with the frozen RPN keeps the macro criterion score within 0.1 points of Vanilla GRPO with full judging, and larger budgets recover more of RRT's gain (Table 4).

Overall, the RPN brings the policy gain and the judge savings, and online updates improve correlation with criterion pass rates and give the highest mean score without additional judge requests.

W4: Performance across policy models

Thank you for raising the question of whether RRT improves consistently across models and for suggesting evaluation with a larger policy. RRT improves macro criterion score on Qwen3.5-4B. While its scores on Qwen3.5-2B and Llama-3.1-8B-Instruct remain close to Vanilla GRPO's, RRT brings other gains. Across all three policies, it scores higher on RubricBench and in the macro mean over HealthBench and ResearchQA (Tables 2, 23, and 25). With adaptive Fisher selection, it also keeps criterion scores close to Vanilla GRPO with full judging while using fewer judge requests (Sec. 4.4).

The reviewer's suggestion motivated an additional comparison with Qwen3.5-9B on Medical, added in App. E.1 (Table 24). RRT gains 1.3 criterion points over Vanilla GRPO.

ConditionMedical
Base policy65.6
Vanilla GRPO71.9
RRT73.2

Thank the authors for your comprehensive response and my concerns are well addressed. I maintain my score of weak acceptance.

Thank you for your kind response. We sincerely appreciate your constructive feedback.

Reviewer_giqQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Reward Aggregation and judging costs in rubric-based reinforcement learning for LLMs. The authors propose Rubric Response Theory (RRT), which uses Item Response Theory to model the difficulty and discrimination of different rubric criteria and estimates the latent quality of responses from multiple criterion verdicts. In addition, the authors introduce a Response Parameter Network (RPN) and use Fisher Information to adaptively select criteria for evaluation, thereby reducing LLM judge costs. Experimental results show that the proposed method can improve rubric satisfaction rates in certain settings.

Critical strengths
  1. The paper addresses an important research problem.

  2. The paper is well written and easy to follow.

  3. The paper develops a reward estimation model based on Item Response Theory and provides relatively comprehensive mathematical derivations and theoretical analyses.

  4. The experiments are extensive, covering multiple datasets and policy models. The paper also compares RRT with existing reward aggregation methods, such as POW3R and DIVA, and includes analyses of parameter estimation, reward stability, and criterion selection.

  5. Fisher-based Criterion Selection achieves promising results. By reducing judging requests by approximately half, it maintains performance close to GRPO with full rubric judging. The paper also provides an analysis of actual training-time overhead.

Critical weaknesses
  1. The main technical components, including Two-parameter Item Response Theory, MAP Estimation, Online EM, and Fisher Information-based Selection, are well-established methods. This paper mainly combines and applies these techniques to rubric-based RL. The authors should better clarify the technical novelty compared with traditional IRT and existing adaptive reward aggregation approaches. Furthermore, Online RPN provides only limited improvements over Frozen RPN, raising questions about its necessity.

  2. RRT assumes that different rubric criteria are monotonic indicators of a single latent quality variable and are conditionally independent given that variable. However, real-world rubrics may involve different or even conflicting dimensions, such as Correctness, Safety, Completeness, and Conciseness. It remains unclear whether these dimensions can be adequately represented by a single latent quality variable. Although the authors analyze criterion dependence, the applicability of this core assumption is not sufficiently validated.

  3. Although RRT improves over Vanilla GRPO by 1.7 percentage points on Qwen3.5-4B, its performance slightly decreases on Qwen3.5-2B and improves by only 0.1 percentage points on Llama-3.1-8B. Therefore, the experimental results do not sufficiently demonstrate consistent improvements across different model scales and architectures.

  4. The paper explicitly states that each experimental condition involves only one training run. Given the sensitivity of reinforcement learning to random seeds, sampling outcomes, and reward estimation noise, a single run is insufficient to establish the stability of relatively small performance improvements. The authors should conduct multiple independent training runs and report the mean and variance.

  5. The code is not publicly available, which limits the reproducibility of the work.

  6. The paper shows that Static Fisher and Adaptive Fisher selection can skip approximately 19.3% and 21.0% of criteria, respectively, under the same reward fidelity requirement. The difference is relatively small, making it unclear whether the additional complexity of adaptive selection is justified. The authors should further analyze its benefits across different training stages and computational budgets.

  7. The comparison in the abstract is somewhat misleading. The proposed method is compared against GRPO with conventional rubric-based rewards, rather than GRPO itself.

Questions for authors

see the weaknesses

5 comments
AuthorsOpenReview ↗

Novelty and RPN (Part 1 of 3)

Thank you for your feedback and positive assessment of the mathematical analysis, the range of experiments, and Fisher based criterion selection.

Changes to the manuscript are shown in blue. Section, appendix, and table numbers below refer to the revised manuscript unless marked as the submission.

W1: Technical novelty and the online RPN

Sec. 2 now clarifies the comparisons with traditional IRT, adaptive testing, and adaptive reward aggregation.

To be more specific, traditional IRT has been used to assess examinees and to select training data by difficulty from neural model responses. RRT contributes a reward formulation for policy training. Its latent variable describes each rollout's quality under its prompt's rubric, and its MAP value becomes the GRPO reward. With a logistic response function, this reward orders rollouts exactly as a weighted sum of verdicts does, so RRT uses the Gaussian CDF, under which criterion difficulty can change the order (Theorems 4 and 5). Rubrics vary across prompts, so the RPN predicts difficulty and discrimination from prompt and criterion text for unseen rubrics. The policy changes the verdict distribution during training, so online EM recalibrates these predictions from current rollouts. Furthermore, the paper analyzes how MAP reward differences become GRPO advantages (App. A.5). Even criteria with identical verdicts across a rollout group can change reward gaps and affect policy updates (Sec. 5).

In addition, RRT contributes criterion selection for a rollout group using the same response model that defines its rewards. It ranks unjudged criteria by total Fisher information across the group and updates inferred qualities after each judged criterion (Sec. 3.3). This adapts Fisher selection from ability assessment to approximating the group's GRPO advantages with fewer judge requests.

The reviewer asks what RRT adds beyond existing adaptive reward aggregation. POW3R rescales human weights using contrast across rollouts, while DIVA uses criterion score variance. Both adapt criterion weights shared across rollouts and aggregate verdicts additively. By comparison, RRT uses MAP aggregation, so a verdict's effect on the reward depends on the other verdicts. The analysis shows that, under RRT's response model, no fixed criterion weights attain the local information bound across quality levels when criterion parameters differ (Theorem 2).

To examine the contribution of reward aggregation, Table 2 now includes Weighted sum + frozen RPN. This policy uses the reward Ri=∑jajGijR_i=\sum_j a_jG_{ij} with aja_j from the frozen RPN, and its other settings match RRT + frozen RPN. The table below reports criterion scores.

ConditionRewardCriterion parametersMedicalScienceRaR ScienceRubricBenchMacro mean
Vanilla GRPORubric pointsNone67.778.370.677.073.4
RRT + batch pass rateMAPaj=1a_j=1, pass rate bjb_j68.078.370.877.373.6
Weighted sum + frozen RPN∑jajGij\sum_j a_jG_{ij}RPN aja_j68.478.670.676.873.6
RRT + frozen RPNMAPRPN aja_j and bjb_j70.179.570.777.974.6
RRT + online RPNMAPRPN aja_j and bjb_j, updated online70.979.870.978.775.1

With the same frozen RPN, RRT exceeds the weighted sum on every dataset, with a gain of 1.0 macro criterion points. This comparison supports RRT's MAP aggregation using both difficulty and discrimination beyond using predicted discrimination as an additive weight.

Beyond the gains from MAP aggregation with a frozen RPN, the reviewer asks what online updates add. These updates improve prediction of criterion pass rates as the policy changes (App. C.6). They reuse existing verdicts, so they need no additional judge requests and add 0.123% to the time of a policy step (Table 34). This calibration benefit is accompanied by the highest mean policy score across three training seeds (W4 below).

These benefits of online calibration complement the frozen variant's reduction in judging costs. With Fisher selection, the frozen RPN keeps policy scores close to Vanilla GRPO with full judging while using fewer judge requests (Sec. 4.4). The average savings persist after accounting for initialization. Both RPN variants are part of RRT, and both exceed Vanilla GRPO in mean macro criterion score over three training seeds (W4 below).

AuthorsOpenReview ↗

Quality and models (Part 2 of 3)

W2: Single latent quality and criterion dependence

The reviewer raises a useful concern about whether one quality variable can represent different or conflicting criteria. In RRT, ziz_i represents quality specific to a prompt and its rubric. Sec. 3.1 assumes that each criterion's pass probability increases with this quality, and the abstract and Introduction state this scope. A rollout can still satisfy some criteria and fail others.

GRPO takes one scalar reward per rollout, so any rubric reward places all criteria on one scale. A weighted sum of verdicts orders rollouts exactly as a logistic model with one quality variable does (Theorem 4), so RRT keeps the single scale of rubric points and makes it explicit and testable.

Correctness, safety, completeness, and conciseness can contribute to this shared target. As a case study, consider the Medical prompt with source index 70651. It asks for the two most widely accepted treatments for insomnia and when to prefer each. Its rubric includes criteria of all four kinds.

KindCriterion
CorrectnessThe response states that major clinical guidelines designate CBT-I as the first-line (gold-standard) treatment for chronic insomnia.
SafetyThe answer includes a brief caution that treatment decisions should be individualized and guided by a healthcare professional, especially given medication risks.
CompletenessThe answer describes core CBT-I components such as sleep restriction, stimulus control, cognitive restructuring, and relaxation training.
ConcisenessThe response provides sufficient detail to capture key distinctions and indications without unnecessary length or excessive technical digression.

In this case study, each criterion describes a feature of the same target answer. The conciseness criterion requires sufficient detail without unnecessary length, so it is defined relative to the content that the other criteria require. In addition, the rubric sources avoid conflicting criteria by design. RubricHub, the source of Medical and Science, constrains rubric generation so that each criterion does not strongly depend on, contradict, or overlap with others (Li et al., 2026), and RubricBench checks each rubric for internal conflicts (Zhou et al., 2026). For rubrics with explicit tradeoffs, Sec. 5 discusses modeling multiple quality targets.

Furthermore, under RRT's model, satisfying an additional criterion, including a safety or conciseness criterion, strictly raises the MAP reward when the other verdicts and criterion parameters are fixed (Corollary 1).

To assess the shared quality assumption empirically, App. D.6 examines dependence between criterion verdicts (Table 22). Fitted quality explains most pairwise dependence, and mean residual correlation is below the parametric bootstrap reference under local independence on every dataset. The remaining positive dependence is concentrated among criteria with the most similar text, while criteria with the least similar text stay within 0.7 points of the bootstrap null rate. In addition, quality inferred from the other criteria predicts an omitted verdict above chance within each criterion (Table 1). As an external check, App. C.8 now compares rubric points and RRT with human preferences on all 150 RubricBench test pairs (Table 12). Using the same verdicts, RRT gains 2.0 points in preference accuracy. Together, the dependence diagnostics, prediction of omitted verdicts, and agreement with human preferences support a shared quality variable for the evaluated rubrics.

W3: Performance across policy models

Thank you for raising the question of whether RRT improves consistently across model sizes and architectures. RRT improves macro criterion score on Qwen3.5-4B, while its scores on Qwen3.5-2B and Llama-3.1-8B-Instruct remain close to Vanilla GRPO's. Across all three policies, RRT scores higher on RubricBench (Tables 2 and 23) and on HealthBench, which is not used in training (Table 25).

One hypothesis is that larger policies with thinking enabled may benefit more from RRT's distinctions between rollouts. The reviewer's concern motivated a small proof of concept with Qwen3.5-9B on Medical, added in App. E.1. RRT gains 1.3 criterion points over Vanilla GRPO, providing evidence of a gain at another model size.

These comparisons provide preliminary evidence.

ConditionMedical
Base policy65.6
Vanilla GRPO71.9
RRT73.2

In addition, a main focus of the paper is reducing judge requests, a major cost of RL training with rubrics. Adaptive Fisher selection retains similar criterion scores to Vanilla GRPO with full judging while reducing calculated judge requests by 19.9% on average, including initialization (Sec. 4.4).

AuthorsOpenReview ↗

Seeds and Fisher selection (Part 3 of 3)

W4: Training seeds

Multiple reviewers raised the need for results across training seeds. App. E.6 now reports the mean and standard deviation of criterion score over three training seeds on Qwen3.5-4B (Table 27).

ConditionMedicalScienceRaR ScienceRubricBenchMacro meanpp value vs. online RPN
Vanilla GRPO67.6 ±\pm 0.578.2 ±\pm 0.470.4 ±\pm 0.477.0 ±\pm 0.273.3 ±\pm 0.20.020
POW3R70.3 ±\pm 0.678.9 ±\pm 0.270.0 ±\pm 0.477.2 ±\pm 0.274.1 ±\pm 0.20.082
RRT + frozen RPN69.9 ±\pm 0.579.4 ±\pm 0.470.5 ±\pm 0.577.8 ±\pm 0.374.4 ±\pm 0.30.361
RRT + online RPN70.4 ±\pm 0.679.4 ±\pm 0.470.4 ±\pm 0.678.2 ±\pm 0.474.6 ±\pm 0.4N/A

The tt tests use macro score differences between each method and RRT + online RPN for runs with the same training seed.

Sec. 4.3 now reports the mean and standard deviation over three training seeds. The abstract and Introduction now report the mean gain over Vanilla GRPO across these seeds.

W5: Code availability

The hyperlink at the end of the abstract could be missed, so the revised manuscript now displays the full code URL: https://anonymous.4open.science/r/rrt-9DC4.

W6: Adaptive and static Fisher selection

Thank you for raising this. Both selectors use RRT's criterion information and the same frozen RPN. Adaptive Fisher ranks unjudged criteria by ∑iIj(z^i)\sum_i I_j(\hat z_i) and updates the MAP estimates z^i\hat z_i after each judged criterion. Fixing every z^i\hat z_i at the prior mean of zero gives static Fisher, NIj(0)N I_j(0). Its ranking for a given prompt depends only on criterion parameters, so it stays fixed as the policy changes. At a fixed criterion budget, adaptation adds no learned parameters, RPN training, or judge requests. Its added computation is MAP inference and ranking between judged criteria. At criterion budget 0.50, judging time still falls by 49.1% to 49.6% for 49.0% fewer judge requests (Tables 32 and 35).

Rollouts from base and trained policies mark the start and end of training. On rollouts from trained policies, static Fisher reduces judging more on average than selection by discrimination alone, and adaptive Fisher matches or exceeds static Fisher on every dataset (Table 3). However, on rollouts from base policies, adaptive Fisher matches or exceeds static Fisher on three datasets but trails on RubricBench, so static Fisher has greater mean savings (Table 26). This reversal from base to trained policies supports updating the ranking as the policy changes.

Figure 9 compares both methods across criterion budgets on rollouts from base policies. In policy training with adaptive Fisher, larger criterion budgets recover more of RRT's gains from full judging (Table 4), while selecting fewer criteria shortens policy steps (Table 35).

W7: GRPO comparison in the abstract

Thank you for pointing this out. Both conditions use the same GRPO implementation. The baseline uses the normalized sum of assigned rubric points as its reward, while RRT uses inferred quality. Accordingly, the abstract now states that replacing the baseline reward with RRT rewards within GRPO gives the reported 1.3 point gain.

Reviewer_WmdWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes new method called Rubric Response Theory (RRT), which employs a reward inferred from 2-parameter item response to replace conventional method (sums the points assigned to satisfied criteria) used in rubric-based GRPO. In RRT, each binary criterion verdict GijG_\text{ij} is modelled as Bernoulli (Pij)(P_{ij}) under local independence (Equation 1), maximizer is found by bisection using the log posterior gradient (Theorem 3 in Appendix A.1), Ri=z^iR_i = \hat{z}_i is employed as the GRPO rewards (Appendix B.2), criterion difficulty bjb_j and discrimination aja_j are predicted from prompt and criterion text by a RPN (Section 3.1), RRT calibrates RPN from new rollout verdicts using 1 online EM sweep per-policy step (Section 3.2). The theoritical contributions are (1) fisher information of a rubric criterion (Theorem 1), (2) a local SNR bound showing that the rubric likelihood score is up to a nonzero affine map, so that no fixed vector of rubric points attains the bound at every quality level once criterion parameters differs (Theorem 2), and (3) the result that under the logistic link the reward order depends only on sum of aj×Gija_j \times G_ij and not on bjb_j (Theorem 4), whereas difficulty can reverse RRT reward order of 2-fixed verdict vectors (Theorem 5). Empirically, with Qwen3.5-4B and a GPT-5.5 judge on Medical, Science, RaR science, and RubricBench. RRT + online RPN reaches 75.1 macro criterion score versus 73.4 for vanilla GRPO and 74.3 for POW3R (Table 2). Adaptive Fisher selection at budget 0.50 cuts judge requests by 49.0% and step time by 23.5-28.7% while matching full judging vanilla GRPO.

Critical strengths

S1. Reward, calibration, and criteria selection is unified by single principled model

  • 2PL probit models yields the reward (MAP Quality), online EM calibration, and fisher-based criterion ranking.

S2. Clear and correct theoritical characterization

  • Theorem 2 provides well-posed argument that additive rubric points cannot attain the local information bound at every quality level once criterion parameters differ

  • Theorems 4-5 give a concrete, testable reason to prefer probit over logistic link

S3. Practical reduction in judge cost

  • At criterion budget 0.50, adaptive FIsher selection reduces judge request by 49.0% and step time by 23.5-28.7% while matching Vanilla GRPO with full judging (Table 4).
Critical weaknesses

W1. One training run per condition

  • Each condition has one training run (Section 4.1)

  • The reported boostrap intervals are computed over prompt groups with the trained policies fixed, so they do not capture training-seed variance

W2. The source of the gain is unclear

  • RRT with estimated parameters (batch pass rate 73.6, cached pass rate 73.4) is distinguishable from vanilla GRPO (73.4)

  • Table 1 shows identical within-criterion ROC-AUC (65.8) for RRT

  • The improvement appears only when the text-conditioned RPN supplies the parameters

  • There is possibility that the gain may come from the RPN's text prior rather than likelihood aggregation (core of the paper)

Questions for authors

Q1. Could you please report at least 3 training seeds for Vanilla GRPO, POW3R, RRT + frozen RPN, and RRT + online RPN on Qwen3.5-4B, with mean +- std and a test of the paired difference?

Q2. How many judge requests produced the cached verdicts used for the RPN warm start (rollouts per training prompt × criteria)? Is the 49.0% saving still positive after amortizing this cost over a training run?

Q3. Selected checkpoint steps differ notably across conditions (e.g., Medical: 145 frozen vs 115 online, Table 28). How sensitive are the Table 2 differences to evaluating every condition at the same step?

4 comments
AuthorsOpenReview ↗

Seeds and gains (Part 1 of 2)

Thank you for your feedback and positive assessment of the theoretical characterization and the reduction in judge cost.

Changes to the manuscript are shown in blue. Section, appendix, and table numbers below refer to the revised manuscript unless marked as the submission.

W1/Q1: Training seeds

Multiple reviewers raised the need for results across training seeds. App. E.6 now reports the mean and standard deviation of criterion score over three training seeds on Qwen3.5-4B (Table 27).

ConditionMedicalScienceRaR ScienceRubricBenchMacro meanpp value vs. online RPN
Vanilla GRPO67.6 ±\pm 0.578.2 ±\pm 0.470.4 ±\pm 0.477.0 ±\pm 0.273.3 ±\pm 0.20.020
POW3R70.3 ±\pm 0.678.9 ±\pm 0.270.0 ±\pm 0.477.2 ±\pm 0.274.1 ±\pm 0.20.082
RRT + frozen RPN69.9 ±\pm 0.579.4 ±\pm 0.470.5 ±\pm 0.577.8 ±\pm 0.374.4 ±\pm 0.30.361
RRT + online RPN70.4 ±\pm 0.679.4 ±\pm 0.470.4 ±\pm 0.678.2 ±\pm 0.474.6 ±\pm 0.4N/A

The tt tests use macro score differences between each method and RRT + online RPN for runs with the same training seed.

Sec. 4.3 now reports the mean and standard deviation over three training seeds. The abstract and Introduction now report the gain over Vanilla GRPO as the mean over three training seeds, 1.3 points.

W2: Source of the gain

As the reviewer notes, all three methods using rollout evidence report 65.8% macro ROC-AUC within each criterion. In pooled prediction, RRT gains 10.1 points over the mean of other verdicts and 5.4 points over the RPN without rollout evidence (Table 1). These comparisons show how criterion parameters and rollout evidence complement each other when ranking verdicts across criteria and rollouts.

The reviewer's thoughtful question led to a clearer account of the source of the gain. To compare how RPN predictions enter the reward, Table 2 now includes Weighted sum + frozen RPN. This policy uses the reward Ri=∑jajGijR_i=\sum_j a_jG_{ij} with aja_j from the frozen RPN, and its other settings match RRT + frozen RPN. The table below reports criterion scores.

ConditionRewardCriterion parametersMedicalScienceRaR ScienceRubricBenchMacro mean
Vanilla GRPORubric pointsNone67.778.370.677.073.4
RRT + batch pass rateMAPaj=1a_j=1, pass rate bjb_j68.078.370.877.373.6
Weighted sum + frozen RPN∑jajGij\sum_j a_jG_{ij}RPN aja_j68.478.670.676.873.6
RRT + frozen RPNMAPRPN aja_j and bjb_j70.179.570.777.974.6
RRT + online RPNMAPRPN aja_j and bjb_j, updated online70.979.870.978.775.1

With the same frozen RPN, RRT exceeds the weighted sum on every dataset, with a gain of 1.0 macro criterion points. Relative to Vanilla GRPO, RRT + batch pass rate and Weighted sum + frozen RPN each gain 0.2 macro points, while RRT + frozen RPN gains 1.2 points. These results support using predicted difficulty and discrimination together in MAP reward inference. Sec. 5 now discusses these findings.

AuthorsOpenReview ↗

Costs and checkpoints (Part 2 of 2)

Q2: Warm start cost and judge savings

The reviewer raises a useful question about the warm start cost. The reported 49.0% saving excludes initialization, which uses eight fully judged rollouts per prompt on up to 2,000 training prompts, or all 847 on RubricBench.

App. G.3 now includes calculated judge requests for initialization and policy training (Table 37). Accordingly, the savings reported in the abstract, Introduction, Sec. 4.4, and Sec. 5 now include the warm start cost.

# Judge requestsMedicalScienceRaR ScienceRubricBenchMacro mean
RPN initialization only481,032428,768119,63239,176267,152
Vanilla GRPO, full judging, total2,404,6722,148,624600,72078,3521,308,092
RRT with adaptive Fisher (0.50), training only1,222,7361,094,880328,86442,432672,228
RRT with adaptive Fisher (0.50), including initialization1,703,7681,523,648448,49681,608939,380
Reduction vs. full Vanilla GRPO, training only49.2%49.0%45.3%45.8%47.3%
Reduction vs. full Vanilla GRPO, including initialization29.1%29.1%25.3%−4.2%19.9%

The saving stays positive after amortization on Medical, Science, and RaR Science, at 25.3% to 29.1% relative to Vanilla GRPO with full judging. On these datasets, the requests saved in one epoch already cover initialization. Across all four datasets, the saving averages 19.9%. However, on RubricBench, initialization judges all 847 training prompts, which equals one epoch of full judging. Consequently, two epochs at this budget do not recover it.

Furthermore, App. G.3 reports the timing comparison in hours (Table 38).

DatasetVanilla GRPO, full judging, training runRRT (0.50), initializationRRT (0.50), training runRRT (0.50), totalReduction vs. Vanilla GRPO
Medical205.314.8157.8172.615.9%
Science214.512.0176.1188.112.3%
Macro mean209.913.4167.0180.414.1%

For the tested Qwen3.5-4B setup, judging is the longest measured stage of policy training. Its median duration is 64% of the median Vanilla GRPO step time (App. G.2). At criterion budget 0.50, RRT reduces this cost while keeping the macro criterion score within 0.1 points of Vanilla GRPO with full judging (Table 4).

Q3: Checkpoint selection

Thank you for raising the effect of checkpoint selection. Each method uses its best validation checkpoint within the same training step limit, since performance can peak at different steps. POW3R also selects the best validation checkpoint for each method (Tyagi et al., 2026). Simoni et al. (2026) report performance degradation during GRPO training and evaluate the checkpoint with the highest accuracy reward for a fair comparison. App. F.3 now gives this rationale and both citations. For example, App. F.3 adds a validation curve (Figure 10) to show why the best validation checkpoint is used. Its validation criterion score drops by 33.5 points from the selected step 100 to step 140.

The selected steps of RRT + online RPN and Vanilla GRPO in Table 2 are close (Table 31, Table 28 in the submission). They differ by at most five policy steps on Medical, RaR Science, and RubricBench, and RRT + online RPN is selected 20 steps earlier on Science. App. F.3 now states this. On RubricBench, both methods use step 50:

ConditionPolicy stepCriterion score (%)
Vanilla GRPO5077.0
RRT + online RPN5078.7

RRT + online RPN gains 1.7 criterion points at the same training step.

Simoni, Marco, et al. "GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control." Transactions of the Association for Computational Linguistics 14 (2026): 2258-2280.

Reviewer_WmdWOpenReview ↗

Official Comment by Reviewer WmdW

Thank you for the clarification and the additional results. They address my questions, I will maintain my score

Thank you for your response and continued support. We appreciate your thoughtful feedback, which helped us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.