- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Rubric Rewards from Item Response Theory (RRT) that treats multiple rubric criteria as noisy measurements of a shared latent response quality, rather than directly aggregating their binary verdicts through weighted summation. It introduces a Response Parameter Network (RPN) to estimate criterion difficulty and discrimination, and infers latent quality scores from the observed verdict patterns to serve as rewards for GRPO training. Additionally, the framework uses Fisher information to adaptively select informative criteria, achieving competitive policy performance while substantially reducing judging costs. Experiments on RLHF demonstrate the effectiveness of RRT.
Critical strengths
-
Simply summing or averaging multiple criterion rewards in rubric reward + GRPO training settings can overlook differences in criterion difficulty and informativeness. The paper offers an interesting perspective on how these signals could be aggregated more meaningfully.
-
The IRT-based formulation is intuitive and well motivated. Treating rubric criteria as noisy measurements of an underlying response quality provides a principled alternative to conventional weighted aggregation. Modeling criterion difficulty and discrimination also allows the framework to make better use of different verdict patterns.
Critical weaknesses
-
The assumption of a shared latent quality may be too strong for general rubric-based RL. RRT assumes that all criteria are monotonic indicators of a single underlying response quality. However, real-world rubrics often involve different objectives, which may have genuine trade-offs.
-
It is unclear whether the inferred latent quality better reflects actual response quality or human preferences. The paper demonstrates that RRT can distinguish different verdict patterns and reduce reward ties, but greater reward discrimination does not necessarily mean better reward quality. In particular, criterion difficulty and statistical informativeness are not necessarily equivalent to criterion importance. Some independent evaluation against human preferences would make this claim more convincing. For example, I suggest that the authors could conduct additional experiments on human preference datasets (like helpsteer3) to examine whether the inferred latent-quality scores better align with human preference judgments than conventional additive reward aggregation.
-
I am also somewhat concerned about the additional complexity introduced by the Response Parameter Network (RPN). Although the authors report relatively low computational overhead for online RPN updates, the framework still requires a separate warm-start procedure and an additional parameterized model that is continuously updated alongside the policy. Since the RPN directly determines the inferred reward, its initialization quality and calibration stability could potentially affect the overall RL training dynamics. Moreover, the improvement from online RPN updates over a frozen RPN appears relatively modest (Table 2). It would be helpful to further investigate sensitivity to RPN initialization, update frequency, and training hyperparameters, as well as whether the additional online calibration consistently improves policy performance across different settings.
-
The downstream RL improvements are relatively modest and not fully consistent across models. (That's fine, but experiments on LLMs with more parameters would be better, like 32B.)
Questions for authors
Please refer to the weaknesses.
4 comments
RPN and policy models (Part 2 of 2)
Continued from Part 1.
W3: RPN cost, sensitivity, and online calibration (continued)
App. E.6 now reports the mean and standard deviation of criterion score over three training seeds on Qwen3.5-4B (Table 27).
| Condition | Medical | Science | RaR Science | RubricBench | Macro mean | value vs. online RPN |
|---|---|---|---|---|---|---|
| Vanilla GRPO | 67.6 0.5 | 78.2 0.4 | 70.4 0.4 | 77.0 0.2 | 73.3 0.2 | 0.020 |
| POW3R | 70.3 0.6 | 78.9 0.2 | 70.0 0.4 | 77.2 0.2 | 74.1 0.2 | 0.082 |
| RRT + frozen RPN | 69.9 0.5 | 79.4 0.4 | 70.5 0.5 | 77.8 0.3 | 74.4 0.3 | 0.361 |
| RRT + online RPN | 70.4 0.6 | 79.4 0.4 | 70.4 0.6 | 78.2 0.4 | 74.6 0.4 | N/A |
The tests use macro score differences between each method and RRT + online RPN for runs with the same training seed.
With the RPN warm start held fixed, RRT + frozen RPN gains 1.1 macro criterion points over Vanilla GRPO across three training seeds, while RRT + online RPN, with one RPN update per policy step, gains 1.3 points over Vanilla GRPO (). Relative to the frozen RPN, the online RPN gains on Medical and RubricBench. Sec. 4.3 now reports the mean and standard deviation over three training seeds.
In addition, the paper's ablations show the RPN's contribution to policy scores. With the same reward, RPN parameters raise the macro criterion score over parameters from empirical pass rates (Table 2). RPN verdict prediction stays between 85.6% and 88.7% macro ROC-AUC across all tested RPN configurations and rollout policies (Table 5).
App. G.3 now reports calculated judge requests for the RPN warm start and policy training (Table 37). Accordingly, the abstract, Introduction, Sec. 4.4, and Sec. 5 now report savings including initialization.
| # Judge requests | Medical | Science | RaR Science | RubricBench | Macro mean |
|---|---|---|---|---|---|
| RPN initialization only | 481,032 | 428,768 | 119,632 | 39,176 | 267,152 |
| Vanilla GRPO, full judging, total | 2,404,672 | 2,148,624 | 600,720 | 78,352 | 1,308,092 |
| RRT with adaptive Fisher (0.50), training only | 1,222,736 | 1,094,880 | 328,864 | 42,432 | 672,228 |
| RRT with adaptive Fisher (0.50), including initialization | 1,703,768 | 1,523,648 | 448,496 | 81,608 | 939,380 |
| Reduction vs. full Vanilla GRPO, training only | 49.2% | 49.0% | 45.3% | 45.8% | 47.3% |
| Reduction vs. full Vanilla GRPO, including initialization | 29.1% | 29.1% | 25.3% | −4.2% | 19.9% |
Including initialization, judge requests fall by 19.9% on average. The requests saved in one epoch cover initialization on Medical, Science, and RaR Science.
Furthermore, App. G.3 reports the timing comparison in hours (Table 38).
| Dataset | Vanilla GRPO, full judging, training run | RRT (0.50), initialization | RRT (0.50), training run | RRT (0.50), total | Reduction vs. Vanilla GRPO |
|---|---|---|---|---|---|
| Medical | 205.3 | 14.8 | 157.8 | 172.6 | 15.9% |
| Science | 214.5 | 12.0 | 176.1 | 188.1 | 12.3% |
| Macro mean | 209.9 | 13.4 | 167.0 | 180.4 | 14.1% |
Including initialization, RRT reduces total elapsed time by 14.1% on average across Medical and Science. At criterion budget 0.50, adaptive Fisher selection with the frozen RPN keeps the macro criterion score within 0.1 points of Vanilla GRPO with full judging, and larger budgets recover more of RRT's gain (Table 4).
Overall, the RPN brings the policy gain and the judge savings, and online updates improve correlation with criterion pass rates and give the highest mean score without additional judge requests.
W4: Performance across policy models
Thank you for raising the question of whether RRT improves consistently across models and for suggesting evaluation with a larger policy. RRT improves macro criterion score on Qwen3.5-4B. While its scores on Qwen3.5-2B and Llama-3.1-8B-Instruct remain close to Vanilla GRPO's, RRT brings other gains. Across all three policies, it scores higher on RubricBench and in the macro mean over HealthBench and ResearchQA (Tables 2, 23, and 25). With adaptive Fisher selection, it also keeps criterion scores close to Vanilla GRPO with full judging while using fewer judge requests (Sec. 4.4).
The reviewer's suggestion motivated an additional comparison with Qwen3.5-9B on Medical, added in App. E.1 (Table 24). RRT gains 1.3 criterion points over Vanilla GRPO.
| Condition | Medical |
|---|---|
| Base policy | 65.6 |
| Vanilla GRPO | 71.9 |
| RRT | 73.2 |
Thank the authors for your comprehensive response and my concerns are well addressed. I maintain my score of weak acceptance.
Thank you for your kind response. We sincerely appreciate your constructive feedback.
Quality and human preferences (Part 1 of 2)
Thank you for your feedback and positive assessment of the motivation and the formulation of RRT. The thoughtful concerns also helped improve the paper.
Changes to the manuscript are shown in blue. Section, appendix, and table numbers below refer to the revised manuscript unless marked as the submission.
W1: Shared latent quality and tradeoffs
The reviewer raises a useful concern about whether one quality variable can represent criteria with different or conflicting objectives. In RRT, zi represents quality specific to a prompt and its rubric. Sec. 3.1 assumes that each criterion's pass probability increases with this quality, and the abstract and Introduction state this scope. A rollout can still satisfy some criteria and fail others.
GRPO takes one scalar reward per rollout, so any rubric reward places all criteria on one scale. A weighted sum of verdicts orders rollouts exactly as a logistic model with one quality variable does (Theorem 4), so RRT keeps the single scale of rubric points and makes it explicit and testable.
Correctness, safety, completeness, and conciseness can contribute to this shared target. As a case study, consider the Medical prompt with source index 70651. It asks for the two most widely accepted treatments for insomnia and when to prefer each. Its rubric includes criteria of all four kinds.
In this case study, each criterion describes a feature of the same target answer. The conciseness criterion requires sufficient detail without unnecessary length, so it is defined relative to the content that the other criteria require. In addition, the rubric sources avoid conflicting criteria by design. RubricHub, the source of Medical and Science, constrains rubric generation so that each criterion does not strongly depend on, contradict, or overlap with others (Li et al., 2026), and RubricBench checks each rubric for internal conflicts (Zhou et al., 2026). For rubrics with explicit tradeoffs, Sec. 5 discusses modeling multiple quality targets.
To assess the shared quality assumption empirically, App. D.6 examines dependence between criterion verdicts (Table 22). Fitted quality explains 72.8% to 85.3% of pairwise mutual information. Furthermore, quality inferred from the other criteria predicts omitted verdicts 15.8 points above chance in macro ROC-AUC within criteria (Table 1). Together, these results support a shared quality variable for the evaluated rubrics.
W2: Criterion importance and human preferences
Thank you for suggesting a comparison with human preferences. Rubric points specify how much each criterion should count, while RRT measures how much its verdict reveals about shared quality (Sec. 1). Sec. 4.2 and App. C.8 now compare rubric points and RRT on all 150 RubricBench test pairs (Table 12). The comparison uses existing human preference labels from RubricBench, which includes response pairs from HelpSteer3, the dataset the reviewer suggested (Zhou et al., 2026).
For this comparison, both rewards use the same released criterion verdicts, with the RPN fitted on training prompts and held fixed for evaluation. Rubric points assign one point per criterion. Human preferences are used only for evaluation. Preference accuracy gives half credit for ties. Values are percentages.
RRT improves human preference accuracy by 2.0 percentage points over rubric points. In the primary run on Qwen3.5-4B, RRT + online RPN also exceeds Vanilla GRPO by 1.7 points in macro normalized points score, which weights criteria by their assigned points and is the reward Vanilla GRPO trains on (Table 2).
W3: RPN cost, sensitivity, and online calibration
The reviewer raises useful questions about the cost of the RPN, its initialization and sensitivity, and what online updates add beyond a frozen RPN. Online updates raise Pearson correlation with criterion pass rates at the next policy step by 1.7 points in the macro mean over Medical and Science, using existing verdicts without additional judge requests (App. C.6, Table 10).
To be continued in Part 2.