- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the heterogeneous aggregration issue in rubric-based reinforcementLearning. It proposes Reinforcement Learning with Verifiable Rubric-based Ranking (RLVR2), which converts raw rubric scores into pairwise comparisons within group and applies BT model to estimate the latent utility. The proposed method improves performance on several benchmarks.
Critical strengths
-
The research topic is meaningful. Scores of different rubric criteria may come from binary score, discrete score or continuous score. These heterogeneous scores may not have same semantic meaning to be compared.
-
The paper conducts extensive experiments on multiple benchmarks and achieves competitive results.
-
Besides aggregration, the paper also investigates systematic correlations between rewards and auxiliary attributes, such as reasoning length and response formatting, and analyzes the corresponding behavioral changes during training.
Critical weaknesses
-
The novelty is not clear compared with existing pairwise rubric reward modeling works, such as [1-3]. Although these papers are not the same as the proposed method exactly, they have covered important part of this paper's motivation. This paper should discuss the advantages and differences compared with these works clearly.
-
The paper lacks some simple while important baselines. First, it should compare with pairwise rubric evaluation. For example, it can ask judge compare two answers for each criterion and then aggregate criterion-level preferences as RL rewards. Second, it should add simple ordinal aggregation as baselines, such as mean rank and pairwise win rate.
-
The BT model may suffer from complete separation, particularly for binary or discrete criteria, leading to non-finite maximum likelihood estimates because the pairwise outcomes are deterministically derived from rubric scores. The paper should clarify how this issue is handled and whether the resulting utilities are sensitive to regularization or optimization choices.
-
The necessity of the proposed attribute adjustment mechanism is not justified sufficiently. The paper argues that introducing additional rubric criteria to control reasoning length and formatting may create new optimization targets and increase the risk of reward hacking. However, these issues could potentially be addressed by explicitly incorporating criteria for reasoning efficiency, conciseness, or formatting appropriateness into the existing rubric framework. Conducting an experiment between these two approaches would be healpful.
[1] Jia R, Yang Y, Wang W, et al. Open rubric system: Scaling reinforcement learning with pairwise adaptive rubric[J]. arXiv preprint arXiv:2602.14069, 2026.
[2] Niu W, He W, Xie Z, et al. From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning[J]. arXiv preprint arXiv:2601.23058, 2026.
[3] Juneja G, Phal S M, She J, et al. Tournament Style RL: Stabilizing Policy Optimization on Non Verifiable Problems[C]//Forty-third International Conference on Machine Learning.
Questions for authors
refer to weakness
6 comments
❓Q4. Is attribute adjustment necessary, or could length and formatting simply be added as ordinary rubric criteria?
💡Thank you for raising this important distinction. We chose reasoning length and formatting as representative attributes because they are two observable properties that may influence quality judgments without necessarily reflecting substantive task quality. They also allow us to demonstrate that the proposed adjustment can address different types of response behavior. The framework can be extended to other measurable attributes according to the needs of a particular task; these two examples are not intended to be an exhaustive or optimized attribute set. Our existing reasoning-efficiency and formatting-control experiments demonstrate this extensibility while preserving the capability-oriented rubric objective.
Adding these attributes as ordinary reward criteria is possible, but it changes them into direct optimization targets. A policy may then obtain higher reward by changing response length or formatting even when the underlying task quality does not improve, creating an additional shortcut for optimization. Our attribute-adjustment mechanism provides an alternative for post-training requirements that concern such observable response properties: the property can be introduced as a covariate, and its association with the observed ranking can be adjusted without adding a new reward dimension or redesigning the underlying rubric. This allows the training process to accommodate additional behavioral requirements while preserving the original quality objective. We will clarify this objective-preserving role in the revised manuscript.
Comments, Suggestions And Typos: We sincerely thank Reviewer pDgi again for the helpful feedback, which has greatly helped us improve the paper. In the revision, we will make the following changes:
- Clarify the distinction from relative-reward methods in Section 5 related work (Q1).
- Add additional baseline into Appendix.
- Expand the discussion of objective-preserving attribute adjustment in Section 3.2.
We are very happy to answer any further questions. We sincerely hope that these improvements will address the concerns and that the reviewer may kindly reconsider the score and support the paper’s presentation at the conference.
Thank you for the detailed rebuttal and additional experiments. While the revisions address some concerns, several key issues remain unresolved, particularly the sensitivity of BT utility estimation to regularization, the lack of a direct rubric-aware pairwise comparison baseline, and the absence of a comparison between attribute adjustment and explicit rubric criteria.
I appreciate the authors' efforts but will maintain my original rating.
We sincerely thank Reviewerfor the thoughtful feedback devoted to reviewing our work. We address each point below and will incorporate the corresponding clarifications and improvements into the manuscript and supplementary material.
❓Q1. How does RLVR$^{2}$ differ from recent pairwise-rubric and relative-reward methods?
💡 Thank you for pointing out these closely related works. They all demonstrate the value of relative signals, but they obtain and use those signals through different reward-construction mechanisms. Open Rubric System dynamically instantiates rubrics and uses an LLM judge to produce criterion-wise pairwise preferences. From Absolute to Relative trains a Ranking Reward Model to produce group-level rankings, introducing an additional learned ranking component. Tournament Style RL compares policy responses against an externally generated set of anchor responses and uses the resulting win rates as rewards.
RLVR$^{2}$ instead starts from the existing criterion-level outputs of verifiable rubric verifiers. For each query, it treats the sampled rollout group as a self-contained comparison set, deterministically converts each criterion's scores into within-group win/tie/loss outcomes, and fits a criterion-specific BT utility before fusing those utilities. RLVR$^{2}$ therefore requires no additional learned ranking model, dynamically generated rubric, or external anchor set. Its contribution is a verifiable aggregation layer for heterogeneous rubric rewards: it preserves within-criterion ordinal information while converting multiple verifier signals into a single scalar reward for policy optimization. As shown in the table below, the performance of the proposed method remains robust even when a different language model is employed as the judge.
❓Q2. Does RLVR$^{2}$ outperform simple ordinal aggregation and direct pairwise rubric evaluation?
💡Thank you for raising this question. As described in our response to Reviewer ZvvJ’s Q1, we additionally include a direct LLM-ranking baseline in the aggregation comparison. This baseline asks an LLM to rank rollout responses without access to an explicit rubric, criterion definitions, or criterion-level verifier outputs. It performs worse than the rubric-informed methods, potentially because its judging standard is underspecified and may rely more heavily on superficial or exploitable attributes, such as response length and formatting. The other baseline methods also achieve lower average performance than RLVR$^{2}$, suggesting that our method is more effective for this complex task.
❓Q3. How does the BT implementation handle complete separation, and how sensitive are the results to regularization and optimization choices?
💡 Thank you for raising this question. We clarify that the BT output is an intermediate per-criterion, per-rollout signal rather than the final policy reward. Each criterion is fitted separately from its within-group comparison outcomes, with ties represented as half wins. The resulting criterion-level signals are aligned by rollout, summed across the active dimensions, and min--max normalized within the group before being written back as the GRPO reward. Thus, the final reward is bounded in [0,1] under the normal fusion path, and policy optimization depends on the resulting group-relative ordering and advantages rather than on the absolute scale of an individual BT utility. We also shared core code of our methods for you to refer https://anonymous.4open.science/r/rlvrr-B001/.