acceptodds
Under review as a conference paper at ICLR 2027

RGC: Global Reward Control for Rubric-Based LLM Post-Training

Abstract

Reinforcement learning (RL) post-training can improve factual accuracy, safety, tone, and other dimensions of model responses. The dimensions relevant to scoring a response vary across prompts, yet developers need global control over their contributions to training. We introduce Rubric-Grounded Coordinates (RGC), a framework for controlling behavior trade-offs through global reward weights. Developers define the dimensions and their global reward weights; prompt-specific rubrics score responses on the applicable dimensions. Our analysis shows how prompt applicability and variation among response scores alter the influence of these weights in group-relative policy optimization (GRPO). RGC uses a small set of training runs to calibrate how global reward weights affect scores across dimensions and guide Pareto search over these weights. We then train new models with the searched weights, achieving distinct behavior trade-offs and improvements on independent benchmarks. RGC achieves the best RubricBench accuracy among the evaluated judges, outperforming OpenRubric by 14.04 percentage points with the same backbone. On Qwen3-8B, our trained models achieve leading results on a broad set of representative public benchmarks among the compared training configurations; one of our trained models improves EQ-Bench3 by 144.8 Elo over training with prompt-specific rubric rewards. https://anonymous.4open.science/r/RGC-reward-system-86B2

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Reinforcement Learning, Rubric-based Reward Modeling, Multi-objective Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_njb5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Rubric-Grounded Coordinates (RGC), a framework that lets developers control behavioral trade-offs in rubric-based RL post-training through global reward weights, even when the dimensions applicable to each prompt vary. It defines 12 shared dimensions, scores each prompt only on its applicable ones using prompt-specific rubrics, and renormalizes the global weights over those dimensions. The paper analyzes how GRPO normalization changes the effective influence of these weights. It then fits a model from 14 calibration runs on Qwen3-8B that predicts how weights affect each dimension’s score, uses it to select three weight configurations, and trains new models with them. As a judge, RGC achieves the best RubricBench accuracy among compared systems. As a training reward, the three models show different per-dimension profiles on a held-out benchmark and modest gains over baselines on public benchmarks and EQ-Bench3.

Critical strengths

The paper shows that per-prompt standardization makes the actual training effect of a weight differ from its nominal value, and constructs examples where all-positive weights decrease every objective. This matters because multi-reward GRPO is widely used: the analysis shows that a larger weight does not guarantee more emphasis on that dimension in training.

Critical weaknesses
  1. ARL-RR already groups prompt-specific rubric criteria into fixed shared dimensions, because rubric criteria vary in number and meaning across prompts. RGC neither cites nor compares with it. This matters because the shared-dimension idea is one of the paper’s two core ideas, so the remaining contribution narrows to the renormalized weighting scheme and the search procedure.
  2. The paper’s main claim is that its search procedure finds weights that steer the trained model toward a desired trade-off. However, it does not report whether the scores predicted for the three searched configurations match the scores the trained models actually obtain. It also does not compare against a simple alternative, such as training with randomly chosen weights. Without this evidence, the paper’s central claim of controllable trade-offs is not fully supported.
Questions for authors

NA

Reviewer_vErMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tries to reconcile two trends in RL for LLMs:

  1. the use of question specific rubrics for fine grained scoring, and
  2. the aim of optimizing multiple 'global' rewards (like coherence or safety) that may trade off against each other The core problem is that the high-level rewards in the second paradigm are only selectively applicable to a diverse collection of prompts, which makes balancing between different goals non-obvious. The idea in this paper is to first define a set a broadly applicable high-level goals (correctness, tone, rigor, usefulness, etc) and then produce question specific rubrics by i. identifying which of the high-level goals are applicable and then ii. producing question-specific rubrics for the applicable goals. Then, tradeoffs among the high-level goals become coherent since each rubric component now belongs to one. They do some further analysis on how GRPO's normalization interacts with the balancing objective and propose a reasonable method for GRPO aware balancing based on this.
Critical strengths
  1. the problem is well-motivated
  2. the exposition is generally clear
  3. the theory is both clear and useful
  4. the experiments are adequate and the results suggest the method is useful
Critical weaknesses

The main weakness that I see is that the difficulties addressed by the analysis and resulting methodology are downstream of a decision to combine the top-level rewards linearly, which seems to ignore prior work suggesting this is already suboptimal. In particular, I'd like to see what happens if the combination rule from https://arxiv.org/abs/2402.00742 was just plugged into the structured rubric strategy that constitutes the first half of the paper. It seems like this would just sidestep a good bit of the difficulty that's induced by GRPO cancelling linear reweighting.

Here's some fleshing out of this from an LLM:

**1. Linear Aggregation of Raw Coordinate Scores**
Both within each coordinate and across coordinates, RGC aggregates rewards via a purely linear weighted sum of raw scores (Section 3.1–3.2, Eqs. 1–2):
$$R_w(x, y) = \sum_{k \in A(x)} \omega_k(x) s_k(x, y), \qquad \text{where } \omega_k(x) = \frac{w_k}{\sum_{j \in A(x)} w_j}.$$
There is no non-linear utility transformation before summing. As noted in Appendix A (p. 13), because this is a linear sum, even "Guardrail" coordinates such as Safety ($D_6$) and Capability Honesty

(D12D_{12}) can be traded off against other dimensions ("including them in a weighted reward does not impose a hard constraint").

**2. Connection to Non-Linear Reward Transformation and Aggregation (Wang et al., ICML 2024)**
Prior work on multi-objective RLHF (*Transforming and Combining Rewards for Aligning Large Language Models*, Wang et al., 2024) shows that summing raw rewards allows strong performance on one property to compensate for failure on another, and instead derives a log-sigmoid-centered (LSC) transformation:
$$U(x, y) = \sum_{k} \log \sigma\!\big(\alpha_k (s_k(x, y) - r_k^{\text{ref}}(x))\big),$$
where $r_k^{\text{ref}}(x)$ is a reference threshold for "good" performance on dimension $k$. Adopting this non-linear combination rule would substantially affect RGC's setup, theoretical analysis, and empirical weight search in four ways:

1. **RGC's scope isolation directly satisfies the conditional independence condition for Logical-AND aggregation:**
   Wang et al. (2024) show that summing $\log \sigma(s_k - r_k^{\text{ref}})$ corresponds to aligning toward the *logical AND* of the response being "good" across all applicable dimensions ($\bigwedge_{k \in A(x)} G_k = 1$), provided the dimension judgments $G_k$ are conditionally independent given $(x, y)$. RGC explicitly engineers this conditional independence via strict "Owns / Does not own" scope isolation (Section 3.1, achieving a within-prompt cross-coordinate correlation of $0.096$), and even defines held-out evaluation pass rates via a binary threshold $G_k = \mathbb{I}[s_k(x,y) \ge 0.8]$ (Section 4.2). Thus, RGC's target objective on each prompt is a logical conjunction over the active coordinates $A(x)$, for which summing log-sigmoid utilities is better suited than summing raw scores.

2. **Explaining the degenerate sparsity of the searched Frontier weights (Table 15):**
   In Table 15, RGC's Pareto search assigns $60\%\text{–}74\%$ of the total weight to just Safety ($D_6$) and Capability Honesty ($D_{12}$)—for example, `Frontier-Delivery` sets $w_6 = 0.333$ and $w_{12} = 0.410$ while assigning $< 0.001$ to Reasoning ($D_4$) and Artifact Usefulness ($D_5$). Under a linear sum, a response that fails Safety or Honesty ($s_6 = 0$) can still achieve a high aggregate reward if it scores well on several other active coordinates; consequently, the linear weight search must assign huge weights to $D_6$ and $D_{12}$ to prevent guardrail violations when those coordinates are active. Under the LSC combination rule, any score below the reference threshold $r_k^{\text{ref}}$ incurs a steep penalty while scores above $r_k^{\text{ref}}$ saturate with near-zero

marginal utility (∂U/∂sk≈0\partial U / \partial s_k \approx 0). Because the Base model already passes D6D_6 (88.3%88.3\%) and D12D_{12} (85.3%85.3\%) on most prompts (Table 2), log-sigmoid aggregation would enforce guardrails automatically without starving the remaining coordinates of optimization weight.

3. **Changes to the Gibbs-policy and cross-coordinate sensitivity analysis (Section 3.3, Eq. 4 & Appendix C.6):**
   In RGC, a large part of the cross-coordinate coupling in Eq. 4 and the second-order curvature in Eq. 17 arises from renormalizing weights by the active weight sum $W_A(x) = \sum_{\ell \in A(x)} w_\ell$:
   $$\frac{\partial m_j}{\partial w_k} = \frac{\text{Cov}_{\pi_w^*}(s_j, s_k - R_w)}{\tau W_A(x)}.$$
   If trade-offs are instead controlled via the reference thresholds $r = (r_1, \dots, r_K)$ in $U_r(x, y) = \sum_{k \in A(x)} \log \sigma\!\big(\alpha(s_k(x, y) - r_k)\big)$:
   - *No active-set weight renormalization is needed:* Because $\log \prod_{k \in A(x)} p(G_k = 1 \mid x, y) = \sum_{k \in A(x)} \log p(G_k = 1 \mid x, y)$, each applicable coordinate contributes additively on the log-probability scale without dividing by $W_A(x)$, eliminating the $1/W_A(x)$ normalization coupling across prompts with different active sets.
   - *Failure-gated cross-coordinate derivatives:* Differentiating the Gibbs expected score $m_j(r, x) = \mathbb{E}_{\pi_r^*}[s_j]$ with respect to $r_k$ yields 

∂mj∂rk∝Covπr∗ ⁣(sj, 1−σ ⁣(α(sk−rk))).\frac{\partial m_j}{\partial r_k} \propto \text{Cov}_{\pi_r^*}\!\Big(s_j,\, 1 - \sigma\!\big(\alpha(s_k - r_k)\big)\Big). Unlike linear weighting—which continues trading off coordinate jj to push sks_k from 0.90→1.000.90 \to 1.00 even after coordinate kk is satisfied—the sensitivity under LSC is gated by 1−σ(α(sk−rk))≈P(Gk=0∣x,y)1 - \sigma(\alpha(s_k - r_k)) \approx \mathbb{P}(G_k = 0 \mid x, y), so coordinate kk only competes with coordinate jj on responses where coordinate kk is currently below its target threshold.

4. **Changes to the GRPO standardization analysis (Appendix C.1–C.5):**
   - *Mitigating single-coordinate amplification (Appendix C.1, Eq. 6):* RGC shows that when only a single active coordinate $k$ varies within a sampled group, GRPO's division by the group standard deviation $\hat{\sigma}_k$ cancels a small weight $\omega_k(x)$ and amplifies minor above-threshold score variations into full-scale updates. Under $\log \sigma(\alpha(s_k - r_k))$, if all sampled responses in a group already exceed $r_k$, their transformed rewards saturate near $0$, so the group variance satisfies $\hat{\sigma}_u^2 \ll \epsilon^2$ and the stabilizer $\epsilon$ in GRPO's denominator suppresses the update.
   - *What remains unchanged:* For responses below the reference threshold ($s_k < r_k$), $\log \sigma(\alpha(s_k - r_k))$ is approximately linear in $s_k$. Thus, in the unsatisfied regime, RGC's core result on GRPO—that per-prompt variance normalization unequally reweights prompts and can induce simultaneous multi-objective descent (Theorem 1)—continues to hold.
Questions for authors

I'd like to see either the experiment suggested in weaknesses or a clear explanation of why it's not an applicable baseline

Reviewer_zSWAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to solve a problem where rubrics are typically written per prompt, so developers have no single knob to say how much a behavior like tone or factuality should matter across the whole training set. RGC fixes a shared set of evaluation dimensions (the paper instantiates them into 12, grouped into guardrails, task delivery, and human-facing quality) and generates prompt-specific rubric items within each dimension that applies to a prompt. The reward is then a combination of the applicable dimension scores using global weights, renormalized per prompt. The authors analyze how applicability and GRPO normalization change the actual influence of these weights. Based on that, they fit a linear-plus-RBF model of how weights shift each dimension’s score, using fourteen calibration runs, and search over this model for weights that hit a target trade-off before training new models with them. For the results, the reward system scores 73.84% on RubricBench, 14.04 points above OpenRubric with the same backbone. Models trained with the searched weights end up with different strengths across the dimensions on a held-out rubric benchmark and improve slightly over the base model on public benchmarks.

Critical strengths

The paper points out that prompt-specific rubric rewards give developers no global control over behavior trade-offs, and it shows that even with global weights, a weight’s real effect depends on which dimensions apply to each prompt and how much scores vary within a GRPO group.

Critical weaknesses

The thing is, the global weights don’t give the control the paper claims. The searched models don’t separate along the axis they were searched for, like for example the Fontier-Human has the best delivery average of the three and the worst conciseness of any trained model, while Frontier-Balanced has the best human-facing average. The searched weights are also hard to interpret, like for example the “human” model puts most of its weight on safety, reasoning and honesty. The paper never checks whether the predictor’s guesses at the new weights matched what training produced, and hand-set weights do just as well or better (see EQ-Bench3 result). In addition, every run gives all twelve dimensions a positive weight and renormalizes per prompt, but there is no uniform-weight baseline. So it is somewhat unclear whether the gains come from the knobs at all or just from the per-dimension rubrics.

Furthermore, the evidence depends on an LLM pipeline that is never validated, i.e. routing accuracy, per-dimension judge scores against humans (human review only covers 364 flagged rubric items), and even the judge comparison in Table 1 is mostly copied numbers from other papers with many entries missing, where RGC uses benchmark-specific decision rules the baselines don’t. Also since Table 2 is scored by the same router and judge used for training, it may mostly show how well each model fits the reward rather than real behavior differences. These make it hard to trust the results coming from the methodology.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.