The main weakness that I see is that the difficulties addressed by the analysis and resulting methodology are downstream of a decision to combine the top-level rewards linearly, which seems to ignore prior work suggesting this is already suboptimal. In particular, I'd like to see what happens if the combination rule from https://arxiv.org/abs/2402.00742 was just plugged into the structured rubric strategy that constitutes the first half of the paper. It seems like this would just sidestep a good bit of the difficulty that's induced by GRPO cancelling linear reweighting.
Here's some fleshing out of this from an LLM:
**1. Linear Aggregation of Raw Coordinate Scores**
Both within each coordinate and across coordinates, RGC aggregates rewards via a purely linear weighted sum of raw scores (Section 3.1–3.2, Eqs. 1–2):
$$R_w(x, y) = \sum_{k \in A(x)} \omega_k(x) s_k(x, y), \qquad \text{where } \omega_k(x) = \frac{w_k}{\sum_{j \in A(x)} w_j}.$$
There is no non-linear utility transformation before summing. As noted in Appendix A (p. 13), because this is a linear sum, even "Guardrail" coordinates such as Safety ($D_6$) and Capability Honesty
(D12) can be traded off against other dimensions ("including them in a weighted reward does not impose a hard constraint").
**2. Connection to Non-Linear Reward Transformation and Aggregation (Wang et al., ICML 2024)**
Prior work on multi-objective RLHF (*Transforming and Combining Rewards for Aligning Large Language Models*, Wang et al., 2024) shows that summing raw rewards allows strong performance on one property to compensate for failure on another, and instead derives a log-sigmoid-centered (LSC) transformation:
$$U(x, y) = \sum_{k} \log \sigma\!\big(\alpha_k (s_k(x, y) - r_k^{\text{ref}}(x))\big),$$
where $r_k^{\text{ref}}(x)$ is a reference threshold for "good" performance on dimension $k$. Adopting this non-linear combination rule would substantially affect RGC's setup, theoretical analysis, and empirical weight search in four ways:
1. **RGC's scope isolation directly satisfies the conditional independence condition for Logical-AND aggregation:**
Wang et al. (2024) show that summing $\log \sigma(s_k - r_k^{\text{ref}})$ corresponds to aligning toward the *logical AND* of the response being "good" across all applicable dimensions ($\bigwedge_{k \in A(x)} G_k = 1$), provided the dimension judgments $G_k$ are conditionally independent given $(x, y)$. RGC explicitly engineers this conditional independence via strict "Owns / Does not own" scope isolation (Section 3.1, achieving a within-prompt cross-coordinate correlation of $0.096$), and even defines held-out evaluation pass rates via a binary threshold $G_k = \mathbb{I}[s_k(x,y) \ge 0.8]$ (Section 4.2). Thus, RGC's target objective on each prompt is a logical conjunction over the active coordinates $A(x)$, for which summing log-sigmoid utilities is better suited than summing raw scores.
2. **Explaining the degenerate sparsity of the searched Frontier weights (Table 15):**
In Table 15, RGC's Pareto search assigns $60\%\text{–}74\%$ of the total weight to just Safety ($D_6$) and Capability Honesty ($D_{12}$)—for example, `Frontier-Delivery` sets $w_6 = 0.333$ and $w_{12} = 0.410$ while assigning $< 0.001$ to Reasoning ($D_4$) and Artifact Usefulness ($D_5$). Under a linear sum, a response that fails Safety or Honesty ($s_6 = 0$) can still achieve a high aggregate reward if it scores well on several other active coordinates; consequently, the linear weight search must assign huge weights to $D_6$ and $D_{12}$ to prevent guardrail violations when those coordinates are active. Under the LSC combination rule, any score below the reference threshold $r_k^{\text{ref}}$ incurs a steep penalty while scores above $r_k^{\text{ref}}$ saturate with near-zero
marginal utility (∂U/∂sk≈0). Because the Base model already passes D6 (88.3%) and D12 (85.3%) on most prompts (Table 2), log-sigmoid aggregation would enforce guardrails automatically without starving the remaining coordinates of optimization weight.
3. **Changes to the Gibbs-policy and cross-coordinate sensitivity analysis (Section 3.3, Eq. 4 & Appendix C.6):**
In RGC, a large part of the cross-coordinate coupling in Eq. 4 and the second-order curvature in Eq. 17 arises from renormalizing weights by the active weight sum $W_A(x) = \sum_{\ell \in A(x)} w_\ell$:
$$\frac{\partial m_j}{\partial w_k} = \frac{\text{Cov}_{\pi_w^*}(s_j, s_k - R_w)}{\tau W_A(x)}.$$
If trade-offs are instead controlled via the reference thresholds $r = (r_1, \dots, r_K)$ in $U_r(x, y) = \sum_{k \in A(x)} \log \sigma\!\big(\alpha(s_k(x, y) - r_k)\big)$:
- *No active-set weight renormalization is needed:* Because $\log \prod_{k \in A(x)} p(G_k = 1 \mid x, y) = \sum_{k \in A(x)} \log p(G_k = 1 \mid x, y)$, each applicable coordinate contributes additively on the log-probability scale without dividing by $W_A(x)$, eliminating the $1/W_A(x)$ normalization coupling across prompts with different active sets.
- *Failure-gated cross-coordinate derivatives:* Differentiating the Gibbs expected score $m_j(r, x) = \mathbb{E}_{\pi_r^*}[s_j]$ with respect to $r_k$ yields
∂rk∂mj∝Covπr∗(sj,1−σ(α(sk−rk))).
Unlike linear weighting—which continues trading off coordinate j to push sk from 0.90→1.00 even after coordinate k is satisfied—the sensitivity under LSC is gated by 1−σ(α(sk−rk))≈P(Gk=0∣x,y), so coordinate k only competes with coordinate j on responses where coordinate k is currently below its target threshold.
4. **Changes to the GRPO standardization analysis (Appendix C.1–C.5):**
- *Mitigating single-coordinate amplification (Appendix C.1, Eq. 6):* RGC shows that when only a single active coordinate $k$ varies within a sampled group, GRPO's division by the group standard deviation $\hat{\sigma}_k$ cancels a small weight $\omega_k(x)$ and amplifies minor above-threshold score variations into full-scale updates. Under $\log \sigma(\alpha(s_k - r_k))$, if all sampled responses in a group already exceed $r_k$, their transformed rewards saturate near $0$, so the group variance satisfies $\hat{\sigma}_u^2 \ll \epsilon^2$ and the stabilizer $\epsilon$ in GRPO's denominator suppresses the update.
- *What remains unchanged:* For responses below the reference threshold ($s_k < r_k$), $\log \sigma(\alpha(s_k - r_k))$ is approximately linear in $s_k$. Thus, in the unsatisfied regime, RGC's core result on GRPO—that per-prompt variance normalization unequally reweights prompts and can induce simultaneous multi-objective descent (Theorem 1)—continues to hold.