acceptodds
Under review as a conference paper at ICLR 2027

Heat-GRPO: Heat-Kernel Group Relative Policy Optimization for Multi-Rubric Reinforcement Learning

Abstract

Training language models to generate useful and reliable responses requires feedback that captures multiple dimensions of quality beyond correctness. Multi-rubric reinforcement learning offers such feedback, yet reducing nuanced judgments to discrete rewards can discard the distinctions needed for learning. We empirically analyze how nonlinear scoring of rubric-level judge margins before aggregation changes group-relative learning. Heat-GRPO realizes this design choice by smoothing hard verdicts under Gaussian margin perturbations, preserving margin distinctions while limiting the influence of extreme judgments. Identical-response comparisons isolate ranking changes; controlled training comparisons under shared scalar normalization demonstrate gains over Hard and Linear scoring. Comparing complete learning configurations across six models—Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Llama-3.2-1B, Llama-3.2-3B, and Llama-3.1-8B—and four benchmarks, Heat-GRPO improves rubric scores by up to 25.0% relative to the strongest baseline and achieves the highest three-seed mean in 23 of 24 conditions. Vectorized scoring and scalar aggregation reduce advantage computation time by 38.9% and 38.2% relative to GDPO and GD2PO, respectively, while retaining a cost comparable to GRPO. Additional evaluation with Qwen3.8-27B, gpt-oss-20b, and Gemma-4-31B ranks Heat-GRPO first in 22, 20, and 21 of the 24 conditions, respectively, supporting the robustness of its overall advantage across evaluators.

Primary area:
reinforcement learning
Keywords:
llm, reinforcement learning, policy optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_eRi1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies information loss caused by discretizing rubric judgments in multi-rubric reinforcement learning. It introduces Heat-GRPO, which transforms the judge's YES/NO log-probability margins through a Gaussian CDF before weighted aggregation and advantage normalization, with a bandwidth that decreases during training. The paper provides scoring-function comparisons, ranking analyses on identical responses, and mathematical explanations of the normalization mechanism. Across six models and four datasets, Heat-GRPO achieves the highest score in 23 of 24 settings under the primary judge, with additional judges broadly supporting its overall advantage.

Critical strengths
  1. The method addresses a concrete and practically relevant problem. Discretizing rubric judgments can assign identical rewards to different responses and weaken within-group learning signals. Heat-GRPO provides a simple, interpretable scoring modification that integrates into an existing GRPO pipeline, with advantage-computation cost comparable to GRPO.

  2. The overall empirical evaluation has useful breadth. Across six models from two families and four datasets, Heat-GRPO ranks first in 23 of 24 settings under the primary judge. Three additional judges rank it first in 22, 20, and 21 settings, respectively, reducing concerns that the results depend entirely on one evaluation judge.

  3. The mechanism analyses provide useful supporting evidence. On Qwen3-1.7B / BeaverTails, experiments with shared normalization show that fixed-bandwidth Heat outperforms Hard and Exact Linear. Rescoring identical responses also clearly demonstrates how the nonlinear transformation changes relative rubric contributions and response rankings, helping explain the observed gains.

Critical weaknesses
  1. The incremental value of Gaussian smoothing over simple alternatives remains unclear. The Gaussian CDF is a standard bounded mapping, and functions such as sigmoid can also preserve continuous distinctions while suppressing extreme values. The Hard/Heat/Linear controls support the usefulness of nonlinear scoring, but the absence of these direct alternatives leaves the independent value of the Heat construction unresolved.

  2. The main comparison provides unequal feedback information. Heat-GRPO uses continuous confidence, whereas GRPO, GDPO, and GD²PO receive binarized feedback, although these methods are not inherently restricted to binary rewards. The authors explicitly describe this as a complete-configuration comparison, but baselines using the same continuous feedback are still missing. Consequently, the benefits of the proposed scoring mechanism are not adequately separated from the benefits of retaining more information.

  3. The central mechanism ablations focus on a single setting. The Hard/Heat/Linear, bandwidth, and annealing comparisons are primarily conducted on Qwen3-1.7B / BeaverTails. This dataset uses 16 fixed criteria, including 14 negative categories, which differs substantially from the question-dependent criteria and weights in HealthBench. The experiments do not sufficiently establish whether the scoring and annealing benefits extend to different rubric structures.

  4. The probability-extraction procedure may affect the interpretation of the mechanism. Appendix B.2 extracts YES/NO evidence only from the top-20 tokens and assigns a log probability of −20 when a verdict is missing, potentially inflating the margins. Suppressing extreme margins is central to Heat, yet missing-verdict frequencies and imputation sensitivity are not reported. It remains unclear whether some gains compensate for this extraction procedure. The method also requires token log probabilities and cannot directly accommodate black-box judges that return only textual verdicts.

  5. Dependence on the training judge's confidence scale is untested. Training rewards always come from Qwen3-32B, while the additional judges are used only for final evaluation. Heat's behavior depends on the ratio between margin and bandwidth, and different judges may produce differently scaled margins. Cross-judge evaluation does not establish whether the bandwidth and annealing configuration transfers to another training judge or what recalibration would be required.

Questions for authors
  1. How does Heat compare with a temperature-scaled sigmoid using the same margins, normalization, and comparable tuning and annealing settings? Does the Gaussian construction provide an additional benefit?

  2. Does Heat-GRPO retain its advantage when GRPO, GDPO, and GD²PO receive the same continuous feedback, with the remaining training conditions held fixed?

  3. Could the authors compare Hard/Heat/Linear and fixed versus annealed bandwidths on a setting such as HealthBench, where criteria and weights vary across questions, to test whether the findings depend on BeaverTails' fixed rubric structure?

  4. How often is either YES or NO absent from the top-20 tokens? When the designated verdict-token log probabilities are obtained directly, avoiding the fixed −20 imputation, do the extreme-margin distribution and Heat's advantage over Linear persist?

  5. Could the authors replace the training judge in a representative setting, hold the remaining training conditions and evaluation judge fixed, and compare Heat's gain over the baselines? Testing both the original bandwidth and a bandwidth calibrated to the new judge's margin scale would help assess dependence on a particular confidence distribution.

2 comments

Thank you for the detailed response and additional experiments. The new results address most of my concerns and strengthen the evidence for the method’s effectiveness. I will therefore raise my rating from 2 to 3.

Thank you for your kind feedback and for taking the time to review our additional experiments.

We are happy to hear that the new results helped address your concerns. We sincerely appreciate your constructive comments and your willingness to reconsider the rating. Your suggestions have been valuable in strengthening our work.

Best Regards,

Authors

Reviewer_94g9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes a new approach to understand how rubric-level LLM-judge feedback is converted into rewards for GRPO in multi-rubric RL. Instead of thresholding each rubric verdict with h(m)=1{m>0}h(m) = 1\lbrace m \gt 0 \rbrace (Section 3.2) on the judge m=log⁡p((YES))−log⁡p((NO))m = \log p(\text(YES)) - \log p(\text(NO)) (Equation 1), Heat-GRPO employs a non-linear function scoring for each rubric with qσ(m)=Φ(m/σ)q_\sigma(m) = \Phi(m/\sigma) (Equation 2), i.e. the hard threshold convolved with a Gaussian (the "heat-kernel" construction), before the signed rubric weighted sum & batch & group normalization. The bandwidth is annealed as σt=0.4047138−t\sigma_t = 0.4047\sqrt{138-t} (Section 3.2; Training Schedule), from about 4.7369 down to the hard scorer at the final update. The paper argues that this preserves margin distinction near the threshold, bounds the influence of extreme margins, and avoids dead group (all advantages zero), and it gives a proposition that batch normalization amplifies surviving advantages by 1ρlive1\sqrt{\rho_{\text{live}}} (Equation 6). The paper utilizes some experiments including train six LLMs (Llama-3.2-1B/3B, Llama-3.1-8B, and Qwen3-1.7B/4B/8B) on 4 benchmarks (RaR-Science, RaR-medicine, BeaverTails, HealthBench; with 1500 train and 611 eval set). For the training and primary evaluation judge they use Qwen3-32B with 3 seeds. Heat-GRPO proved to be the highest 3 seeds in 23 / 24 conditions (Table 2) against open-weight judges. Additionally, the paper adds a scorer ablation on Qwen3-1.7B / BeaverTails (Table 4) compares hard (.5510), fixed bandwidths (best: Broad σ=\sigma = 4.75, .6900), exact linear (.3895), and the annealed schedule (.7403). FUthermore, the paper also reports an advantage regarding computation time to be 38.9% (compared to GDPO) and 38.2% (compared to GD$^2$PO), Heat-GRPO is faster than three other candidates, even in just small gap with GRPO (only 1.4%) (Table 3).

Critical strengths

S1. Broad and repoducible empirical study

  • 6 LLMs from 2 families (Llama & Qwen), 4 benchmarks with different rubric structures (uniform$\pm$ weights, fixed safety rubric, and −10...+10-10...+10 weights), 3 seeds, shared verl pipeline that evaluated with 4 judges from 3 families (Qwen, GPT, and Gemma)
  • Heat-GRPO rank first in 20-23 of 24 conditions under all judges (Appendix D, Table 12)
  • Result could be reproduced easily with all available description in Appendix B.2 -B.5

S.2 Clear and insightful information

  • Table 4 isolates the scoring under shared scalar normalization. With raw margins (Exact Linear; .3895) below hard verdicts (.5510) & bounded smoothing well above both (Broad, .6900)
Critical weaknesses

W1. The result does not isolate the scorer

  • In Table 2, baselines receive binary verdicts 1{u>0.5}1 \lbrace u \gt 0.5 \rbrace, whereas Heat-GRPO receives continuous margins (Appendix A.3). The authores acknowledge these are "configuration-level gains". Nevertheless, Table 2 measures value of giving the polici optimizer judge log-probabilities, not of the Heat transform. The obvious baseline (GRPO with the soft probability p(YES)=sigmoid(m)p(\text{YES}) = sigmoid(m) per-rubric) is missing. Table 4 suggests this baseline would recover most of the gain: σ=1.5\sigma = 1.5 (roughly p(YES)p (\text{YES})) already gives .6487 vs .5510 for Hard.

W.2 Statistical support is missing

  • No standard deviation, CIs, or paired tests, so there is a chance that the 23/24 is may not interpretable
  • Extra judges only use 1-2 runs per-condition (Appendix D), the only controlled evidence for the scorer itself (Table 4) is a single model-benchmark pair

W.3 Hyperparameter selection

  • There is no validation split: the same 611 beaverTails evaluation prompts are also used for training-time monitoring (Appendix B.1, 8)
  • The origin of σbase=0.4047\sigma_\text{base} = 0.4047 and of the (138−t)\sqrt{(138-t)} schedule is not stated and explained at all, maybe there will be good to show some experiment with different value; mathematical proof; or explanation about intuition to get the number
  • The bandwidth sweep (Table 4) runs on Qwen3-1.7B / BeaverTails, which is itself becoming one of the 24 test conditions, and the results are very sensitive to σ\sigma (.3734 - .6900)

W.4 Evaluation protocol allow some optimistic biases

  • Qwen3-32B is used as training and primary evaluation judge, with no KL penalty (Table 5), Heat-GRPO directly optimizes that judge's YES log-probabilites (which aligns its training signal more tightly with the evaluator's verdict than binary-input baselines) so over-optimization to the judge is plausible
  • The largest gain is on BeaverTails (+0.093), where the rubric rewards helpfulness and engagement on safety-dataset prompts, there is no breakdown of whether gains come from positive vs harm (negative) category
Questions for authors

Q1. Can you add GRPO with per-rubric p(YES)=sigmoid(m)p(\text{YES}) = sigmoid(m) (without bandwidth & annealing) and GDPO with continuous channels, with 3 seeds on at least one condition per-benchmark, as I mentioned before in W1

Q2. How and on which data, were σbase=0.4047\sigma_\text{base} = 0.4047 and the schedule ((138−t)\sqrt{(138-t)}) chosen? Please re-select them on a split disjoint from the 611 evaluation prompts and report sensitivity to σbase(×0.5,×2)\sigma_\text{base} (\times 0.5, \times 2) (Also please looks again at point W3 )

Q3. Please give per-condition std over seeds or paired bootstrap CIs for Table 2 (How many of the 24 conditions does the gain exceed seed noise? )

Q4. Related to point W4, please calculates the harm-category YES rates, refusal rate, helpfulness and engagement rates (ideally also with an independent safety classifier). Do the gains come from more engagement with harmful prompts, fewer violations, or both?

Q.5 Do response length or margin distributions shift differently under Heat-GRPO? A small human or held-out-rubric evaluation would help

Q6. About 109 ms is saved per-advantage computation, against rougly 100-175 s per-update (Table 6). Please report the saving as a fraction of step time, the 38.9% in the abstract overstates its relevance, it will make the dataset becomes easier to understand

Reviewer_zYgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the problem of combining multiple reward signals---provided as log likelihood ratios of positive vs negative outcomes. The construction seems straightforward and the analysis is reasonable. My main concern in the relationship w/ prior work.

Critical strengths

see summary

Critical weaknesses

I'm concerned with the relationship to prior work. In particular Transforming and Combining Rewards for Aligning Large Language Models (arXiv:2402.00742) seems to offer a very similar (and possibly better motivated) solution to the same problem---take the log likelihood ratio scale rewards, transform them to a suitable scale, and combine them. Apparently, the main difference between the two methods is that the prior work caps the contribution of already strongly satisfied criterion by allows failures on any aspect to be unbounded, while the Gaussian transformation in the present paper bounds the contribution of both the negative and positive reward tails.

(I also see there's quite a few papers citing the Transforming and Combining paper; I haven't done a proper literature review, but the others really ought to!)

Gemini makes the following point as well that I'd like to hear a response to:

"In Section 3.1 (p. 4, line 201) and Appendix A.3 (p. 15, line 771), the authors reveal that the underlying verl/rubric training interface already computes continuous sigmoidal utilities pₓᵢₖ = sigmoid(mₓᵢₖ/8) from the judge's YES/NO logits (and Heat-GRPO literally inverts mₓᵢₖ = 8 logit(pₓᵢₖ) before applying Φ(mₓᵢₖ/σ)!). Yet they neither test pₓᵢₖ = sigmoid(mₓᵢₖ/a) as a baseline (only acknowledging in Appendix B.7, p. 22 that logistic scoring shares all of Heat-GRPO's boundary and symmetry properties), nor cite recent rubric-reward papers that use continuous judge probabilities/scores to avoid binary ties (e.g., Hashemi et al., ACL 2024, LLM-Rubric; Viswanathan et al., NeurIPS 2025, Checklists Are Better Than Reward Models)."

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.