- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies information loss caused by discretizing rubric judgments in multi-rubric reinforcement learning. It introduces Heat-GRPO, which transforms the judge's YES/NO log-probability margins through a Gaussian CDF before weighted aggregation and advantage normalization, with a bandwidth that decreases during training. The paper provides scoring-function comparisons, ranking analyses on identical responses, and mathematical explanations of the normalization mechanism. Across six models and four datasets, Heat-GRPO achieves the highest score in 23 of 24 settings under the primary judge, with additional judges broadly supporting its overall advantage.
Critical strengths
-
The method addresses a concrete and practically relevant problem. Discretizing rubric judgments can assign identical rewards to different responses and weaken within-group learning signals. Heat-GRPO provides a simple, interpretable scoring modification that integrates into an existing GRPO pipeline, with advantage-computation cost comparable to GRPO.
-
The overall empirical evaluation has useful breadth. Across six models from two families and four datasets, Heat-GRPO ranks first in 23 of 24 settings under the primary judge. Three additional judges rank it first in 22, 20, and 21 settings, respectively, reducing concerns that the results depend entirely on one evaluation judge.
-
The mechanism analyses provide useful supporting evidence. On Qwen3-1.7B / BeaverTails, experiments with shared normalization show that fixed-bandwidth Heat outperforms Hard and Exact Linear. Rescoring identical responses also clearly demonstrates how the nonlinear transformation changes relative rubric contributions and response rankings, helping explain the observed gains.
Critical weaknesses
-
The incremental value of Gaussian smoothing over simple alternatives remains unclear. The Gaussian CDF is a standard bounded mapping, and functions such as sigmoid can also preserve continuous distinctions while suppressing extreme values. The Hard/Heat/Linear controls support the usefulness of nonlinear scoring, but the absence of these direct alternatives leaves the independent value of the Heat construction unresolved.
-
The main comparison provides unequal feedback information. Heat-GRPO uses continuous confidence, whereas GRPO, GDPO, and GD²PO receive binarized feedback, although these methods are not inherently restricted to binary rewards. The authors explicitly describe this as a complete-configuration comparison, but baselines using the same continuous feedback are still missing. Consequently, the benefits of the proposed scoring mechanism are not adequately separated from the benefits of retaining more information.
-
The central mechanism ablations focus on a single setting. The Hard/Heat/Linear, bandwidth, and annealing comparisons are primarily conducted on Qwen3-1.7B / BeaverTails. This dataset uses 16 fixed criteria, including 14 negative categories, which differs substantially from the question-dependent criteria and weights in HealthBench. The experiments do not sufficiently establish whether the scoring and annealing benefits extend to different rubric structures.
-
The probability-extraction procedure may affect the interpretation of the mechanism. Appendix B.2 extracts YES/NO evidence only from the top-20 tokens and assigns a log probability of −20 when a verdict is missing, potentially inflating the margins. Suppressing extreme margins is central to Heat, yet missing-verdict frequencies and imputation sensitivity are not reported. It remains unclear whether some gains compensate for this extraction procedure. The method also requires token log probabilities and cannot directly accommodate black-box judges that return only textual verdicts.
-
Dependence on the training judge's confidence scale is untested. Training rewards always come from Qwen3-32B, while the additional judges are used only for final evaluation. Heat's behavior depends on the ratio between margin and bandwidth, and different judges may produce differently scaled margins. Cross-judge evaluation does not establish whether the bandwidth and annealing configuration transfers to another training judge or what recalibration would be required.
Questions for authors
-
How does Heat compare with a temperature-scaled sigmoid using the same margins, normalization, and comparable tuning and annealing settings? Does the Gaussian construction provide an additional benefit?
-
Does Heat-GRPO retain its advantage when GRPO, GDPO, and GD²PO receive the same continuous feedback, with the remaining training conditions held fixed?
-
Could the authors compare Hard/Heat/Linear and fixed versus annealed bandwidths on a setting such as HealthBench, where criteria and weights vary across questions, to test whether the findings depend on BeaverTails' fixed rubric structure?
-
How often is either YES or NO absent from the top-20 tokens? When the designated verdict-token log probabilities are obtained directly, avoiding the fixed −20 imputation, do the extreme-margin distribution and Heat's advantage over Linear persist?
-
Could the authors replace the training judge in a representative setting, hold the remaining training conditions and evaluation judge fixed, and compare Heat's gain over the baselines? Testing both the original bandwidth and a bandwidth calibrated to the new judge's margin scale would help assess dependence on a particular confidence distribution.
2 comments
Thank you for your kind feedback and for taking the time to review our additional experiments.
We are happy to hear that the new results helped address your concerns. We sincerely appreciate your constructive comments and your willingness to reconsider the rating. Your suggestions have been valuable in strengthening our work.
Best Regards,
Authors
Thank you for the detailed response and additional experiments. The new results address most of my concerns and strengthen the evidence for the method’s effectiveness. I will therefore raise my rating from 2 to 3.