- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of reward hacking in LLM post-training when the surrogate reward function is inaccurate. The authors consider using a limited number of ground-truth reward labels to recalibrate the original judge before policy optimization. The main question is how to choose the sampling distribution for collecting these labels.
For the theoretical analysis, the authors first connect the regret upper bound with the reward calibration error and coverage coefficient. They then propose Envelope Sampling and prove that it minimizes the worst-case coverage coefficient for a given reward function class. For a clipped (L_2) reward class, they derive a closed-form expression for the envelope distribution and propose rejection sampling and additional reward optimization to sample from this distribution. They also establish the optimal label complexity in a simplified rare-bad-modes setting.
For the experiments, the authors evaluate their method on clinical note generation and controlled sycophancy tasks. The results show that using Envelope Sampling for reward recalibration can reduce reward hacking compared with sampling from the base policy.
Critical strengths
-
The paper studies reward hacking caused by the difference between surrogate rewards and true rewards. How to select useful samples to correct the reward function under a limited labeling budget is an important problem.
-
The authors establish a theoretical connection between reward calibration error, the coverage coefficient, and policy regret. They also derive a closed-form envelope distribution for a specific reward class, which provides theoretical motivation for the proposed sampling method. The label complexity analysis in Section 5 further helps explain why Envelope Sampling may perform better.
-
The authors evaluate the method on a real LLM clinical note generation task and a controlled sycophancy task. In the latter task, policy optimization can be computed exactly, which avoids the influence of optimization errors and allows the authors to better study the effect of the sampling distribution.
Critical weaknesses
Theoretical Weaknesses
-
Minimizing the coverage coefficient does not necessarily minimize the regret upper bound.
Theorem 1 gives a regret upper bound that depends on both the coverage coefficient and the reward calibration error . However, Theorem 2 only proves that Envelope Sampling minimizes . It does not consider how different sampling distributions affect the recalibrated reward function and its error. Therefore, a smaller coverage coefficient does not necessarily lead to a smaller overall regret bound or better policy performance. I think the authors should explain more clearly why minimizing only the coverage coefficient is a suitable objective for sampling distribution design.
-
There is a lack of general theoretical guarantees for improving reward calibration.
The main idea of this paper is to use Envelope Sampling to collect labels and then correct the reward function. However, Theorems 1–3 do not analyze how the sampling distribution affects the accuracy of the recalibrated reward function. Although Theorem 6 provides a label complexity guarantee, it is based on a simplified finite output space, where the reward for each labeled state is directly replaced by the true reward. This is quite different from reward model learning or LLM rubric editing in practice. Therefore, it remains unclear whether Envelope Sampling can improve reward calibration accuracy under more general reward learning methods.
Experimental Weaknesses
-
The reward calibration quality is not directly evaluated.
In the clinical experiment, the authors do not directly optimize the recalibrated scalar reward. Instead, they use reward scores to rank a fixed set of responses, construct preference pairs, and then use offline DPO for training. In this process, the magnitude information of the reward scores is lost, which differs from directly optimizing the scalar reward as considered in the theory.
Moreover, the experiments mainly report the final policy performance, but do not directly compare the accuracy of the reward functions obtained from different sampling methods. I think it would be useful to compare the reward MSE, error variance, and pairwise preference accuracy of and on the same test dataset. It would also be helpful to show how the coverage coefficient and reward calibration error change under different sampling distributions. Furthermore, experiments using methods that directly use scalar rewards, such as PPO, would help to better connect the theory with the experiments.
-
The main experiments do not clearly show the additional benefit of Envelope Reweighting
The main text compares sampling from the original policy with sampling from the envelope distribution . However, the envelope distribution is constructed based on the judge-optimized policy . Therefore, this comparison involves two changes: sampling from a judge-optimized policy and applying Envelope Reweighting.
Although Appendix F provides a comparison with directly sampling from , Table 5 shows that, in the clinical experiment, sampling from achieves a reference quality of 0.680, while Envelope Sampling only achieves 0.666. This suggests that the main improvement may come from sampling from the judge-optimized policy rather than from Envelope Reweighting itself.
Although the synthetic heterogeneous bad-mode experiments demonstrate the advantage of Envelope Sampling, there is still limited evidence of this advantage in real open-ended LLM tasks. I suggest that the authors move the comparison in Appendix F to the main text and include more active sampling or active reward calibration baselines to better demonstrate the benefit of the proposed method.
-
The clinical experiment is relatively small.
The clinical experiment only uses 16 training prompts and 16 test prompts, and the final DPO training uses only 32 preference pairs. Considering that reward hacking in real applications may involve many different types of errors, such a small experiment may not be sufficient to demonstrate the generalization of the proposed method. It would be better to evaluate the method with more training and test prompts, larger preference datasets, and longer RLHF training runs.
Questions for authors
-
The eight responses selected by Envelope Sampling are only used to update the judge's rubric. Could the authors explain why they choose this indirect way of using the feedback? Have they considered directly using these labeled responses to construct or update the preference dataset?
-
Could the authors report the changes in judge scores, preference reversal rates, or pairwise agreement before and after reward recalibration on the fixed candidate responses? How much does reward recalibration change the final preference dataset?
-
The envelope distribution depends on M, S, and . Since the true reward (h) is unknown in practice, how do the authors choose these parameters and determine whether the assumed reward class is reasonable?
3 comments
I thank the authors for their detailed response and additional clarifications. I have carefully considered their arguments and appreciate the theoretical motivation and controlled experiments. However, my central concerns remain insufficiently addressed. So, although I feel sorry about this, I remain unconvinced that the current experimental evidence sufficiently supports the paper's central claims, and I may maintain my original rating.
In particular, while the authors emphasize evaluating reward calibration through downstream optimization, the lack of direct evaluation of the recalibrated rewards still leaves their actual improvements unclear (My earlier suggestion to include PPO was also motivated by the desire to evaluate the effects of reward through downstream optimization more directly, in line with the authors' own evaluation rationale). Moreover, the key comparison with Judge-optimized Sampling in Appendix F does not demonstrate additional benefits from Envelope Reweighting in the main clinical experiment. Although the controlled experiments illustrate its potential advantages, they do not fully resolve this concern in realistic LLM settings.
We thank the reviewer for clarifying the motivation for the PPO comparison. Although measuring the judge’s accuracy on held-out notes would tell us how well it scores those notes, our central question is which examples to label so that the model makes fewer errors when we train it against the judge. In fact, a judge can score most notes accurately yet consistently reward notes that invent vital signs, giving the model an incentive to invent them. This is why we evaluate the notes the model generates after training. With the same label budget and DPO procedure, we observe severe fabrication in 19.8% of notes with base-model sampling and 1.0% with envelope sampling (Table 2). We also study scalar-reward optimization directly in our controlled tasks, where we compute the optimal policy exactly.
On reweighting, we find that sampling directly from the judge-optimized policy already improves on base-model sampling in the clinical task, and we see no further clinical gain from reweighting. We show where reweighting helps in Figure 13. When the judge overvalues several types of outputs, sampling its preferred outputs can reveal the most overvalued one while missing others that the model later exploits. We also observe a gain in our controlled sycophancy experiment, where reweighting reduces error from 12.2% to 8.9% with the same 128 label draws and 27B judge. We will move the clinical comparison (Table 5) into the main text to make this distinction clear.
We sincerely thank the reviewer for their helpful and constructive feedback. We have included our responses below -- we are happy to take further comments and request the reviewer to increase their scores if they feel that we have adequately addressed some of their concerns.
Reward accuracy + the role of coverage
We thank the reviewer for the opportunity to clarify our contribution. A judge can be accurate on typical outputs while making rare mistakes that optimization exploits (Section 5, Appendix B.4), as also studied in [1]. We ask which notion of reward accuracy controls optimization, and connect error under the query distribution to regret in Theorem 1.
We agree that minimizing coverage need not minimize the product for every learner. However, Section 4.1 (lines 203–207) already gives our justification: bounded rewards give bounded variance, while coverage can be arbitrarily large. If h,r lie in a common interval of width L, then Varρ(h−r)≤L2 for every ρ. We minimize the bound βL2C(ρ)/2 with the envelope. For a finite realizable family H⊆F in this interval, consistent learning from n i.i.d. noiseless labels under full-support ρ also gives, with probability at least 1−δ,
J(πh)−J(πr)≤2nβL2C(ρ)logδ∣H∣.
We obtain this corollary by a union bound and Theorem 1, and will include it as a guarantee beyond local repair. We agree that MSE and pairwise agreement are useful diagnostics, but their interpretation depends on the evaluation distribution. Furthermore, a constant score offset changes MSE without changing the policy. We therefore test whether the revised judge is safer to optimize (Table 2), which is our original goal. Figure 8 reports judge scores during training, and Appendix D.1 describes how the feedback changes fabrication penalties.
Comparison with πj and active sampling
We already include the πj comparison in Appendix F and will move it into the main text. Note that πj is itself a limiting case of our method as S→0. Sampling from it can reveal errors that base-model samples miss; reweighting additionally helps when πj still assigns little probability to some exploitable errors. We obtain:
Table 5 gives the clinical results; Figure 13 gives heterogeneous results at β=2 across all three supports. Although we see no additional clinical gain, the heterogeneous experiment shows why correcting only the errors that πj reveals can be insufficient. We cover these additional errors by rejecting proposals from πj (Proposition 4). We discuss active preference learning in Section 2 and would appreciate the specific additional baselines the reviewer recommends.
Rubric editing + DPO
Rubric editing is a common practical way to calibrate judges [2, Section 2], and rubric-based rewards are increasingly used for post-training [3]. We use eight reference labels to improve a rubric that can then score all 256 candidates. To our knowledge, we are the first to formulate the choice of these calibration examples as a minimax coverage problem for subsequent KL-regularized optimization. We will also compare against direct preference training on reference-labeled pairs in the final paper, using the same label budget.
We agree that DPO discards reward gaps when forming 32 hard preference pairs (Appendix D.1), and use it as a practical approximation. However, we also observe the sampling effect with exact scalar-reward optimization in sycophancy and the finite experiments (Section 6.2, Appendix C).
Parameters and scale
We discuss β,M,S in Appendix B.3 and give the clinical protocol in Appendix D.1. Here, M bounds correction standard deviation and S its range. Scores in [0,1] justify M=1,S≥2 without observing h, and we choose S using independent pilot probabilities to target acceptance. In Eq. (1), 1/β multiplies KL; we use envelope β=10 and DPO coefficient 0.1. We will make these choices more prominent.
We use a small panel to isolate sampling under fixed feedback and training budgets. We will expand prompts, preference data, and training duration in the final paper. Given the same failure across three seeds and an earlier model generation (Appendices D.2 and G), we expect the mechanism to remain relevant at larger scale.
References
[1] Zhang et al. Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training. ICLR, 2026.
[2] Kong et al. The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations. arXiv:2608.18300, 2026.
[3] Gunjal et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746, 2025.