acceptodds
Under review as a conference paper at ICLR 2027

How to post-train on a surrogate: Envelope sampling provably mitigates reward hacking

Abstract

Large language models (LLMs) are commonly post-trained against LLM judges and other cheap surrogates because the true reward, such as human preference, is too expensive to query at scale. This practice often leads to reward hacking, where reinforcement learning against a miscalibrated surrogate leads to undesirable side effects. In this work, we study a setting in which a small number of model outputs are annotated with ground-truth labels (e.g., from expert review) and used to recalibrate the LLM judge before optimizing against it. Prior approaches to judge recalibration are costly or heuristic, and it is known that on-policy sampling fails when the surrogate is miscalibrated on a rare set of outputs. In this work, we propose envelope sampling, a theoretically-grounded method for judge recalibration that seeks to minimize an upper bound on the regret of the post-trained model under the assumption that the human reward and re-calibrated reward lie in an ball around the judge. We give practical algorithms to sample from the envelope by rejection or by fine-tuning against a modified reward, and experiments on clinical note generation and on a controlled sycophancy task show that recalibrating on envelope samples mitigates reward hacking where recalibrating on base-model samples does not.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
reward hacking, language model post-training, LLM-as-a-judge, reward recalibration, label-efficient learning, envelope sampling, minimax analysis, KL-regularized reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VLAMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of reward hacking in LLM post-training when the surrogate reward function is inaccurate. The authors consider using a limited number of ground-truth reward labels to recalibrate the original judge before policy optimization. The main question is how to choose the sampling distribution for collecting these labels.

For the theoretical analysis, the authors first connect the regret upper bound with the reward calibration error and coverage coefficient. They then propose Envelope Sampling and prove that it minimizes the worst-case coverage coefficient for a given reward function class. For a clipped (L_2) reward class, they derive a closed-form expression for the envelope distribution and propose rejection sampling and additional reward optimization to sample from this distribution. They also establish the optimal label complexity in a simplified rare-bad-modes setting.

For the experiments, the authors evaluate their method on clinical note generation and controlled sycophancy tasks. The results show that using Envelope Sampling for reward recalibration can reduce reward hacking compared with sampling from the base policy.

Critical strengths
  1. The paper studies reward hacking caused by the difference between surrogate rewards and true rewards. How to select useful samples to correct the reward function under a limited labeling budget is an important problem.

  2. The authors establish a theoretical connection between reward calibration error, the coverage coefficient, and policy regret. They also derive a closed-form envelope distribution for a specific reward class, which provides theoretical motivation for the proposed sampling method. The label complexity analysis in Section 5 further helps explain why Envelope Sampling may perform better.

  3. The authors evaluate the method on a real LLM clinical note generation task and a controlled sycophancy task. In the latter task, policy optimization can be computed exactly, which avoids the influence of optimization errors and allows the authors to better study the effect of the sampling distribution.

Critical weaknesses

Theoretical Weaknesses

  1. Minimizing the coverage coefficient does not necessarily minimize the regret upper bound.

    Theorem 1 gives a regret upper bound that depends on both the coverage coefficient C(ρ)C(\rho) and the reward calibration error Var⁡ρ(h−r)\operatorname{Var}_{\rho}(h-r). However, Theorem 2 only proves that Envelope Sampling minimizes C(ρ)C(\rho). It does not consider how different sampling distributions affect the recalibrated reward function and its error. Therefore, a smaller coverage coefficient does not necessarily lead to a smaller overall regret bound or better policy performance. I think the authors should explain more clearly why minimizing only the coverage coefficient is a suitable objective for sampling distribution design.

  2. There is a lack of general theoretical guarantees for improving reward calibration.

    The main idea of this paper is to use Envelope Sampling to collect labels and then correct the reward function. However, Theorems 1–3 do not analyze how the sampling distribution affects the accuracy of the recalibrated reward function. Although Theorem 6 provides a label complexity guarantee, it is based on a simplified finite output space, where the reward for each labeled state is directly replaced by the true reward. This is quite different from reward model learning or LLM rubric editing in practice. Therefore, it remains unclear whether Envelope Sampling can improve reward calibration accuracy under more general reward learning methods.

Experimental Weaknesses

  1. The reward calibration quality is not directly evaluated.

    In the clinical experiment, the authors do not directly optimize the recalibrated scalar reward. Instead, they use reward scores to rank a fixed set of responses, construct preference pairs, and then use offline DPO for training. In this process, the magnitude information of the reward scores is lost, which differs from directly optimizing the scalar reward as considered in the theory.

    Moreover, the experiments mainly report the final policy performance, but do not directly compare the accuracy of the reward functions obtained from different sampling methods. I think it would be useful to compare the reward MSE, error variance, and pairwise preference accuracy of rbaser_{\mathrm{base}} and renvr_{\mathrm{env}} on the same test dataset. It would also be helpful to show how the coverage coefficient and reward calibration error change under different sampling distributions. Furthermore, experiments using methods that directly use scalar rewards, such as PPO, would help to better connect the theory with the experiments.

  2. The main experiments do not clearly show the additional benefit of Envelope Reweighting

    The main text compares sampling from the original policy π\pi with sampling from the envelope distribution ρ⋆\rho^\star. However, the envelope distribution is constructed based on the judge-optimized policy πj\pi_j. Therefore, this comparison involves two changes: sampling from a judge-optimized policy and applying Envelope Reweighting.

    Although Appendix F provides a comparison with directly sampling from πj\pi_j, Table 5 shows that, in the clinical experiment, sampling from πj\pi_j achieves a reference quality of 0.680, while Envelope Sampling only achieves 0.666. This suggests that the main improvement may come from sampling from the judge-optimized policy rather than from Envelope Reweighting itself.

    Although the synthetic heterogeneous bad-mode experiments demonstrate the advantage of Envelope Sampling, there is still limited evidence of this advantage in real open-ended LLM tasks. I suggest that the authors move the comparison in Appendix F to the main text and include more active sampling or active reward calibration baselines to better demonstrate the benefit of the proposed method.

  3. The clinical experiment is relatively small.

    The clinical experiment only uses 16 training prompts and 16 test prompts, and the final DPO training uses only 32 preference pairs. Considering that reward hacking in real applications may involve many different types of errors, such a small experiment may not be sufficient to demonstrate the generalization of the proposed method. It would be better to evaluate the method with more training and test prompts, larger preference datasets, and longer RLHF training runs.

Questions for authors
  1. The eight responses selected by Envelope Sampling are only used to update the judge's rubric. Could the authors explain why they choose this indirect way of using the feedback? Have they considered directly using these labeled responses to construct or update the preference dataset?

  2. Could the authors report the changes in judge scores, preference reversal rates, or pairwise agreement before and after reward recalibration on the fixed candidate responses? How much does reward recalibration change the final preference dataset?

  3. The envelope distribution depends on M, S, and β\beta. Since the true reward (h) is unknown in practice, how do the authors choose these parameters and determine whether the assumed reward class is reasonable?

3 comments

We sincerely thank the reviewer for their helpful and constructive feedback. We have included our responses below -- we are happy to take further comments and request the reviewer to increase their scores if they feel that we have adequately addressed some of their concerns.

Reward accuracy + the role of coverage

We thank the reviewer for the opportunity to clarify our contribution. A judge can be accurate on typical outputs while making rare mistakes that optimization exploits (Section 5, Appendix B.4), as also studied in [1]. We ask which notion of reward accuracy controls optimization, and connect error under the query distribution to regret in Theorem 1.

We agree that minimizing coverage need not minimize the product for every learner. However, Section 4.1 (lines 203–207) already gives our justification: bounded rewards give bounded variance, while coverage can be arbitrarily large. If h,rh,r lie in a common interval of width LL, then Var⁡ρ(h−r)≤L2\operatorname{Var}_{\rho}(h-r)\leq L^2 for every ρ\rho. We minimize the bound βL2C(ρ)/2\beta L^2 C(\rho)/2 with the envelope. For a finite realizable family H⊆F\mathcal H\subseteq\mathcal F in this interval, consistent learning from nn i.i.d. noiseless labels under full-support ρ\rho also gives, with probability at least 1−δ1-\delta,

J(πh)−J(πr)≤βL2C(ρ)2nlog⁡∣H∣δ.J(\pi_h)-J(\pi_r)\leq\frac{\beta L^2 C(\rho)}{2n}\log\frac{|\mathcal H|}{\delta}.

We obtain this corollary by a union bound and Theorem 1, and will include it as a guarantee beyond local repair. We agree that MSE and pairwise agreement are useful diagnostics, but their interpretation depends on the evaluation distribution. Furthermore, a constant score offset changes MSE without changing the policy. We therefore test whether the revised judge is safer to optimize (Table 2), which is our original goal. Figure 8 reports judge scores during training, and Appendix D.1 describes how the feedback changes fabrication penalties.

Comparison with πj\pi_j and active sampling

We already include the πj\pi_j comparison in Appendix F and will move it into the main text. Note that πj\pi_j is itself a limiting case of our method as S→0S\to0. Sampling from it can reveal errors that base-model samples miss; reweighting additionally helps when πj\pi_j still assigns little probability to some exploitable errors. We obtain:

QueryClinical quality, T=64T=64Heterogeneous regret, 64 labels
Base π\pi0.589—
πj\pi_j0.680>3.4>3.4
Envelope0.666<0.2<0.2

Table 5 gives the clinical results; Figure 13 gives heterogeneous results at β=2\beta=2 across all three supports. Although we see no additional clinical gain, the heterogeneous experiment shows why correcting only the errors that πj\pi_j reveals can be insufficient. We cover these additional errors by rejecting proposals from πj\pi_j (Proposition 4). We discuss active preference learning in Section 2 and would appreciate the specific additional baselines the reviewer recommends.

Rubric editing + DPO

Rubric editing is a common practical way to calibrate judges [2, Section 2], and rubric-based rewards are increasingly used for post-training [3]. We use eight reference labels to improve a rubric that can then score all 256 candidates. To our knowledge, we are the first to formulate the choice of these calibration examples as a minimax coverage problem for subsequent KL-regularized optimization. We will also compare against direct preference training on reference-labeled pairs in the final paper, using the same label budget.

We agree that DPO discards reward gaps when forming 32 hard preference pairs (Appendix D.1), and use it as a practical approximation. However, we also observe the sampling effect with exact scalar-reward optimization in sycophancy and the finite experiments (Section 6.2, Appendix C).

Parameters and scale

We discuss β,M,S\beta,M,S in Appendix B.3 and give the clinical protocol in Appendix D.1. Here, MM bounds correction standard deviation and SS its range. Scores in [0,1][0,1] justify M=1,S≥2M=1,S\geq2 without observing hh, and we choose SS using independent pilot probabilities to target acceptance. In Eq. (1), 1/β1/\beta multiplies KL; we use envelope β=10\beta=10 and DPO coefficient 0.10.1. We will make these choices more prominent.

We use a small panel to isolate sampling under fixed feedback and training budgets. We will expand prompts, preference data, and training duration in the final paper. Given the same failure across three seeds and an earlier model generation (Appendices D.2 and G), we expect the mechanism to remain relevant at larger scale.

References

[1] Zhang et al. Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training. ICLR, 2026.

[2] Kong et al. The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations. arXiv:2608.18300, 2026.

[3] Gunjal et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746, 2025.

I thank the authors for their detailed response and additional clarifications. I have carefully considered their arguments and appreciate the theoretical motivation and controlled experiments. However, my central concerns remain insufficiently addressed. So, although I feel sorry about this, I remain unconvinced that the current experimental evidence sufficiently supports the paper's central claims, and I may maintain my original rating.

In particular, while the authors emphasize evaluating reward calibration through downstream optimization, the lack of direct evaluation of the recalibrated rewards still leaves their actual improvements unclear (My earlier suggestion to include PPO was also motivated by the desire to evaluate the effects of reward through downstream optimization more directly, in line with the authors' own evaluation rationale). Moreover, the key comparison with Judge-optimized Sampling in Appendix F does not demonstrate additional benefits from Envelope Reweighting in the main clinical experiment. Although the controlled experiments illustrate its potential advantages, they do not fully resolve this concern in realistic LLM settings.

We thank the reviewer for clarifying the motivation for the PPO comparison. Although measuring the judge’s accuracy on held-out notes would tell us how well it scores those notes, our central question is which examples to label so that the model makes fewer errors when we train it against the judge. In fact, a judge can score most notes accurately yet consistently reward notes that invent vital signs, giving the model an incentive to invent them. This is why we evaluate the notes the model generates after training. With the same label budget and DPO procedure, we observe severe fabrication in 19.8% of notes with base-model sampling and 1.0% with envelope sampling (Table 2). We also study scalar-reward optimization directly in our controlled tasks, where we compute the optimal policy exactly.

On reweighting, we find that sampling directly from the judge-optimized policy already improves on base-model sampling in the clinical task, and we see no further clinical gain from reweighting. We show where reweighting helps in Figure 13. When the judge overvalues several types of outputs, sampling its preferred outputs can reveal the most overvalued one while missing others that the model later exploits. We also observe a gain in our controlled sycophancy experiment, where reweighting reduces error from 12.2% to 8.9% with the same 128 label draws and 27B judge. We will move the clinical comparison (Table 5) into the main text to make this distinction clear.

Reviewer_WvBZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses miscalibration in LLM judges. It proposes envelope sampling, a recalibration method with theoretical support. The goal is to minimize an upper bound on the post-trained model’s regret. The theory assumes that the human reward and the recalibrated reward both lie within an L² ball around the judge’s reward.

The authors give two practical ways to sample from the envelope: rejection sampling and fine-tuning with a modified reward. They test the method on clinical note generation and a controlled sycophancy task. In both tasks, recalibration with envelope samples reduces reward hacking, while recalibration with base-model samples does not.

Critical strengths
  1. The theory is clear and well structured. It first bounds regret by recalibration error times a coverage factor. It then shows that the envelope distribution p minimizes the worst-case coverage factor. Finally, it gives a closed-form solution for the L² ball. The authors also explain how to sample from this distribution in practice.

  2. The experiments show that envelope sampling works well in practice. On both tasks, reward hacking remains after recalibration with on-policy samples. Using envelope samples helps reduce it.

Critical weaknesses
  1. My main concern is the role of the radius of F. It is both part of the theoretical assumption and a tunable hyperparameter. The paper does not clearly explain how to handle these two roles. What happens when the judge is badly miscalibrated? A large radius M would make the envelope broader and require more labels. A smaller radius, however, could make the assumption invalid.
Questions for authors
  1. In the clinical experiment, recalibration relies almost entirely on another LLM. GPT-5.6 Sol is given just eight clinical notes and asked to rewrite the scoring rubric. This suggests that Var may depend more on whether Sol can learn a better rubric from eight examples than on the query distribution. Does the recalibrator’s ability limit how much the method can help? I think this needs more discussion.
2 comments

We sincerely thank the reviewer for their helpful and constructive feedback. We have included our responses below -- we are happy to take further comments and request the reviewer to increase their scores if they feel that we have adequately addressed some of their concerns.

Selection of the uncertainty radius

We thank the reviewer for pointing out that we should explain the two roles of the radius more clearly. We discuss parameter selection in Appendix B.3 and give our clinical protocol in Appendix D.1. Here, MM bounds the correction's standard deviation under πj\pi_j, and SS bounds its range (maximum minus minimum). Although we discuss estimating tighter radii from labeled discrepancies, note that we can justify our clinical choices from the score ranges alone. Since j,h,r∈[0,1]j,h,r\in[0,1], both corrections lie in [−1,1][-1,1], so

\qquad \operatorname{osc}(h-j),\operatorname{osc}(r-j)\leq2$$ Thus, $M=1$ and every $S\in[2,256]$ used in Appendix D.1 contain both rewards, even when the judge is badly miscalibrated. We choose $S$ from 16 independent pilot draws per encounter to target rejection acceptance; this uses sequence probabilities and does not require reference labels. For sycophancy, the true correction lies in $[-2,1]$, and each recalibrated correction is either zero or the true correction. The same argument therefore justifies $M=1.5,S=3$ (Appendix E.1). We will add this justification to Section 4.3 and make the practical choices more prominent. More generally, we agree that allowing a broader reward class can require more labels. We quantify this dependence through $Z_{\mathcal F}$. However, $M$ and $S$ also interact: once $M\geq S/2$, clipping determines the envelope everywhere, so increasing $M$ further has no effect. We already compare acceptance targets of 80% and 50%, with the latter corresponding to a larger $S$, in Appendix D.3 (Figure 9). At $T=64$, we obtain: | Query | Reference quality | Severe fabrications / 192 | |---|---:|---:| | Base | 0.589 | 29 | | Envelope, 80% target | 0.666 | 4 | | Envelope, 50% target | 0.630 | 11 | We therefore find that the improvement over base-query recalibration persists under both choices. We will highlight this comparison alongside the distinction between conservative radii that guarantee containment and heuristic estimates of tighter ones. ## Dependence on the recalibrator We agree with the reviewer that the recalibrator must be able to learn from its feedback. In fact, we explicitly account for this in Theorem 1 through the factor $\operatorname{Var}_{\rho}(h-r)$. However, note that this does not prevent us from justifying the choice of query independently of a particular recalibrator. As we discuss in Section 4.1 (lines 203–207), bounded rewards give a bounded error term, whereas coverage can be arbitrarily large. If $h,r$ take values in a common interval of width $L$, then $h-r\in[-L,L]$, and hence $$\operatorname{Var}_{\rho}(h-r)\leq L^2,\qquad J(\pi_h)-J(\pi_r)\leq\frac{\beta L^2}{2}C(\rho).$$ The variance bound holds for every query distribution, so we minimize this relaxed regret bound by choosing the envelope. A better recalibrator can further reduce the actual variance. We will add these inequalities after lines 203–207 to make the argument explicit. Furthermore, we can combine our result with a learning guarantee. For a finite realizable family $\mathcal H\subseteq\mathcal F$ whose rewards lie in the same interval, any consistent learner using $n$ independent noiseless labels from a query $\rho$ covering the base model's support satisfies, with probability at least $1-\delta$, $$J(\pi_h)-J(\pi_r)\leq\frac{\beta L^2 C(\rho)}{2n}\log\frac{|\mathcal H|}{\delta}.$$ We obtain this additional corollary by bounding disagreement mass with a union bound and applying Theorem 1. The learned reward can generalize from one example to many outputs, so this extends beyond the local replacement rule in Theorem 6. We will include the corollary with its assumptions. Theorem 1 also applies to the clinical editor without this additional learning assumption. In the clinical experiment, we hold GPT-5.6 Sol, the eight encounters, feedback budget, and candidate pool fixed. We require the editor to justify its changes using the supplied disagreements. With envelope feedback, we obtain rubrics assigning the minimum score to invented numeric vitals; with base-query feedback, we obtain rubrics that cap or reduce the score (Appendix D.1). Thus, even the same powerful editor makes different corrections depending on which examples we show it. We also demonstrate the sampling effect without an editor in sycophancy, where recalibration and optimization are exact (Section 6.2, Appendix E). We believe that studying both settings helps explain why the query matters, and that our formulation gives a useful starting point for analyzing other recalibration methods.

Thank you to the authors for their rebuttal. I have no further questions.

Reviewer_ZzVaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studied which outputs should receive ground-truth labels when an LLM judge is recalibrated before post-training against it. The paper proposes envelope sampling, a theoretically-grounded method for judge recalibration that seeks to minimize an upper bound on the regret of the post-trained model. Experiments on clinical note generation and on a controlled sycophancy task show that recalibrating on envelope samples mitigates reward hacking.

Critical strengths
  1. The problem is well-motivated and cleanly formulated: where to spend a small expert-labeling budget before optimizing against a cheap judge.
  2. The experiments document a real and important failure: on-policy labels miss rare outputs that the judge overrates, such as invented vital signs, and post-training then exploits them.
Critical weaknesses

The experiments do not show that the envelope itself is what helps. The main text compares only against on-policy sampling. In the clinical task, labeling samples from πjπ_j directly does at least as well. In the two-answer sycophancy task, the envelope is effectively uniform labeling, and its advantage over πjπ_j is modest (e.g., 8.9% vs 12.2% error at 128 draws under the 27B judge). As a result, the paper's central claim, that the minimax envelope is the right query, is not supported by the real-task experiments.

Questions for authors
  1. Can you add the judge-optimized query πjπ_j and a uniform-over-candidates query as main-text baselines? Can you show a real LLM task where the envelope's reweighting gives a significant gain over πjπ_j?

  2. How should M and S be chosen without access to h? Because the weights use raw sequence probabilities, is the envelope sensitive to response length or tokenization?

  3. In the clinical task, the envelope keeps quality near that of the untrained model (0.658 vs 0.651) rather than improving it. With 3 seeds, 16 evaluation encounters and an LLM reference as h, would physician labels on a subset confirm the result?

2 comments

We sincerely thank the reviewer for their helpful and constructive feedback. We have included our responses below -- we are happy to take further comments and request the reviewer to increase their scores if they feel that we have adequately addressed some of their concerns.

Comparison with πj\pi_j + uniform sampling

First, we would like to point out that we include the πj\pi_j baseline in Appendix F. We will move this comparison into the main text, as suggested. In the clinical task, we find that both envelope and πj\pi_j sampling substantially improve on sampling from the base model.

Although we do not observe an additional clinical gain from reweighting, note that πj\pi_j is itself a limiting case of our method: as S→0S\to0, the envelope reduces to πj\pi_j (Appendix F). Our motivation for reweighting is that πj\pi_j can still miss errors that matter after recalibration. For example, if the judge overvalues several modes differently, sampling from πj\pi_j may only reveal the most overvalued one. Correcting that mode leaves the others available for exploitation. We demonstrate this in Appendix F (Figure 13), where at β=2\beta=2 and 64 labels we obtain the following across all three support sizes:

QueryRegret
πj\pi_j>3.4>3.4
(π+πj)/2(\pi+\pi_j)/2>3.4>3.4
Envelope<0.2<0.2

Furthermore, we observe an improvement over πj\pi_j in controlled sycophancy (Section 6.2, actor: Qwen3.8-27B). With the 27B judge, we obtain the following existing results (Appendix F, Figure 12):

Label drawsπj\pi_j errorEnvelope error
12812.2%8.9%
2564.8%2.1%

We agree that our envelope is effectively uniform over the two answers. However, for open-ended generation, it is unclear what a uniform query should mean. There is no uniform distribution over all finite strings, and uniform token strings under a length cap are not useful language outputs. Selecting uniformly from a finite pool is possible, but depends on how we generate that pool: for an i.i.d. πj\pi_j pool, we recover πj\pi_j marginally. We instead sample from πj\pi_j and apply the rejection step of Proposition 4, giving a tractable query with optimal worst-case coverage.

Selection of MM and SS

We discuss parameter selection in Appendix B.3 and give our clinical protocol in Appendix D.1. Here, MM bounds the correction's standard deviation under πj\pi_j, and SS bounds its range (maximum minus minimum). Note that we can justify conservative choices without observing hh: since j,h,r∈[0,1]j,h,r\in[0,1], both corrections h−j,r−j∈[−1,1]h-j,r-j\in[-1,1], giving variance at most 11 and range at most 22. Thus, M=1,S≥2M=1,S\geq2 contain both rewards even if the judge is badly miscalibrated. We choose S∈[2,256]S\in[2,256] from 16 independent pilot draws per encounter to target 80% rejection acceptance, using only sequence probabilities. We also compare 80% and 50% targets in Appendix D.3 (Figure 9), and improve on base-query recalibration under both. We will bring this discussion into the main text and distinguish these conservative choices from heuristic estimates of tighter radii.

Response length and tokenization

We agree with the reviewer that sequence probabilities depend on length and tokenization. This is one reason clipping matters: we cap the reweighting at eβSe^{\beta S}. In the clipped regime, acceptance is [seβS+1−s]−1[s e^{\beta S}+1-s]^{-1} for s=πj(x)s=\pi_j(x), and approaches one when seβS≪1s e^{\beta S}\ll1. Therefore, making an already unlikely response even less likely does not increase its relative weight indefinitely. We do not claim tokenization invariance. In our downstream evaluation, we actually obtain shorter notes with envelope-query training at T=64,128T=64,128 (Appendix D.2, lines 1497–1499). We will clarify this dependence in the manuscript.

Clinical completeness and quality

We agree that quality stays close to the untrained model's. However, we would like to emphasize that we improve the judge's documentation-completeness score while preserving reference quality. We report both quantities in Table 2 and Figure 8:

ModelCompleteness (judge)Quality (reference)Severe fabrication
Untrained0.8300.6511.6%
Base query, T=128T=1280.8750.56319.8%
Envelope, T=128T=1280.8520.6581.0%

Thus, both runs improve what the judge rewards, but base-query training also introduces fabrications that reduce quality. With envelope feedback, we obtain the completeness-score improvement without this degradation. We observe the same behavior across seeds (Figure 7) and with an earlier model generation (Appendix G). Our reference is an LLM, and we agree that physician review would provide valuable independent validation. Still, we believe these results demonstrate the practical importance of choosing which examples to label before post-training.

Thanks for your response. My concerns are mostly solved. I raised my rating.

Comments on the paper

1 comment

We sincerely thank the area chair and reviewers for their helpful and constructive feedback. We address the shared concerns below and highlight results that we will make more prominent in the main text.

Choosing the uncertainty class

All three reviewers ask how to choose the radius without knowing the true reward. We discuss parameter selection in Appendix B.3 and give our clinical protocol in Appendix D.1. Here, MM bounds the correction's standard deviation and SS its range. Note that our clinical score ranges already justify conservative choices: j,h,r∈[0,1]j,h,r\in[0,1] imply corrections in [−1,1][-1,1], with variance at most 11 and range at most 22. Thus, M=1,S≥2M=1,S\geq2 contain both rewards even when the judge is badly miscalibrated. We choose SS from independent pilot sequence probabilities to target rejection acceptance, without reference labels.

We already compare 80% and 50% acceptance targets in Appendix D.3 (Figure 9), improving on base-query recalibration under both. We will explain more clearly when the score ranges suffice and when we use pilot data to choose tighter radii.

Calibration accuracy and the role of the recalibrator

Reviewers 2 and 3 ask whether the recalibrator's ability limits the improvement. We agree, and account for its error explicitly in Theorem 1. However, we already explain in Section 4.1 (lines 203–207) that bounded rewards give bounded error variance, whereas coverage can be arbitrarily large. If h,rh,r lie in a common interval of width LL, then Var⁡ρ(h−r)≤L2\operatorname{Var}_{\rho}(h-r)\leq L^2 for every ρ\rho, so choosing the envelope minimizes the resulting bound βL2C(ρ)/2\beta L^2 C(\rho)/2.

We can also combine our result with a learning guarantee. For a finite realizable family H⊆F\mathcal H\subseteq\mathcal F in this interval, any consistent learner using nn independent noiseless labels from a query covering the base model's support satisfies, with probability at least 1−δ1-\delta,

J(πh)−J(πr)≤βL2C(ρ)2nlog⁡∣H∣δ.J(\pi_h)-J(\pi_r)\leq\frac{\beta L^2 C(\rho)}{2n}\log\frac{|\mathcal H|}{\delta}.

We will include this corollary in the paper. More generally, we would like to emphasize why we study calibration through subsequent optimization. A judge can agree with the reference on typical outputs and still make rare errors that optimization exploits. To our knowledge, we are the first to formulate the choice of calibration examples as a minimax coverage problem for subsequent KL-regularized optimization, and to derive a practical sampler from this formulation.

Comparison with πj\pi_j and uniform sampling

Reviewers 1 and 3 ask us to emphasize the πj\pi_j baseline; we already include this in Appendix F but will move the comparison into the main text for clarity. Note that the envelope reduces to πj\pi_j as S→0S\to0; reweighting helps cover errors that πj\pi_j still treats as rare. We obtain:

Comparisonπj\pi_jEnvelope
Clinical quality, T=64T=64 (Table 5)0.6800.666
Sycophancy error, 128 draws, 27B judge (Figure 12)12.2%8.9%
Heterogeneous regret, β=2\beta=2, 64 labels (Figure 13)>3.4>3.4<0.2<0.2

The last row holds across all three supports. Although we see no additional clinical gain over πj\pi_j, the heterogeneous experiment shows why correcting only the most overvalued mode leaves other errors available for exploitation. In two-answer sycophancy, the envelope is effectively uniform. For open-ended text, uniform selection from a finite pool depends on its construction: an i.i.d. πj\pi_j pool recovers πj\pi_j marginally. There is no uniform distribution over all finite strings. We would appreciate specific references for the active-learning baselines Reviewer 3 recommends.

Interpreting the clinical results

We would like to emphasize that preserving reference quality does not mean that our post-training has no effect. We improve the original judge's documentation-completeness score, as does base-query training. The difference is that base-query training also introduces substantial fabrication (Table 2, Figure 8):

ModelCompleteness (judge)Quality (reference)Severe fabrication
Untrained0.8300.6511.6%
Base query, T=128T=1280.8750.56319.8%
Envelope, T=128T=1280.8520.6581.0%

We observe the same reward hacking and improvement across seeds and with an earlier model generation. Our reference is an LLM, and physician review would provide independent validation. We will expand the clinical prompts, preference data, and training duration, and compare rubric editing with direct preference training using the same label budget. We also demonstrate the sampling benefit under exact scalar-reward optimization in our controlled tasks. We agree that length and tokenization can affect the query; clipping limits how much we upweight rare outputs. We believe these results show why choosing which outputs to label matters for making a judge safer to optimize.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.