acceptodds
Under review as a conference paper at ICLR 2027

RLHF without Reward Clipping: Near-Optimal Distortion at Every KL Budget

Abstract

Reinforcement learning from human feedback (RLHF) fits a single reward model to comparisons from users with heterogeneous preferences and then optimizes a policy under a Kullback–Leibler (KL) constraint. Distortion measures the ratio between the optimal average user utility and that attained by the learned policy. Gölz et al. (2025) show that RLHF can suffer distortion exponential in the Bradley–Terry temperature β, and Oko et al. (2026) trace this growth to distribution mismatch between the preference data and the KL reference policy. Without such mismatch, they prove distortion linear in β, which is the optimal order. However, their guarantee at finite KL budgets relies on clipping the fitted reward, and the best known bound for the unclipped reward is quadratic in β. We show that unclipped RLHF attains this optimal order. Without distribution mismatch and with the reward fitted to population comparisons, unclipped RLHF has worst-case distortion at most C₀κ(β) at every KL budget, where C₀ < 2.526 and κ(β) = β/[2 tanh(β/2)] is the known algorithm-independent lower bound. The main difficulty is that a KL-constrained policy depends on reward gaps as well as reward rankings. We address it by comparing reward maximization with win-probability maximization inside the same KL ball. In population simulations on MovieLens and KuaiRec, the unclipped reward has mean relative utility loss below 0.15% over each dataset's evaluation grid and lower mean loss than the clipping rule of Oko et al. (2026). These results indicate that clipping is also unnecessary on the evaluated user profiles.

Primary area:
learning theory
Keywords:
reinforcement learning from human feedback, AI alignment, preference learning, utilitarian distortion, social choice, KL-constrained optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZRcxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the utilitarian distortion of RLHF in the population setting of Gölz et al. (2025) and Oko et al. (2026). Users have heterogeneous utilities in [0,1]m[0,1]^m and give Bradley–Terry comparisons with temperature β\beta. A single BT reward is fitted to the exact population comparison probabilities, and the policy maximizes this reward inside a KL ball around a reference μ\mu. The comparison distribution equals the KL reference (no distribution mismatch). Prior work obtains optimal-order (O(β)O(\beta)) distortion at finite KL budgets only for a clipped reward; the best known general bound for the unclipped reward is quadratic in β\beta.

Critical strengths

1. A clean, optimal-order guarantee for the objective practitioners actually optimize.

Removing clipping from the linear-in-β\beta guarantee closes a gap explicitly left open by Oko et al. (2026), and addresses the reward-magnitude obstacle highlighted by Gölz et al. (2025).

2. A reusable technical idea.

The proof never bounds reward magnitudes. It uses the reward only through the exponential tilt and the inequality 4Y≥Eqr−J(1)4Y\ge \mathbb E_q r - J(1), and works on the bounded win-score scale everywhere else.

Critical weaknesses

1. The guarantee holds in a strongly idealized setting, and the paper does not examine where it breaks.

The analysis assumes a single prompt, a tabular reward with one free parameter per alternative, a reward fitted to exact population comparison probabilities, and exact equality between the comparison distribution and the KL reference. The first two are inherited from Gölz et al. (2025) and Oko et al. (2026), so I do not count them against this paper. The last two, however, are precisely the conditions under which clipping was introduced, and the paper does not discuss whether its main message survives without them.

  • Finite samples. Appendix C notes that the proof never bounds reward magnitudes and relies only on win scores lying strictly inside (−1/2,1/2)(-1/2, 1/2). With finitely many comparisons this fails: an alternative that wins every comparison it appears in has an empirical win score of exactly 1/21/2, the Bradley–Terry MLE does not exist (separation), and the unclipped reward is undefined. Even short of separation, near-boundary win scores produce very large fitted rewards that a binding KL constraint would amplify. This is a concrete mechanism by which clipping may be statistically necessary even when it is unnecessary at the population level. The conclusion lists finite samples as future work, but given that the title and the clipping comparison are framed around "unnecessary", a brief discussion of this failure mode belongs in the paper.

  • Reference mismatch. Corollary 5 is itself a mismatch result (reward fitted from μ⊗μ\mu \otimes \mu, optimized around ν\nu), with a factor R2R^2 that is unbounded as the density ratio spreads. The paper does not say whether R2R^2 is tight, nor how it relates to the exponential lower bound of Gölz et al. (2025, Theorem 6) for RLHF under mismatch. Since the exponential growth in β\beta is exactly the phenomenon Oko et al. traced to mismatch, placing Corollary 5 relative to that lower bound seems necessary to understand what the corollary buys.

The scope qualifiers in the abstract and conclusion are adequate; the title "at every KL budget" is what overstates.

2. The experiments occupy a large share of the paper but provide little evidence for or against the theory.

  • Headline means are diluted. Of the ten budgets, τ∈{4,8,∞}\tau \in \{4, 8, \infty\} are saturated for panels with 5–20 candidates (saturation occurs around log⁡5\log 5–log⁡20≈1.6\log 20 \approx 1.6–33) and τ∈{10−4,10−3}\tau \in \{10^{-4}, 10^{-3}\} are essentially the reference, so roughly half the grid contributes near-zero loss by construction. The reported 0.005% / 0.07% means should be read accordingly. Figure 5 does isolate the binding-budget regime, which I appreciate, but the peaks there (about 0.02% and 0.5%) are still far too small to distinguish the linear bound from the quadratic one of Oko et al. The honest summary is that the evaluated preference profiles are too homogeneous to exercise the theorem, which is a limitation of the datasets rather than of the method, and the paper should say so.

  • The clipping baseline is a proof device, not a practical rule. The interval c≤1/316c \le 1/316 is the sufficient range of Oko et al.'s theorem, not a value anyone would use; the paper itself calls clipping a "technical device" in Appendix J. Losing utility against it is therefore expected and uninformative, and at c=1c = 1 the gap largely closes (0.029% vs. 0.006% on MovieLens). To the paper's credit, Table 2 shows that the ordering is robust to tie-breaking (clipping loses even under utility-optimal tie selection and on settings with a unique clipped optimum), so the conclusion is not fragile — it is just not very interesting. I would shorten this comparison substantially and state its purpose more modestly.

  • The released-model experiment measures no utility. Section 6.2 reports total variation between the reward and win-score policies on HelpSteer but never connects it to utility or distortion, so it only shows that the allocation gap in Lemma 2 is nonzero on real reward models. HelpSteer ships with human-rated attribute scores (helpfulness, correctness, etc.) that could serve as a utility proxy; computing distortion against those would turn this section into an actual test of the paper's claim.

None of this undermines the theory, which is the paper's real contribution. But Sections 6.1–6.2 and Appendices I–L could be cut by half without losing information.

Questions for authors
  1. The paper describes the existing unclipped finite-budget bound of Oko et al. (2026, Thm. 4.4) as "quadratic in β\beta" and says linear bounds there require "an additional condition on the utility distribution", without stating either precisely. Please state the exact bound and the exact condition. This would let readers judge the delta: is the improvement solely the removal of that condition, or also a better constant or wider scope?

  2. Is the R2R^2 factor in Corollary 5 tight in order? Can you construct instances where distortion grows with RR (as RR or as R2R^2)? Corollary 5 is itself a mismatch setting (reward fitted from μ⊗μ\mu \otimes \mu, optimized around ν\nu), so how does it interpolate toward the exponential-in-β\beta lower bound of Gölz et al. (2025, Thm. 6) as RR grows?

  3. Does the argument extend beyond the exact population reward? Two cases matter:

    • Bounded misspecification r^=r+e\hat r = r + e with ∥e∥∞≤δ\|e\|_\infty \le \delta. Does the bound degrade gracefully, or can the error be amplified by the exponential tilt at intermediate budgets in a way that clipping prevents?
    • Finite samples. With finitely many comparisons an alternative can win every comparison it appears in, its empirical win score is exactly 1/21/2, and the Bradley–Terry MLE does not exist. The unclipped objective is then undefined, which no bounded-δ\delta analysis captures. How should the unclipped policy be defined in this case, and does clipping play a role here that it does not play at the population level?

    If clipping retains a statistical role in either case, the title and the framing of the clipping comparison should be adjusted (the scope qualifiers in the abstract and conclusion are already adequate).

  4. For the real-profile simulations, Figure 5 reports mean loss at interior budgets. Could you also report the worst-case and upper-quantile (e.g. 90th/99th percentile) distortion restricted to interior (binding) budgets, across panels, references and temperatures? The grid means and the maxima attained at saturated budgets do not describe the regime the theorem targets.

  5. For the released reward models (Section 6.2), can the observed TV gaps be related to a utility or welfare proxy, for instance the HelpSteer attribute scores (helpfulness, correctness, etc.) that ship with the dataset? Otherwise, what conclusion should readers draw from this experiment beyond the fact that tilts depend on reward gaps?

  6. Is there a clipping rule used in an actual RLHF pipeline that would serve as a more meaningful baseline than c≤1/316c \le 1/316, which the paper itself calls a technical device?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZRcx

Thank you for the detailed review.

Our stability bound extends the population guarantee to sufficiently accurate fitted rewards at every KL budget. Comparison candidates are drawn from the KL reference, as in Oko et al.'s (2026) no-mismatch setting. Smoothing handles separation, and sampled-comparison loss approaches the population benchmark.

Q1. Exact improvement over Oko et al.'s unclipped bounds.

We improve the general unclipped guarantee from quadratic to linear in β\beta without a utility-mass condition. With matching references, our bound is C0κ(β)C_0\kappa(\beta), C0<2.526C_0<2.526, for all β>0\beta>0 and every maximizer at every KL budget. Oko et al.'s Theorem 4.4 gives C3β2+4C_3\beta^2+4 for β≥1\beta\ge1 and positive budgets. Their Theorem A.15 gives 16(c3+16)β/c716(c^3+16)\beta/c^7 under Pr⁡u∼D,x∼μ[βu(x)>c]≥c2,0<c≤1.\Pr_{u\sim\mathcal D,x\sim\mu}[\beta u(x)>c]\ge c^2,\qquad 0<c\le1.

Q2. Is R2R^2 necessary, and how does it relate to Gölz et al.?

We prove a C0R2κ(β)C_0R^2\kappa(\beta) upper bound at binding budgets, with tightness in RR still open. At reward saturation, it sharpens to C0κ(β)C_0\kappa(\beta), independent of RR, including infinite budget.

For fixed RR, our bound is linear in β\beta. In Gölz et al. (2025, Theorem 6), R=(m−2)(1−ε)/(2ε)R=(m-2)(1-\varepsilon)/(2\varepsilon) grows at least as exp⁡(Ω(eβ))\exp(\Omega(e^\beta)). Thus their exponential-in-β\beta lower bound is compatible with ours, without proving R2R^2 necessary.

Q3. Fitting error, finite-sample separation and clipping.

Small fitting error has a controlled effect at every KL budget. Smoothing handles separation. With matching comparison/KL reference μ\mu, define

P=\mathbb E_{u\sim\mathcal D,y\sim\mu}[\sigma(\beta u_y)-1/2],$$ where $r$ is the population reward and $\operatorname{osc}(e)=\max e-\min e$. If $\delta<2P$, every maximizer of a finite fitted reward $\hat r$ satisfies $$\mathrm{Dist}(\hat q)\le C_0\kappa(\beta)\frac{P+\delta/2}{P-\delta/2}.$$ The multiplier tends to one as error vanishes. Your sup-norm assumption $\max_x|\hat r_x-r_x|\le d<2P$ suffices, with $d$ replacing $\delta$. Specifically, we add 0.5 per pair/direction and maximize the resulting reward in the KL ball. We record ordinary MLE separately. Across 32,000 comparison datasets on 100 panels, ordinary MLE is absent in 46.7% of samples at $5m$ and none at $500m$, where $m$ is the candidate count. One further $5m$ sample has a finite, non-identifiable MLE. |$N/m$|5|20|100|500| |---|---:|---:|---:|---:| |Smoothed mean relative loss (%)|9.4|4.7|1.4|0.43| |$\delta<2P$ met (%)|43.3|60.6|92.4|99.96| Larger samples reduce loss toward the 0.08% population benchmark and increase bound applicability. The $5m$ mean uses 7,967 of 8,000 estimates, excluding 33 unresolved near-tie policy evaluations. Clipping can help individual fits: at $c=1$, utility improves in 13.3%–34.4% of computable settings. Bounding the unresolved cases gives a full-sample $5m$ mean change of [−0.11,+0.03] percentage points, so its sign is undetermined. For $N\ge20m$, clipped-minus-smoothed mean utility changes are −0.05 to −0.12 points, with paired 95% intervals below zero. Our revised title is **Linear Distortion of Unclipped Population RLHF**. **Q4 / W2. Loss tails at binding budgets and experimental focus.** Binding-budget distortion peaks at about 1.004 on MovieLens and 1.05 on KuaiRec. We pool panels, references and temperatures where both reward-policy and utility-oracle KL constraints bind and $\tau\ge0.01$. For fractional loss $\ell$, $\mathrm{Dist}=1/(1-\ell)$: |Data|P90 (%)|P99 (%)|Maximum (%)| |---|---:|---:|---:| |MovieLens|1.6×10⁻²|6.9×10⁻²|0.36| |KuaiRec|0.35|1.06|4.80| The table shows binding-budget losses far below both worst-case bounds. The proof and rare-alternative construction establish the linear order. We shortened discussion of Oko's theoretical $c\le1/316$ truncation. Wider thresholds reduce this population clipping gap. **Q5. Do HelpSteer TV gaps imply changes in human-rated utility?** Small signed means partly reflect cancellation across prompts. On all 800 HelpSteer answers, raw-versus-win-score mean helpfulness/correctness changes are below 0.06 percentage points. Mean absolute per-prompt changes are 0.35–0.37 points for Llama and about 0.10 for Qwen. Three of four 95% prompt-bootstrap intervals include zero. Qwen helpfulness favors raw by 0.03 points [0.005,0.052]. Raw policies are 4.9–5.7 points below same-budget attribute oracles. We average seven interior budgets per prompt before either summary. **Q6. Is there a clipping baseline from a real RLHF pipeline?** Yes. We test OpenRLHF's default [−10,10] clamp on frozen native scores in the same KL balls. It leaves Qwen unchanged and lowers Llama correctness (mean change −0.15 points, 95% CI [−0.26,−0.05]). Helpfulness changes by −0.12 points [−0.25,+6.3×10⁻⁴], spanning zero. Some prompts benefit: 26% for correctness and 27.5% for helpfulness.
Reviewer_jBacOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

To analyze the limits of unclipped RLHF, this submission studies a procedure that fits a single reward function to population comparisons from users with different utilities and then maximizes the expected fitted reward subject to a KL constraint. Distortion is the ratio of the best achievable average user utility to the average user utility attained by the resulting policy, with both policies subject to the same constraint. When the outputs used in comparisons are sampled from the reference distribution in the KL constraint, Theorem 1 improves the general distortion bound for unclipped RLHF from quadratic to linear dependence on the Bradley–Terry temperature, at every KL budget. The paper further provides lower-bound constructions and a distortion bound for using a different KL reference distribution.

Critical strengths
  • Theorem 1 improves unclipped bound to C0κ(β)C_0\kappa(\beta), with C0<2.526C_0<2.526, at every KL budget. At every positive budget, this is within a constant factor of the known lower bound κ(β)\kappa(\beta). It establishes the optimal order of temperature dependence for the unclipped population objective without requiring an additional assumption on the utility distribution beyond the stated model.

  • Lemma 2 compares reward maximization with win-score maximization within the same KL ball. Because it applies to any finite reward vector and accounts for the different probability allocations caused by reward gaps, Lemmas 3 and 4 can connect this comparison to utility without introducing a second factor of κ(β)\kappa(\beta).

  • Proposition 6 provides a three-alternative construction approaching the lower-bound order at every fixed positive finite KL budget. Additionally, Proposition 10 yields an explicit finite instance with distortion approximately 60.94 at β=64\beta=64 and τ=8\tau=8, exceeding 1.9κ(64)1.9\kappa(64). This provides a concrete example of substantial distortion for unclipped RLHF, while NLHF attains distortion one on the same instance.

Critical weaknesses
  • While Theorem 1 establishes a linear distortion bound, it assumes an exact maximizer of the unrestricted population likelihood in Equation 2 and uses the resulting identity in Equation 9. Because the analysis omits any bound on the effect of reward-fitting error or a restricted reward class, the conclusion regarding the removal of clipping applies to the population objective and does not establish the same guarantee for practical rewards estimated from finitely many comparisons.

  • The bound in Corollary 5 introduces a factor of R2R^2, but the analysis does not establish whether this dependence is necessary.

  • MovieLens and KuaiRec use utility proxies and exact comparison probabilities generated from the assumed BT model, with the selected panels containing users with complete observations of popular items. Section 6.2 reports total variation between qrq_r and qsq_s on released reward models, with win scores computed from the same reward outputs, without measuring user utility. These experiments support the population analysis but do not establish the effect of clipping rewards learned from finitely many comparisons.

  • The practical significance of the improved guarantee needs clearer justification. The average-utility objective and distortion framework are inherited from prior work, and the paper’s main contribution is a sharper bound within that framework. The authors should clarify what practical conclusions follow from removing clipping from the population guarantee, beyond closing the quadratic-to-linear gap in the existing analysis.

  • Some sentences are difficult to follow and should be revised for clarity. I would ask the authors to revisit the main text and improve the flow of the presentation.

Questions for authors
  • Can Lemma 3 be extended when Equation 9 holds only approximately, and what conditions would allow the resulting error to be incorporated into the distortion bound?

  • Is R2R^2 in Corollary 5 necessary, or can the guarantee be strengthened?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jBac

Thank you for your positive and insightful comments.

Q1 / W1. Does approximate Equation 9 preserve the guarantee for restricted rewards?

Yes. The distortion guarantee extends through a controlled induced-score residual. With matching comparison/KL reference μ\mu, let rr be the population reward and r^\hat r any finite fitted reward. Define sx=∑yμy[σ(rx−ry)−1/2]s_x=\sum_y\mu_y[\sigma(r_x-r_y)-1/2] and s^\hat s analogously from r^\hat r, with logistic σ\sigma. Put

P=\mathbb E_{u\sim\mathcal D,y\sim\mu}[\sigma(\beta u_y)-1/2].$$ If both score vectors are at most $1/2-\gamma$, where $0<\gamma\le1/2$, define $L_\gamma=[4\gamma(1-\gamma)]^{-1}$. For $L_\gamma\eta<P$, $$\mathrm{Dist}(\hat q)\le C_0\kappa(\beta)\frac{P+\eta}{P-L_\gamma\eta}.$$ Every maximizer $\hat q$ of $\hat r$ satisfies this bound at every KL budget, including zero and saturation. Here $C_0<2.526$ and $\kappa(\beta)=\beta/[2\tanh(\beta/2)]$. Thus finite rewards from restricted classes retain a controlled guarantee when their induced scores satisfy this condition. The margin controls sensitivity near score $1/2$. Alternatively, bounded reward error gives a guarantee uniform over KL budgets. With $\delta=\tfrac12\operatorname{osc}(\hat r-r)<2P$, where $\operatorname{osc}(e)=\max e-\min e$, the multiplier on $C_0\kappa(\beta)$ is $(P+\delta/2)/(P-\delta/2)$. Both multipliers approach one as fitting error vanishes. **Q2 / W2. Is the $R^2$ mismatch factor necessary?** We prove a $C_0R^2\kappa(\beta)$ upper bound at binding budgets, with tightness in $R$ still open. At reward saturation, it sharpens to $C_0\kappa(\beta)$, independent of $R$, including infinite budget. For fixed $R$, our bound is linear in $\beta$. In Gölz et al. (2025, Theorem 6), $R=(m-2)(1-\varepsilon)/(2\varepsilon)$ grows at least as $\exp(\Omega(e^\beta))$. Thus their exponential-in-$\beta$ lower bound is compatible with ours, without proving $R^2$ necessary. **W3. What do the experiments show about utility and finite-sample clipping?** The small signed human-attribute effects partly reflect cancellation across prompts. For finite-sample rewards, clipping helps some fits, while lowering mean utility at the tested $N\ge20m$ levels. On all 800 original HelpSteer answers, raw-versus-win-score helpfulness/correctness differences are below 0.06 percentage points. Three of four 95% prompt-bootstrap intervals include zero. Raw policies remain about 4.9–5.7 points below same-budget attribute oracles. After averaging seven interior budgets per prompt, mean absolute changes are 0.35–0.37 points for Llama and about 0.10 for Qwen. Separately, we tested estimation noise on MovieLens and KuaiRec utility proxies. With $m$ candidates, mean loss falls from 9.4% at $5m$ comparisons to 0.43% at $500m$, toward the 0.08% population benchmark. We use 32,000 datasets on the original 100 complete-observation panels of popular items, with BT probabilities generated from ratings/watch ratios. Adding 0.5 per pair/direction gives the finite smoothed reward maximized in the KL ball. Ordinary MLE is recorded separately: it is absent in 46.7% of samples at $5m$, with one further finite but non-identifiable sample. For these finite estimates, clipping at $c=1$ improves utility in 13.3%–34.4% of computable settings, while lowering mean utility at the tested $N\ge20m$ levels. The $5m$ mean uses 7,967 of 8,000 estimates. Bounding the other 33 estimates with unresolved near-tie policy evaluations gives a full-sample mean clipping effect of [−0.11,+0.03] percentage points, so its sign is undetermined. For $N\ge20m$, clipped-minus-smoothed mean utility changes are −0.05 to −0.12 points, with paired 95% intervals below zero. **W4. What does removing clipping from the population guarantee mean in practice?** With matching references, fitted rewards satisfying our error condition can be optimized directly with a controlled distortion guarantee, without additional reward truncation. Q1 quantifies how fitting error changes the bound. In our finite-sample experiment, $\delta<2P$ holds for 99.96% of smoothed estimates at $500m$, versus 43.3% at $5m$. All 135,600 applicable ordinary-MLE/smoothed policy checks satisfy the bound. These instance-level results link the sufficient condition to fitted-reward accuracy. **W5. Improving difficult sentences and the flow of the main text.** Thank you for pointing out the difficulty of following the presentation. We have revised the main text to guide readers through the question, the reward-to-win-score mechanism, the resulting utility guarantee and its empirical meaning. In doing so, we clarified the comparison with prior bounds, shortened repeated explanations, and made the transitions between the population result, finite-sample analysis and experiments explicit.

I thank the authors for their detailed response. I maintain my positive recommendation.

Reviewer_Zn3COpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper shows that RLHF with unclipped has linear distortion. Experimental validation shows that the tools that Oko et al used to fix the unbounded distortion are not practical.

Critical strengths

The paper's technique relaxes reward clipping used by Oko et al. (20) provides an interesting way of bounded the mismatch and reference ball. The MovieLens experiment shows that Oko's rule can hurt total reward.

Critical weaknesses

Finite sample noise for reward estimate is not considered.

Questions for authors

In the LLama vs Qwen experiments, what explain the large gap between the two models? Is the shape of Figure 3 unsurprising?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Zn3C

Thank you for the review. With matching comparison and KL-reference distributions, sufficiently small reward-fitting error changes the population guarantee by a controlled factor that approaches one as the error vanishes. Sampled-comparison experiments show utility loss approaching the population benchmark as sample size grows. The Llama–Qwen TV gap is largely explained by reward scale: matching mean reward spans reduces it by about 96.7%. We agree that Figure 3's broad shape follows from the zero-budget and saturation endpoints. Our main result improves Oko et al.'s (2026) general unclipped bound from quadratic to linear in β\beta in the same population model.

W1. How does finite-sample reward noise affect the result?

With matching comparison and KL-reference distributions, sufficiently small fitting error increases the bound by a controlled factor that tends to one as the error vanishes. Our experiments show utility loss decreasing toward the population benchmark as comparison counts grow. We draw 32,000 comparison datasets on the original 100 utility panels, with 8,000 datasets at each sample size. A fixed pseudocount of 0.5 per pair/direction defines a finite smoothed reward, which we maximize in the same KL ball. For mm candidates:

Comparisons N/mN/m520100500
Smoothed mean relative utility loss (%)9.44.71.40.43
Ordinary MLE absent (%)46.74.10.040

As the table shows, increasing the comparison count reduces utility loss toward the population mean of about 0.08% on the same budgets and makes ordinary-MLE failure rarer. One additional 5m5m sample has a finite but non-identifiable ordinary MLE. At 100m100m, the 0.04% absence rate corresponds to 3 of 8,000 samples. The 5m5m utility mean uses 7,967 of 8,000 estimates with resolved policy evaluations.

The stability analysis explains how fitting accuracy preserves the guarantee. For matching comparison/KL reference μ\mu, let rr be the population reward, r^\hat r a finite estimate, δ=12[max⁡(r^−r)−min⁡(r^−r)]\delta=\tfrac12[\max(\hat r-r)-\min(\hat r-r)], and P=Eu∼D,y∼μ[σ(βuy)−1/2]P=\mathbb E_{u\sim\mathcal D,y\sim\mu}[\sigma(\beta u_y)-1/2], with uu drawn from the utility population and logistic σ\sigma. If δ<2P\delta<2P, the multiplier on the population distortion bound is (P+δ/2)/(P−δ/2)(P+\delta/2)/(P-\delta/2) at every KL budget. This condition holds for 99.96% of smoothed estimates at 500m500m. An induced-score residual and margin condition gives a second route for finite fits from restricted reward classes. Thus the theory links fitting accuracy to the sample-size trend above.

Q1. Why does the allocation gap differ between Llama and Qwen?

The observed between-model TV gap is largely explained by reward scale: matching either mean reward span reduces the gap by about 96.7%. We rescale each model once by the ratio of the other model's mean within-prompt reward span to its own. Each intervention uses all 200 original prompts, four candidates and seven interior KL budgets.

Reward scalesLlama mean TV (%)Qwen mean TV (%)
Native2.390.65
Both matched to Qwen's mean span0.710.65
Both matched to Llama's mean span2.392.33

As the table shows, either common scale reduces the between-model gap from 1.74 to about 0.06 percentage points. Positive scaling preserves the raw-reward policy, while the nonlinear win-score map changes the allocations. This supports the role of scale in the observed gap and complements the within-model span–TV correlations of 0.69 and 0.79.

The small signed utility effects partly reflect cancellation across prompts. At native scale, mean raw-versus-win-score changes are below 0.06 percentage points. After averaging seven interior budgets per prompt, mean absolute changes are 0.35–0.37 points for Llama and about 0.10 for Qwen. These raw-versus-win-score comparisons use helpfulness/correctness annotations on all 800 original answers, with three of four 95% prompt-bootstrap intervals including zero. Qwen helpfulness favors raw by 0.03 points [0.005,0.052].

Q2. Is the shape of Figure 3 unsurprising?

Yes. At zero budget, both policies equal the reference. At saturation, reward and induced win score share the same maximizers because they preserve rankings. These endpoints explain the broad rise-and-fall shape.

Accordingly, the empirical finding concerns the size and consequences of the interior gap: Figure 3 quantifies its variation across prompts, and the added attribute evaluation shows that visibly different allocations coexist with small average utility effects.

Difficult to follow despite careful reading.

Thank you for pointing this out. We have revised the whole paper to make it clearer and easier to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.