acceptodds
Under review as a conference paper at ICLR 2027

LLM Alignment with Biased Proxy Preferences under Heterogeneous Utilities

Abstract

Aligning a language model in a specialized domain such as medicine raises the question of whose judgment should count. Expert comparisons are on target but scarce, a general population's are plentiful but systematically biased, and pooling the two learns a utility between the two populations. We ask when the crowd's comparisons still help, which way of using them helps most, and why. Under heterogeneous utilities, we propose a two-headed estimator (THE) of the direction of the experts' utility center from a small gold and a large proxy sample, which fits one utility per population with the sign loss and couples the two through a penalty on their difference; re-weighting the pooled samples is its infinite-penalty limit. We prove that proxy comparisons speed up estimation while their bias is below the statistical error of pooling, and that once the bias exceeds the gold-only error the gain of any re-weighting over gold-only vanishes as the bias grows. For controlled comparisons, which vary one group of criteria at a time, with few contentious criteria and precisely estimated consensus criteria, a variant of THE has a smaller asymptotic error than every re-weighting. With a smoothed loss, a finite penalty improves on the best re-weighting whenever the bias lies, on average, in more curved directions of the gold loss than the error of the best re-weighting. For a bias sparse in of features, THE's two-step special case with an penalty reduces the gold sample size that our sign-loss bounds require by a factor of order , under local conditions. In simulations with the exact sign loss, THE improves on gold-only by 15% at a bias twice the gold-only error, where the best re-weighting improves by 3%; on real answer embeddings with simulated preferences, it improves on the best re-weighting by up to 20% when the bias lies along high-variance features.

Primary area:
learning theory
Keywords:
Alignment, RLHF, Bradley-Terry-Luce model, Penalized Estimation, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_txHkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how abundant, biased proxy preferences can improve estimation of a small gold population's utility direction under heterogeneous annotator preferences. Its two-headed estimator (THE) fits separate population directions and penalizes their difference, borrowing less along directions where gold data are precise.

The main contributions are cube-root limits and baseline lower bounds for exact sign loss, a curvature criterion for improvement under smoothing, and local sparse-correction guarantees. Synthetic experiments support the proposed mechanism. Real-label results are more qualified: natural-population shifts generally produce a tie with re-weighting, while stronger gains appear with reward-model or rule-based relabeling.

Critical strengths
  • Clear problem formulation: The paper distinguishes within-population heterogeneity from between-population bias.
  • Substantial theory: The main novelty lies in analyzing transfer with discontinuous preference losses, rather than the coupling architecture itself.
  • Careful validation: Cross-fitted oracle tuning, isotropic controls, smoothing corrections, and grid sensitivity checks strengthen the synthetic evidence.
  • Transparent reporting: Negative results and limitations are disclosed, and the headline numerical claims agree with the tables.
Critical weaknesses

W1: Practical tuning can cause substantial negative transfer. Under large proxy bias, the implementable rule produces errors 10.6%-20.0% worse than gold-only at three-quarter relabeling and 44.5%-98.4% worse at complete relabeling. These disclosed failures limit practical applicability without invalidating the oracle theory. Highlight them in the main text and evaluate a reliable gold-only fallback.

W2: Real-label evaluation differs from the theoretical target. Headline experiments use ridge logistic likelihood and measure angle to a held-out logistic fit. Because that fit can be inconsistent under heterogeneous preferences, these results do not establish recovery of the gold utility center. Smoothed-sign reruns retain the same reference and often show smaller gains. Add held-out preference performance and sensitivity to references fitted with the same loss.

W3: The informal theorem omits an assumption. Informal Theorem 2 attributes small-bias improvement to reverse ordering of curvature and noise. Formal Theorem 6 additionally requires the nonzero smoothing offset to point along the least-curved direction. Include this condition in the introduction. This is a confirmed statement mismatch, not a failure of the formal theorem.

W4: An auxiliary proof needs clarification. Theorem 13 assumes a localization rate while allowing curvature to diverge, but Lemma 6's bound also depends on c_max * r^2 / c_q. Explain how changing design constants remain controlled, or add sufficient growth conditions. This is an unresolved proof-completeness question concerning an appendix upper-bound comparison.

W5: Small real-label gains need stronger calibration. Half-sample reference fits disagree by 15-35 degrees, and joint-bootstrap intervals are median-recentered. Check coverage or sensitivity to standard interval constructions. Smoothed-sign experiments also encountered local minima 30-52 degrees apart before selecting a bandwidth and using one warm-started path; multiple-start diagnostics would strengthen confidence.

W6: Reproduction requires more complete specifications. The PDF omits the synthetic tilt coefficient and exact real-label grids and split sizes. Supplementary code is promised but was unavailable for this review. Provide a configuration manifest and clarify whether Appendix K.6's abstract matrix regimes are realizable under the original preference model.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer txHk (1/3): W1, W2

Thank you for a precise review. We accept W1, W3 and W4 as stated; the corresponding changes are given below. W2, W5 and W6 are answered with new analyses.

W1. Negative transfer under the implementable rule; a gold-only fallback. Agreed on both requests. The failure belongs in the main text, next to the oracle results and together with the fallback evaluated below. Its cause is the risk estimate of the implementable rule, which is too optimistic when the bias is large: gold-only (weight zero) was in its grid and was not selected.

We evaluated a gold-only fallback (Table R2 in general comment (3/4)) and state it as a two-stage algorithm (cross-validation, then fallback).

Algorithm 1THE with a gold-only fallbackInput: gold and proxy comparisons; ridge values K of gold-only; tuning values G of THEStage 1 (cross-validation). Split the gold annotators into five folds.for each fold k and each candidate c∈K∪G dofit c on the other four folds, and on all proxy comparisons if c∈Grecord its logistic loss on fold kL(c)←mean loss of c over all gold comparisonsStage 2 (fallback). Let g and t minimize L over K and over G.if L(t)<L(g) then c^←t else c^←gFinal fit. Refit c^ on all gold comparisons, and on all proxy comparisons if c^∈G.Output: the refitted c^\begin{array}{l} \hline \textbf{Algorithm 1}\quad \text{THE with a gold-only fallback} \cr \hline \textbf{Input: } \text{gold and proxy comparisons; ridge values } K \text{ of gold-only; tuning values } G \text{ of THE} \cr \textbf{Stage 1 (cross-validation).} \text{ Split the gold annotators into five folds.} \cr \quad \textbf{for } \text{each fold } k \text{ and each candidate } c \in K \cup G \textbf{ do} \cr \qquad \text{fit } c \text{ on the other four folds, and on all proxy comparisons if } c \in G \cr \qquad \text{record its logistic loss on fold } k \cr \quad L(c) \leftarrow \text{mean loss of } c \text{ over all gold comparisons} \cr \textbf{Stage 2 (fallback).} \text{ Let } g \text{ and } t \text{ minimize } L \text{ over } K \text{ and over } G. \cr \quad \textbf{if } L(t) \lt L(g) \textbf{ then } \hat c \leftarrow t \textbf{ else } \hat c \leftarrow g \cr \textbf{Final fit.} \text{ Refit } \hat c \text{ on all gold comparisons, and on all proxy comparisons if } \hat c \in G. \cr \textbf{Output: } \text{the refitted } \hat c \cr \hline \end{array}

No reference, held-out label or oracle quantity enters. The rule was fixed before any of its outcomes were computed.

Under this rule THE's angular error on Community Alignment is 0.2% below gold-only's at three-quarter relabeling and 1.7% above at complete relabeling (implementable rule: 16.9% and 49.1% above); on PRISM it is 9.1% below and 3.7% above (implementable rule: 10.6% and 44.5% above). The fallback therefore removes most of the negative transfer, and the benefit remains where there is one: with the judge on half of the proxy THE is 19.1% below gold-only's on Community Alignment and 27.3% below on PRISM. Intervals and the figures for WE are in general comment (3/4).

W2. The real-label target. We agree that the angle to a held-out logistic fit measures the error for the loss in use and is not by itself evidence of recovery of the utility center. Both additions you ask for follow.

Held-out preference performance. Table R1 in general comment (2/4) reports the error on held-out target comparisons and best-of-four selection; these outcomes involve no reference fit. Under the oracle rule the hyperparameters are still chosen by the angle to the tuning reference. Under cross-validation no reference enters the tuning either, and with the judge on half THE's held-out error is 0.50 points below WE's on Community Alignment [0.27, 0.74] and 0.51 points below on PRISM [0.37, 0.66] (general comment (3/4)). In the reward-model representation THE and WE are ordered as for the angle; with sentence-embedding features their held-out errors are within 0.2 points of each other. For WE and gold-only the two outcomes differ: on Community Alignment and HH-RLHF, WE at its angle-optimal weight has a larger held-out error than gold-only, which general comment (2/4) traces to the curved directions.

References fitted with the same loss. The smoothed-sign runs had stored the angle to a smoothed-sign reference of the same held-out annotators. On Community Alignment with the judge on 10 to 25% of the proxy, THE's gain over WE is 4.6 to 6.8% against that reference (logistic reference: 4.2 to 6.0%). This reference is noisier than the logistic one. At 20 and 25% the centered, percentile and basic joint intervals all exclude zero; at 10 and 15% not all three do. On PRISM the gain is 0.5 to 1.2% (logistic: 0.8 to 1.5%). With the judge on half of the proxy it is −0.4% against the smoothed-sign reference and +2.9% against the logistic one; at that share the judge pulls the target of the sign loss almost onto its own direction (Appendix K.5), which is why the paper added the smaller shares. The four intervals and the held-out errors are in general comment (3/4).

AuthorsOpenReview ↗

Reply to Reviewer txHk (2/3): W3, W4

W3. Informal Theorem 2. You are right. The corrected sentence reads: "When curvature and noise share eigenvectors with eigenvalues in reverse order, neither being a multiple of the identity, and the nonzero smoothing offset lies along the least curved eigenvector, this holds for small bias. Under the same reverse ordering, with one bandwidth for both samples and a nonzero offset along any common eigenvector, the ratio of THE's gain over gold-only to that of re-weighting tends to a constant larger than one as the bias grows." The smoothing offset is the shift of the population minimizer caused by smoothing. The second sentence states the conditions of Theorem 28 for large bias. The appendix states somewhat weaker sufficient conditions than Theorem 6: Theorem 24(2) allows the offset along any common eigenvector whose curvature is at most a weighted mean of the curvatures, and Theorem 24(3) gives the exact inequality for a general offset.

W4. Theorem 13 and Lemma 6. You are right. Lemma 6(a) bounds the localization radius through cmax⁡r2/cqc_{\max} r^2/c_q. The bracket assumption gives cq≤(1+γ)cmin⁡c_q \le (1+\gamma) c_{\min}, so the factor cmax⁡/cqc_{\max}/c_q is at least chi/((1+γ)cmin⁡)c_{hi}/((1+\gamma) c_{\min}) and grows in the limit of Theorem 13, while the step "ηTHE≍(Λ/N1)1/4\eta^{THE} \asymp (\Lambda/N_1)^{1/4}" needs cmax⁡r2=O((Λ/N1)1/2)c_{\max} r^2 = O((\Lambda/N_1)^{1/2}).

The repair is in Lemma 6. Its proof compares the minimizer with the pair in which both heads sit at the gold center. Comparing also with the pair in which each head sits at its own center, where both losses are at their minima and only the penalty is paid, replaces ρcmax⁡r2\rho c_{\max} r^2 in the radius by the smaller of ρcmax⁡r2\rho c_{\max} r^2 and λ/2\lambda/2 times the squared distance between the two centers. In Theorem 13 the penalty is a fixed multiple of cmin⁡c_{\min}, so the squared radius is of order ϱ2=(cmin⁡r2+(Λ/N1)1/2)/cq\varrho^2 = (c_{\min} r^2 + (\Lambda/N_1)^{1/2})/c_q, which does not involve cmax⁡c_{\max}.

The corrected theorem states its conditions with the constants explicit: (γ+η02)ϱ2(\gamma+\eta_0^2)\varrho^2 is small relative to κ12\kappa_1^2, which makes the gap terms of Theorem 10 negligible, and η0\eta_0 is at least a constant times ϱ\varrho, which is the localization condition (Loc) that the statement had left implicit. The other terms of Theorem 10 are of lower order once cmin⁡Λ/N1c_{\min}\Lambda/N_1 is small relative to Γ2\Gamma^2. When cmin⁡c_{\min}, cqc_q and Γ\Gamma are fixed, this holds, and the two conditions are the rate already stated and a lower bound on η0\eta_0 of order (Λ/N1)1/4(\Lambda/N_1)^{1/4}. The two are compatible, and no growth condition on the curvature ratio is needed. (With Γ\Gamma fixed, the bracket assumption and Lemma 1(b) already bound chic_{hi} by a constant times 1/η01/\eta_0.)

We checked the limit numerically from the formulas of Lemma 6 and Theorem 10, on a sequence that the assumptions allow and that satisfies the stated rate (η0=(Λ/N1)1/5\eta_0 = (\Lambda/N_1)^{1/5}, γ=(Λ/N1)1/6/log⁡N1\gamma = (\Lambda/N_1)^{1/6}/\log N_1, chic_{hi} proportional to 1/η01/\eta_0; script thm13_growth_check.py in rebuttal/). With the lemma as printed, the left side of (30) divided by κ1\kappa_1 grows: 1.9, 2.5, 3.9 and 12.1 at N1=1040,1060,1080,10120N_1 = 10^{40}, 10^{60}, 10^{80}, 10^{120}. With the sharper lemma it decreases toward the claimed limit 0.89: 1.18, 1.13, 1.10 and 1.06, slowly because on this sequence the gap terms vanish at a logarithmic rate. Theorem 13 and Proposition 4 serve only the comparison of upper bounds inside Appendix B; no result of the main text uses them.

AuthorsOpenReview ↗

Reply to Reviewer txHk (3/3): W5, W6

W5. Calibration of the small real-label gains.

Interval construction. Table 9 has eleven rows on PRISM and Community Alignment in which the reported interval for THE's gain over WE under the oracle rule excludes zero. From the stored joint-bootstrap draws we recomputed each interval in three ways: centered at the point estimate (as reported), percentile, and basic (pivotal). In ten rows all three exclude zero; for Community Alignment with the judge on half they are [4.5, 12.1], [2.8, 10.4] and [5.9, 13.4]. The exception is Community Alignment at three quarters, whose percentile interval is [−0.02, 11.75] (97% of the joint draws favor THE). The percentile interval is shifted toward zero because a resampled reference is noisier, which is why the paper centers it; in one row (PRISM, rule on 20%) it lies wholly below the estimate.

Reference noise. Each joint-bootstrap draw refits the reference on resampled annotators, so Table 9's intervals include its noise. Table R1's held-out error and best-of-four selection involve no reference; their intervals resample held-out annotators and seeds. On the reward-model representation they confirm the ordering of THE and WE; with sentence-embedding features the differences are at most 0.2 points (zero excluded in three of seven designs).

Local minima of the smoothed sign loss. We ran the diagnostic you suggest on Community Alignment (judge on 20%, the reported bandwidth, 120 gold samples), refitting gold-only, WE and THE from five to six starting points each. Replacing every estimate by the lowest-objective solution over the starts leaves THE's gain over WE in place: it is 6.1, 8.9, 8.1 and 8.4% for 10, 20, 40 and 80 gold annotators, against 5.2, 8.4, 7.6 and 6.4% with the paper's starts (hyperparameters fixed at each gold size's most frequent oracle choice). Local optima do occur at this bandwidth: the starts reach 2.2 to 2.9 distinct optima on average, and the paper's start attains the lowest objective found in 96% of the gold samples for gold-only, 81% for WE and 65% for THE. We recommend several starts for the smoothed loss.

W6. Reproducibility.

Manifest. The tilt coefficient is 0.8 in the smoothed designs of K.1 and 0.4 in the sparse smoothed design of K.4; the exact-sign design has none. Grids of the reward-model designs on Community Alignment: 13 ridge values from 10−610^{-6} to 10; 18 weights for WE from 0 (gold-only) to 1 (proxy-only); for THE 13 weights from 0.1 to 0.9999 and 21 penalty levels from 10−610^{-6} to 10410^{4}. Splits on Community Alignment: the 647 US annotators are divided once, without labels, into 259 for training (gold samples of 10 to 80 annotators), 129 for the tuning reference and 259 for evaluation, of whom 230 rated response groups seen in both countries; the proxy is about 15,000 comparisons of a random subset of the 625 annotators in India; 30 seeds. PRISM, reward-model designs: three folds of 182 to 225 participants per cohort, one for evaluation; gold samples of 20 to 80 participants; 12 seeds per fold. HH-RLHF: 1,000 gold pairs, 15,000 proxy pairs, 9,912 evaluation pairs. The grids of the remaining designs are in the scripts and configuration files of the supplementary code.

Code. The code was uploaded with the submission, under "Supplementary Material" on the submission page (a zip; its README gives the commands, experiment by experiment). It may not have displayed on your side; we downloaded it again and it opens. The updated zip adds a folder rebuttal/ with the scripts behind the new numbers.

Appendix K.6. That subsection evaluates the limit of Proposition 2 at abstract triples of curvature HH, noise covariance VV and offset, varying the spectra of HH and VV independently. The preference model ties them together and gives the reverse order outside special cases; other cells only check the limit. By the formulas of Appendix D, HH and the score covariance are second moments of the design on the hyperplane orthogonal to the target, and HH carries an extra weight from the heterogeneity. This gives V⪰23H−1V \succeq \tfrac23 H^{-1}, with equality when there is no heterogeneity in the tangent directions. For the simulated design of Figure 2(a) the formulas give five curved and two flat directions with a curvature ratio of about 249 and a noise ratio of about 242 in reverse order; for the exact-sign design, about 58 and 52. With isotropic heterogeneity the order stayed reverse in every design and at each strength we tried. The same order requires disagreement concentrated along the directions in which the design varies most: with tangent half-widths 3 and 1 and heterogeneity of standard deviation 50 times the norm of the center along the wide direction only, that direction is both more curved (ratio 2.4) and noisier (ratio 1.6).

Thanks for the response and the extra experiments.

The two-stage CV fallback in Table R2 addresses my main practical concern in W1 well, bringing the severe negative transfer under complete relabeling down to a few percent while keeping the gains at moderate bias. The held-out comparison and best-of-four metrics in Table R1, the same-loss smoothed-sign reference check, and the DPO results in Table R3 also address W2, though the real-data gains over WE remain modest in absolute terms and mainly appear when the proxy shift lies along curved directions rather than between unmodified natural populations.

On the theory and calibration side, adding the separate-center comparison point β⋆\beta^\star and β2\beta_2 in Lemma 6 removes the cmax⁡c_{\max} dependence in Theorem 13 and resolves W4, and the revised wording of Informal Theorem 2 fixes W3. The alternative bootstrap interval checks, the multi-start diagnostic for the smoothed sign loss, and the reproducibility details and Appendix K.6 clarification also settle W5 and W6.

For the revised version, please make sure to include the Lemma 6 and Theorem 13 fix, the updated Informal Theorem 2, the negative-transfer results with the CV fallback as the main practical recommendation, Tables R1–R3, and the experiment manifest. Unifying the loss and penalty normalization across the main text and Appendices B–E would also make the paper much easier to follow.

Overall, my main concerns are addressed, so I am raising my rating from 2 to 3.

Reviewer_vUKNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to make better use of a large amount of biased proxy preference data when only a small amount of gold preference data is available. The authors propose a two-headed estimator (THE), which estimates the preferences of the two populations separately while connecting them through a penalty term. The main idea is that proxy preferences can be given different weights in different directions depending on the curvature of the gold loss. The paper provides theoretical analysis of when proxy preferences can help, when THE can outperform simple re-weighting, and how sparse differences between the two populations can be handled. The authors also conduct experiments on simulated preference data and real preference datasets, including PRISM, Community Alignment, and HH-RLHF. The experiments mainly evaluate how accurately the methods estimate the target preference direction, and show improvements over re-weighting in some settings.

Critical strengths
  1. The problem is clearly defined and meaningful. The paper studies an important problem in LLM alignment: how to effectively use a large amount of proxy preference data when gold preference data is limited. This is a practical problem, since high-quality human preferences are often expensive and difficult to collect. I think this is a valuable research direction.

  2. The theoretical analysis is quite detailed. The paper does not simply propose a new method and show that it works, but also tries to explain why and under what conditions it works. In particular, the analysis of how the curvature of the gold loss affects the weights assigned to proxy preferences is interesting. The theoretical results help us better understand when proxy preferences are useful and when they may introduce too much bias.

Critical weaknesses
  1. There is a gap between the theoretical analysis and the actual implementation. The theoretical results are mainly based on the sign loss and several assumptions about the preference distributions. However, the experiments on real preference data mainly use the Bradley-Terry logistic loss instead. Also, some assumptions in the theoretical analysis may not always hold in real LLM alignment settings. This makes it somewhat unclear how well the theoretical results can explain the performance in practice.

  2. The experimental validation is relatively limited. A large part of the paper is devoted to theoretical analysis, while the experimental section is quite compressed. More importantly, the experiments mainly focus on the accuracy of estimating preference directions. The paper does not further investigate whether using these estimated preferences for reward modeling, DPO, or other LLM alignment methods can actually improve the quality of generated responses. I think this is an important missing part, since the final goal is to improve LLM alignment, rather than just estimate preference parameters more accurately.

Questions for authors
  1. How well do the theoretical results apply to the actual implementation? The theoretical analysis is mainly based on the sign loss, but the real-data experiments mostly use the Bradley-Terry logistic loss. Under heterogeneous utilities, the Bradley-Terry model may not correctly recover the direction of the population's utility center. Can the authors explain whether the main theoretical conclusions, especially the curvature-based advantage of THE over re-weighting, still hold for the loss used in the experiments?

  2. How reliable is THE when the gold preference data is very limited? The main motivation is that gold preferences are scarce, but the advantage of THE depends on the curvature of the gold loss and how the proxy bias is distributed across different directions. When the gold sample is very small, how reliable is this curvature information? Also, Appendix K.5 shows that the implementable tuning rule can sometimes make THE perform worse than gold-only when the proxy bias is large. How can we determine when to trust the proxy data without relying on oracle tuning?

  3. Does a smaller preference estimation error actually lead to better LLM alignment? The experiments mainly evaluate the angular error between the estimated preference direction and a reference direction. However, it is not clear how much this metric reflects actual preference prediction or the quality of LLM outputs. For example, does a smaller angular error consistently lead to better accuracy on held-out human preference comparisons? And is there any evidence that using THE for reward modeling or DPO would lead to better aligned LLM outputs? Without this connection, I feel the practical benefit of the method is still somewhat unclear.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer vUKN (1/3): W1, Q1

Thank you for the review.

W1, Q1. Sign loss in the theory, logistic loss in the experiments. You are right that under heterogeneous utilities the Bradley–Terry fit need not recover the center; this is why the theory uses the sign loss. The real-label experiments use the logistic likelihood because it is much more precise at these sample sizes (the smoothed-sign gold-only errors are 10° to 18° larger against the logistic reference, and 6° to 15° larger when each fit is measured against a reference of its own loss). They measure the error relative to the logistic target of the gold population and test the transfer mechanism. On our data the two targets cannot be told apart: the logistic and smoothed-sign references of the same held-out annotators are 17° to 34° apart on average, the order of their own noise.

Does the curvature-based advantage hold for the loss in the experiments? The mechanism carries over to the logistic loss. In words, THE borrows less from the proxy along the feature directions in which the gold loss is most curved. This follows from the first-order conditions of the two-headed objective and holds for any smooth loss: the weight on the proxy fit has the form of (4), with the Hessians of the two losses in place of HH, and equals (4) when the populations share a Hessian. For the logistic likelihood the limiting error of each fit is a centered Gaussian: there is no smoothing offset and no bandwidth to tune. For limits of this form Theorem 24(1) gives the sign of the small-bias criterion when curvature and noise share eigenvectors: positive when curved directions have low noise, negative when they have high noise. The theorems of the paper are stated for the sign loss; for the logistic loss the support is this mechanism and the empirical record of Appendix K.5. Predictions computed from the fitted logistic Hessians and covariances gave 7.9 to 8.7% for the judge on half on Community Alignment (observed 8.1%) and 0.3 to 5.1% on PRISM (observed 3.5%), and across designs they rank-correlate with the observed gains at 0.69 to 0.83.

With the smoothed sign loss and a reference fitted by the same loss, THE's gain over WE on Community Alignment is 4.6 to 6.8% for judge shares of 10 to 25%, with joint intervals that exclude zero at 20 and 25%; on PRISM it is 0.5 to 1.2%. All shares, including one half, are in general comment (3/4).

Assumptions in real data. We agree that some of the assumptions need not hold exactly in LLM alignment data. Whether or not they do, the method was tested on such data. The estimators were applied to public preference datasets, and every gain is measured on human labels held out from the fit. With the judge on half of the proxy and tuning by cross-validation, which uses no oracle, THE's held-out error is 0.49 points below gold-only's on Community Alignment [0.25, 0.75] and 0.51 points below on PRISM [0.36, 0.66] (general comment (3/4)).

Two points can be checked on our data. Assumption 2 asks that both populations have the same law of features. On Community Alignment the two populations judge the same response groups by design, and on PRISM a linear classifier cannot tell the two samples apart (AUC 0.51 to 0.53 on sentence features). Theorems 4 to 6 take the bias to be of the order of the noise. The estimated shift between the gold and proxy fits has medians of 0.8 to 1.6 standard errors of the gold fit unless the judge relabels the whole proxy, so the bias is at most of that order. Symmetry of the heterogeneity (Assumption 1) cannot be tested from choices alone. It matters for reading the estimated direction as the utility center. How well the estimate predicts is measured directly on held-out annotators (Table R1 in general comment (2/4)).

The theory also accounts for what the data show. It predicts no gain of THE over WE when the shift between the populations lies in flat directions, and between natural populations the two are tied. It predicts that THE borrows less where the gold loss is curved, and the split of each estimate's error along the curvature shows this (general comment (2/4)).

AuthorsOpenReview ↗

Reply to Reviewer vUKN (2/3): W2, Q3

W2, Q3. Does a smaller angular error give better preference prediction and better outputs?

Prediction on held-out comparisons. Table R1 in general comment (2/4) reports two outcomes on held-out target annotators that need no reference: the held-out error, and on Community Alignment best-of-four selection, in which the estimated utility chooses among four candidate responses. In these designs the features are the representation of an existing reward model, so the estimate is a linear reward head on a frozen representation: reward modeling in its simplest form. The comparison is between ways of training that head from gold and proxy labels. With the judge on half of the proxy, THE's held-out error on Community Alignment is 0.56 points below gold-only's under the oracle rule and 0.49 points below [0.25, 0.75] under cross-validation; its best-of-four accuracy is 0.61 points above gold-only's ([0.04, 1.20]) under the oracle rule and 0.78 points above ([0.40, 1.17]) under cross-validation.

Angle and prediction error. In most cases a smaller angle goes with a smaller prediction error: within a gold sample the two outcomes are rank-correlated across candidate estimates (median 0.96 on Community Alignment and 0.89 on PRISM; 0.55 to 0.84 with sentence-embedding features). The exception is WE with a judge. On Community Alignment and HH-RLHF, WE at its angle-optimal weight has a smaller angle than gold-only and a larger held-out error, because it accepts bias in the curved directions, which matter most for prediction; THE lowers both on Community Alignment. With sentence-embedding features THE's angle is 2.6 to 7.8% below WE's and its held-out error is within 0.2 points of WE's. Table R1 reports prediction and Table R2 the angle.

DPO. We ran a DPO experiment (Table R3 in general comment (4/4)). Each estimate labels preference pairs, and a language model is fine-tuned on them with DPO. The experiment measures which candidate fine-tuning makes more probable, the reward that DPO fits implicitly: the fine-tuned model is correct on a four-way choice of a held-out annotator if the candidate whose log-probability rose most is the one the annotator chose. It scores existing candidate responses and generates no text, since scoring generated text would need new human ratings or a judge model. The design and the analyses were fixed in writing before the first run, and all 342 runs are reported.

  1. A more accurate reward gives a more accurate model. Across runs, one point of reward accuracy corresponds to 0.89 points of accuracy of the fine-tuned model ([0.66, 1.09]; added after the runs: 0.83, [0.55, 0.98], among the rewards from gold-only, WE and THE alone).
  2. THE's gain carries over to the implicit reward. With the judge on half of the proxy and 10 gold annotators, the model trained with THE is more accurate than the models trained with pooling, WE and gold-only, by 5.59 ([3.94, 7.40]), 0.93 ([0.38, 1.48]) and 0.81 points ([0.22, 1.39]). With no judge, where THE and WE tie as rewards, the model trained with THE is 1.70 points more accurate than the one trained with gold-only ([0.88, 2.58]); this is the gain from the proxy. For scale: a reward fitted on 221 training annotators is 2.8 points more accurate than gold-only with 10 annotators; THE (judge on half) recovers 1.1 of these points, and 0.8 of 2.2 after fine-tuning.
  3. The models keep the order of their rewards also with the IPO loss in place of DPO (six seeds), where THE is 2.64 points above pooling. Every fine-tuned model selects the annotators' choice more often than the model before fine-tuning: 39.6 to 42.3% against 38.2%, by the model's own log-probability per token (added after the runs). General comment (4/4) reports every pre-registered outcome and marks what was added.

Within this scope (one language model, one dataset, no generated text), the answer is yes for an estimate that predicts held-out choices better: fine-tuning with it moves the model toward the choices of held-out target annotators more often. (Gold-only's best-of-four accuracy is 41.9% in Table R1 and 38.26% in Table R3: the first uses the oracle rule, 10 to 80 gold annotators and 3,516 choices, the second cross-validation, 10 gold annotators and 4,003 choices.)

Compressed experiments. We agree that the experimental section of the submission is compressed. The protocol of the real-label experiments and Tables R1 to R3 belong in the main text.

AuthorsOpenReview ↗

Reply to Reviewer vUKN (3/3): Q2

Q2. Very small gold samples; when to trust the proxy without an oracle.

How reliable is the curvature information with little gold data? THE needs no estimate of the curvature: the gold loss itself resists moving the gold fit where it is curved, so no Hessian enters the estimator. What a small gold sample affects is the tuning, and the table below shows the result at each gold size. It also affects the diagnostics. We recomputed them with the 10 to 80 annotators of each gold sample, instead of the 388 pooled annotators of Appendix K.5. On Community Alignment the curvature of the gold loss along the estimated shift, relative to its mean, has median 0.7, 0.4, 0.2 and 0.1 without a judge for 10, 20, 40 and 80 gold annotators (10 annotators are about 410 comparisons), and 4.5, 4.4, 2.7 and 2.3 with the judge on half. It is above the mean in every gold sample with a judge and in 27, 7, 0 and 0% of the gold samples without one. So the direction of the shift is largely visible even with 10 annotators. Its size is less informative: with small gold samples the estimated shift is about one standard error of the gold fit with or without a shift, because gold noise alone produces that much.

The implementable rule of the submission estimates the risk too optimistically when the bias is large and then selects a large weight.

Cross-validation on the gold sample (Table R2 in general comment (3/4)) uses no oracle and has gold-only among its candidates. Community Alignment, judge on half of the proxy, by number of gold annotators:

gold annotators10204080
THE under cross-validation, angular error relative to gold-only (%; positive: below)29.022.117.28.4
THE under the oracle rule, relative to oracle-tuned gold-only (%)35.926.221.37.1
gold-only under cross-validation, held-out error (%)37.736.435.635.2
THE under cross-validation, held-out error (%)36.735.735.435.2

The gain is largest where gold data are scarcest. WE gains little (1.3% on average over the four sizes, Table R2), because cross-validation of the gold loss rejects the weights at which WE's angle would improve (median weight 0.1, against 0.5 for the smallest angle). When the judge relabels three quarters or all of the proxy, THE under cross-validation is 0.2% below and 1.7% above gold-only's in angular error on Community Alignment (implementable rule: 16.9% and 49.1% above), and 9.1% below and 3.7% above on PRISM (10.6% and 44.5% above). The other designs are in Table R2.

We recommend tuning by cross-validation on the gold sample with gold-only among the candidates; this decides whether the proxy is used. The two diagnostics (the estimated shift in standard errors of the gold fit, and the curvature along it) help to interpret the outcome: on Community Alignment THE's gains over WE occur where the curvature along the shift is above the mean.

Thank you for the responses. I appreciate the authors' substantial effort in addressing my concerns. I am satisfied with the clarifications and will maintain my original rating of 3 (Weak Acceptance).

Reviewer_DMLbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how a small amount of preference data from a target population can benefit from abundant but systematically biased proxy preferences. It proposes a two-headed estimator (THE) that learns separate preference directions for the two populations while encouraging them to remain close. Unlike scalar reweighting, THE allows different degrees of borrowing across directions. The theoretical analysis characterizes when proxy data help, when THE can outperform reweighting, and how sparse differences can reduce the required gold data. Experiments with synthetic preferences, real answer embeddings, and human preference datasets support these conditional benefits, although THE largely matches reweighting in unmodified natural-population settings.

Critical strengths
  1. The paper addresses a practically meaningful problem: how to use abundant but biased proxy preferences to improve learning from scarce target-population data without shifting away from the intended target.
  2. The paper provides extensive theoretical analysis and empirical validation, characterizing when proxy data are beneficial and when the proposed estimator can outperform scalar reweighting.
Critical weaknesses
  1. The scope of the proposed approach appears limited. The paper assumes a predefined target population and estimates its utility-center direction using biased proxy preferences. Although within-population disagreement is allowed, it is unclear whether a single center with symmetric heterogeneity adequately captures more complex preference structures. Practical applications may also involve multiple, partially overlapping gold target populations: a population treated as a biased proxy for one target may itself be the intended target for another. Can the approach be naturally extended to accommodate these distinct targets without collapsing them into a single center?
  2. The practical applicability of the theoretical assumptions remains unclear. Much of the paper develops theoretical guarantees under different sets of assumptions. However, it is not sufficiently clear how representative these assumptions are of real preference data, how practitioners can assess whether they approximately hold, or how robust the conclusions are when they are violated. The existing diagnostics and robustness tests are useful, but do not yet provide a clear practical boundary between beneficial transfer and potential failure.
  3. The empirical advantage over the simpler WE baseline is limited in natural-population settings. Without additional proxy relabeling, THE essentially matches WE, while larger gains mainly occur in relabeled or structurally favorable settings. This raises concern that the conditions required for meaningful gains may be restrictive in practice.
Questions for authors
  1. Could you clarify the distinction between the existing sign estimator and the proposed THE? My understanding is that the sign estimator estimates each population’s utility-center direction using that population’s own data—for example, gold data to estimate the gold direction, or proxy data to estimate the proxy direction. In contrast, THE uses abundant but systematically shifted proxy comparisons to improve estimation of the gold direction from scarce gold data. Is this interpretation correct?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer DMLb (1/2): W1, W3, Q1

Thank you for the review.

W1. Scope: one center per population; multiple targets.

One center. Assumption 1 asks that the heterogeneity be symmetric about the center and put some mass near it in every direction. It imposes no parametric form, so multimodal symmetric laws are allowed as long as some users are close to the center. The rates in Theorems 4 to 6 also use a finite mean or second moment. Asymmetric structure inside a population is outside the model. If the subgroups are known, each can be given its own direction. Unknown subgroups are the personalization problem, a different question from ours.

Multiple targets. The estimator does not collapse the populations into one center; it fits one direction per population. What the paper analyzes is one target per fit: Sections 3 to 5 with one proxy, and Appendices I and J with several proxies, each tied to the target's direction. With several targets, the direct use is one fit per target, where the other populations serve as proxies. The PRISM experiments already do this: each of 18 overlapping cohorts (for example women, religious participants, non-native English speakers) is the target in turn and the remaining participants are the proxy, which reduces the gold-only error by 36% (Table 9). A population can therefore be the target of one fit and a proxy in another, and nothing forces two targets to share a center.

For one target with several proxies, Appendix J gives the asymptotic weights of the linearized problem and shows that they match the optimal linear combination under squared loss with independent isotropic biases and noise proportional to the inverse curvature. A joint fit of several targets at once goes beyond this analysis. In PRISM each participant is on one side of a fit only, so the two samples of a fit are independent.

W3. Limited advantage over WE between natural populations. We agree that between natural populations THE and WE perform alike. You ask whether the conditions for a meaningful gain are restrictive in practice. The framework of the paper, a small gold sample combined with a larger biased proxy, gives such gains under realistic conditions (at most 80 gold annotators, no oracle for tuning, a proxy that is cheaply available). The figures below use cross-validation on the gold sample (Table R2 in general comment (3/4)) and held-out human labels.

The proxy is another human population: annotators in India for a target in the United States (Community Alignment), or the remaining participants for a demographic cohort (PRISM). THE's angular error is 48.6% and 43.8% below gold-only's, and its held-out error is 0.90 and 0.93 points lower ([0.46, 1.37] and [0.73, 1.12]). A language model fine-tuned on pairs labeled by THE's estimate is 1.70 points more accurate on held-out annotators than with gold-only's ([0.88, 2.58]; general comment (4/4)). WE does as well (47.4% and 43.8%), as the theory predicts when the shift between the populations is small and lies in flat directions, those along which the gold loss changes slowly: THE then reduces to WE.

Part of the proxy is labeled by a model. Cheap preference labels are often produced by a model instead of people (AI feedback) and carry its bias. In our designs a public reward model, the judge of general comment (1/4), relabels half of the proxy comparisons, and the features are its own representation. THE's angular error is then 19.1% and 27.3% below gold-only's, and its held-out error is 0.49 and 0.51 points lower ([0.25, 0.75] and [0.36, 0.66]); the fine-tuned language model is 0.81 points more accurate than with gold-only ([0.22, 1.39]). WE's angular error is 1.3% below and 0.1% above gold-only's. The judge's bias lies along curved directions, those along which the gold loss rises steeply. THE borrows less from the proxy there and keeps the gain; re-weighting has one weight for all directions, and cross-validation keeps it small.

When the judge relabels all of the proxy, the proxy no longer helps; cross-validation has gold-only among its candidates, and THE's angular error stays within 1.7% and 3.7% of gold-only's. A practitioner therefore need not know in advance which case holds.

Q1. Sign estimator and THE. Your reading is correct. We add two refinements. First, THE consists of two sign-loss fits coupled by a penalty on their difference. With penalty zero its output is exactly the sign estimator on the gold sample; with an infinite penalty it is the sign estimator on the re-weighted pooled sample; in between, the gold fit is moved toward the proxy fit by a different fraction in each direction, less where the gold loss is curved. Second, the coupling itself follows data-enriched regression. Our contribution is the analysis for the discontinuous sign loss: when the proxy helps, and when THE improves on WE.

AuthorsOpenReview ↗

Reply to Reviewer DMLb (2/2): W2

W2. What the assumptions mean in practice. The assumptions are idealizations, and we agree that some need not hold exactly in real preference data. The empirical conclusions do not rest on their holding exactly: the estimators were applied to public preference datasets, and every gain is measured on human labels held out from the fit. Under tuning by cross-validation, which a practitioner can run, THE's angular error is 48.6% below gold-only's on Community Alignment and 43.8% below on PRISM when the proxy keeps its own population's labels, and 19.1% below and 27.3% below with the judge on half of the proxy (Table R2 in general comment (3/4)).

A practitioner can check two things on data. The first is that both populations have the same law of features: on Community Alignment they judge the same response groups by design, and on PRISM a linear classifier cannot tell the two samples apart (AUC 0.51 to 0.53 on sentence features). The second is the size of the bias relative to the noise, which the table below reports. Symmetry of the heterogeneity cannot be tested from choices alone. It matters for reading the estimated direction as the utility center; how well the estimate predicts is measured directly on held-out annotators (Table R1 in general comment (2/4)).

The practical boundary. It is set by cross-validation on the gold sample: the proxy is used only when it lowers the validation loss of gold comparisons (Table R2 in general comment (3/4)). Two diagnostics that a practitioner can compute from the training data describe which case one is in: the estimated shift between the gold and proxy fits in units of the estimated standard error of the gold fit, and the curvature of the gold loss along that shift relative to its mean (how steeply the gold loss rises in that direction). Community Alignment, medians over seeds for 10 and 80 gold annotators:

proxy labelsestimated shift (10 / 80)curvature along it (10 / 80)oracle: THE relative to gold-onlycross-validation: THE relative to gold-only
no judge0.8 / 1.00.7 / 0.154.1% below48.6% below
judge on half1.0 / 1.14.5 / 2.322.6% below19.1% below
judge on three quarters1.5 / 1.67.3 / 4.04.8% below0.2% below
judge on all3.6 / 7.410.0 / 11.70.3% above1.7% above

An estimated shift of about one standard error is what gold noise alone produces, and a judge on all of the proxy stands out with 3.6 to 7.4. The curvature along the shift is above the mean in every gold sample with a judge, and in 27, 7, 0 and 0% of the gold samples without one for 10, 20, 40 and 80 annotators, so the direction of the shift is largely visible even in the smallest gold samples. (The diagnostics of Appendix K.5 were computed from all 388 training and tuning annotators; the ones above use only the annotators of each gold sample.)

The robustness checks of the paper vary the direction of the bias, the features and the loss, and for the sparse correction the heterogeneity law among symmetric laws (Appendix K.3).

Thanks for your careful responses. These have addressed some of my concerns, while I still hold the concern about its minor performance improvement on real data. I maintain my original rating of 3 (Weak Acceptance).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.