- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how abundant, biased proxy preferences can improve estimation of a small gold population's utility direction under heterogeneous annotator preferences. Its two-headed estimator (THE) fits separate population directions and penalizes their difference, borrowing less along directions where gold data are precise.
The main contributions are cube-root limits and baseline lower bounds for exact sign loss, a curvature criterion for improvement under smoothing, and local sparse-correction guarantees. Synthetic experiments support the proposed mechanism. Real-label results are more qualified: natural-population shifts generally produce a tie with re-weighting, while stronger gains appear with reward-model or rule-based relabeling.
Critical strengths
- Clear problem formulation: The paper distinguishes within-population heterogeneity from between-population bias.
- Substantial theory: The main novelty lies in analyzing transfer with discontinuous preference losses, rather than the coupling architecture itself.
- Careful validation: Cross-fitted oracle tuning, isotropic controls, smoothing corrections, and grid sensitivity checks strengthen the synthetic evidence.
- Transparent reporting: Negative results and limitations are disclosed, and the headline numerical claims agree with the tables.
Critical weaknesses
W1: Practical tuning can cause substantial negative transfer. Under large proxy bias, the implementable rule produces errors 10.6%-20.0% worse than gold-only at three-quarter relabeling and 44.5%-98.4% worse at complete relabeling. These disclosed failures limit practical applicability without invalidating the oracle theory. Highlight them in the main text and evaluate a reliable gold-only fallback.
W2: Real-label evaluation differs from the theoretical target. Headline experiments use ridge logistic likelihood and measure angle to a held-out logistic fit. Because that fit can be inconsistent under heterogeneous preferences, these results do not establish recovery of the gold utility center. Smoothed-sign reruns retain the same reference and often show smaller gains. Add held-out preference performance and sensitivity to references fitted with the same loss.
W3: The informal theorem omits an assumption. Informal Theorem 2 attributes small-bias improvement to reverse ordering of curvature and noise. Formal Theorem 6 additionally requires the nonzero smoothing offset to point along the least-curved direction. Include this condition in the introduction. This is a confirmed statement mismatch, not a failure of the formal theorem.
W4: An auxiliary proof needs clarification. Theorem 13 assumes a localization rate while allowing curvature to diverge, but Lemma 6's bound also depends on c_max * r^2 / c_q. Explain how changing design constants remain controlled, or add sufficient growth conditions. This is an unresolved proof-completeness question concerning an appendix upper-bound comparison.
W5: Small real-label gains need stronger calibration. Half-sample reference fits disagree by 15-35 degrees, and joint-bootstrap intervals are median-recentered. Check coverage or sensitivity to standard interval constructions. Smoothed-sign experiments also encountered local minima 30-52 degrees apart before selecting a bandwidth and using one warm-started path; multiple-start diagnostics would strengthen confidence.
W6: Reproduction requires more complete specifications. The PDF omits the synthetic tilt coefficient and exact real-label grids and split sizes. Supplementary code is promised but was unavailable for this review. Provide a configuration manifest and clarify whether Appendix K.6's abstract matrix regimes are realizable under the original preference model.
Questions for authors
Please see weaknesses.
4 comments
Reply to Reviewer txHk (2/3): W3, W4
W3. Informal Theorem 2. You are right. The corrected sentence reads: "When curvature and noise share eigenvectors with eigenvalues in reverse order, neither being a multiple of the identity, and the nonzero smoothing offset lies along the least curved eigenvector, this holds for small bias. Under the same reverse ordering, with one bandwidth for both samples and a nonzero offset along any common eigenvector, the ratio of THE's gain over gold-only to that of re-weighting tends to a constant larger than one as the bias grows." The smoothing offset is the shift of the population minimizer caused by smoothing. The second sentence states the conditions of Theorem 28 for large bias. The appendix states somewhat weaker sufficient conditions than Theorem 6: Theorem 24(2) allows the offset along any common eigenvector whose curvature is at most a weighted mean of the curvatures, and Theorem 24(3) gives the exact inequality for a general offset.
W4. Theorem 13 and Lemma 6. You are right. Lemma 6(a) bounds the localization radius through . The bracket assumption gives , so the factor is at least and grows in the limit of Theorem 13, while the step "" needs .
The repair is in Lemma 6. Its proof compares the minimizer with the pair in which both heads sit at the gold center. Comparing also with the pair in which each head sits at its own center, where both losses are at their minima and only the penalty is paid, replaces in the radius by the smaller of and times the squared distance between the two centers. In Theorem 13 the penalty is a fixed multiple of , so the squared radius is of order , which does not involve .
The corrected theorem states its conditions with the constants explicit: is small relative to , which makes the gap terms of Theorem 10 negligible, and is at least a constant times , which is the localization condition (Loc) that the statement had left implicit. The other terms of Theorem 10 are of lower order once is small relative to . When , and are fixed, this holds, and the two conditions are the rate already stated and a lower bound on of order . The two are compatible, and no growth condition on the curvature ratio is needed. (With fixed, the bracket assumption and Lemma 1(b) already bound by a constant times .)
We checked the limit numerically from the formulas of Lemma 6 and Theorem 10, on a sequence that the assumptions allow and that satisfies the stated rate (, , proportional to ; script thm13_growth_check.py in rebuttal/). With the lemma as printed, the left side of (30) divided by grows: 1.9, 2.5, 3.9 and 12.1 at . With the sharper lemma it decreases toward the claimed limit 0.89: 1.18, 1.13, 1.10 and 1.06, slowly because on this sequence the gap terms vanish at a logarithmic rate. Theorem 13 and Proposition 4 serve only the comparison of upper bounds inside Appendix B; no result of the main text uses them.
Reply to Reviewer txHk (3/3): W5, W6
W5. Calibration of the small real-label gains.
Interval construction. Table 9 has eleven rows on PRISM and Community Alignment in which the reported interval for THE's gain over WE under the oracle rule excludes zero. From the stored joint-bootstrap draws we recomputed each interval in three ways: centered at the point estimate (as reported), percentile, and basic (pivotal). In ten rows all three exclude zero; for Community Alignment with the judge on half they are [4.5, 12.1], [2.8, 10.4] and [5.9, 13.4]. The exception is Community Alignment at three quarters, whose percentile interval is [−0.02, 11.75] (97% of the joint draws favor THE). The percentile interval is shifted toward zero because a resampled reference is noisier, which is why the paper centers it; in one row (PRISM, rule on 20%) it lies wholly below the estimate.
Reference noise. Each joint-bootstrap draw refits the reference on resampled annotators, so Table 9's intervals include its noise. Table R1's held-out error and best-of-four selection involve no reference; their intervals resample held-out annotators and seeds. On the reward-model representation they confirm the ordering of THE and WE; with sentence-embedding features the differences are at most 0.2 points (zero excluded in three of seven designs).
Local minima of the smoothed sign loss. We ran the diagnostic you suggest on Community Alignment (judge on 20%, the reported bandwidth, 120 gold samples), refitting gold-only, WE and THE from five to six starting points each. Replacing every estimate by the lowest-objective solution over the starts leaves THE's gain over WE in place: it is 6.1, 8.9, 8.1 and 8.4% for 10, 20, 40 and 80 gold annotators, against 5.2, 8.4, 7.6 and 6.4% with the paper's starts (hyperparameters fixed at each gold size's most frequent oracle choice). Local optima do occur at this bandwidth: the starts reach 2.2 to 2.9 distinct optima on average, and the paper's start attains the lowest objective found in 96% of the gold samples for gold-only, 81% for WE and 65% for THE. We recommend several starts for the smoothed loss.
W6. Reproducibility.
Manifest. The tilt coefficient is 0.8 in the smoothed designs of K.1 and 0.4 in the sparse smoothed design of K.4; the exact-sign design has none. Grids of the reward-model designs on Community Alignment: 13 ridge values from to 10; 18 weights for WE from 0 (gold-only) to 1 (proxy-only); for THE 13 weights from 0.1 to 0.9999 and 21 penalty levels from to . Splits on Community Alignment: the 647 US annotators are divided once, without labels, into 259 for training (gold samples of 10 to 80 annotators), 129 for the tuning reference and 259 for evaluation, of whom 230 rated response groups seen in both countries; the proxy is about 15,000 comparisons of a random subset of the 625 annotators in India; 30 seeds. PRISM, reward-model designs: three folds of 182 to 225 participants per cohort, one for evaluation; gold samples of 20 to 80 participants; 12 seeds per fold. HH-RLHF: 1,000 gold pairs, 15,000 proxy pairs, 9,912 evaluation pairs. The grids of the remaining designs are in the scripts and configuration files of the supplementary code.
Code. The code was uploaded with the submission, under "Supplementary Material" on the submission page (a zip; its README gives the commands, experiment by experiment). It may not have displayed on your side; we downloaded it again and it opens. The updated zip adds a folder rebuttal/ with the scripts behind the new numbers.
Appendix K.6. That subsection evaluates the limit of Proposition 2 at abstract triples of curvature , noise covariance and offset, varying the spectra of and independently. The preference model ties them together and gives the reverse order outside special cases; other cells only check the limit. By the formulas of Appendix D, and the score covariance are second moments of the design on the hyperplane orthogonal to the target, and carries an extra weight from the heterogeneity. This gives , with equality when there is no heterogeneity in the tangent directions. For the simulated design of Figure 2(a) the formulas give five curved and two flat directions with a curvature ratio of about 249 and a noise ratio of about 242 in reverse order; for the exact-sign design, about 58 and 52. With isotropic heterogeneity the order stayed reverse in every design and at each strength we tried. The same order requires disagreement concentrated along the directions in which the design varies most: with tangent half-widths 3 and 1 and heterogeneity of standard deviation 50 times the norm of the center along the wide direction only, that direction is both more curved (ratio 2.4) and noisier (ratio 1.6).
Thanks for the response and the extra experiments.
The two-stage CV fallback in Table R2 addresses my main practical concern in W1 well, bringing the severe negative transfer under complete relabeling down to a few percent while keeping the gains at moderate bias. The held-out comparison and best-of-four metrics in Table R1, the same-loss smoothed-sign reference check, and the DPO results in Table R3 also address W2, though the real-data gains over WE remain modest in absolute terms and mainly appear when the proxy shift lies along curved directions rather than between unmodified natural populations.
On the theory and calibration side, adding the separate-center comparison point and in Lemma 6 removes the dependence in Theorem 13 and resolves W4, and the revised wording of Informal Theorem 2 fixes W3. The alternative bootstrap interval checks, the multi-start diagnostic for the smoothed sign loss, and the reproducibility details and Appendix K.6 clarification also settle W5 and W6.
For the revised version, please make sure to include the Lemma 6 and Theorem 13 fix, the updated Informal Theorem 2, the negative-transfer results with the CV fallback as the main practical recommendation, Tables R1–R3, and the experiment manifest. Unifying the loss and penalty normalization across the main text and Appendices B–E would also make the paper much easier to follow.
Overall, my main concerns are addressed, so I am raising my rating from 2 to 3.
Reply to Reviewer txHk (1/3): W1, W2
Thank you for a precise review. We accept W1, W3 and W4 as stated; the corresponding changes are given below. W2, W5 and W6 are answered with new analyses.
W1. Negative transfer under the implementable rule; a gold-only fallback. Agreed on both requests. The failure belongs in the main text, next to the oracle results and together with the fallback evaluated below. Its cause is the risk estimate of the implementable rule, which is too optimistic when the bias is large: gold-only (weight zero) was in its grid and was not selected.
We evaluated a gold-only fallback (Table R2 in general comment (3/4)) and state it as a two-stage algorithm (cross-validation, then fallback).
Algorithm 1THE with a gold-only fallbackInput: gold and proxy comparisons; ridge values K of gold-only; tuning values G of THEStage 1 (cross-validation). Split the gold annotators into five folds.for each fold k and each candidate c∈K∪G dofit c on the other four folds, and on all proxy comparisons if c∈Grecord its logistic loss on fold kL(c)←mean loss of c over all gold comparisonsStage 2 (fallback). Let g and t minimize L over K and over G.if L(t)<L(g) then c^←t else c^←gFinal fit. Refit c^ on all gold comparisons, and on all proxy comparisons if c^∈G.Output: the refitted c^
No reference, held-out label or oracle quantity enters. The rule was fixed before any of its outcomes were computed.
Under this rule THE's angular error on Community Alignment is 0.2% below gold-only's at three-quarter relabeling and 1.7% above at complete relabeling (implementable rule: 16.9% and 49.1% above); on PRISM it is 9.1% below and 3.7% above (implementable rule: 10.6% and 44.5% above). The fallback therefore removes most of the negative transfer, and the benefit remains where there is one: with the judge on half of the proxy THE is 19.1% below gold-only's on Community Alignment and 27.3% below on PRISM. Intervals and the figures for WE are in general comment (3/4).
W2. The real-label target. We agree that the angle to a held-out logistic fit measures the error for the loss in use and is not by itself evidence of recovery of the utility center. Both additions you ask for follow.
Held-out preference performance. Table R1 in general comment (2/4) reports the error on held-out target comparisons and best-of-four selection; these outcomes involve no reference fit. Under the oracle rule the hyperparameters are still chosen by the angle to the tuning reference. Under cross-validation no reference enters the tuning either, and with the judge on half THE's held-out error is 0.50 points below WE's on Community Alignment [0.27, 0.74] and 0.51 points below on PRISM [0.37, 0.66] (general comment (3/4)). In the reward-model representation THE and WE are ordered as for the angle; with sentence-embedding features their held-out errors are within 0.2 points of each other. For WE and gold-only the two outcomes differ: on Community Alignment and HH-RLHF, WE at its angle-optimal weight has a larger held-out error than gold-only, which general comment (2/4) traces to the curved directions.
References fitted with the same loss. The smoothed-sign runs had stored the angle to a smoothed-sign reference of the same held-out annotators. On Community Alignment with the judge on 10 to 25% of the proxy, THE's gain over WE is 4.6 to 6.8% against that reference (logistic reference: 4.2 to 6.0%). This reference is noisier than the logistic one. At 20 and 25% the centered, percentile and basic joint intervals all exclude zero; at 10 and 15% not all three do. On PRISM the gain is 0.5 to 1.2% (logistic: 0.8 to 1.5%). With the judge on half of the proxy it is −0.4% against the smoothed-sign reference and +2.9% against the logistic one; at that share the judge pulls the target of the sign loss almost onto its own direction (Appendix K.5), which is why the paper added the smaller shares. The four intervals and the held-out errors are in general comment (3/4).