acceptodds
Under review as a conference paper at ICLR 2027

Optimal Design for Active Preference Learning with Biased LLM Judges

21 views

Abstract

Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank–Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-Judge, Preference Learning, Active Learning, Experimental Design, LLM Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YjATOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper studies active preference acquisition with systematically biased LLM judges. It proposes Nuisance-Adjusted Optimal Design (NAOD), which selects comparisons using effective information about human preferences after accounting for judge-specific nuisance parameters, weighted by downstream policy relevance. The paper provides a conditional local asymptotic minimax characterization, analyzes the additional risk caused by learned nuisance representations, and develops a finite-pool selection algorithm. Synthetic experiments examine the theoretical mechanisms, while a Chatbot Arena evaluation reports improvements in proxy policy regret and held-out human-preference prediction.

Critical strengths
  1. The paper explicitly models human preferences and judge-specific deviations as separate components. This formulation allows the acquisition criterion to account for uncertainty in the nuisance parameters and the sensitivity of the downstream policy to reward estimation errors.

  2. The representation-learning analysis identifies an additional risk term and conditions under which representation error can reverse the ranking of acquisition methods. The synthetic experiments include examples of this reversal, documenting a limitation of the proposed approach.

Critical weaknesses
  1. My main concern is the choice of evaluation reference. The primary Arena metric is DF(θref,θ^)D_F(\theta_{\mathrm{ref}},\hat{\theta}), where θref\theta_{\mathrm{ref}} is trained on upstream human labels and is already used to construct the nuisance representation. Simply retaining this model would therefore give zero proxy regret, yet its held-out human cross-entropy and accuracy are not included in the comparison. Without this baseline, the reported 29.1% reduction tells us how NAOD compares with Target-info, but leaves a more basic question unanswered: does collecting additional judge labels improve on the human preference model already available? This matters particularly because the upstream stage uses 8,575 human-labeled clusters (Appendix F.2; supplementary experiment.py and representation.py).

  2. There is a gap between the estimator analyzed in the theory and the one used in Arena. Appendix B.2 constructs preliminary estimates from the current observations, with an error that decreases as the sample size grows. In the released code, the preliminary estimates are fitted once on Init128 and reused across budgets, followed by a single Newton update. A fixed initialization sample does not generally provide the shrinking error required by the proof. The paper therefore needs to explain how the attainment result relates to this implementation and establish whether the empirical advantage survives a change in the final estimator (Appendices B.2 and F.1; supplementary estimator.py).

  3. The paper's own analysis shows that representation error can reverse the ranking of acquisition methods, but the Arena experiments do not examine this dependence. Section 5 derives a risk of the form Φ0/n+CP/m\Phi_0/n+C_P/m, while NAOD optimizes only the fitted information criterion. The synthetic experiments illustrate the resulting limitation clearly. On real data, however, the upstream sample size is fixed, so there is little guidance on how much historical data is needed for NAOD to help, or when representation error makes its selections worse than those of simpler methods.

  4. The role of the learned target representation needs closer examination. Section 5 studies errors in WW with XX fixed, whereas the Arena pipeline learns both from upstream data. Moreover, XX is constructed separately for each judge using human and judge preference heads. It is unclear how the learning-cost analysis accounts for this additional source of uncertainty. The experiments also fix both target and nuisance dimensions at two. Results with a shared target space and larger dimensions would help establish whether the gains depend on this particular construction. A shared target space would also make the cross-judge relationship between coupling and performance easier to interpret (Section 5; Appendix F.1).

Questions for authors
  1. How does NAOD compare with simply using θref\theta_{\mathrm{ref}}? Please report this model's held-out human cross-entropy and accuracy, together with a human-only model fitted on the upstream and current trusted labels. Using the same target representation and test split, does additional judge acquisition improve either metric beyond these baselines?

  2. Do the acquisition rankings change with the final estimator? Keeping the selected subsets fixed, please compare the Init128-based one-step update with the estimator constructed in Appendix B.2 and with a joint likelihood fit run to convergence. Paired comparisons of NAOD, Target-info, and D-opt, with uncertainty intervals, would help determine whether the reported gains depend on the update rule. Please also clarify how the current implementation satisfies the assumptions used for attainment.

  3. What happens when less historical data is available to learn WW? Please vary this sample size on Arena while keeping XX, the evaluation reference, current budgets, and test set fixed. Report both proxy regret and human-label metrics. Does the ranking reversal predicted in Section 5.2 occur, and over what range of sample sizes does NAOD retain an advantage?

  4. Are the results robust to the representation choices? Please test a modest range of target and nuisance dimensions, and include a target space shared across judges and learned without judge-specific labels. Do the performance gains and the reported relationship between coupling and improvement persist?

  5. What exactly does Theorem 5.1 cover when the target representation is also learned? Is the result conditional on the realized XX, and does CP/mC_P/m account for any error from learning XX? Please clarify how the theorem should be interpreted for the Arena pipeline, where both XX and WW are learned from the historical sample.

3 comments

Dear Reviewer,

Thank you again for your valuable feedback. As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our rebuttal and additional experiments have addressed your concerns, or if any further clarification is needed.

Thank you for your time and consideration.

Thanks for clarifying the assumptions in Appendix B.2. I’ll raise Soundness to 2. I’m keeping my overall rating at 1 because the gains at small human-label budgets still depend on the estimator. I still find the paper hard to follow.

Dear Reviewer,

Thank you for acknowledging our clarification of Appendix B.2 and raising the Soundness rating to 2. We would like to further clarify the two remaining concerns.

1. Estimator dependence at small human-label budgets.

We acknowledge the observed finite-sample dependence, but its interpretation is important. This behavior is consistent with the conditions underlying Appendix B.2, rather than evidence against our theoretical results or acquisition principle.

The asymptotic efficiency of the Newton one-step estimator requires sufficiently accurate preliminary estimates. With only 32 or 64 current human labels, a current-only preliminary fit can be unstable, and a single Newton update is not guaranteed to achieve asymptotic efficiency.

Our additional experiments exhibit behavior consistent with this mechanism. The B.2-style estimator's absolute proxy regret decreases substantially as the human-label budget increases. In the exploratory subgroup with H = 128, 256, and 512, NAOD achieves an 18.39% mean regret reduction over Target-info using this estimator.

Our original Arena protocol explicitly specifies the shared historical Init128 initialization, providing a more stable starting point under limited current-label budgets. Importantly, all acquisition methods use the same estimator within each comparison, isolating the effect of the acquisition strategy.

Furthermore, NAOD retains a 14.40% mean regret reduction over Target-info under converged joint MLE across the full budget grid, with a positive paired confidence interval. Thus, its gains are not exclusive to Init128.

We do not claim estimator-independent improvements or asymptotic attainment for every practical initialization. The observed small-budget behavior highlights precisely why preliminary estimation accuracy matters in Appendix B.2. As the budget increases, the B.2-style estimator improves markedly, consistent with the theoretical mechanism. These findings clarify the finite-sample conditions for effective implementation rather than contradicting our conditional guarantees or invalidating the proposed acquisition criterion.

2. Readability and presentation.

Our methodology builds on the established statistical optimal experimental design framework, extended to accommodate judge-specific nuisance parameters. The central derivation follows a standard statistical progression from joint target–nuisance Fisher information, through Schur-complement effective information, to a policy-weighted acquisition criterion. We then develop a practical finite-pool selection algorithm.

We have further clarified the assumptions, estimator construction, and experimental protocol throughout our rebuttal. We would appreciate clarification of any specific definitions, derivations, or proof steps that remain difficult to follow, so that we can address them directly.

Thank you again for your time and consideration.

Reviewer_Vq2rOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLM judges can supplement scarce human preference labels but deviate from human preferences. Calibration only balances these errors on the distribution it was fitted to, while in active selection residual bias can reappear on the chosen comparisons. The authors therefore account for judge deviation during selection, modelling human preference and judge deviation jointly. Their method, NAOD, is supported by theoretical guarantees and by synthetic and Chatbot Arena experiments.

Critical strengths

Calibrating a judge against trusted human data only balances its errors on the distribution used for calibration, while active selection deliberately samples a different distribution. Errors that cancel on average can therefore reappear on the comparisons chosen for feedback. Recognising that selection and judge bias interact, rather than as separate steps, is the paper’s main conceptual contribution. The paper also identifies its limitations: Section 5 shows that when judge deviation is learned from limited historical data, NAOD can perform worse than the simpler Target-info baseline. Establishing when the method should not be used makes the conditions for its advantage explicit.

Critical weaknesses

The method relies on learning judge deviation, which requires historical data. When this is learned from limited data, the resulting error can make the method perform worse than the simpler Target-info design. The authors demonstrate this on synthetic data in Figure 2(c), where the threshold is around m/n = 1. However, it is not clear whether this threshold holds on real data.

In Table 1, NAOD reduces mean proxy policy regret over Target-info by 29.11%, with 14 of 15 splits favouring NAOD. The corresponding improvement in held-out human choice accuracy is +0.058 percentage points (Table 10). If the predicted winner rarely changes, it is unclear whether the regret improvement reflects a meaningful change in policy behaviour.

Questions for authors

The reported gains are concentrated at smaller budgets. For example, as shown in Figure 4(b), the advantage over D-opt narrows as the judge budget grows and the mean difference becomes negative at the two largest settings. This is a reasonable operating regime, since acquisition design matters most when labels are scarce. Nonetheless it would be helpful to reflect this scope in the main body, and to comment on whether the narrowing is expected to continue at larger budgets.

Could the authors report real-data results when the judge deviation is learned from fewer upstream comparisons? This would indicate whether the degradation shown in Figure 2(c) also appears outside the synthetic setting, and would give a clearer sense of how much historical feedback the method requires in practice.

It would be helpful if the authors could justify why the reduction in proxy policy regret is practically meaningful, given the small accompanying difference in human choice accuracy (Table 1).

Reviewer_s6T2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies active preference acquisition when inexpensive feedback is provided by an LLM judge whose preferences may systematically differ from those of the target human population. The authors formulate human reward as the target parameter and the judge deviation as a nuisance parameter. They propose Nuisance-Adjusted Optimal Design (NAOD), which selects comparisons by minimizing a policy-weighted inverse effective-information criterion based on the Schur complement of the joint target–nuisance information matrix. A Frank–Wolfe relaxation, followed by feasible rounding and exchanges, makes the criterion applicable to finite candidate pools. The paper establishes a conditional local asymptotic minimax lower bound for policy risk and constructs a projected one-step estimator attaining the bound. It also derives an additional risk term caused by learning the nuisance representation and shows that this term can reverse the oracle ordering of acquisition designs. Synthetic experiments validate these mechanisms. On Chatbot Arena, across 17 judges, 15 budget settings, and 15 cluster-level splits, NAOD reduces mean proxy policy regret by 29.1% relative to the matched Target-info design and yields smaller, but positive overall, improvements in held-out human cross-entropy and choice accuracy.

Critical strengths

The Schur-complement criterion cleanly removes target-score variation that is reproducible by nuisance-score variation, while the policy curvature assigns importance only to reward directions that affect the induced policy.

Critical weaknesses
  1. The 29.1% relative reduction is measured in proxy policy regret within a learned, judge-specific two-dimensional target space. The direct held-out human improvements are directionally supportive but small in absolute terms: relative to Target-info, the overall gains are approximately 0.000531 nats in cross-entropy and 0.058 percentage points in choice accuracy. This limits how strongly the current experiment establishes practically meaningful human-preference improvement beyond the fitted proxy geometry.

  2. The bounded soft-feedback result requires mean correctness of the joint model; under mean misspecification, the theory describes fluctuations around a potentially displaced population root. Positive out-of-fold predictive gain for the nuisance representation does not by itself establish mean correctness or small target-score residual moments. Moreover, the reported t-intervals use 15 repeated partitions of the same Arena archive. They characterize split-to-split variability conditional on that archive, but not full dataset-level sampling uncertainty. This is particularly relevant because the improvements over D-opt and the direct human-metric gains are small.

Questions for authors
  1. Could the authors report held-out calibration or residual-moment diagnostics for the fitted joint model, particularly quantities related to E[(YJ−qθ^,a^(Z))X(Z)]\mathbb{E}[(Y_J-q_{\hat{\theta},\hat{a}}(Z))X(Z)] and E[(YH−pθ^(Z))X(Z)]\mathbb{E}[(Y_H-p_{\hat{\theta}}(Z))X(Z)]? Positive out-of-fold predictive gain for the nuisance representation does not necessarily establish the mean-correctness condition used in the soft-feedback theory. A sensitivity analysis over the nuisance rank, residual learner, or target dimension would also help determine how robust the empirical gains are to representation misspecification.

  2. Could the authors clarify the intended inferential interpretation of the split-level tt-intervals? Since the 15 repeated splits reuse the same underlying Arena archive, these intervals appear to quantify variability over random partitions conditional on this archive rather than archive-level sampling uncertainty. Do the main NAOD-versus-Target-info and NAOD-versus-D-opt conclusions remain unchanged under a cluster bootstrap or another procedure that accounts for reuse of the underlying comparisons?

  3. In the released experimental protocol, PA D-opt receives the 128 initialization comparisons as past information, whereas NAOD and Target-info also incorporate the information matrix of the current human reservoir during acquisition. Could the authors report a past-aware D-opt variant that additionally includes the planned current human comparisons, or explain why such a variant would not constitute an appropriate matched baseline?

1 comment

Dear Reviewer,

Thank you again for your valuable feedback. As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our rebuttal and additional experiments have addressed your concerns, or if any further clarification is needed.

Thank you for your time and consideration.

Comments on the paper

1 comment

Additional Clarification on Evaluation Metrics and Practical Significance

We sincerely thank all reviewers for their valuable feedback and constructive suggestions, and the Area Chairs, Senior Area Chairs, and Program Chairs for their time and efforts in overseeing and coordinating the review process. As all three reviewers raised concerns about the modest improvements in held-out human accuracy and cross-entropy (CE), we provide a consolidated clarification and additional probability diagnostics.

1. Why are improvements in Accuracy and CE modest?

Hard-choice accuracy only records whether the predicted preference probability crosses the 0.5 threshold. Predictions of 0.51 and 0.90 receive the same accuracy credit when the human label favors A, despite indicating very different preferences. CE instead scores the full predicted probability against the observed label.

To understand the scale of CE, we analyzed the upstream human reference and the test-set fitted reference under the original (d,r)=(2,2)(d,r)=(2,2) representation.

  • For the upstream reference, 61.10% of predictions lie in [0.4, 0.6], and 88.58% in [0.3, 0.7]. Mean predicted-winner confidence is 59.71%, mean predictive entropy is 0.659788, and held-out human CE is 0.669858.
  • For the test-set fitted reference, which directly uses evaluation labels, 68.90% of predictions lie in [0.4, 0.6], and 93.00% in [0.3, 0.7]. Mean predicted-winner confidence is 58.16%, mean predictive entropy is 0.669402, and in-sample CE is 0.667837.
  • Central confidence bins exhibit reasonable group-level calibration. In the 55%–60% bin, the upstream reference predicts 57.35% confidence versus 55.96% observed human agreement. The test-set fitted reference yields 57.28% versus 57.51%, respectively.
  • 29.35% of evaluation weight consists of human tie labels in Chatbot Arena. Under tie-aware CE, a tie with p=0.5p=0.5 contributes ln⁡2≈0.693147\ln 2\approx0.693147. NAOD CE is 0.723478 on ties and 0.647667 on non-ties, giving 0.669915 overall.

Together, these results help explain why CE is around 0.67 under the current representation and Bradley–Terry model family. Even fitting directly to the evaluation labels yields probabilities predominantly near 0.5, for which each CE contribution is near −ln⁡(0.5)≈0.693147-\ln(0.5)\approx0.693147. Tie labels also contribute to this pattern. Thus, the small differences between acquisition methods should be understood in the context of the limited additional fit observed within the adopted representation and model family.

2. Why is Policy Regret more practically meaningful?

The purpose of preference acquisition is not simply to make a classifier more accurate. It is to estimate a reward function that induces a better downstream policy.

In actual LLM use, the model must generate accurate, helpful, human-aligned responses to complex requests, not merely identify the winner among two given answers. Pairwise comparisons are a training signal, not the final task.

Accuracy measures predicted winners and CE evaluates predicted preference probabilities. Neither directly measures the value of the induced policy. In contrast, policy regret measures the downstream KL-regularized policy-value loss. Two reward models can predict the same winner yet induce different Gibbs policy distributions and values, so small gains in accuracy need not correspond to small policy changes.

This distinction is central to NAOD. Our acquisition rule prioritizes information that matters for downstream policy decisions, rather than classification accuracy alone.

3. Further Evidence from Held-Out Human-Preference Utility

We additionally evaluate negative policy updates using held-out human labels without using the proxy reward reference as the evaluation target.

Let pi=σ(Xi⊤θ^)p_i=\sigma(X_i^\top\hat\theta) be the probability of selecting A and hi∈{0,0.5,1}h_i\in\{0,0.5,1\} be the observed human preference. We compute expected human-preference utility as

U^(θ^)=1G∑g=1G1∣Tg∣∑i∈Tg[pihi+(1−pi)(1−hi)].\widehat U(\hat\theta)=\frac1G\sum_{g=1}^{G}\frac1{|T_g|}\sum_{i\in T_g}\left[p_i h_i+(1-p_i)(1-h_i)\right].

Scores are averaged within each held-out cluster, then equally over clusters. A negative update occurs when

ΔU=U^(θ^)−U^(θInit128)<0.\Delta U=\widehat U(\hat\theta)-\widehat U(\theta_{\mathrm{Init128}})<0.

NAOD reduces the negative-update rate to 27.11% overall, versus 30.20% for Target-info and 36.26% for D-opt. In the high-bias judge group, the rates are 21.85%, 26.00%, and 36.37%, respectively.

These results indicate more reliable policy updates on held-out human-preference utility. The improvement is not restricted to proxy policy regret and is corroborated by a separate evaluation against actual human labels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.