acceptodds
Under review as a conference paper at ICLR 2027

DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

Abstract

Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human preferences. We introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
uncertainty quantification, pairwise comparison, LLM evaluation, LLM-as-a-judge, adaptive calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bLxoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a statistical framework that corrects the position biases of LLM judges and calibrates them toward human preferences. It fits the preferences of each judge model into a BTL model with an order-effect parameter bkb_k. The recovered preferences from different judges are then decomposed into a consensus space and disagreement directions. Human preferences are incorporated via a combination of human and LLM likelihoods. Finally, the paper proposes a GVAC-based selection of the LLM/human weighting parameter. Experiments show that the framework is robust against position biases under Chatbot Arena, MT-Bench, and PandaLM, and can achieve high human-label agreement with few human labels.

Critical strengths
  • Experiments are extensive, covering multiple judges and test benchmarks.
  • The paper proposes a unified framework for both position biases and human preference calibration. These two issues are usually addressed separately in prior work.
Critical weaknesses
  • The human-alignment assumption is theoretically flawed. The paper assumes that human preference vectors are well-approximated by a subspace learned by LLM judges. This does not hold when LLMs and humans disagree systematically. The experiments use consensus calibration with r=1, instead of demonstrating that human preferences can be well captured across different tasks or LLM preferences.
  • The observed improvement over baselines is not strong enough to justify the complexity of the framework. Simpler methods such as AtC remain competitive in the human-label efficiency experiments.
Questions for authors
  • What if the human preferences have a large orthogonal component to the LLM's learned preference spaces? How large can this approximation be before your method loses its edge?
  • Can you do the same label efficiency experiment with a wider budget coverage, ranging from zero budget to full budget? How will AtC compare to your method in a high-budget setting?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bLxo (R3)

Thank you for recognizing that the experiments are extensive and that the framework unifies position debiasing and human preference calibration, which are usually addressed separately in prior work. DIAL is designed for evaluations in which both kinds of judgment are costly: human labels are few, and querying a panel of LLM judges on every record in both display orders is expensive. DIAL saves on both:

  1. it corrects order effects in the model, so one display order per LLM comparison suffices (Proposition 1, Figure 3);
  2. judges share a preference space, so each judge needs fewer comparisons (Proposition B.2); and
  3. the human comparisons estimate a few calibration coefficients and set how strongly to rely on the LLM evidence (Figures 4 and G8). When LLM judgments are abundant and well aligned, the debiased consensus alone already performs well.

We address your comments about weaknesses and questions below:

[W1(i), Q1] (Human-LLM alignment and its limits) We agree that the calibration restriction can fail under systematic human-LLM disagreement, and our theory and estimator cover that case. Theorem 3 allows the human scores to lie outside the LLM space, and the adaptive estimator ranges from the anchored fit to the human-only fit, which uses no alignment assumption (Section 5.1). Exact calibration is assumed for the fixed-weight intervals (Theorem 4). Below we give the empirical evidence for partial alignment and quantify how estimation changes as the mismatch grows.

  • Evidence for partial alignment. Post-training with human preference feedback provides a practical reason to expect some alignment between LLMs and humans. Empirically, in all three real datasets, Figure 1(b) shows substantial agreement between human labels and order-consistent LLM verdicts, while Figure 4(a) shows strong held-out human ranking agreement for the position-debiased LLM consensus.
  • Motivation for DIAL. For the overall human-preference target, the shared LLM preference information is useful, as demonstrated by DIAL's strong performance with few human labels (Figure 4(a)). However, overall alignment can mask systematic disagreement within subgroups: on non-English Arena prompts, both the exact and the tolerance calibration-space tests reject the global consensus in most splits, whereas a consensus refitted on those prompts is not rejected and improves held-out performance (Figure G7(b,c)). We discuss this subgroup-specific misalignment in Section 7, where subgroup-aware calibration is a future direction.
  • Quantified limits and robustness. Under local misspecification, Theorem 3 shows that oracle anchoring has lower first-order risk than unrestricted human-only BTL when nHΔM<(N−1−d)/2n_{\mathrm{H}}\Delta_M<(N-1-d)/2 and the tolerance version of the calibration-space test takes this condition as its null, so it can be checked on data (following Proposition D.11; Figure G7(c)). Our new Figure F4 adds a human component orthogonal to the whole LLM space, of size ρ\rho relative to the consensus component, and confirms the predicted loss of advantage near ρ≈0.31\rho\approx0.31 at nH=200n_{\mathrm{H}}=200 and 0.150.15 at 800800. DIAL-Ada moves toward the human fit as ρ\rho grows, and near the boundary its risk can slightly exceed that of the human-only fit. We also examine robustness to injected anti-consensus judges (Figure G6(c,f)).
AuthorsOpenReview ↗

Response to Reviewer bLxo (R3) - Cont.

[W1(ii)] (Choice of r=1r=1) Here rr counts the judge-disagreement directions beyond the consensus μ\mu in the LLM score matrix, and the method allows any rr (Figure F1 uses r=2r=2). On the full LLM panels, BIC, which consistently recovers this rank (Yu et al., 2026), selects r=1r=1 on Chatbot Arena and r=0r=0 on MT-Bench and PandaLM; we added a rank sensitivity analysis showing that refitting with r=0r=0 or r=2r=2 changes the held-out log loss and Kendall's τ\tau of DIAL-Ada by less than one standard error, so we report r=1r=1 throughout (Table G5). Across tasks and LLM panels, consensus calibration predicts held-out human preferences with few labels on all three benchmarks, the label gains persist on smaller judge panels, and the oracle gains from adding disagreement directions are negligible (Figures G8 and G9, Section G.4). When human preferences depart from the consensus, DIAL-Ada (W) calibrates along the disagreement directions and attains nominal coverage in the misaligned simulation (Figure F2). On real data, the tolerance test at the boundary of Theorem 3 rarely rejects the consensus on English Chatbot Arena prompts, whereas both the exact and the tolerance tests reject the global consensus on non-English prompts in most splits (Figure G7(b,c)).

[W2, Q2] (AtC across the full budget range) We have extended the experiments as suggested, and the results further clarify DIAL's main benefit.

  • Full-budget comparison. The revised Figures 4 and G8 cover the full range from zero human labels to the full human training pool, including Firth-regularized AtC at small budgets. At the full pool, AtC slightly exceeds DIAL in Kendall's τ\tau on Chatbot Arena and PandaLM, while DIAL-Ada retains the lowest held-out log loss on Chatbot Arena; all the methods become increasingly similar as human labels grow.
  • Intended use case. As motivated in Figure 1, DIAL addresses the practical challenge of learning human preferences from limited human judgments, where our experiments show its largest gains (Table GR): Kendall's τ\tau for DIAL-Ada against AtC (Firth) is 0.870.87 against 0.640.64 with 100100 labels on Chatbot Arena, and 0.930.93 against 0.810.81 on MT-Bench and 0.720.72 against 0.680.68 on PandaLM with 2020 labels. AtC needs the human comparisons to identify all N−1N-1 score contrasts and keeps the LLM consensus fixed; DIAL needs rank⁡(DHM)=d\operatorname{rank}(D_{\mathrm{H}}M)=d, lets the human comparisons update an imperfect consensus (Figure 4(b)), and provides confidence intervals (Proposition 2 and Theorem 4). DIAL is thus most useful when both human labels and LLM judgments are costly, since it needs one display order per LLM comparison, fewer comparisons per judge, and few human labels.
Reviewer_ujDgOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DIAL to learn human ranking scores using limited human data and abundant LLM data. However, LLM data may be biased, and the proposed algorithm makes a trade-off between bias and variance under the inductive bias that human scores are close to a low-dimensional latent space of multiple LLM scores after adjusting for position bias. The paper provides theoretical guarantees and shows good empirical performance.

Critical strengths
  1. Solid theoretical guarantees. The theory is solid, and the authors are clear about what they are estimating. The paper provides theoretical guarantees for the proposed method.

  2. Good empirical performance and presentation. The experimental results are promising. The paper is also generally well written and easy to follow.

Critical weaknesses

Justification for the low-rank model. I think the most important question the paper needs to answer is why the authors choose this particular low-rank model. There are many ways to leverage LLM data to improve the estimation of human ranking scores. For example, one could use standard transfer learning, prediction-powered inference (PPI), or joint training with an L2 penalty that encourages human and LLM scores to be close. Why is the proposed low-rank model preferable to these alternatives? I think the authors need to better justify this modeling choice.

Limited empirical comparisons and demonstration of inference. There are many approaches to using LLM judges to assist human score estimation, but the paper does not provide sufficient comparisons with these alternatives. In addition, although the proposed method supports statistical inference, the real-data experiments do not demonstrate the practical value of this capability.

There are also a few minor presentation issues. Some parts of the paper are not sufficiently clear, and a few sentences have traces of AI-generated writing. I encourage the authors to improve these passages and specify which words or sentences have been revised in their response. Overall, however, the writing is quite good and the paper is generally clear.

Also, GACV is used without giving its full name when it first appears.

Questions for authors
  1. Why is position bias the main bias the authors want to correct? LLMs may have other biases, such as preferences for particular output formats or longer responses. Could these biases also be important? Do they need to be corrected as well?

  2. What would happen if we did not correct position bias? For example, could we simply treat position bias as an additional dimension of the latent space? How would this affect the proposed method and its theoretical guarantees?

  3. Do the authors actually observe position bias in their data? Is there empirical evidence for its presence, and how can it be tested or quantified?

  4. How sensitive is the proposed method to model misspecification? In particular, what happens if the assumed model for the LLM scores or their relationship with human scores is incorrect?

  5. Why do the authors choose this particular low-rank model to leverage LLM judges? As mentioned above, there are several alternatives, including standard transfer learning, PPI, and joint training with an L2 penalty. What is the main advantage of the proposed model over these approaches?

  6. Could the authors provide comparisons with alternative approaches that use LLM judges to improve human score estimation? Such comparisons would help clarify the empirical benefits of the proposed method.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ujDg (R2)

Thank you for the positive assessment of the theoretical guarantees, for noting that we are clear about what we are estimating, and for finding the empirical results promising and the writing generally clear. Your comments led us to compare the low-rank model with the alternatives you listed, in both modeling and experiments, and to show the practical use of the intervals on real data.

[W1, Q5] (Why the low-rank model) The low-rank model reduces human calibration to a few coefficients in a preference space shared by the position-debiased LLM judges; transfer learning, PPI, and ℓ2\ell_2-penalized joint training use the LLM information differently.

  • Human-label efficiency. DIAL learns a low-dimensional preference space from the position-debiased scores of multiple LLM judges. For a fixed space of dimension dd, calibration estimates only dd human coefficients instead of N−1N-1 scores, which allows identification under incomplete human coverage (Proposition 2) and gives lower first-order risk when human preferences are sufficiently aligned with the space (Theorem 3). See also [GR2], benefits (i) and (ii).

  • Differences from alternatives. Transfer learning can use the LLM scores as an initialization or a regularization target, and Joint-ℓ2\ell_2, the ℓ2\ell_2 version we added, shrinks an unrestricted human-score fit toward the fixed LLM consensus space. DIAL's adaptive weighted likelihood also lets the human comparisons update the learned preference space (Section 5.1). AtC isotonic-calibrates the LLM consensus to an ordering aggregated from the human comparisons, and PPI++ uses matched proxy verdicts for variance reduction. See also [GR2], [R1W1, R1Q1] for PPI++, and [R2W2(i), R2Q6] for the empirical comparisons.

[W2(i), Q6] (Comparisons with alternative estimators) We added the requested comparisons: DIAL gains most when human labels are limited, and the alternatives match it as the labels grow.

  • Added baselines. We added Joint-ℓ2\ell_2 and Bradley-Terry PPI++, together with Firth-regularized Human and AtC, on the synthetic design and all three benchmarks (Table GR in our general response; Figures 2, F3, 4, and G8). All methods receive the same human records and LLM sample, and PPI++ is reported where every human-labeled record carries an LLM verdict. SAJA and PRECISE do not estimate an item-level latent score (SAJA calibrates record-level LLM scores; PRECISE estimates ranking metrics such as Precision@KK by prediction-powered estimation), so PPI++ is the closest comparable instance (Section A). See also [GR1] and [R1W2, R1Q2] for the regularized baselines, and [R1W1, R1Q1] for the PPI++ design.

  • Empirical results. On Chatbot Arena with 100100 human labels, Kendall's τ\tau is 0.870.87 for DIAL-Ada, 0.820.82 for Joint-ℓ2\ell_2, 0.640.64 for AtC (Firth), 0.580.58 for PPI++ (over its finite fits), and 0.530.53 for Human (Firth). The gains are largest when the LLM preferences are informative and human labels are limited; with more human labels or weaker human-LLM alignment, the alternatives match or exceed DIAL (Figures F4 and G8). See also [GR2] and [R1W1, R1Q1] for the comparison with PPI++, and [R3W2, R3Q2] for AtC across the human budgets.

[W2(ii)] (Practical value of inference) Estimated human-preference scores and the rankings they induce are subject to sampling uncertainty, so confidence intervals for pairwise score differences help assess which model orderings the data support. We added a real-data demonstration (Section 6.2 and Table G6). With 200200 human labels, the 9595% cluster-robust intervals of DIAL-Ada at the selected weight exclude zero for 2828% of the pairs of models adjacent in the estimated ranking on Chatbot Arena and 8484% on MT-Bench, against 2828% for the human-only fit on MT-Bench, whose MLE does not exist in some Chatbot Arena splits. These pairs agree with the held-out human ranking in 9696-9797% of cases, and repeating the weight selection when resampling the records leads to similar conclusions. The intervals thus tell a practitioner which model comparisons the available labels support and which remain uncertain.

AuthorsOpenReview ↗

Response to Reviewer ujDg (R2) - Cont.

[W3] (Clarity and AI-writing traces) Thank you for your suggestions! As the comment does not name specific words or sentences, we reread the main text and reworded or removed phrasing that may read as awkward or as jargon. The changes are:

  • (i) in the abstract, "benchmarks, showing that DIAL remains robust" →\to "benchmarks. DIAL remains robust" and "providing a resource for future studies" →\to "which can be used in future studies";
  • (ii) in Section 1, "providing a practical complement to" →\to "as a practical complement to", "Moreover, LLM and human evaluations can differ" →\to "LLM and human evaluations can also differ", "two distinct but coupled challenges" →\to "two challenges", "What remains less understood is how to connect these" →\to "It is less understood how to connect them", and "becomes especially valuable" →\to "helps most";
  • (iii) for clarity, the two introduction sentences on the learned representation and adaptive weighting now read "With this representation, the human comparisons estimate a few calibration coefficients along the learned preference directions instead of one score per item. Adaptive weighting controls how much the estimator relies on LLM evidence, between human-only and LLM-anchored estimation"; and
  • (iv) in Section 7, "DIAL treats position debiasing and human-preference alignment as distinct but complementary" →\to "DIAL separates two tasks", and we deleted "as reflected in both theory and experiments".

We would be glad to revise further if you could point out any remaining passages.

[W4] (GACV acronym) Thank you for pointing this out; we now spell out Generalized Approximate Cross-Validation (GACV) at its first use in Section 1.

AuthorsOpenReview ↗

Response to Reviewer ujDg (R2) - Cont..

[Q1] (Why position bias; other biases) Thank you for this comment. We acknowledge that LLM judges may exhibit other biases, such as verbosity and formatting preferences. Our focus on position bias is motivated by:

  • Why position bias. Position bias is relatively straightforward to quantify, as display order can be manipulated without changing response content, allowing us to identify and estimate position effects separately from latent preferences (Proposition 1). Moreover, Figure 3 demonstrates substantial performance degradation when position bias is ignored (see [R2Q3]).

  • Other biases. Unlike position bias, verbosity and formatting biases are often entangled with response quality, making them harder to isolate. Nevertheless, DIAL could accommodate other systematic biases through additional covariates, subject to appropriate identifiability conditions.

We have added this extension to Section 7.

[Q2] (Uncorrected position bias) We answer the two parts of the question in turn.

  • Impact of ignoring position bias. Without the correction, the fitted LLM scores absorb the order effect: even with balanced display, a BTL fit to order-averaged judgments is misspecified (Section B.2). Empirically, the order-agnostic Pooled and DIAL-noPos fits deteriorate under unbalanced display, while the order-aware DIAL fits remain stable (Figures 2 and 3); see also [R2Q3].

  • Position bias as an additional dimension. Yes, when that dimension is the observed display order: the linear predictor is then $

Dk,zkD_k,z_k

(s_k^{\top},b_k)^\top=s_{ki}-s_{kj}+ab_k$, which is our model, so the method and all guarantees are unchanged (Proposition 1). We keep bb outside the low-rank structure because the human model has no order term. An additional latent dimension on the item side, however, cannot capture position bias: it contributes uk(vi−vj)u_k(v_i-v_j) under either display order, whereas abkab_k changes sign with the order (remark added in Section B.1).

[Q3] (Evidence of position bias) Thank you for this question. We provide empirical evidence of position sensitivity and its impact on evaluation performance:

  • Observed position sensitivity. Across 21 LLM judges and three benchmarks (Chatbot Arena, MT-Bench, and PandaLM), order-inconsistency rates in Figure 1(b) range from approximately 8% to 80%, with some judges reaching 75-80%. This demonstrates substantial and heterogeneous sensitivity to response order.

  • Impact on evaluation reliability. Under one-sided display with eight injected pure-position judges in Figure 3, Pooled's excess held-out human log loss reaches approximately 0.155 on Chatbot Arena and 0.25 on MT-Bench, while position-aware DIAL remains near 0.015 on Chatbot Arena and substantially more stable across all three benchmarks. DIAL-noPos also degrades, confirming the importance of explicit position modeling.

We additionally provide judge-level estimates of position effects on real-world data in Section G.3, further characterizing systematic position bias beyond order-inconsistency rates.

[Q4] (Misspecification sensitivity) Both kinds are evaluated.

  • For the LLM side, Figure 2 (bottom) misspecifies the LLM likelihood by judge-pair overdispersion, Figure F5 lets the order effect vary across pairs, and Figures 3 and G6 use one-sided display and injected pure-position or anti-consensus judges; the order-aware fits stay stable, and the adaptive weight limits the damage when the consensus is corrupted.
  • For the Human-LLM relationship, Figure F2 (human component inside the LLM space but outside the consensus direction) and the new Figure F4 (component orthogonal to the whole LLM space) quantify the cost through Theorem 3, and Proposition D.11 and the tolerance test following it check the calibration space and detect the mismatch on real data (Figure G7(b,c)). The failure mode is DIAL-Anc, which keeps the LLM ranking; DIAL-Ada falls back to the human fit, see [R3W1(i), R3Q1].

Thanks for the response. I'll maintain my original rating.

Reviewer_S3fzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies position bias and human preference calibration for LLM judges. DIAL (Debiasing and human-preference Informed Alignment for LLM judges ) separates position effects from judge preferences and learns a shared representation across judges. A limited number of human comparisons are used to calibrate this representation. The authors provide identification and inference results, and further conduct simulations and three benchmark studies.

Critical strengths
  1. The problem is interesting. The paper distinguishes position bias from disagreement with human preferences.
  2. The identification results are very useful. They clarify when preferences can be recovered from incomplete comparisons.
  3. The theoretical and empirical analyses are also substantial. The simulations also examine several departures from the model assumptions.
Critical weaknesses
  1. The comparison with existing methods needs further clarification. Boyeau et al. (2025) already estimate human Bradley-Terry scores using LLM judgments and limited human comparisons. Thus, the distinction between latent scores and aggregate quantities seems not sufficient. The empirical advantage over their Bradley-Terry PPI++ estimator remains unclear.

  2. The comparisons at small human-label budgets are limited. Figure 4 omits Human and AtC below 800 labels because their unregularized BTL estimates are not always finite. A regularized human-only baseline would make the comparison more informative.

  3. The role of human calibration in the ranking results is unclear. In the default anchored model, calibration multiplies the LLM consensus by one coefficient. A positive coefficient leaves the ranking unchanged. The anchored ranking results may therefore mainly reflect the initial consensus.

  4. The uncertainty estimates need further justification. Appendix E.2 acknowledges omitted dependence among judgments on the same record. Theorem 4 assumes a fixed weight and exact calibration. It does not directly cover the intervals after weight selection. The undercoverage in Figure F2 may also add to this concern.

Questions for authors
  1. How does DIAL compare with the Bradley-Terry PPI++ estimator under the same human and LLM budgets?
  2. Do the gains at small human-label budgets persist against regularized Human and AtC estimators?
  3. In Figure G6, how much does human calibration improve held-out log loss relative to the LLM-only consensus?
  4. How sensitive are the intervals to a record-level bootstrap that repeats weight selection?
4 comments
AuthorsOpenReview ↗

Response to Reviewer S3fz (R1)

Thank you for recognizing that the problem is interesting, that the identification results clarify when preferences can be recovered from incomplete comparisons, and that the theoretical and empirical analyses are substantial. Your comments led us to correct the comparison with Bradley-Terry PPI++, add regularized baselines at small budgets, clarify the role of human calibration, and extend the inference to dependent records.

[W1, Q1] (Comparison with BT PPI++) We agree that the distinction between latent scores and aggregate quantities does not separate DIAL from Bradley-Terry PPI++ (Boyeau et al., 2025), since both estimate item-level human scores; and we have corrected the comparison with it in Section 2 and Section A accordingly. The key difference is how the LLM judgments are used.

  • How the LLM judgments enter. PPI++ and DIAL differ in three respects.

    • (i) Data. PPI++ uses an LLM verdict as a record-level proxy for the human verdict, so it requires matched records: every human-labeled comparison must also carry the proxy's verdict on the same record. DIAL requires no matching: it fits the LLM judgments on their own records, so the human and LLM comparisons may cover different records or different pairs. This lets DIAL pair existing human labels with LLM judgments on fresh records, which can be preferable as recent judges may have been trained on published human labels.

    • (ii) Use of the LLM judgments. PPI++ uses them only through how well the proxy predicts the human verdict on each labeled record. DIAL estimates judge-specific position effects, debiased judge scores, and their shared preference space col⁡(M)\operatorname{col}(M), and calibrates that space to the human comparisons.

    • (iii) Target. PPI++ keeps the unrestricted human score with N−1N-1 free coordinates, whereas DIAL estimates the human score within col⁡(M)\operatorname{col}(M), so in a fixed learned space the human labels determine only d=dim⁡col⁡(M)d=\dim\operatorname{col}(M) coordinates (DIAL-Ada also lets them update the space).

  • Different trade-offs. PPI++ is consistent for sHs_{\mathrm{H}} and gives asymptotically valid intervals for any proxy when the labeled records are a random subsample, but its gain is a variance reduction that depends on the covariance of the proxy and human loss gradients, and it still estimates all N−1N-1 coordinates. For oracle anchoring, Theorem 3 gives first-order excess risk (δM+d/2)/nH(\delta_M+d/2)/n_{\mathrm{H}} against (N−1)/(2nH)(N-1)/(2n_{\mathrm{H}}) for the human-only fit, so anchoring helps when δM<(N−1−d)/2\delta_M<(N-1-d)/2; PPI++ lowers the human-only constant only by that variance reduction, which explains DIAL's potential advantage over PPI++, and the comparison below confirms it; otherwise the adaptive weight moves DIAL toward the human fit (Figure F4). Beyond matching, PPI++ needs a human design that identifies all N−1N-1 contrasts, whereas DIAL needs only rank⁡(DHM)=d\operatorname{rank}(D_{\mathrm{H}}M)=d, which can hold for a disconnected human graph (Proposition 2).

  • Comparison under the same budgets. In each experiment both methods receive the same human records and LLM sample, with the panel-majority verdict as the PPI++ proxy; the Table GR in our general response reports Kendall's τ\tau.

    • (i) Accurate LLM consensus: synthetic, Chatbot Arena with all LLM data, MT-Bench (Figures F3 and 4(a), Figure G8). DIAL leads at small budgets: at 100100, 100100, and 2020 labels, τ\tau is 0.980.98, 0.870.87, and 0.930.93 for DIAL-Anc against 0.750.75, 0.580.58, and 0.720.72 for PPI++. PPI++ with nHn_{\mathrm{H}} labels falls between Human with nHn_{\mathrm{H}} and 2nH2n_{\mathrm{H}} labels on Chatbot Arena, consistent with a variance reduction on all N−1N-1 coordinates; in the simulation it follows the (N−1)/(2nH)(N-1)/(2n_{\mathrm{H}}) risk scale of Human, and DIAL-Anc the 1/(2nH)1/(2n_{\mathrm{H}}) scale. The matched simulation provides the records PPI++ requires, but its human and judge verdicts are independent given the item pair, so the proxy carries no information on the human verdict beyond the pair and PPI++ gains little there (Section F.2). The DIAL-Anc ranking is the LLM consensus ranking (see [R1W3]), and its excess log loss at 100100 labels on Chatbot Arena is 0.0160.016, against 0.1280.128 for Human (Firth) and 0.1640.164 for PPI++.

    • (ii) Imperfect LLM consensus: Chatbot Arena with 2,0002{,}000 LLM comparisons, PandaLM (Figure 4(b), Figure G8). On Chatbot Arena the consensus reaches only τ=0.78\tau=0.78, and DIAL-Ada moves toward the human data, reaching 0.830.83 at 800800 labels; PPI++ is not applicable in this design, where the LLM sample is drawn separately and most labeled records carry no LLM verdict. On PandaLM at 160160 labels, AtC exceeds DIAL in τ\tau (0.810.81 against 0.790.79) and PPI++ is close (0.770.77), as the trade-off above predicts once the human sample can estimate all coordinates.

    Overall, DIAL-Ada performs best in most settings of Table GR, with the largest margins at small human budgets.

AuthorsOpenReview ↗

Response to Reviewer S3fz (R1) - Cont.

[W2, Q2] (Regularized baselines at small budgets) We added Firth-regularized (Firth, 1993) Human and AtC fits, which exist at every budget (Figures 4 and G8); ridge or box penalties give similar rankings, with Kendall's τ\tau within 0.040.04 at nH≤100n_{\mathrm{H}}\le100 on Chatbot Arena (Sections G.2 and G.4).

  • Result. Against the regularized baselines, the gains of DIAL persist at the smallest budgets on all three benchmarks (Table GR). With more human labels the gap narrows: the human ordering that AtC builds on becomes accurate, so AtC (Firth) matches DIAL, and exceeds it on PandaLM, where the LLM consensus is imperfect.

  • Why regularization does not close the gap. Regularization makes the fits finite, but it adds no information to the labels: a penalized Human or AtC fit still estimates all N−1N-1 scores from the same few comparisons. At small budgets the Chatbot Arena human comparison graph is often disconnected (3333 and 55 of the 5050 splits at 5050 and 100100 labels; Section G.4), so the contrasts between components are set by the penalty, not by the data. In contrast, DIAL estimates a single calibration coefficient and needs only rank⁡(DHμ)=1\operatorname{rank}(D_{\mathrm{H}}\mu)=1 (Proposition 2).

[W3] (Calibration and the anchored ranking) You are right for DIAL-Anc: its ranking is that of DIAL's position-debiased consensus, and the human comparisons calibrate only its probability scale, so its Kendall's τ\tau reflects the quality of that consensus (Figure 4(a)). This is not the case for DIAL-Ada: its consensus is updated and aligned toward the human preferences, because it is estimated jointly with the human comparisons through the weighted likelihood, so the human labels change the ranking itself. With nL=2,000n_{\mathrm{L}}=2{,}000 and nH=800n_{\mathrm{H}}=800 on Chatbot Arena, DIAL-Ada reaches mean τ=0.83\tau=0.83 against 0.780.78 for DIAL-Anc, showing how human comparisons improve the ranking when the consensus is imperfect (Figure 4(b)). We clarified this distinction in Section 6.2 and summarized the use cases in Section 7.

[Q3] (Benefit of human calibration) The benefit of human calibration depends on how much LLM evidence is available: with abundant LLM judgments the debiased consensus is already accurate, whereas with a moderate LLM budget, as when querying many judges is costly, calibrating toward the human comparisons matters. We added the uncalibrated LLM-only consensus, the output of DIAL's first stage, to Figure G8 (previous Figure G6), evaluated on the same held-out human records.

  • With abundant judgments from all 2121 judges, the consensus already achieves low excess log loss (0.0080.008-0.0090.009; Figure G8, top). Scalar calibration matches it at the full human pool (0.0080.008-0.0110.011) and is higher at the smallest budgets (0.0210.021, 0.0380.038, and 0.0830.083 on the three benchmarks), where the estimated coefficient adds variance; the ranking is the same with and without it (Figure 4(a)).

  • With limited LLM judgments, DIAL performs substantially better in excess log loss: on Chatbot Arena with the five order-sensitive judges subsampled to nL=500n_{\mathrm{L}}=500 and nH=1,000n_{\mathrm{H}}=1{,}000, it is 0.250.25 for the LLM-only consensus, 0.110.11 for DIAL-Anc, and 0.0160.016 for DIAL-Ada (Figure G8, bottom); in a separate experiment with nL=2,000n_{\mathrm{L}}=2{,}000, Figure 4(b) also shows the ranking benefit of adaptive updating when the LLM consensus is imperfect.

DIAL is therefore most useful when both human labels and LLM judgments are limited; as LLM judgments become abundant and accurate, the GACV-selected weight moves toward anchoring and DIAL converges to the consensus-based fit (Section G.4).

AuthorsOpenReview ↗

Response to Reviewer S3fz (R1) - Cont..

[W4, Q4] (Uncertainty under record dependence and weight selection) We added cluster-robust inference and, as a check, resampling of whole records (B=200B=200), both at the selected weight and with GACV weight selection repeated (Section E.2; Tables F1 and G6).

  • Record-level dependence. Thank you for raising this important question! Judges and the human annotator judge each record separately, but they respond to the same prompt and responses, so their verdicts on the same record can be correlated, which our original sandwich omitted. We added Proposition D.6, which extends Theorem 4 at a fixed weight to cluster-robust inference that allows dependence among judgments on the same record, and implemented the cluster-robust sandwich with record, judge-pair, and two-way kernels (Section E.2); the reported intervals use the two-way kernel. Empirically, the two-way standard errors on the real data are within 11% of the record-clustered ones, so the two kernels lead to the same conclusions (Section G.4). In simulation its coverage is nominal without a record effect and slightly below nominal with one, approaching nominal as the human labels grow (Table F1, Section F.2).

  • Uncertainty after weight selection. Theorem 4 and Proposition D.6 are fixed-weight results, as we state in the abstract, Section 5.2, and Section E.2, and Section 7 lists post-selection inference as an extension. At the GACV-selected weight, the cluster-robust standard errors are within 33-88% of those from resampling at that weight, in simulation and on real data. In simulation the coverage at the selected weight is similar to that at a prespecified weight (Table F1). To repeat the selection we draw half-samples of records without replacement, because a bootstrap sample keeps a copy of each held-out record in the fit, which invalidates GACV. Repeating the selection widens the intervals modestly, more at the smaller human budget, and the model pairs distinguished by the cluster-robust and the subsampling intervals agree with the held-out human ranking equally often (Tables F1 and G6). If a guarantee after selection is needed, a Bonferroni correction over all grid weights gives one, with critical value 2.772.77 instead of 1.961.96 for our 99 grid weights (intervals 1.411.41 times wider); we describe it, with the conditions it requires, in Section E.2.

  • Undercoverage in Figure F2 under misspecification. This design draws sH=WcH=μ+VcVs_{\mathrm{H}}=Wc_{\mathrm{H}}=\mu+Vc_V with W=[μ,V]W=[\mu,V] and cV∼N(0,0.52)c_V\sim\mathcal{N}(0,0.5^2), so sH∈col⁡(W)s_{\mathrm{H}}\in\operatorname{col}(W) but sH∉span⁡(μ)s_{\mathrm{H}}\notin\operatorname{span}(\mu). The default DIAL-Ada calibrates towards M=μM=\mu and is thus misspecified: its estimand stays near the best consensus-only approximation μcμ⋆\mu c_\mu^\star, with cμ⋆∈argmin⁡cRH(μc)c_\mu^{\star}\in \operatorname{argmin}_{c} R_{\mathrm{H}}(\mu c), which differs from sHs_{\mathrm{H}}, and its interval does not account for this bias. The undercoverage is due to this misspecification: at a fixed weight, DIAL-Ada covers even less, and the correctly specified DIAL-Ada (W) (M=WM=W) is near nominal at both the selected and the fixed weight (Section F.2). In practice, the calibration-space test (Proposition D.11) checks a chosen calibration space, M=μM=\mu or M=WM=W, against the unrestricted human fit, and its tolerance version tests whether the misalignment is large enough for anchoring to stop reducing the risk in Theorem 3 (Figure G7(b,c)).

Thank you for your detailed response and additional results. I will raise my score from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.