- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a statistical framework that corrects the position biases of LLM judges and calibrates them toward human preferences. It fits the preferences of each judge model into a BTL model with an order-effect parameter . The recovered preferences from different judges are then decomposed into a consensus space and disagreement directions. Human preferences are incorporated via a combination of human and LLM likelihoods. Finally, the paper proposes a GVAC-based selection of the LLM/human weighting parameter. Experiments show that the framework is robust against position biases under Chatbot Arena, MT-Bench, and PandaLM, and can achieve high human-label agreement with few human labels.
Critical strengths
- Experiments are extensive, covering multiple judges and test benchmarks.
- The paper proposes a unified framework for both position biases and human preference calibration. These two issues are usually addressed separately in prior work.
Critical weaknesses
- The human-alignment assumption is theoretically flawed. The paper assumes that human preference vectors are well-approximated by a subspace learned by LLM judges. This does not hold when LLMs and humans disagree systematically. The experiments use consensus calibration with r=1, instead of demonstrating that human preferences can be well captured across different tasks or LLM preferences.
- The observed improvement over baselines is not strong enough to justify the complexity of the framework. Simpler methods such as AtC remain competitive in the human-label efficiency experiments.
Questions for authors
- What if the human preferences have a large orthogonal component to the LLM's learned preference spaces? How large can this approximation be before your method loses its edge?
- Can you do the same label efficiency experiment with a wider budget coverage, ranging from zero budget to full budget? How will AtC compare to your method in a high-budget setting?
2 comments
Response to Reviewer bLxo (R3) - Cont.
[W1(ii)] (Choice of ) Here counts the judge-disagreement directions beyond the consensus in the LLM score matrix, and the method allows any (Figure F1 uses ). On the full LLM panels, BIC, which consistently recovers this rank (Yu et al., 2026), selects on Chatbot Arena and on MT-Bench and PandaLM; we added a rank sensitivity analysis showing that refitting with or changes the held-out log loss and Kendall's of DIAL-Ada by less than one standard error, so we report throughout (Table G5). Across tasks and LLM panels, consensus calibration predicts held-out human preferences with few labels on all three benchmarks, the label gains persist on smaller judge panels, and the oracle gains from adding disagreement directions are negligible (Figures G8 and G9, Section G.4). When human preferences depart from the consensus, DIAL-Ada (W) calibrates along the disagreement directions and attains nominal coverage in the misaligned simulation (Figure F2). On real data, the tolerance test at the boundary of Theorem 3 rarely rejects the consensus on English Chatbot Arena prompts, whereas both the exact and the tolerance tests reject the global consensus on non-English prompts in most splits (Figure G7(b,c)).
[W2, Q2] (AtC across the full budget range) We have extended the experiments as suggested, and the results further clarify DIAL's main benefit.
- Full-budget comparison. The revised Figures 4 and G8 cover the full range from zero human labels to the full human training pool, including Firth-regularized AtC at small budgets. At the full pool, AtC slightly exceeds DIAL in Kendall's on Chatbot Arena and PandaLM, while DIAL-Ada retains the lowest held-out log loss on Chatbot Arena; all the methods become increasingly similar as human labels grow.
- Intended use case. As motivated in Figure 1, DIAL addresses the practical challenge of learning human preferences from limited human judgments, where our experiments show its largest gains (Table GR): Kendall's for DIAL-Ada against AtC (Firth) is against with labels on Chatbot Arena, and against on MT-Bench and against on PandaLM with labels. AtC needs the human comparisons to identify all score contrasts and keeps the LLM consensus fixed; DIAL needs , lets the human comparisons update an imperfect consensus (Figure 4(b)), and provides confidence intervals (Proposition 2 and Theorem 4). DIAL is thus most useful when both human labels and LLM judgments are costly, since it needs one display order per LLM comparison, fewer comparisons per judge, and few human labels.
Response to Reviewer bLxo (R3)
Thank you for recognizing that the experiments are extensive and that the framework unifies position debiasing and human preference calibration, which are usually addressed separately in prior work. DIAL is designed for evaluations in which both kinds of judgment are costly: human labels are few, and querying a panel of LLM judges on every record in both display orders is expensive. DIAL saves on both:
We address your comments about weaknesses and questions below:
[W1(i), Q1] (Human-LLM alignment and its limits) We agree that the calibration restriction can fail under systematic human-LLM disagreement, and our theory and estimator cover that case. Theorem 3 allows the human scores to lie outside the LLM space, and the adaptive estimator ranges from the anchored fit to the human-only fit, which uses no alignment assumption (Section 5.1). Exact calibration is assumed for the fixed-weight intervals (Theorem 4). Below we give the empirical evidence for partial alignment and quantify how estimation changes as the mismatch grows.