- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
CaLLM applies the MCGrad multicalibration algorithm to debias LLM-judge scores, with theoretical guarantees and gains in calibration and accuracy across five benchmarks.
Critical strengths
The empirical results are promising, improving both calibration and predictive accuracy. The paper also offers theoretical analysis showing how multicalibration in representation space controls group-wise errors.
Critical weaknesses
-
Presentation is hard to follow. I was lost by page 2. Many definitions appear before the problem and method are explained, so their purpose is unclear. I suggest restructuring to first present the practical problem, available data, and the algorithm.
-
Novelty and necessity are unclear. The method seems a straightforward application of an existing multicalibration algorithm to LLM scores and embeddings. With human labels available, why not train a supervised model on the same features? The experiments do not show why multicalibration is needed. Also, lower group-wise calibration error does not necessarily mean position or verbosity bias is removed.
Questions for authors
-
Could the authors compare against standard supervised baselines (e.g., XGBoost or an MLP) trained on the same features and human labels?
-
How does the group-wise calibration guarantee translate into mitigating actual LLM judge biases, especially under counterfactual changes such as swapping answer positions?
4 comments
Tables
Comparison of CaLLM with supervised classification and residual-correction baselines in terms of multicalibration error and predictive performance.
| Dataset | Method | MCE_{\\%} | MCEσ | ACC | |
|---|---|---|---|---|---|
| PKU | f0 | 22.46 ± 0.28 | 32.91 ± 0.47 | 0.630 ± 0.002 | 0.730 ± 0.005 |
| CaLLM | 5.55 ± 0.28 | 7.68 ± 0.38 | 0.692 ± 0.001 | 0.572 ± 0.002 | |
| XGB$_{resid}$ | 10.45 ± 0.18 | 15.40 ± 0.26 | 0.679 ± 0.001 | 0.607 ± 0.004 | |
| XGB$_{base}$ | 5.87 ± 0.47 | 7.56 ± 0.61 | 0.658 ± 0.001 | 0.618 ± 0.001 | |
| MLP$_{resid}$ | 9.57 ± 0.29 | 14.22 ± 0.42 | 0.683 ± 0.001 | 0.599 ± 0.003 | |
| MLP$_{base}$ | 5.47 ± 0.47 | 7.24 ± 0.65 | 0.663 ± 0.001 | 0.612 ± 0.001 | |
| AEX | f0 | 16.11 ± 0.31 | 20.81 ± 0.38 | 0.633 ± 0.001 | 0.760 ± 0.004 |
| CaLLM | 5.09 ± 0.19 | 5.83 ± 0.21 | 0.677 ± 0.002 | 0.598 ± 0.002 | |
| XGB$_{resid}$ | 12.14 ± 0.26 | 15.70 ± 0.31 | 0.664 ± 0.002 | 0.649 ± 0.004 | |
| XGB$_{base}$ | 6.26 ± 0.25 | 6.88 ± 0.28 | 0.640 ± 0.001 | 0.625 ± 0.001 | |
| MLP$_{resid}$ | 12.67 ± 0.28 | 16.36 ± 0.34 | 0.664 ± 0.002 | 0.650 ± 0.003 | |
| MLP$_{base}$ | 6.33 ± 0.13 | 7.01 ± 0.13 | 0.638 ± 0.002 | 0.626 ± 0.002 | |
| PPE | f0 | 16.23 ± 0.44 | 9.03 ± 0.25 | 0.639 ± 0.004 | 0.741 ± 0.008 |
| CaLLM | 7.97 ± 0.43 | 3.87 ± 0.20 | 0.670 ± 0.005 | 0.612 ± 0.002 | |
| XGB$_{resid}$ | 13.97 ± 0.31 | 7.79 ± 0.18 | 0.663 ± 0.004 | 0.664 ± 0.007 | |
| XGB$_{base}$ | 9.02 ± 0.39 | 4.23 ± 0.18 | 0.636 ± 0.003 | 0.640 ± 0.002 | |
| MLP$_{resid}$ | 15.18 ± 0.54 | 8.49 ± 0.31 | 0.663 ± 0.005 | 0.665 ± 0.006 | |
| MLP$_{base}$ | 9.31 ± 0.48 | 4.40 ± 0.23 | 0.641 ± 0.005 | 0.637 ± 0.001 | |
| RWB | f0 | 13.07 ± 0.41 | 5.97 ± 0.14 | 0.860 ± 0.009 | 0.357 ± 0.018 |
| CaLLM | 7.29 ± 0.92 | 3.92 ± 0.47 | 0.930 ± 0.002 | 0.208 ± 0.006 | |
| XGB$_{resid}$ | 8.57 ± 1.11 | 3.80 ± 0.48 | 0.922 ± 0.004 | 0.231 ± 0.010 | |
| XGB$_{base}$ | 18.74 ± 1.00 | 6.21 ± 0.28 | 0.818 ± 0.004 | 0.377 ± 0.011 | |
| MLP$_{resid}$ | 9.63 ± 1.27 | 4.20 ± 0.53 | 0.925 ± 0.005 | 0.232 ± 0.006 | |
| MLP$_{base}$ | 24.91 ± 2.52 | 9.52 ± 1.23 | 0.814 ± 0.007 | 0.398 ± 0.014 | |
| MTB | f0 | 19.59 ± 0.56 | 3.50 ± 0.12 | 0.788 ± 0.006 | 0.485 ± 0.021 |
| CaLLM | 16.36 ± 0.84 | 2.70 ± 0.11 | 0.814 ± 0.015 | 0.405 ± 0.021 | |
| XGB$_{resid}$ | 18.36 ± 0.93 | 3.30 ± 0.17 | 0.817 ± 0.006 | 0.429 ± 0.019 | |
| XGB$_{base}$ | 16.04 ± 1.80 | 2.27 ± 0.25 | 0.757 ± 0.013 | 0.560 ± 0.019 | |
| MLP$_{resid}$ | 17.97 ± 0.93 | 3.12 ± 0.22 | 0.800 ± 0.007 | 0.450 ± 0.011 | |
| MLP$_{base}$ | 20.24 ± 1.97 | 2.94 ± 0.22 | 0.746 ± 0.010 | 0.546 ± 0.006 |
The authors have addressed my concern about the missing supervised baselines by adding XGBoost and MLP experiments, which I appreciate.
However, my main concern about novelty remains. The method relies on the existing MCGrad algorithm, and the main theoretical contribution concerns transferring calibration guarantees from embeddings to the original input space. Since using embeddings is itself a design choice, it is unclear how much this result contributes to the original problem. For known biases such as position and verbosity, group indicators can be used directly. For unknown biases, the guarantee depends on whether the embeddings preserve the relevant information.
Overall, while the empirical results are promising, I still have concerns about the novelty and the broader claims of bias mitigation. I will maintain my rating.
We acknowledge the reviewer decision. With the goal to improve our work, we would kindly ask if it would be possible to better clarify which of the raised concerns was not addressed by our rebuttal.
Rebuttal to reviewer VpTw
We thank the reviewer. We are glad they found our results promising and acknowledges the value of our theoretical analysis. We address concerns in the following.
Clarity We understand the reviewer's concerns about clarity of the presentation. We followed the reviewer's suggestion and restructured the introduction and methodological section of our manuscript. We restated our introduction to more clearly identify the problem we tackle and the shortcomings of existing methodologies. We present our goal (lines 103-106) in the preliminary section of the manuscript, followed by the proposed methodology (lines 177-197), all before the theoretical section. We further improve the interpretability of the first bound (Theorem 1) and extend the discussion on its components and their meaning.
Novelty and Necessity. Our goal is to mitigate multiple sources of bias in LLM-as-a-judge within a unified framework, including biases that are not known in advance. Our main theoretical contribution is to address the unexplored problem of transferring multicalibration guarantees from a learned representation to groups defined in input space. Traditional multicalibration approaches operate on the same features that characterize groups. Attributes such as age can be provided as group indicators or used to discover miscalibration within ranges of their values. In contrast, LLM judges operate on unstructured textual inputs, where potentially relevant bias groups cannot always be directly identified or specified. While known biases can be represented through explicit group indicators, unknown biases must be addressed through information preserved in a generic representation, such as text embeddings. We bridge this gap by establishing conditions under which multicalibration in representation space controls the residual bias of groups defined over the original inputs, even when their membership cannot be recovered from the representation. The results go beyond existing multicalibration guarantees, which do not characterize this interaction between representation quality, prediction error, and input-space group membership. We additionally derive sufficient conditions under which multicalibration guarantees an improvement in predictive accuracy, even if it changes decisions. Unlike existing approaches, CaLLM provides a unified mechanism for mitigating multiple documented and potentially unknown biases simultaneously, without requiring a separate correction procedure for each. Thus, our contribution is not a new multicalibration algorithm, but the theoretical characterization and empirical validation of multicalibration as a general framework for mitigating biases in LLM judges operating on learned representations.
Supervised baselines. We add XGBoost and MLP models trained on the same representations and labels, considering both direct prediction and residual correction strategies (Table 12). Since MCGrad itself relies on gradient-boosted trees, these comparisons can help assess the benefits of the multicalibration procedure. Results show that simple supervised approaches can recover a portion of the improvements, highlighting the importance of the representation used by CaLLM. CaLLM achieves the lowest log loss on all five datasets, the highest accuracy (except for MTB where it is second best within a standard deviation), and consistently strong group-wise calibration. Notably, CaLLM achieves lower \mathrm{MCE}_{\\%} than both residual-correction baselines on all five datasets, while classifiers fitted on labels occasionally achieve competitive or lower multicalibration error, but consistently exhibit worse predictive performance. Overall, these findings support the choice of MCGrad as a well-rounded calibration strategy, offering a favorable balance between predictive accuracy, probabilistic quality, and multicalibration.
Group-wise calibration and counterfactual bias As noted in line 132, our framework adopts a distributional notion of bias, measuring systematic discrepancies between the judge's confidence and human preferences within groups associated with bias-inducing attributes. While this differs from counterfactual notions such as position-swap consistency, the two are related. In particular, assuming that human preferences are invariant to response ordering and that both orderings follow the same underlying distribution of comparisons, perfect calibration within the corresponding position groups implies E[f(X)+f(Xswap)−1]=0 where f(X) denotes the probability assigned to the first response and Xswap denotes the same comparison with reversed response order. This guarantee holds in expectation, rather than for individual comparisons: positive and negative counterfactual discrepancies may cancel out. Thus, our approach controls systematic position bias at the population level without necessarily enforcing counterfactual invariance for every question.