- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This article studies post-processing calibration of frozen LLM-as-a-judge: using a small number of paired human and model ratings to adjust the model output to a human rating scale. The author rewrites the classic quantile matching into a streaming method SQM, maintains the marginal empirical distribution of the model and human ratings, and uses (g(j)=\widehat F_H^{-1}(\widehat F_J(j))) for mapping; and also proposes an extension to handle the complete predicted rating distribution. The authors compared 17 calibration methods on five datasets and four evaluation models, evaluating sample-by-sample MAE, marginal distribution Wasserstein distance, exact matching rate, and annotation budget efficiency. The results show that SQM performs well on the distribution matching index and has competitive sample-by-sample error; post-processing calibration can also reduce the scoring error gap between large and small models
Critical strengths
- The method is simple and low-cost, addressing a practical and well-defined problem. It offers immediate practical value by correcting rating scale bias without requiring the re-training of the evaluation model. For finite rating sets, the empirical distribution can be updated by maintaining only a small number of counts, resulting in low deployment and maintenance costs. As shown in Table 2, the median Wasserstein distance for SQM is 0.26 (0.24 for the predicted distribution version), while its MAE is 0.53—close to the best-performing 0.52. This indicates that the method serves as a lightweight baseline worthy of serious consideration.
- The experiments are comprehensive, examining calibration performance across multiple dimensions. Comparing five datasets, four models, and various calibration methods provides greater insight than validation on a single task alone. The paper reports marginal distribution errors, per-sample errors, and exact match rates, while also considering annotation budgets. This approach helps avoid the pitfall of claiming an overall improvement in evaluation quality simply because a histogram appears closer to the human distribution.
Critical weaknesses
- The formula of SQM cannot support "one-dimensional optimal transmission exact closed-form solution": when the discrete score contains quality points, the deterministic mapping cannot accurately push the target distribution, which may worsen the Wasserstein-1 distance (such as 0.1→0.5 in the counterexample); if randomization or quality splitting is used, it must be clearly stated and re-certified.
- Non-decreasing mapping does not necessarily preserve Spearman/Kendall (may introduce parallel values), nor is preserving weights sufficient to preserve spread, skew, or multimodality.
- Insufficient evidence for "overall best": There is selection bias in selecting the best checkpoint from the hold-out set, and it is not compared with batch implementation, making it impossible to distinguish the benefits of classic quantile matching and streaming processing.
- SQM is not the best in MAE or exact matching rate; "overall best" needs to be supported by budget, independent point selection, and paired testing between methods.
Questions for authors
-
What is the specific implementation of the discrete mapping? Please explain the actual code output for the aforementioned Bernoulli distribution counterexample. Does it employ the standard empirical CDF, generalized inverse, quantile interpolation, randomization, or mass splitting? Section 4.1 mentions both estimation and exact counting; which method was actually used in the experiments? If the formulas differ from the code, which results or theoretical claims require revision?
-
Was independent validation data used for checkpoint selection? Please specify whether the final checkpoint, fixed-budget checkpoint, or best checkpoint was used for Table 2, Figure 2, Figures 3–4, and the overall ranking. If the best checkpoint was used, were the selection set and the final test set kept separate? If not, do the main conclusions still hold given the pre-fixed budget?
-
Does the advantage persist when compared against relatively straightforward baselines that align with the objective? Please provide a comparison using batch empirical quantile matching with the same set of anchor points, as well as paired difference and uncertainty estimates against the strongest sample-wise calibration methods. Does SQM’s contribution primarily lie in distributional metrics, sample efficiency, or computational overhead?
-
What constitutes the independent unit for data splitting and statistical testing? Are multiple responses and multi-attribute scores derived from the same article, prompt, or original task grouped together during the split? What exactly changes across the three repetitions—the data split, the anchor point order, or the model sampling? This distinction affects the interpretation of the effective sample size and statistical significance.
8 comments
Weakness 2: Monotonicity, ties and distribution shape
Weakness 2: Monotonicity, ties and distribution shape
Ties and rank correlation. We agree that a weakly monotone map does not guarantee that Spearman or Kendall correlations are preserved, because merging distinct scores creates ties. We measured this on the real held-out sets. Calibration merges about one score level on a 5-point scale (median distinct values 5 → 4), and rank correlation is essentially unchanged (Kendall τb 0.422 → 0.439). We will remove the “by construction” wording in Section 5.1 and state that correlations are preserved empirically, with this tie analysis reported in the paper.
Distribution shape in the predictive-distribution variant. Keeping the weights w preserves the judge’s relative uncertainty across candidate scores, but the nonlinear map moves the support points, so spread, skew and modality are reshaped toward the human distribution. We will rephrase Section 4.1 accordingly and remove any suggestion that the original shape is kept.
On Checkpoint selection
Weakness 3 and Question 2: Checkpoint selection
Held-out data was never used to fit any method. However, we acknowledge that parts of our analysis select a method’s best checkpoint on the same held-out set that scores it:
Figures 3 and 4 use each group’s best-MAE checkpoint for streaming and mini-batch methods. The Wasserstein and exact-match comparisons, the overall ranking, and the significance tests against the raw judge use each method’s best checkpoint. MAE rankings use the final checkpoint. Figure 2 uses pre-specified fractions of the pool (10%, 25%, 50%, 75%, 100%) with no selection, and the “Best” column of Table 2 denotes a method’s best group, not its best checkpoint.
This is an oracle analysis. It overstates achievable performance, and it favours methods with many checkpoints (about 1,400 for streaming methods, about 70 for mini-batch, one for full-batch). We therefore reran the comparison at the final checkpoint on the four judges (48 groups):
-- EMD. SQM has the lowest mean EMD (0.225, versus 0.326 for the next best method) and ranks first in 42 of 48 groups. At the best checkpoint its mean rank is 1.54 and it wins 25 groups, so selection narrows its lead.
-- MAE. SQM has the lowest mean rank among 15 methods (5.17) and ranks first in 18 groups. At the best checkpoint few-shot ranks first (33 wins), because it gains most from selection (MAE 0.672 → 0.553, EMD 0.471 → 0.219, against 0.596 → 0.567 and 0.225 → 0.218 for SQM).
Best-checkpoint selection therefore does not favour SQM. It helps other streaming methods more, and it disadvantages full-batch methods such as isotonic regression, whose results do not change.
Code and formula clarifications
We thank the reviewer for the rigorous assessment, insightful counterexample, and constructive feedback. Below, we address each point and detail our planned revisions. We provide answers in separate comments to allow for easier readability and length constraints.
Weakness 1 and Question 1: Discrete mapping implementation, counterexample, and theory
The discrete formulation in Section 4.1 did not match our code, and we will correct it. In the implementation: Anchor scores are stored exactly in sorted arrays. The mention of P² in Section 4.1 does not describe the experiments, which use exact counts. For a raw judge score j, the percentile is the empirical midrank: u(j) = (F_J(j⁻) + F_J(j)) / 2. This percentile is mapped through the empirical human quantile function with linear interpolation (np.quantile, method="linear"). There is no randomization and no mass splitting. Because of the interpolation, calibrated predictions g(j) are real-valued, within the range of the scale. Held-out MAE and EMD (W1) are computed directly on these real values.
The reviewer’s counterexample. The judge distribution is 60% at 2 and 40% at 4, the human distribution is 50% at 2 and 50% at 4, and the initial W1 is 0.2.
The printed formula F_H⁻¹(F_J(j)) with a generalized inverse gives F_J(2) = 0.6 and F_J(4) = 1.0. Since F_H(2) = 0.5 < 0.6, both scores map to 4. All mass collapses onto 4 and W1 rises to 1.0, as the reviewer states. In our code the midranks are u(2) = 0.3 and u(4) = 0.8. Applied to the human anchors, the interpolated quantiles are 2.0 and 4.0. The code therefore acts as the identity on {2, 4}, and W1 stays at 0.2.
We confirmed that the midrank form is what the experiments used. Replaying the stored SQM streams reproduces the logged calibrated scores at 5,216 of 5,216 sampled steps across the four judges in the paper. Refitting on all anchors reproduces the logged held-out MAE and EMD (median difference 0.000; a few BiGGen Bench and UPHELD groups differ slightly because of score-validity filtering).
Empirical effect of the correction. On the real held-out sets of the four judges (48 dataset-attribute-model groups, all anchors), the printed formula worsens EMD relative to the raw judge in 21 groups, with a median EMD improvement of only 6%. The implemented version worsens EMD in 2 groups, with a median improvement of 41% (median MAE improvement 18%). The midrank form is still not guaranteed to help everywhere. With only 100 anchors, EMD worsens in 11 of 48 groups, most often for Kimi-K3 and HelpSteer2 attributes. We will report this in the limitations.
The reviewer is right that the closed-form transport map is exact only for atomless distributions. On a finite discrete scale, no deterministic function can in general push one discrete marginal exactly onto another without randomization or mass splitting. Our implementation uses a continuous relaxation (midranks plus interpolated quantiles). It produces a monotone real-valued approximation that avoids the bin collapse in the counterexample. We claim this as an empirical improvement, not as optimal transport on a discrete scale.
Revisions.
Replace the Section 4.1 equation with g(j) = F_H⁻¹((F_J(j⁻) + F_J(j)) / 2), with linear interpolation. Remove the P² statement for the experiments. Update Section 3 so that g outputs real values within the scale range. Qualify “closed-form optimal transport” in Sections 4 and 5.5 as exact for atomless distributions and an approximation on discrete scales.