acceptodds
Under review as a conference paper at ICLR 2027

Quantiles, Not Points: Streaming Quantile Matching for Ordinal-Aware LLM-as-a-Judge Calibration

Abstract

LLM-as-a-judge scores are often distributionally shifted from human ratings on any given evaluation task, and post-hoc calibration (mapping judge scores to human scores in a small labeled anchor set) has become a popular method to correct for this shift. We present streaming quantile matching (SQM), a method previously unexplored for LLM-as-a-judge calibration that wins outright over every other method by directly computing the optimal-transport map between the judge and human score distributions. We develop a novel framework for evaluating post-hoc calibration performance and within it we show that unlike point-wise calibrations, SQM respects the ordinality of the human anchor set and offers superior performance regardless of model or data budget. We also show that in general, post-hoc calibration narrows the raw error gap between smaller and larger judge models, which opens the possibility of more efficient LLM judges.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
llm as a judge, evaluation, post-policy calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NgZ5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This article studies post-processing calibration of frozen LLM-as-a-judge: using a small number of paired human and model ratings to adjust the model output to a human rating scale. The author rewrites the classic quantile matching into a streaming method SQM, maintains the marginal empirical distribution of the model and human ratings, and uses (g(j)=\widehat F_H^{-1}(\widehat F_J(j))) for mapping; and also proposes an extension to handle the complete predicted rating distribution. The authors compared 17 calibration methods on five datasets and four evaluation models, evaluating sample-by-sample MAE, marginal distribution Wasserstein distance, exact matching rate, and annotation budget efficiency. The results show that SQM performs well on the distribution matching index and has competitive sample-by-sample error; post-processing calibration can also reduce the scoring error gap between large and small models

Critical strengths
  1. The method is simple and low-cost, addressing a practical and well-defined problem. It offers immediate practical value by correcting rating scale bias without requiring the re-training of the evaluation model. For finite rating sets, the empirical distribution can be updated by maintaining only a small number of counts, resulting in low deployment and maintenance costs. As shown in Table 2, the median Wasserstein distance for SQM is 0.26 (0.24 for the predicted distribution version), while its MAE is 0.53—close to the best-performing 0.52. This indicates that the method serves as a lightweight baseline worthy of serious consideration.
  2. The experiments are comprehensive, examining calibration performance across multiple dimensions. Comparing five datasets, four models, and various calibration methods provides greater insight than validation on a single task alone. The paper reports marginal distribution errors, per-sample errors, and exact match rates, while also considering annotation budgets. This approach helps avoid the pitfall of claiming an overall improvement in evaluation quality simply because a histogram appears closer to the human distribution.
Critical weaknesses
  1. The formula of SQM cannot support "one-dimensional optimal transmission exact closed-form solution": when the discrete score contains quality points, the deterministic mapping cannot accurately push the target distribution, which may worsen the Wasserstein-1 distance (such as 0.1→0.5 in the counterexample); if randomization or quality splitting is used, it must be clearly stated and re-certified.
  2. Non-decreasing mapping does not necessarily preserve Spearman/Kendall (may introduce parallel values), nor is preserving weights sufficient to preserve spread, skew, or multimodality.
  3. Insufficient evidence for "overall best": There is selection bias in selecting the best checkpoint from the hold-out set, and it is not compared with batch implementation, making it impossible to distinguish the benefits of classic quantile matching and streaming processing.
  4. SQM is not the best in MAE or exact matching rate; "overall best" needs to be supported by budget, independent point selection, and paired testing between methods.
Questions for authors
  1. What is the specific implementation of the discrete mapping? Please explain the actual code output for the aforementioned Bernoulli distribution counterexample. Does it employ the standard empirical CDF, generalized inverse, quantile interpolation, randomization, or mass splitting? Section 4.1 mentions both (P2)(P^2) estimation and exact counting; which method was actually used in the experiments? If the formulas differ from the code, which results or theoretical claims require revision?

  2. Was independent validation data used for checkpoint selection? Please specify whether the final checkpoint, fixed-budget checkpoint, or best checkpoint was used for Table 2, Figure 2, Figures 3–4, and the overall ranking. If the best checkpoint was used, were the selection set and the final test set kept separate? If not, do the main conclusions still hold given the pre-fixed budget?

  3. Does the advantage persist when compared against relatively straightforward baselines that align with the objective? Please provide a comparison using batch empirical quantile matching with the same set of anchor points, as well as paired difference and uncertainty estimates against the strongest sample-wise calibration methods. Does SQM’s contribution primarily lie in distributional metrics, sample efficiency, or computational overhead?

  4. What constitutes the independent unit for data splitting and statistical testing? Are multiple responses and multi-attribute scores derived from the same article, prompt, or original task grouped together during the split? What exactly changes across the three repetitions—the data split, the anchor point order, or the model sampling? This distinction affects the interpretation of the effective sample size and statistical significance.

8 comments
AuthorsOpenReview ↗

Code and formula clarifications

We thank the reviewer for the rigorous assessment, insightful counterexample, and constructive feedback. Below, we address each point and detail our planned revisions. We provide answers in separate comments to allow for easier readability and length constraints.


Weakness 1 and Question 1: Discrete mapping implementation, counterexample, and theory

The discrete formulation in Section 4.1 did not match our code, and we will correct it. In the implementation: Anchor scores are stored exactly in sorted arrays. The mention of P² in Section 4.1 does not describe the experiments, which use exact counts. For a raw judge score j, the percentile is the empirical midrank: u(j) = (F_J(j⁻) + F_J(j)) / 2. This percentile is mapped through the empirical human quantile function with linear interpolation (np.quantile, method="linear"). There is no randomization and no mass splitting. Because of the interpolation, calibrated predictions g(j) are real-valued, within the range of the scale. Held-out MAE and EMD (W1) are computed directly on these real values.

The reviewer’s counterexample. The judge distribution is 60% at 2 and 40% at 4, the human distribution is 50% at 2 and 50% at 4, and the initial W1 is 0.2.

The printed formula F_H⁻¹(F_J(j)) with a generalized inverse gives F_J(2) = 0.6 and F_J(4) = 1.0. Since F_H(2) = 0.5 < 0.6, both scores map to 4. All mass collapses onto 4 and W1 rises to 1.0, as the reviewer states. In our code the midranks are u(2) = 0.3 and u(4) = 0.8. Applied to the human anchors, the interpolated quantiles are 2.0 and 4.0. The code therefore acts as the identity on {2, 4}, and W1 stays at 0.2.

We confirmed that the midrank form is what the experiments used. Replaying the stored SQM streams reproduces the logged calibrated scores at 5,216 of 5,216 sampled steps across the four judges in the paper. Refitting on all anchors reproduces the logged held-out MAE and EMD (median difference 0.000; a few BiGGen Bench and UPHELD groups differ slightly because of score-validity filtering).

Empirical effect of the correction. On the real held-out sets of the four judges (48 dataset-attribute-model groups, all anchors), the printed formula worsens EMD relative to the raw judge in 21 groups, with a median EMD improvement of only 6%. The implemented version worsens EMD in 2 groups, with a median improvement of 41% (median MAE improvement 18%). The midrank form is still not guaranteed to help everywhere. With only 100 anchors, EMD worsens in 11 of 48 groups, most often for Kimi-K3 and HelpSteer2 attributes. We will report this in the limitations.

The reviewer is right that the closed-form transport map is exact only for atomless distributions. On a finite discrete scale, no deterministic function can in general push one discrete marginal exactly onto another without randomization or mass splitting. Our implementation uses a continuous relaxation (midranks plus interpolated quantiles). It produces a monotone real-valued approximation that avoids the bin collapse in the counterexample. We claim this as an empirical improvement, not as optimal transport on a discrete scale.

Revisions.

Replace the Section 4.1 equation with g(j) = F_H⁻¹((F_J(j⁻) + F_J(j)) / 2), with linear interpolation. Remove the P² statement for the experiments. Update Section 3 so that g outputs real values within the scale range. Qualify “closed-form optimal transport” in Sections 4 and 5.5 as exact for atomless distributions and an approximation on discrete scales.

AuthorsOpenReview ↗

Weakness 2: Monotonicity, ties and distribution shape

Weakness 2: Monotonicity, ties and distribution shape

Ties and rank correlation. We agree that a weakly monotone map does not guarantee that Spearman or Kendall correlations are preserved, because merging distinct scores creates ties. We measured this on the real held-out sets. Calibration merges about one score level on a 5-point scale (median distinct values 5 → 4), and rank correlation is essentially unchanged (Kendall τb 0.422 → 0.439). We will remove the “by construction” wording in Section 5.1 and state that correlations are preserved empirically, with this tie analysis reported in the paper.

Distribution shape in the predictive-distribution variant. Keeping the weights w preserves the judge’s relative uncertainty across candidate scores, but the nonlinear map moves the support points, so spread, skew and modality are reshaped toward the human distribution. We will rephrase Section 4.1 accordingly and remove any suggestion that the original shape is kept.

AuthorsOpenReview ↗

On Checkpoint selection

Weakness 3 and Question 2: Checkpoint selection

Held-out data was never used to fit any method. However, we acknowledge that parts of our analysis select a method’s best checkpoint on the same held-out set that scores it:

Figures 3 and 4 use each group’s best-MAE checkpoint for streaming and mini-batch methods. The Wasserstein and exact-match comparisons, the overall ranking, and the significance tests against the raw judge use each method’s best checkpoint. MAE rankings use the final checkpoint. Figure 2 uses pre-specified fractions of the pool (10%, 25%, 50%, 75%, 100%) with no selection, and the “Best” column of Table 2 denotes a method’s best group, not its best checkpoint.

This is an oracle analysis. It overstates achievable performance, and it favours methods with many checkpoints (about 1,400 for streaming methods, about 70 for mini-batch, one for full-batch). We therefore reran the comparison at the final checkpoint on the four judges (48 groups):

-- EMD. SQM has the lowest mean EMD (0.225, versus 0.326 for the next best method) and ranks first in 42 of 48 groups. At the best checkpoint its mean rank is 1.54 and it wins 25 groups, so selection narrows its lead.

-- MAE. SQM has the lowest mean rank among 15 methods (5.17) and ranks first in 18 groups. At the best checkpoint few-shot ranks first (33 wins), because it gains most from selection (MAE 0.672 → 0.553, EMD 0.471 → 0.219, against 0.596 → 0.567 and 0.225 → 0.218 for SQM).

Best-checkpoint selection therefore does not favour SQM. It helps other streaming methods more, and it disadvantages full-batch methods such as isotonic regression, whose results do not change.

Reviewer_4AwROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper talks about how post-hoc calibration for LLM Judges can make them more reliable by improving LLM-human alignment. They adapt the classical setting of quantile matching to map the distribution between human and LLM-scores. Different ways of mapping are possible and the authors have done a good analysis by considering options such as streaming one at a time, minibatch and full-batch; various map functions and measuring the improvements of LLM-Judges via various metrics - MAE on held out dataset, Wasserstein distance for ordinality measurement and exact match rate to understand where the (human, LLM) scores are identical. Their experiments on post-hoc calibration brings smaller LLMs closer in performance to larger LLMs as judges thus bringing in possibility of smaller/better/efficient judges.

Critical strengths
  • The motivation is well established - understanding how we can calibrate LLM-Judges with minimal effort - either via small curated labeled dataset or fine-tuning or multi-judge aggregation efforts as different means if made easy, accessible and effective has potential impact on industry-scale applications.
  • The writing is clear to follow and well structured. Please refer to Questions to Authors, On formatting for some suggestions on presentation.
Critical weaknesses
  • Claim of contribution/framing presentation: While the paper calls about SQM as a method (in title), it's an extension of the classical quantile matching to LLMJ (as mentioned in abstract; L43) which makes the claim bit weaker and I think the main contribution is more on the extensive evaluation framework (L46-53), not the method itself. If the authors agree with this view, I suggest to polish relevant sections accordingly as I think this is not a method's paper; but is more an analysis/framework paper.
    • To be clear and fair, the paper is interesting enough and I am not concerned about novelty because bringing tools from one domain to another is valid research, just the framing seems to be slightly off at places.
    • If the authors feel differently, that's also equally valid point and I don't mind and it again requires polishing at relevant sections. Whichever way the authors pursue, the paper needs some polishing because at current state it is claimed as a method-oriented paper but the paper story is not aligned that way.
  • (Experiment - Multi-judge aggregation setup as baselines): The authors cite multi-judge aggregation as related work (L84-85) and recent research on aggregation methods such as [1, 2, 3] is basically to remove noise from these judges and improve reliability. When viewed from different lens this achieves post-hoc calibration as a side effect i.e if aggregated judges are more accurate, they are more aligned with humans. So, I believe these needs to be considered as relevant baselines. I would like to see if the authors can expand their study with (1) Aggregation methods as relevant baselines (2) Whether calibration + aggregation steps combined helps further to be aligned. Please make relevant assumptions based on chosen method and 1-2 methods should be fine (need not be specifically these ones, any recent conference/journal published work is fine).
    • (Clarity on human and LLM scores aggregations) From L215 - if the scores are averaged, they need not be integers right. So, how does it work because in "Frozen-weight calibration" (L125) it's denoted as ZZ which is integers? Are the values rounded? What about (averaged) human scores that are non-integers? Overall, the clarity needs to be improved. This also suggests to multi-judge aggregation as baseline comparison (above experiment) i.e averaging is not the only possible aggregation.
    • [Optional Experiment] While experiments on fine-tuned model + calibration is equally interesting, I acknowledge it's addressed in limitations (L463-496) which also requires GPU resources, but nevertheless it's an interesting setting where understanding how much the base -> finetuning -> post calibration improves is a good tradeoff to understand.
  • (Discussion, Potential experiment on varying and distributional-shift held out set): (1) A sweep on held-out set would inform the tradeoffs between different modes considered in study. For eg; full-batch might not work if the held out is large as the available labeled data for one-time mapping function is less! (2) Understanding the distributional shift of calibrated/held-out set experiment is also important - If you calibrated on dataset A, in real-world data drift happens so understanding whether the post-hoc calibration holds is a valid experiment.
    • I believe some parts are closely addressed in 5.2 and Appendix B (Fig 2 esp); but would like to understand whether sweep and distributional-shift on held-out dataset is a relevant and interesting setting to study -- if not possible due to any reason, atleast having a discussion would suffice on why and what the authors think about.
  • (Clarity on Table 2): Table 2 has both Method and mode confounded and understanding individual contributions is not super clear. Ideally, the possible combination of method x mode should be reported to strengthen the claims - consider atleast for some settings and expand. And fig 6 has per model but all datasets combined and only MAE - maybe similar plots for Wasserstein and exact match rates might inform better.
  1. CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (ICML 2026): https://openreview.net/forum?id=3WPDFjZ1UT
  2. SkillAggregation: Reference-free LLM-Dependent Aggregation (ACL 2025): https://aclanthology.org/2025.acl-long.756/
  3. Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks, (NAACL 2025): https://aclanthology.org/2025.naacl-long.617/
Questions for authors

On formatting

  • L37-38: "last approach" -> "latter approach"
  • L45: "clasical" -> "classical"
  • L81 - 93: Can have a table in Appendix which can easily summarize what the previous papers have done vs this paper - the differentiator
  • Sec 3, L125-133: I think the notations can to be explained better - for eg, a, b, n was not clear at first. While at it, consider adding a notation table in Appendix so it's easy to refer back.
  • L220: Sentence formation seems to be abrupt - "Formal definitions ..."
  • Because the paper has lot of settings, consider adding a motivation figure which will help understand the idea better for reader.

Others

  • (Discussion, Optimal transport map): Because Optimal transport can be an identity function, please provide more details in Appendix on L16 "optimal transport-map"
  • Sanity check on Table 1. SummEval has 100 articles × 16 summaries × 4 attributes × 4 judges = 25,600 judgments. Table 1 has 24,149 which is about 5.66% reduction. That’s inconsistent with the stated “<=2% of judge calls fail” (L205). Was the value averaged across all datasets? Maybe Table 1 can be improved with an additional column?
    • Analysis on judge failures; L204-205: Why did the judge calls fail? Did the authors do any root-cause analysis on that? Is it API call failure, or the question, response pair is difficult to get a rating?
  • Discussion on Non Likert settings: While the paper focused mostly on the ordinal/Likert style datasets, LLM-Judges can be used in other settings as well - float ratings, binary comparisons among many, so understanding how the method can be extended to those settings strengthens the paper
  • (Connection to relevant literature, L470-476): Having few human labels to calibrate LLM Judge is closely connected to confidence-driven and label-efficient estimation [1, 2, 3 ,4] - This is reflected in results such as Fig 2; L354-358 where SQM works better than full-batch because intuitively the map/method is learning something useful about the distribution as it starts looking at data.
  • Settings used for models: What's the temperature, reasoning effort of the model during decoding? All this information should be there in Appendix
  • How is the rank determined from Table 2 last column on W/Ls. Please have relevant details in Appendix.
  1. Can Unconfident LLM Annotations Be Used for Confident Conclusions? (NAACL 2025): https://aclanthology.org/2025.naacl-long.179/
  2. Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL (ICML 2025): https://openreview.net/forum?id=w4c5bLkhsz
  3. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement (ICLR 2025): https://arxiv.org/abs/2407.18370
  4. Active statistical inference (ICML 2024): https://arxiv.org/abs/2403.03208.
7 comments

We thank the reviewer for there for a careful and genuinely engaged review. It is clear the paper was read, including the appendix and the line-level details, and the suggestions are constructive and specific. We also want to acknowledge that, in cutting the paper to the page limit under time pressure, we made some preemptive style changes, and we are aware that several stylistic and clarity issues remain, many of which you pointed out. We have already started on them and list them below.

1. Framing: method vs. framework

We agree the framing needs to be more precise. Our work began as a question about distribution-based, streaming, and distance-aware calibration, because in practice human labels often arrive one at a time (for example from a rarely completed feedback form) rather than in full batches. Once we found that quantile matching works well in that setting, we also found there was no standard way to compare judge-to-human calibration methods across judges, data budgets and metrics. We therefore see two roughly equal contributions: (i) SQM, an adaptation of classical equipercentile matching to streaming LLM-judge calibration, and (ii) a framework for evaluating any calibration method.

We will revise the abstract and contribution list to say this explicitly and to state that SQM is a classical method applied to a new setting, not a new algorithm. We will also state the streaming motivation in the introduction, which is currently missing. We will make the SQM claims more precise: its advantages are distributional fidelity (lowest median Wasserstein) and data efficiency (it reaches the full-batch +5% band at roughly 4% of the labeled pool on average, versus 19–24% for other streaming and batch methods). It is not uniformly best on MAE, where it is within 0.01 of the best median.

2. Multi-judge aggregation as a baseline, and calibrating after aggregating

We agree this is a relevant comparison and ran it during the rebuttal period. We re-implemented the calibrators offline on the stored judge outputs. We used all items scored by all four judges (9,340 item-attribute pairs across 12 dataset-attribute groups), with 10 random 70/30 splits per group, and report held-out averages with the full anchor pool. Aggregators were mean, median, Dawid–Skene (reference-free, in the spirit of SkillAggregation), and a ridge stack fit on the anchors (joint aggregation and calibration).

Held-out MAERaw+ Isotonic+ SQM
GPT-5 (single)0.7100.5210.591
Kimi-K3 (single)0.6910.5290.564
Mean of GPT-5 + Kimi-K30.6640.4980.524
Mean of all 4 judges0.7050.4960.501

Dawid–Skene + isotonic gives 0.494, and the ridge stack + SQM gives 0.499. The findings are as follows.

  • Aggregation alone is not enough. The raw mean of four judges (0.705) is essentially no better than raw GPT-5 (0.710). The aggregate still needs calibration: calibration cuts its MAE by about 30%.
  • Calibration and aggregation combine, but the added gain is modest. Compared with the better of GPT-5 and Kimi-K3 chosen per group after the fact (a strong baseline), the two-strong-judge mean lowers MAE by about 0.010 with isotonic regression and 0.013 with SQM. It wins in 7 and 8 of 12 groups respectively. The ordering is the same with 50 anchors.
  • The small judges add almost nothing. Using the two small judges only to break disagreements between the strong ones does not help (0.505 with isotonic vs. 0.498 for the plain strong mean).e first, the Wasserstein distance rises to 0.394, close to the single-judge value.

So calibration is complementary to aggregation, not a substitute for it. In our setting the combined benefit over a good single judge is small. We will add this experiment, with both aggregation baselines, to the appendix. We also note that our calibration targets a single judge because that is the most common practice, and that calibrating an aggregate is not equivalent to calibrating a single judge, because the aggregate has a different and smoother score distribution.

3. Clarity on human and LLM score aggregation (L215)

Our formula is stated for a single judge, whose scores are integers. We will fix the notation by defining the human label space as the finite set of observed values on the scale, so that the calibrated output lies in that set. MAE and Wasserstein distance are well defined on it, and exact match compares integer-rounded values.

We will also report inter-annotator agreement, which varies a lot across datasets: Krippendorff's α is 0.54 for FLASK and 0.25–0.29 for UPHELD, and per-item rater disagreement correlates with error (Spearman ρ of about 0.26 for the raw judge on FLASK, falling to about 0.12 after calibration).

4. Table 2: separating method and mode

All Table 2 values are taken at the final checkpoint, so every method has seen the same data, and every method is scored against the same held-out set. Each run shuffles the data, reserves a random 30% as held-out, shares it across all methods and modes within the run, and the results are averaged over 3 runs (the split is not fixed across runs). We will state this in the caption and in Sec. 3, where the current wording ("a fixed held-out fraction" was used to denote that every method sees the same validation data in the single run) is misleading.

We agree the two factors are not separable in the current table. Figure 2 already gives a matched-budget comparison: the full-batch average moves only from about 0.645 at 10% of the pool to about 0.618 at 100%, and SQM's streaming trajectory is at or below the full-batch range from about 55% of the pool on. We will also extend Fig. 6 to Wasserstein distance and exact match (the pooled per-method versions are already in Appendix Figs. 3–4).

5. Held-out sweep and distribution shift

Because every run draws a fresh random split, split variance is already averaged over runs. We agree a held-out-size sweep is informative: the calibrators run in milliseconds on cached judge outputs, so we will add a sweep from 10% to 50% held-out. Figure 2 already suggests full-batch methods are not strongly limited by calibration-set size.

For distribution shift, we will add a discussion and, where the data allows, a cross-attribute or cross-dataset calibration test. We will say plainly that SQM matches marginals, so it is sensitive to shifts in the human label distribution, whereas methods that learn a mapping conditional on the judge score may be less sensitive. That trade-off deserves study, and we will not claim robustness we have not tested.

6. Formatting and smaller points

  • Typos and notation. We will fix "last"→"latter" and "clasical"→"classical", add a notation table to the appendix (a, b, n and the others), and rewrite the abrupt sentence at L220.
  • Related work and motivation. We will add an appendix table contrasting prior work with ours, and a motivation figure.
  • Optimal transport. If the judge and human distributions are equal, the map is the identity. We will add an appendix section on the map. On a discrete scale a deterministic quantile map cannot split mass across scores, so it gives a monotone (comonotone) coupling that approximates the pushforward, and we will say so rather than call it exact OT.
  • Table 1 and judge failures. The "≤2%" figure was averaged over all datasets. You are right that SummEval is higher: 24,149 of 25,600 expected judgments is a 5.7% loss. We will add per-dataset expected, parsed and dropped columns, with an error breakdown by cause (API/transport errors vs. unparseable outputs). We also exclude four BiGGen Bench items whose source human rating is a −1 sentinel, which are data errors and not judge failures. What we have at the moment is that our 2% initial number is based on the non-successful API calls alone. The real numbers a slightly larger, but we still belive that the size of the dataset we analyized is representative.
DatasetExpectedCollectedDropped% dropped
BiGGen Bench11,1209,9161,20410.8
FLASK8,0046,4291,57519.7
HelpSteer220,76017,3043,45616.6
SummEval25,60024,1491,4515.7
UPHELD (content only)1,6321,63110.1
All67,11659,4297,68711.5
  • Decoding settings. All models run at temperature 0, except GPT-5, where 0 is not allowed and we used minimal reasoning effort. We will list this in the appendix.
  • Rank column. The Table 2 rank is built from the method's ranks on the three metrics. We will add the exact procedure and tie handling, and per-metric rank columns, to the appendix.
  • Non-Likert settings. We will add a discussion: continuous scores work directly with quantile functions, binary labels reduce to threshold matching, and pairwise comparisons need a score model such as Bradley–Terry first.
  • Label-efficient estimation. Thank you for the pointers [1–4]. We will cite them. They give statistical guarantees for estimates from few labels, while we learn a score-level mapping, so the settings are related but not interchangeable.
  • Fine-tuning. We agree base → fine-tuned → calibrated is an interesting trade-off. We were in this work interested mostly in how calibrated methods work in isolation, for a common use case where GPU for a training procedure is not allowed.

Thank you again. The review has already improved the paper's clarity and its claims, a rare occurance in today's overburdened review space.

Reviewer_4AwROpenReview ↗

Response to rebuttal

Hi authors,

Thanks a lot for the detailed response. Most of my questions were addressed and I appreciate the positive comments.

On Multi-judge aggregation as a baseline, and calibrating after aggregating: Thanks for this experiment. It's interesting to see some of the findings such as Aggregation alone is not enough and Calibration and aggregation combine, but the added gain is modest.

That said, the aggregation method chosen matters a lot here and from what I understand the current ones are primitive (mean, median, Dawid-Skene); thus I suggested initially to pick from recent conference papers (for 1,2,3 - repo links attached now for quick reference).

  • Option 1: If possible, pick recent conference published papers (maybe these or any others) and add those as well by expanding the current results - A good idea is to usually pick the latest ones as those benchmark against previous ones as standard baselines thus transitively showing the results.
  • Option 2: If it's time consuming, please try to make an effort to add this later
  • Option 3: At the very least if adding more nuanced aggregators is out of scope for your study (which I don't think), say something like "more sophisticated aggregation methods such as [x, y, ...] can change this finding"

For Option 2 and Option 3, I say it because with current data, I think the conclusion seems to be very strong!

With all that said, this will only strengthen or shed some new light on the claims and I am ok with any option the authors choose based on their time.

For the rest of the points: Sure, please update your manuscript as I believe it will be more reader-friendly and can help understand the main paper better. I do believe for ICLR you can update the manuscript during rebuttal, so please share it if possible.

All the very best with your submission

  1. CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation (ICML 2026): https://openreview.net/forum?id=3WPDFjZ1UT; http://github.com/SprocketLab/CARE
  2. SkillAggregation: Reference-free LLM-Dependent Aggregation (ACL 2025): https://aclanthology.org/2025.acl-long.756/; https://github.com/BriansIDP/Crowdsourcing/tree/crowdlayer_dev
  3. Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks, (NAACL 2025): https://aclanthology.org/2025.naacl-long.617/; https://www.justinxzhao.com/demos/language-model-council-paper/
Reviewer_2zZ9OpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies post-hoc calibration of frozen LLM judges using small human-labeled anchor sets. It adapts classical quantile matching into Streaming Quantile Matching (SQM) and compares 17 methods across five datasets and four judges. SQM achieves strong distributional alignment while remaining competitive on per-item scoring error.

Critical strengths
  1. The method has a clear motivation and a straightforward formulation. Correcting systematic scoring differences without updating the judge is practical, and the lightweight mapping is easy to maintain.

  2. The experiments cover multiple datasets, judges, and calibration methods. Reporting both per-item error and distributional distance provides useful evidence about the strengths and trade-offs of calibration.

Critical weaknesses
  1. The connection between numerical calibration and practical evaluation capability remains insufficiently established. Lower MAE and EMD indicate closer agreement with human scores. However, a monotone scalar mapping cannot correct reversed sample-level preferences and may merge distinct scores. Consequently, a smaller–larger judge gap in scoring error does not establish a corresponding reduction in their ability to distinguish response quality or human preferences.

  2. The stated mapping can worsen an already close score distribution. For example, suppose judge scores place 60% of their mass at 2 and 40% at 4, while human scores place 50% at each. Under the stated formula and standard empirical-CDF definitions, both judge scores map to 4, increasing EMD from 0.2 to 1.0. The discrete mapping procedure therefore needs clarification to establish how it reliably improves distributional alignment.

Questions for authors
  1. How are the number and selection of anchors determined? Were results evaluated across independently sampled anchor sets, particularly when early anchors disproportionately contain high- or low-scoring examples?

  2. How does the implementation handle the counterexample above? Does it directly apply the stated formula, or include additional interpolation, smoothing, or other processing?

  3. What practical evaluation benefits does calibration provide when it leaves sample-level ordering unchanged and may even lose discriminative information by mapping distinct scores to ties? In particular, how do the numerical improvements support the claim that smaller judges approach larger ones in evaluation capability?

2 comments

We thank the reviewer for the insightful feedback and for the time and attention given to the details. Our answers below aim to clarify the work and ease understanding of the paper.

Q1: Anchor number and selection.

A1:We do not tune the anchor count per run. We use the fixed horizon of 100–200 labeled examples from Appendix B.1, and Figure 1 shows held-out MAE at every checkpoint along the stream. SQM’s state is K running counts per distribution. At a given anchor count it depends only on which items were seen, and at the full pool it is independent of order. Every checkpoint is scored on the same held-out 30% split, so a skewed early prefix shows up as slower convergence in the curves, not as a hidden bias.

Q2: Formula clarification and negative results.

A2: The reviewer’s arithmetic is correct for the formula as printed in Section 4.1. Our implementation does not apply that formula. It assigns each score its midrank percentile and reads off an interpolated human quantile. On the reviewer’s example this keeps 2 at 2 and 4 at 4, so EMD stays at 0.2. The printed equation did not match the code, and we will correct Section 4.1 accordingly. We also acknowledge that on a discrete scale a deterministic monotone map is not guaranteed to be the optimal transport plan. That guarantee holds for atomless distributions, so we will qualify the “closed-form optimal transport” wording in Sections 4 and 5.5. The midrank version is not guaranteed to improve EMD in every case, and we see this empirically. EMD improves in most groups (a median reduction of 27.9%), but it was worse than the raw judge in (7 of 55 groups). These groups are concentrated on skewed or low-variance attributes such as consistency, coherence and fluency. We will report this failure rate and these cases in the revision, and soften the claim that SQM reliably improves distributional alignment to “improves it in most settings”.

Q3: Calibration vs. discriminative ability.

A3: We agree that a monotone map cannot reorder samples and can only merge scores. We do not claim that calibration improves a judge’s ability to rank responses or predict preferences. Section 5.1 reports that rank correlations with human scores are unchanged (median Spearman ρ = 0.494 and Kendall τb = 0.450, before and after).

Our claim concerns absolute score fidelity. This matters when scores are consumed directly: thresholds, pass rates, reported means, score deltas in points, and comparisons between judges on a common scale. MAE, EMD and exact match against human scores are the right measures for this, and calibration is complementary to methods that improve ranking. We will reword Section 5.4 and the abstract to claim only a narrower error gap against human scores, not equivalent evaluation capability. We will also report per-model Spearman and Kendall values before and after calibration, so the unchanged rank-correlation gap between small and large judges is shown openly. On tie merging, SQM does not collapse scores in practice: the raw 5-point judge already has heavy ties, and rank correlations are essentially unchanged [median τ 0.254 → 0.244].

Final remarks

We thank the reviewer once again for a constructive discussion that helped sharpen our framing. In the revision we will make clear that calibration preserves the ordering of responses while substantially improving agreement with human scores, using only 100–200 labels and no fine-tuning. No calibrator can guarantee gains on every dataset, so we will document SQM’s limitations and the settings where gains shrink. Even with those boundaries, SQM is a lightweight, practical way to align LLM judge scores with human judgments, and we hope this clarifies the scope of our contribution.

Thank you for the thoughtful response and clarifications. I accept that some of my earlier concerns reflected a mismatch between my expectations and the paper’s intended focus on absolute-score and distributional calibration. The implementation clarification and additional analyses address most of my technical concerns.

I appreciate the method’s simplicity, low cost, and improvements in distributional alignment while remaining competitive on per-item error. However, I still have reservations about the significance of its contribution relative to existing calibration approaches.

I am therefore raising my soundness score while maintaining my recommendation of Weak rejection. I am also lowering my confidence, reflecting my uncertainty about the contribution’s significance. I would appreciate the AC weighing the other reviewers’ assessments alongside mine on this point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.