acceptodds
Under review as a conference paper at ICLR 2027

Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory

Abstract

Rerankers decide which documents users and LLMs see, yet their standard metric, nDCG, relies on human relevance labels that are costly, sparse, noisy, and discretely graded. As rerankers approach each other in quality, nDCG on these labels therefore increasingly fails to separate them. LLM judges could supply dense labels. Relative judgments within one query tell even close candidates apart, yet their scores share no scale across queries. Absolute grades share one scale but are too coarse to distinguish documents of similar relevance. In this work, we propose Rubric-Calibrated Preferences (RCP), which combine both kinds of judgment. A listwise Bradley-Terry tournament orders each query's documents, and a rubric of yes/no criteria of increasing stringency provides an absolute standard. Item Response Theory (IRT), which scores test-takers based on their answers to common questions, then uses the shared criteria to put all queries' tournament scores on one scale. RCP's retrieval metric, RCP-nDCG, replaces nDCG's discrete labels with the resulting calibrated relevance probabilities. Against blind grades from 46 external annotators, calibration raises the correlation between a query's mean score and its mean human grade from 0.538 to 0.795. The probabilities rank a useful document above a non-useful one with probability 0.910 (AUC, chance 0.5), versus 0.651 for the benchmark labels. When the annotators' grades prefer one of two rerankers and exactly one metric agrees, that metric is RCP-nDCG in 72.4% of 185 comparisons (chance about 53%). On TREC-DL, RCP-nDCG sides with NIST assessors' grades on every reranker pair that these grades separate significantly. RCP-nDCG also resolves many of nDCG's ties and separates 1.9 times as many reranker pairs on NanoBEIR. Rubric calibration thus turns relative LLM judgments into dense relevance labels that are comparable across queries and agree with human judgment.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-Judge, Item Response Theory, Retrieval Evaluation, Reranking, Relevance Judgments, Calibration, Evaluation Metrics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2siVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Rubric-Calibrated Preferences (RCP), a method for constructing dense relevance labels from LLM judgements. Stage A aggregates listwise judgements into within-query Bradley-Terry scores, while Stage B evaluates documents against shared binary relevance criteria. A two-parameter logistic (2PL) model from Item Response Theory (IRT) learns query-specific scales and offsets, mapping the tournament scores onto a scale based on shared criteria. The discrimination-weighted mean of the resulting criterion pass probabilities defines the continuous gain for RCP-nDCG.

The paper evaluates the approach across multiple retrieval benchmarks and validates its labels and reranker comparisons using a blinded external annotation study, NIST judgments, and re-assessments. The results support improved agreement with human judgments relative to the benchmark qrels and useful ordering of some documents that the rubric-only Count-nDCG baseline ties. The additional aggregate benefit of the complete calibrated metric over Count-nDCG is less clearly established.

Critical strengths

S1. Substantial validation against human judgements.

The paper goes beyond demonstrating increased metric sensitivity and checks whether its labels and reranker preferences agree with human assessments. The external annotation study hides system identities, list membership, qrels, and metric scores, and the analysis accounts for query-level dependence and includes relevant robustness checks. The NIST and re-assessment analyses provide additional references beyond the authors' annotation protocol. This is consequential because higher sensitivity alone would not establish that a new evaluation metric is useful.

S2. A coherent construction with evidence for useful fine-grained ordering.

The separation between within-query tournament ordering and rubric-based cross-query calibration is clear, and the mathematical properties are consistent with the construction. Importantly, Table 18 shows that the tournament resolves exact Count-nDCG ties in agreement with the human panel, alongside positive NIST and ViDoRe v3 results. This demonstrates that the additional ordering information can capture meaningful distinctions.

Critical weaknesses

W1. The incremental value of the complete IRT-calibrated metric remains insufficiently isolated.

Most headline human comparisons are against sparse benchmark qrels. The paper acknowledges that Count-nDCG accounts for most of the increased sensitivity and that removing calibration reverses the preferred reranker in only 7 of 289 human-study contests with a verdict. Although Table 18 provides meaningful evidence that the Stage A tournament resolves Count-nDCG ties in agreement with human judgments, and Table 16 reports favorable comparisons by external LLM judges, the incremental benefit for agreement with human reranker verdicts remains less clearly established. A direct, paired comparison of RCP-nDCG, Count-nDCG, and raw-BT-nDCG on the same human verdicts, with consistent treatment of ties and uncertainty intervals, would help isolate the contributions of the tournament and calibration. Additionally, a comparison with a simpler method combining the same absolute and relative judgments would help clarify the specific benefit of the proposed IRT merge.

W2. The correctness of the additional reranker separations remains incompletely validated.

Agreement on all NIST-decisive pairs in the primary per-year pool-panel setting with linear NIST gains is reassuring, as is the agreement with all 25 pairs separated by the DL19 re-assessment (Appendix H.2), but these results do not validate all additional pairs that RCP-nDCG separates and qrel-nDCG does not. Furthermore, Appendix H.1 reports agreement on 36 of 39 NIST-decisive pairs when the two TREC-DL years are pooled, with only one pair significantly separated by RCP-nDCG in the opposite direction. This suggests that the perfect agreement reported in the primary per-year evaluation does not necessarily extend to alternative evaluation settings. Appendix H.1 also notes that neither NIST nor the available re-assessment significantly orders the five rerankers ranked highest by RCP-nDCG on DL19. The external human study provides useful complementary evidence, but it uses a sample enriched for disagreements between RCP-nDCG and qrel-nDCG and an annotation rubric closely related to Stage B. These limitations constrain how broadly the results can establish more accurate comparisons of strong rerankers.

Questions for authors

Q1. Incremental benefit over Count-nDCG.

Could the authors provide a direct, paired comparison of RCP-nDCG, Count-nDCG, and raw-BT-nDCG on the same human-study contests, including agreement with human verdicts, uncertainty intervals, and a consistent treatment of ties? While the paper shows that the tournament can meaningfully resolve Count-nDCG ties, it remains unclear whether this leads to improved overall agreement with human reranker preferences. Such a comparison would help clarify the incremental contributions of the tournament and IRT calibration, particularly given the additional Stage A judging cost.

Q2. Necessity of the IRT-based merge.

Appendix E.3 compares the 2PL model with per-criterion logistic regression on raw BT scores, using the judge’s Stage B responses as the reference. Could the authors extend this comparison to calibration against human grades and agreement with human reranker verdicts? Evaluating this existing simpler baseline against the external human reference would help clarify whether the proposed 2PL calibration provides additional benefits for human alignment relative to per-criterion logistic regression.

Q3. Robustness of the IRT assumptions.

The 2PL formulation relies on a single calibrated document ability θ~ij\tilde{\theta}_{ij} and conditional independence of criterion responses given that ability. However, Appendix E.2 and Table 8 report several criterion pairs with Yen’s $ |Q_3|>0.2$, the paper’s stated warning threshold for local dependence. Could the authors clarify whether this residual dependence materially affects the estimated criterion parameters, cross-query calibration, or agreement with human grades? The rubric robustness experiments in Appendix G.6 support leaderboard stability, but do not directly establish robustness of the probability calibration.

Q4. Independent validation and protocol sensitivity of additional reranker separations.

RCP-nDCG agrees with all NIST-decisive pairs in the primary per-year pool-panel setting with linear NIST gains, but Appendix H.1 reports agreement on 36 of 39 decisive pairs when the two TREC-DL years are pooled, with only one pair significantly separated by RCP-nDCG in the opposite direction, as well as some differences under alternative pool or gain settings. How should these results qualify the headline agreement claim? Given the positive re-assessment results in Appendix H.2, which of the additional pairs separated by RCP-nDCG but not by NIST-based qrel-nDCG are independently supported, and which remain inconclusive, particularly among the strongest rerankers?

3 comments

Thank you for your thoughtful review.

W1, Q1: paired comparison on the human contests. We add this table to Appendix H. All metrics use one tie rule, and a metric tie counts as a miss.

MetricAgrees with verdict (289 contests)RCP minus metric [95% CI]
RCP-nDCG75.8%
Count-nDCG79.6%-3.8 [-8.4, +1.0]
raw-BT-nDCG72.7%+3.1 [+0.3, +6.1]
per-criterion logistic regression76.1%-0.3 [-2.5, +2.1]
qrel-nDCG47.1%+28.7 [+20.2, +37.3]

Count-nDCG is at least as good as RCP-nDCG on verdicts. We will present it as RCP's rubric-only variant (Proposition I.3). Raw-BT-nDCG leaves 30 contests tied. Where it names a winner, it agrees in 81.1%, against 77.9% for RCP-nDCG.

The tournament improves the order at the top over the rubric alone. References independent of our judge confirm this. As you note, Table 18 shows this for the documents that Count ties. A new test covers all queries. We take each label's 10 best documents per query and score them with the benchmark qrels. RCP's top 10 beats Count's by 0.041 nDCG@10 on NanoBEIR, 0.032 on BRIGHT and 0.024 on ViDoRe v3, with all 95% intervals above zero. Both labels face the same qrels. Thus, qrel noise therefore cannot favor either.

Calibration acts on a different axis. A verdict compares two rerankers on the same query, and nDCG normalizes each query. Calibration thus cannot change verdicts much. Its benefit is the comparability of labels across queries. There, RCP matches Count's agreement with human grades (AUC at Useful 0.910 against 0.913). Importantly, a simpler merge does not (Q2).

W1: a simpler combination. The per-criterion logistic regression of Appendix E.3 is such a method. The regression combines the same judgments with curves fitted once per suite, without per-query parameters. On verdicts, the regression matches RCP (76.1%). Across queries, it loses the comparability (Q2).

Q2: logistic regression against humans. Within a query, the regression is an increasing function of the raw tournament score. The regression thus orders each query as RCP. However, it cannot tell how relevant a pool is as a whole. Against the human grades (2,268 documents):

MeasureRCPLogistic regression
AUC at Useful0.9100.841
Query-level Spearman (query mean gain vs mean grade)0.8160.560
Brier score (lower is better)0.1590.181
ECE at Useful (lower is better)0.0540.036

RCP is ahead on AUC (+0.068 [0.048, 0.089]), query-level correlation (+0.257 [0.182, 0.339]) and Brier score (-0.023 [-0.030, -0.015]). The ECE difference is not significant. The query-level value of 0.816 is for the gain g, whereas Figure 4a reports 0.795 for the score. The per-query scale and offset of the 2PL thus provide the comparability across queries. We will use this regression as the comparator in Section 4.2.

Q3: local dependence. C1 and C2, the two easiest criteria, form the pair flagged in every fit. We will report this dependence and no longer describe the assumption as met. We tested its effect with two refits of the 2PL, one without C1 and one without C2.

  • The difficulty order of the other criteria stays the same.
  • The calibrated scores correlate with the full fit at 0.96 or more.
  • In the pooled data of the three suites, neither refit worsens AUC at Useful, query-level Spearman or ECE against human grades beyond the 95% interval. We compute ECE through the full fit's gain function.

The dependence thus has little effect on the agreement with human grades.

Q4, W2: additional separations. We will scope the abstract and Section 4.5 to each TREC-DL year and report the pooled test there (36 of 39 pairs, Appendix H.1). On DL19, RCP-nDCG separates 47 reranker pairs that NIST does not.

  • The re-assessment separates 12 of these pairs, each with the same winner as RCP-nDCG.
  • One re-assessor alone separates 7 more, again with the same winner.
  • Neither the re-assessment nor a single re-assessor separates any of the 47 pairs with the opposite winner.
  • The other pairs remain open, including those among the five strongest rerankers.

With GPT-OSS-120B as judge, the re-assessment supports 11 pairs and contradicts none. The contests oversample disagreements because only disagreements can tell two metrics apart. The 90 concordant contests serve as controls (Appendix D.1).

Thank you for the detailed and constructive rebuttal. I appreciate the additional paired comparisons with Count-nDCG and raw-BT-nDCG, the human-grade validation against per-criterion logistic regression, and the analyses of local dependence and DL19 re-assessments. These substantially address my original questions and provide useful evidence for the value of the proposed calibration.

My main remaining concern is the distinction between improvements in document-level ordering and cross-query calibration versus improvements in reranker-level evaluation. The new human-verdict comparison does not establish an advantage of RCP-nDCG over Count-nDCG (75.8% vs. 79.6%), although the additional results support the benefits of fine-grained ordering and cross-query label quality. Similarly, the DL19 re-assessments support some of the additional reranker separations, while comparisons among the strongest rerankers remain inconclusive.

Could the authors ensure that the revised main text and conclusion explicitly distinguish these contributions and limitations, particularly when discussing the advantages of RCP-nDCG over Count-nDCG? I do not think further extensive experiments are necessary, but clearer framing of the supported claims would help resolve my remaining concern.

Thank you for your reply. We agree with your distinction and have now revised the paper along it. The revision marks every change.

The main paper now separates three levels.

  • Documents. The tournament orders the top of a pool better than the rubric alone (Section 4.4, with the new test under the benchmark qrels).
  • Queries. Section 4.4 now states: "Across queries, the gains of Count-nDCG agree with human grades as well as those of RCP-nDCG."
  • Rerankers. RCP-nDCG compares rerankers better than qrel-nDCG (Section 4.5). We claim no advantage over Count-nDCG.

The main text and the conclusion state both limitations.

  • Section 4.4: "Count-nDCG also agrees with our annotators at least as often as RCP-nDCG. [...] Between rerankers and across queries, RCP-nDCG thus shows no advantage over its rubric-only variant."
  • Conclusion: "Between rerankers, our evidence shows no advantage of RCP-nDCG over Count-nDCG, which is at least as good on the human verdicts. On DL19, a re-assessment supports 12 of the 47 reranker pairs that RCP-nDCG separates and NIST does not, and it does not separate the others."
  • Section 4.5 adds that these pairs include those among the five rerankers that RCP-nDCG ranks highest.

Further changes. The contributions (Section 1) and Appendix A make the same distinction. Count-nDCG now appears in Tables 9, 10 and 21 and in the new paired Table 21a. No table or appendix of the submission changes its number.

Reviewer_qJo4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an LLM-based method for constructing relevance labels for offline reranking evaluation, and a metric built on those labels. Two kinds of LLM judgments are collected for each query's candidate pool. In Stage A, the judge reranks windows of 10 documents and assigns each a continuous relevance score on [−5, 5]; only within-window score differences are kept, converted to soft pairwise preferences σ(si−si′)\sigma(s_i - s_{i'}), and fit with a Bradley–Terry model per query. In Stage B, the judge answers five binary rubric criteria for each document. A two-parameter logistic IRT model then maps each query's BT scores onto a shared scale: criterion parameters (γc,βc)(\gamma_c, \beta_c) are shared across all queries, and a per-query affine map (τj,αj)(\tau_j, \alpha_j) from BT scores to abilities is estimated from the Stage B answers. Equivalently, the model is a logistic regression of the binary answers on the BT score with query-specific slope and intercept and criterion-specific parameters. The gain of a document is the discrimination-weighted mean of its predicted pass probabilities (Eq. 4), and RCP-nDCG is nDCG with this gain and the Stage A order as the ideal ranking. The paper claims (1) a calibration method that places within-query LLM preferences on a cross-query scale, (2) a continuous metric, and (3) validation of both against grades from 46 external annotators and against NIST grades on TREC-DL.

Critical strengths
  1. No human relevance labels enter the fit (App. E.3). The evidence that the resulting labels can replace qrels for comparing rerankers is reasonably strong: in 185 contests where exactly one metric agrees with the annotators' verdict, it is RCP-nDCG in 72.4% (95% CI [66.0, 78.9], chance 52.7%; Table 21), and RCP-nDCG agrees with NIST on all 14 and 37 NIST-decisive pairs in the pool panel (Table 20). Two caveats on "no human input": the rubric wording was refined on 300 documents using judgments by two authors (App. J), and the experimental pools insert qrel positives missed by the first-stage retrievers (App. C.1).

  2. The evaluation is broad and the human study is carefully designed. Three suites plus TREC-DL; two or three judges; an external human study in which annotators see neither the lists, the qrels, nor any scores (App. D.2); NIST grades and the DL19 re-assessments of Parry et al. as references that share no annotations with the study; a rubric × prompt factorial (Table 15); and a budget replay (App. C.4).

Critical weaknesses

W1. The novel component, the IRT calibration that links Stage A and Stage B, is not shown to add value under either of the paper's two framings.

(a) As a metric. The paper's own appendices indicate that the calibration and the gain shape have no measurable effect on RCP-nDCG. Scoring the human-study contests with the gain applied to uncalibrated BT scores reverses the winner in 7 of 289 contests (App. E.3). Five alternative gain shapes give leaderboards with Spearman ≥ 0.859 to the paper's (App. E.5). A per-query min-max rescaling of the BT scores, which uses Stage A only and no rubric, separates more reranker pairs than RCP-nDCG (68.1% vs 63.2% on NanoBEIR; App. E.5). Conversely, Count-nDCG, which uses Stage B only, yields a nearly identical NanoBEIR leaderboard (Table 14) and separates 60.4% of pairs against 63.2% for RCP-nDCG (App. I.3); on TREC-DL, 56/64 pairs against 60/70 (Table 20). Either stage alone therefore produces the same reranker comparisons as both stages combined. The paper's evidence for the tournament's contribution beyond Count (Table 18; Fig. 6b) concerns document pairs that Count ties and preferences of LLM judges, not reranker comparisons against human verdicts. The improvement over qrel-nDCG (34.2% → 63.2% sensitivity) follows from replacing sparse binary labels with dense LLM labels, not from the calibration and the gain.

(b) As a label. The paper validates the calibration against raw BT scores (Table 9: pooled AUC 0.829 → 0.939 on NanoBEIR; Fig. 4a: query-level Spearman 0.538 → 0.795). Raw BT scores are not usable as a cross-query label, since they are centered within each query. Because the criteria are shared, the Count gain is already comparable across queries, and it is obtained with roughly one third of the judge budget (Stage B: 100 windows per query versus Stage A: 216 calls; Table 4). Count-nDCG does not appear in Table 9, Table 10, or Table 21.

(c) Under either reading, the comparison that would establish the value of the calibration step, against Count rather than against raw BT or against qrels, is absent.

(d) The estimand is not stated and changes across sections. The construction targets the judge's expected rubric score (Eq. 2, Prop. I.3, ECE against the judge's answers). The validation targets human perceived relevance, with the threshold for the probability reading chosen post hoc among three (App. E.3: ECE 0.054 at Useful, 0.167 at Topical, 0.187 at Answers). The IR claim targets reranker rankings. This explains why §4.2 compares against raw BT, §4.3 against qrels, and Count appears in neither.

W2. Several simple baselines and basic diagnostics are missing.

(a) Several simple baselines are missing. In addition to Count-nDCG in the human-verdict and label-AUC analyses: (i) the uncentered Stage A score level, averaged over windows, as a direct estimate of αj\alpha_j (the Stage A prompt asks for absolute anchors and context independence, App. J.1, so this information is collected and then discarded); (ii) a pointwise graded LLM judge (e.g., grades 0-3, repeated and averaged), the standard baseline in the LLM-as-assessor literature; (iii) Count with the Stage A budget reallocated to Stage B (the paper has only a reproducibility proxy on ViDoRe v3, App. E.4, and no validity comparison); (iv) a fixed total budget with the Stage A fraction varied. The budget replay (App. C.4) shows that 50.2% of the judgments reproduce the full leaderboard at Kendall 0.987, which suggests that the marginal value of the remaining judgments is small.

(b) The basic diagnostics of the fit of Eq. (2) are not reported. Two quantities should be shown first: the within-query Spearman or Kendall correlation between g(θ~ij)g(\tilde\theta_{ij}) and the Count gain ∑cSijc/(Cnij)\sum_c S_{ijc}/(C n_{ij}), and the distribution of the residual rij=1C∑c(Sijc/nij−P^ijc)r_{ij} = \frac{1}{C}\sum_c \big(S_{ijc}/n_{ij} - \hat P_{ijc}\big). The reported ECE of 0.017 (App. E.3) measures calibration only and does not substitute for either.

Questions for authors
  1. What are the main measure (Table 21), margin AUC, and AUC at Useful (Table 10) for Count-nDCG?
  2. What is the within-query rank correlation between the tournament order and the Count order, excluding Count ties?
  3. What does RCP-nDCG give if αj\alpha_j is replaced by the mean uncentered Stage A score of the query?
  4. With a fixed total judge budget, how do validity and sensitivity change as the Stage A fraction varies from 0 to the current value?
3 comments

Thank you for your review.

W1(a, b, c), Q1: what RCP adds over Count. We add Count-nDCG to Tables 9, 10 and 21.

Requested measureRCPCountqrels
AUC at Useful (Table 10)0.9100.9130.651
Main measure (Table 21)72.4% of 18577.0% of 174
Margin AUC (Table 21)0.8100.8220.521

Here, Count is at least as good as RCP. We will present it as RCP's rubric-only variant (Proposition I.3). We also agree that calibration barely changes reranker comparisons, because nDCG normalizes each query. RCP adds two things over Count.

  • Tournament improves the order at the top. We take the 10 best documents per query of both and score them with the qrels. RCP beats Count by 0.041 nDCG@10 on NanoBEIR, 0.032 on BRIGHT and 0.024 on ViDoRe v3, with all 95% CI above zero. The reference is human and qrel noise cannot favor either label.
  • Calibration makes this order usable across queries. Count is comparable across queries. RCP matches its agreement with human grades (AUC 0.910 against 0.913). Against Count, calibration thus gives parity and not a higher AUC.

W1(d): estimand. We agree and will state the estimand in Section 3.4. RCP estimates one quantity, the gain g of Equation 4: the share of the rubric that the judge expects a document to pass, weighted by discrimination. The human grades and the reranker comparisons test this quantity and are not further estimands. Reading g as the probability of a grade of Useful or better is empirical (ECE 0.054). The annotators' Useful threshold lies between the difficulties of C2 and C3 (Appendix F.1).

W2(a)(i), Q3: Stage A level as the offset. We agree that Stage A carries unused level information, since the prompt asks for scores on an absolute scale (Appendix J.1). The query's mean uncentered score correlates with alpha_j at Pearson 0.83 to 0.89 across suites. The rubric's offset remains the better choice for two reasons.

1. Agreement with human grades. An offset keeps the order within each query and only places the queries. We replaced alpha_j with the Stage A level, mapped with two constants per suite that we fit to alpha_j. Against the human grades on NanoBEIR and BRIGHT, the Stage A offset does worse on three measures. Its mean gain misses the share of Useful grades by 3.6 points (RCP: 0.3). Its Brier score is 0.160 against 0.149. Its query-level Spearman (query mean gain vs mean grade) is 0.787 against 0.829. All three 95% CIs exclude zero. ECE is level (0.052 against 0.050). RCP-nDCG barely changes (leaderboard Kendall 0.978 and 0.934).

2. Meaning of the scale. A Stage A score is a number on the judge's own sigmoid logit scale from -5 to 5, grounded in context (prompt, query, window documents). A gain from this score still needs tau_j, the criterion parameters and two constants from the rubric fit. Even then, the Stage A level misses alpha_j by 1.1 to 1.4 logits for a typical query, more than the distance between criteria C3 and C4 (0.96 logits on NanoBEIR). The rubric states the scale in named criteria.

W2(a)(ii): a pointwise graded judge. We ran no separate graded judge, because this response uses no new judge calls. Count is the closest baseline: it averages about 6.7 Stage B gradings per document against five fixed criteria. We will cite UMBRELA.

W2(a)(iii, iv), Q4: budget. RCP needs less than half of the calls of the paper's protocol. Replays of the logged judgments (cells: NanoBEIR / BRIGHT):

Calls per queryLabelSensitivityTop 10 vs qrels (nDCG@10)AUC at Useful
100Count60.4% / 88.0%0.688 / 0.4540.937 / 0.886
100RCP (85 tournament, 15 rubric)63.9% / 88.3%0.710 / 0.4790.920 / 0.872
138RCP (108 tournament, 30 rubric)63.6% / 88.3%0.720 / 0.4860.936 / 0.886
316RCP (paper protocol)63.2% / 88.0%0.729 / 0.4860.939 / 0.889
  • 138 calls stay within 0.01 of the paper protocol on every measure, beat Count's top 10 by 0.032 on both suites and match Count's AUC at Useful.
  • Equal budget (iv, Q4). At 100 calls, a tournament share of 85% still beats Count's top 10, with a slightly lower AUC at Useful. Below a share of one half, RCP does not beat Count.
  • Count at RCP's budget (iii) needs new judge calls: the logs hold only 6.7 gradings per document.

W2(b), Q2: fit diagnostics. Within a query, the median Spearman correlation between the tournament order and Count is 0.70 (NanoBEIR), 0.79 (BRIGHT) and 0.82 (ViDoRe, TREC-DL). Without Count's ties, the two orders agree on 86% to 90% of pairs. The residual r_ij has an SD of 0.064 to 0.071 across suites. On NanoBEIR, its 5th and 95th percentiles are -0.10 and 0.12.

Some of the new baselines come close to RCP. Where the task is harder, such as ordering several good documents at the top of a pool, the two stages together work best.

I thank the authors for the new analyses, which answer my four questions and confirm that Count-nDCG is at least as good as RCP-nDCG on each of the three measures I requested (AUC at Useful 0.913 vs 0.910; main measure 77.0% vs 72.4%; margin AUC 0.822 vs 0.810). The Stage A level experiment (Q3) answers my question, and I withdraw that suggestion. The budget table shows that the protocol can be reduced to 138 calls per query, and the changes listed for the final version, in particular Count in Tables 9, 10 and 21, address the other points of my review.

The response also states the estimand: the share of the rubric that the judge expects a document to pass, weighted by discrimination. I take this statement as the basis of my assessment, because it determines what the paper has to show. Under this estimand, the discrimination-weighted Count (Proposition I.3) is not a baseline. It is the direct, unbiased estimator of the stated quantity, obtained from Stage B alone. RCP is an indirect estimator of the same quantity through the affine map of Eq. (2), and it can improve on Count only by lowering the estimation error at a given judge budget. The claim the paper must support is therefore that Stage A, at its cost (216 of 316 calls per query on NanoBEIR, Table 4), reduces this error more than the same calls spent on further Stage B placements would.

None of the reported experiments tests this claim. The results against raw BT scores (Table 9, Fig. 4a) compare RCP with a quantity that does not estimate the estimand. The comparison with the per-criterion logistic regression (shared response, claim 3) answers which map from Stage A to the estimand is better, not whether Stage A is needed. Count requires no such map and reaches the same agreement. The results against human grades and qrels (Tables 10 and 21) and against NIST (Table 20) test whether the estimand agrees with human relevance judgments. On the human-study measures, Count matches RCP with one third of the budget; this is the only comparison at the paper's own budgets, and it is evidence against the claim. The tie-resolution test (Table 18) covers only pairs that Count ties and uses unequal budgets, since further placements would resolve many of these ties. The ideal-list tests (Tables 16, 17 and 19, and the new top-10 result) use unequal budgets and break Count's ties at random (App. H.5), by identifier (Table 19), or by an unstated rule (the new result). The reproducibility test (App. E.4) is the only comparison in the paper at equal budgets, and it measures stability rather than error. The new budget table adds one point at equal budgets (100 calls), where RCP is higher by about 0.02 on the top-10 nDCG and lower by about 0.015 on AUC at Useful, with no confidence intervals. The new fit diagnostics are in-sample and show that RCP disagrees with Count on 10 to 14% of the pairs that Count orders, but they do not show which of the two is closer to the estimand.

The experiment that would test the claim directly can be run on the released judgments: hold out part of each query's Stage B windows, predict the held-out pass shares from the remaining windows alone (Count) and from fewer remaining windows plus Stage A calls, so that the two use the same number of calls (RCP), and compare the errors. A second experiment, which requires new judge calls, is Count at 138 and 316 calls per query. Stating the estimand in the final version is an improvement, but it does not change my assessment, because the experiments do not address the claim that follows from it.

Thank you for your reply.

The estimand. Your argument reveals that our wording lacked nuance. The gain g of Equation 4 is a function of the calibrated tournament score. That is the share of the rubric that the 2PL expects a document with this score to pass. A document's own pass share, which Count estimates, differs from the unweighted share that the 2PL expects by the residual r_ij that you asked us to report. We will state this in the revision of Section 3.4.

Your held-out experiment. We ran it on the supplied judgments. We held out 30% of each query's Stage B windows and gave each label the same 70 calls per query. This test asks which label predicts the judge's held-out rubric answers. Crucially, Count is itself an average of such answers and converges to this target. RCP on the other hand converges to the 2PL's expectation. Unsurprisingly, Count predicts them better. Count's Brier score is 0.038 on NanoBEIR and 0.044 on BRIGHT. The Brier score of RCP is higher by 0.005 to 0.009 on NanoBEIR and by 0.005 to 0.007 on BRIGHT, depending on which logged tournament calls RCP keeps. The paper does not claim that RCP estimates the judge's rubric answers better.

What Stage A adds. A label can agree with the judge's repeated answers and still be wrong about relevance. References outside the judge therefore decide between the two labels. At 100 calls per query for each label:

  • Top 10 under the benchmark qrels. RCP is higher by 0.022 [0.010, 0.032] on NanoBEIR and by 0.025 [0.016, 0.035] on BRIGHT. Documents that Count ties receive their expected credit under a random order (Appendix G.7).
  • AUC at Useful against human grades. Count is higher by 0.017 [0.003, 0.030] on NanoBEIR. On BRIGHT, the interval of the difference includes zero.

The revision claims exactly this (Sections 1, 4.4 and 6). The tournament orders the top of a pool better than the rubric alone. Across queries and between rerankers, RCP shows no advantage over Count.

Count at 138 and 316 calls. The logs hold about 6.7 gradings per document. The released judgments therefore cannot trivially compute this comparison.

Reviewer_ZyvROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Rubric-Calibrated Preferences (RCP), a methodology designed to evaluate retrieval systems by projecting isolated, within-query LLM preferences onto a shared cross-query scale. By combining listwise Bradley-Terry tournaments with a fixed Item Response Theory (IRT) rubric, the authors aim to replace discrete, sparse human labels with continuous relevance probabilities. The method is evaluated on a large scale across 13 NanoBEIR datasets, 12 BRIGHT tasks, and ViDoRe v3, utilizing multiple LLMs as judges.

Critical strengths
  • The scale of the empirical validation is highly commendable. Testing the metric across multiple benchmarks and evaluating 14 different rerankers demonstrates the robust applicability of the approach.

  • The methodology provides a highly pragmatic solution to the saturation and compression issues plaguing standard nDCG metrics. Translating relative judgments into a unified, absolute scale is highly useful for benchmarking modern recommendation and retrieval systems at scale.

Critical weaknesses
  • The technical core of the paper relies entirely on established methods—specifically the Bradley-Terry model and the 2-Parameter Logistic IRT model. While the pipeline composition is effective, the fundamental algorithmic contribution is thin for an ICLR submission.

  • Despite prompting the LLM for a "reasoning" string during the listwise tournament and a "justification" during the rubric evaluation, these outputs are entirely ignored in the final metric calculation. A dense scalar score is insufficient for automated LLM judges; redefining governance, control, and accountability is critical when deploying these systems. Without a structured explainability framework to audit the underlying reasoning, there is effectively no mechanism for keeping a human in the oversight.

  • The paper relies heavily on aggregate metrics like Expected Calibration Error and sensitivity, but lacks a granular investigation of its own failure modes. For instance, the multi-hop failure on HotpotQA is noted quantitatively, but the mechanical causes are not systematically diagnosed. A detailed qualitative breakdown of false positives and LLM hallucinations is necessary to understand when and why the metric misaligns with human judgment.

Questions for authors

1- Can the authors provide a qualitative analysis of the failure cases, specifically examining instances where the LLM judge confidently passes an irrelevant document?

2- How might the proposed metric incorporate the generated reasoning strings to provide verifiable confidence bounds or accountability mechanisms?

3- Given that the ZeroEntropy rerankers are trained on LLM preferences, how do the authors mitigate or measure the risk of the RCP-nDCG metric inherently biasing toward LLM-trained rankers over other architectures?

1 comment

Thank you for the encouraging review and the concrete questions.

W1: novelty. We agree that BT and the 2PL are established. The novel step is the 2PL merge. RCP treats the rubric criteria as test items that all queries share. The shared items place separate tests on one scale (Lord, 1980). As a result, RCP puts every query's tournament scores without human involvement on one scale. Both parts of the merge matter (see shared response). The tournament improves the order at the top over the rubric alone. Crucially, calibration makes this order comparable across queries.

The merge also inherits a critical property of IRT. The 2PL estimates how sharply each criterion's pass probability rises with a document's relevance. The gain weights the criterion by this discrimination (Equation 4). A criterion that does not track relevance thus gets almost no weight. In a new test on NanoBEIR and BRIGHT, a criterion that answers at random receives under 3% of the weight. A plain rubric count gives it one sixth. We also release ~7K blind grades from 46 external annotators.

W2, Q2: reasoning strings and oversight. We agree that LLM judges need human oversight. In RCP, human judgment enters at three points.

  • Before judging: we refined the wording of the criteria to raise the agreement between the judge's answers and human judgments on 300 NanoBEIR documents (Appendix J). All other data are held out.
  • During judging: RCP keeps a record that people can audit. The record holds named yes/no answers to five criteria for every document, and we release it.
  • After judging: we test the scores against people who never saw them, namely our blind annotators, NIST and the DL19 re-assessors.

We do not build the score on the rationales, because a free-text rationale need not reflect how the model reached its answer (Turpin et al., 2023). The rationales can still guide audits. The fit shows where the rubric answers and the calibrated score disagree (2% of NanoBEIR documents by more than 0.2). If needed, a reviewer can read the rationales of these documents first.

W3, Q1: failure cases. We add a failure-case appendix with two new analyses.

  • Confident false passes are rare. Among human-study documents with a gain of at least 0.7, annotators rate 2.9% below Topical (16 of 544). For comparison, annotators rate 14.5% of the documents with a positive qrel below Topical.
  • Multi-hop queries are a known failure case. On HotpotQA, RCP's top two documents are the two qrel documents in 42 of 50 queries (44 with GPT-OSS-120B). In the other queries, a document that looks relevant on its own displaces the document for the second fact. One example is a second document on the first fact (Appendix F.3).

Q3: bias toward LLM-trained rerankers. We measure this risk with references that use no LLM (Section 4.4, Appendix G.5). Our annotators never see system names. In 66 contests, RCP-nDCG favors a zerank model and qrel-nDCG favors the other reranker. Annotators prefer the zerank model in 78.8% of these contests. On TREC-DL, NIST does not separate zerank-2 from each year's leader. Part of its lower NIST rank arises because NIST never graded many of the passages that zerank-2 retrieves. Such passages fill 8.9% of its top-10 slots, against 3.7% to 5.4% for the systems from other families that NIST ranks above it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Shared response

We thank all three reviewers for their detailed feedback. The reviewers find the human study "carefully designed" (qJo4), the validation against humans "substantial" (2siV), and RCP "a highly pragmatic solution" to nDCG's saturation (ZyvR). We summarize our three claims and the changes for the final version.

1. Rubric-anchored dense labels agree with blind humans far better than qrels. RCP ranks a document that annotators grade Useful above one that they do not with probability 0.910 (AUC, chance 0.5). The qrels reach 0.651 (Table 10).

2. The tournament improves the order at the top over the rubric alone. References independent of our judge confirm this.

  • New: we take each label's ten best documents per query and score them with the benchmark qrels. RCP's top 10 beats the top 10 of the rubric-only label (Count) by 0.041 nDCG@10 on NanoBEIR, 0.032 on BRIGHT and 0.024 on ViDoRe v3. All 95% intervals lie above zero. Both labels face the same qrels, and qrel noise therefore cannot favor either.
  • At the top, Count often ties documents. The tournament orders these tied pairs like our annotators in 0.680 of cases, like NIST in 0.671 and like the ViDoRe v3 qrels in 0.660 (Table 18, chance 0.5).
  • On DL19, the consensus of NIST and two re-assessors prefers RCP's top 10 on 72.1% of the queries where the two lists differ (Table 19).

3. Calibration places this order on one scale across queries and keeps the rubric's agreement with human grades. Count has a shared scale by construction, because all queries share its criteria. RCP matches Count's agreement with human grades (AUC at Useful 0.910 against 0.913) and adds the tournament's order. Simpler routes to a shared scale lose this agreement. Raw tournament scores reach an AUC of 0.829 on NanoBEIR, against 0.939 after calibration (Table 9). New: a regression on the same judgments without per-query parameters reaches 0.841 on all graded documents, against 0.910 for RCP.

Changes for the final version. We add Count-nDCG to Tables 9, 10 and 21 and present it as RCP's rubric-only variant. We state the estimand, scope the NIST claim to each TREC-DL year, and report the local dependence between criteria C1 and C2. We add a failure-case appendix and the new analyses of this response. All new numbers use the released supplement and the public DL19 re-assessments, with no new judge calls. Brackets are 95% query-cluster bootstrap intervals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.