- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Rubric-Calibrated Preferences (RCP), a method for constructing dense relevance labels from LLM judgements. Stage A aggregates listwise judgements into within-query Bradley-Terry scores, while Stage B evaluates documents against shared binary relevance criteria. A two-parameter logistic (2PL) model from Item Response Theory (IRT) learns query-specific scales and offsets, mapping the tournament scores onto a scale based on shared criteria. The discrimination-weighted mean of the resulting criterion pass probabilities defines the continuous gain for RCP-nDCG.
The paper evaluates the approach across multiple retrieval benchmarks and validates its labels and reranker comparisons using a blinded external annotation study, NIST judgments, and re-assessments. The results support improved agreement with human judgments relative to the benchmark qrels and useful ordering of some documents that the rubric-only Count-nDCG baseline ties. The additional aggregate benefit of the complete calibrated metric over Count-nDCG is less clearly established.
Critical strengths
S1. Substantial validation against human judgements.
The paper goes beyond demonstrating increased metric sensitivity and checks whether its labels and reranker preferences agree with human assessments. The external annotation study hides system identities, list membership, qrels, and metric scores, and the analysis accounts for query-level dependence and includes relevant robustness checks. The NIST and re-assessment analyses provide additional references beyond the authors' annotation protocol. This is consequential because higher sensitivity alone would not establish that a new evaluation metric is useful.
S2. A coherent construction with evidence for useful fine-grained ordering.
The separation between within-query tournament ordering and rubric-based cross-query calibration is clear, and the mathematical properties are consistent with the construction. Importantly, Table 18 shows that the tournament resolves exact Count-nDCG ties in agreement with the human panel, alongside positive NIST and ViDoRe v3 results. This demonstrates that the additional ordering information can capture meaningful distinctions.
Critical weaknesses
W1. The incremental value of the complete IRT-calibrated metric remains insufficiently isolated.
Most headline human comparisons are against sparse benchmark qrels. The paper acknowledges that Count-nDCG accounts for most of the increased sensitivity and that removing calibration reverses the preferred reranker in only 7 of 289 human-study contests with a verdict. Although Table 18 provides meaningful evidence that the Stage A tournament resolves Count-nDCG ties in agreement with human judgments, and Table 16 reports favorable comparisons by external LLM judges, the incremental benefit for agreement with human reranker verdicts remains less clearly established. A direct, paired comparison of RCP-nDCG, Count-nDCG, and raw-BT-nDCG on the same human verdicts, with consistent treatment of ties and uncertainty intervals, would help isolate the contributions of the tournament and calibration. Additionally, a comparison with a simpler method combining the same absolute and relative judgments would help clarify the specific benefit of the proposed IRT merge.
W2. The correctness of the additional reranker separations remains incompletely validated.
Agreement on all NIST-decisive pairs in the primary per-year pool-panel setting with linear NIST gains is reassuring, as is the agreement with all 25 pairs separated by the DL19 re-assessment (Appendix H.2), but these results do not validate all additional pairs that RCP-nDCG separates and qrel-nDCG does not. Furthermore, Appendix H.1 reports agreement on 36 of 39 NIST-decisive pairs when the two TREC-DL years are pooled, with only one pair significantly separated by RCP-nDCG in the opposite direction. This suggests that the perfect agreement reported in the primary per-year evaluation does not necessarily extend to alternative evaluation settings. Appendix H.1 also notes that neither NIST nor the available re-assessment significantly orders the five rerankers ranked highest by RCP-nDCG on DL19. The external human study provides useful complementary evidence, but it uses a sample enriched for disagreements between RCP-nDCG and qrel-nDCG and an annotation rubric closely related to Stage B. These limitations constrain how broadly the results can establish more accurate comparisons of strong rerankers.
Questions for authors
Q1. Incremental benefit over Count-nDCG.
Could the authors provide a direct, paired comparison of RCP-nDCG, Count-nDCG, and raw-BT-nDCG on the same human-study contests, including agreement with human verdicts, uncertainty intervals, and a consistent treatment of ties? While the paper shows that the tournament can meaningfully resolve Count-nDCG ties, it remains unclear whether this leads to improved overall agreement with human reranker preferences. Such a comparison would help clarify the incremental contributions of the tournament and IRT calibration, particularly given the additional Stage A judging cost.
Q2. Necessity of the IRT-based merge.
Appendix E.3 compares the 2PL model with per-criterion logistic regression on raw BT scores, using the judge’s Stage B responses as the reference. Could the authors extend this comparison to calibration against human grades and agreement with human reranker verdicts? Evaluating this existing simpler baseline against the external human reference would help clarify whether the proposed 2PL calibration provides additional benefits for human alignment relative to per-criterion logistic regression.
Q3. Robustness of the IRT assumptions.
The 2PL formulation relies on a single calibrated document ability and conditional independence of criterion responses given that ability. However, Appendix E.2 and Table 8 report several criterion pairs with Yen’s $ |Q_3|>0.2$, the paper’s stated warning threshold for local dependence. Could the authors clarify whether this residual dependence materially affects the estimated criterion parameters, cross-query calibration, or agreement with human grades? The rubric robustness experiments in Appendix G.6 support leaderboard stability, but do not directly establish robustness of the probability calibration.
Q4. Independent validation and protocol sensitivity of additional reranker separations.
RCP-nDCG agrees with all NIST-decisive pairs in the primary per-year pool-panel setting with linear NIST gains, but Appendix H.1 reports agreement on 36 of 39 decisive pairs when the two TREC-DL years are pooled, with only one pair significantly separated by RCP-nDCG in the opposite direction, as well as some differences under alternative pool or gain settings. How should these results qualify the headline agreement claim? Given the positive re-assessment results in Appendix H.2, which of the additional pairs separated by RCP-nDCG but not by NIST-based qrel-nDCG are independently supported, and which remain inconclusive, particularly among the strongest rerankers?
3 comments
Thank you for the detailed and constructive rebuttal. I appreciate the additional paired comparisons with Count-nDCG and raw-BT-nDCG, the human-grade validation against per-criterion logistic regression, and the analyses of local dependence and DL19 re-assessments. These substantially address my original questions and provide useful evidence for the value of the proposed calibration.
My main remaining concern is the distinction between improvements in document-level ordering and cross-query calibration versus improvements in reranker-level evaluation. The new human-verdict comparison does not establish an advantage of RCP-nDCG over Count-nDCG (75.8% vs. 79.6%), although the additional results support the benefits of fine-grained ordering and cross-query label quality. Similarly, the DL19 re-assessments support some of the additional reranker separations, while comparisons among the strongest rerankers remain inconclusive.
Could the authors ensure that the revised main text and conclusion explicitly distinguish these contributions and limitations, particularly when discussing the advantages of RCP-nDCG over Count-nDCG? I do not think further extensive experiments are necessary, but clearer framing of the supported claims would help resolve my remaining concern.
Thank you for your reply. We agree with your distinction and have now revised the paper along it. The revision marks every change.
The main paper now separates three levels.
- Documents. The tournament orders the top of a pool better than the rubric alone (Section 4.4, with the new test under the benchmark qrels).
- Queries. Section 4.4 now states: "Across queries, the gains of Count-nDCG agree with human grades as well as those of RCP-nDCG."
- Rerankers. RCP-nDCG compares rerankers better than qrel-nDCG (Section 4.5). We claim no advantage over Count-nDCG.
The main text and the conclusion state both limitations.
- Section 4.4: "Count-nDCG also agrees with our annotators at least as often as RCP-nDCG. [...] Between rerankers and across queries, RCP-nDCG thus shows no advantage over its rubric-only variant."
- Conclusion: "Between rerankers, our evidence shows no advantage of RCP-nDCG over Count-nDCG, which is at least as good on the human verdicts. On DL19, a re-assessment supports 12 of the 47 reranker pairs that RCP-nDCG separates and NIST does not, and it does not separate the others."
- Section 4.5 adds that these pairs include those among the five rerankers that RCP-nDCG ranks highest.
Further changes. The contributions (Section 1) and Appendix A make the same distinction. Count-nDCG now appears in Tables 9, 10 and 21 and in the new paired Table 21a. No table or appendix of the submission changes its number.
Rebuttal
Thank you for your thoughtful review.
W1, Q1: paired comparison on the human contests. We add this table to Appendix H. All metrics use one tie rule, and a metric tie counts as a miss.
Count-nDCG is at least as good as RCP-nDCG on verdicts. We will present it as RCP's rubric-only variant (Proposition I.3). Raw-BT-nDCG leaves 30 contests tied. Where it names a winner, it agrees in 81.1%, against 77.9% for RCP-nDCG.
The tournament improves the order at the top over the rubric alone. References independent of our judge confirm this. As you note, Table 18 shows this for the documents that Count ties. A new test covers all queries. We take each label's 10 best documents per query and score them with the benchmark qrels. RCP's top 10 beats Count's by 0.041 nDCG@10 on NanoBEIR, 0.032 on BRIGHT and 0.024 on ViDoRe v3, with all 95% intervals above zero. Both labels face the same qrels. Thus, qrel noise therefore cannot favor either.
Calibration acts on a different axis. A verdict compares two rerankers on the same query, and nDCG normalizes each query. Calibration thus cannot change verdicts much. Its benefit is the comparability of labels across queries. There, RCP matches Count's agreement with human grades (AUC at Useful 0.910 against 0.913). Importantly, a simpler merge does not (Q2).
W1: a simpler combination. The per-criterion logistic regression of Appendix E.3 is such a method. The regression combines the same judgments with curves fitted once per suite, without per-query parameters. On verdicts, the regression matches RCP (76.1%). Across queries, it loses the comparability (Q2).
Q2: logistic regression against humans. Within a query, the regression is an increasing function of the raw tournament score. The regression thus orders each query as RCP. However, it cannot tell how relevant a pool is as a whole. Against the human grades (2,268 documents):
RCP is ahead on AUC (+0.068 [0.048, 0.089]), query-level correlation (+0.257 [0.182, 0.339]) and Brier score (-0.023 [-0.030, -0.015]). The ECE difference is not significant. The query-level value of 0.816 is for the gain
g, whereas Figure 4a reports 0.795 for the score. The per-query scale and offset of the 2PL thus provide the comparability across queries. We will use this regression as the comparator in Section 4.2.Q3: local dependence. C1 and C2, the two easiest criteria, form the pair flagged in every fit. We will report this dependence and no longer describe the assumption as met. We tested its effect with two refits of the 2PL, one without C1 and one without C2.
The dependence thus has little effect on the agreement with human grades.
Q4, W2: additional separations. We will scope the abstract and Section 4.5 to each TREC-DL year and report the pooled test there (36 of 39 pairs, Appendix H.1). On DL19, RCP-nDCG separates 47 reranker pairs that NIST does not.
With GPT-OSS-120B as judge, the re-assessment supports 11 pairs and contradicts none. The contests oversample disagreements because only disagreements can tell two metrics apart. The 90 concordant contests serve as controls (Appendix D.1).