- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
RankJudge generates document-grounded conversation pairs with a designated flawed turn and failure category, then checks them through automated verification. A Bradley–Terry/Rasch model jointly estimates judge ability and item difficulty, supporting both ranking and benchmark curation. The paper evaluates 21 judges across three domains. Further experiments examine generator dependence, scoring criteria, incomplete coverage, prompting, pointwise evaluation, and fine-tuning on items of different difficulty.
Critical strengths
-
The paper addresses the practical need to keep multi-turn judge benchmarks current.
-
The additional experiments give a fuller picture than a single leaderboard.
-
The method is tested in different domains and the experiments reveals substantial differences in judge ability.
Critical weaknesses
-
The audit used to justify difficulty-based filtering depends on the judges being evaluated. A blinded audit with overlapping annotations is needed to establish this relationship and assess the quality of the released set.
-
The fine-tuning study does not separate label noise from genuine difficulty or differences in failure categories. Each difficulty band contributes one 100-pair training set, while both difficulty and retention vary substantially by category.
-
The connection between synthetic diagnostic accuracy and broader judge quality needs independent validation.
-
The incomplete-coverage experiment needs clearer uncertainty estimates and a comparison with simpler estimators.
Questions for authors
Please refer to the Weakness section.
We thank the reviewer for the feedback. All new results below are in the revised PDF.
W1. Blinded audit with overlapping annotations. We ran this audit. To break the dependence on the judges, we repeated the audit without use of judge outputs, per-judge accuracy, or Elo ratings. The same 120 pairs were relabeled by an annotator who is not an author and took no part in building the benchmark, so each pair now has two independent labels (new Appendix C.4).
The blinded audit therefore shows that the dependence on the judges is weak, and the relation between flagged pairs and Elo holds without access to judge outputs.
W2. Fine-tuning study. We reran the study with the failure categories matched across bands. Each training set has 40 pairs, with the same number of pairs per category in every band (the original used 100 pairs per band without matching). We drew 5 training sets per band, fine-tuned one model on each (15 models in total), and evaluated them on the held-out clean pairs of the same categories.
The top band is the lowest in all 5 draws, so the gap is not explained by category composition or sampling variation. If the top band held only valid hard pairs, its labels would be correct, and training on it would not lower accuracy. The drop therefore indicates that part of the top band is label noise, which agrees with the audit in W1.
W3. Independent validation of the judge ranking. The revision adds two independent checks.
Judge Elo therefore tracks model quality as rated by humans, and the labeled failures are ones that a human finds independently.
W4. Elo and simpler estimators under incomplete coverage. We now report the mean and 95% confidence interval over 200 random subsamples per setting, and compare Elo with two simpler estimators. Raw accuracy is the fraction correct on the pairs each judge was observed on. Centered accuracy subtracts, on each pair, the mean accuracy of the other judges observed on it. Entries are Spearman rho with the full-data ranking.
Under uniform coverage, the three estimators are within 0.02 of each other. They separate under uneven coverage, where each judge is observed more often on easy or on hard pairs. This is the setting of a new judge that is run on part of the pool. Raw accuracy then falls to 0.75 at 10% coverage, while Elo stays at 0.91, and centered accuracy is below Elo at every fraction (new Appendix A.13). Figure 5(c) and Figure 12 now show the 95% confidence intervals.