acceptodds
Under review as a conference paper at ICLR 2027

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator

Abstract

As interactive LLM-based applications are created and refined, model developers need to evaluate the quality of generated text along many possible axes. LLM judges make this evaluation scalable, but the judges themselves must also be evaluated to ensure that their assessments are reliable. Benchmarks built around isolated responses do not test failures that depend on conversation history, and fixed benchmarks can provide less differentiation among increasingly capable models. Testing judges for such interactions requires multi-turn benchmarks that can be renewed as models and applications evolve. We introduce RankJudge, an end-to-end generator of document-grounded, multi-turn judge benchmarks. Its fully synthetic construction produces labeled conversation pairs through controlled generation and automated verification, without per-item human annotation. A judge-versus-problem formulation then jointly estimates judge ability and pair difficulty using the Bradley-Terry model. These estimates support difficulty-based benchmark curation and judge ranking under incomplete evaluation coverage. We instantiate RankJudge in machine learning, biomedicine, and finance, and evaluate 21 LLM judges, revealing substantial differences in judge ability across models. RankJudge thus provides a reusable construction and calibration pipeline, rather than only a fixed evaluation set.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-Judge, Multi-Turn Conversations, Benchmark, Synthetic Data Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K9wCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

RankJudge generates document-grounded conversation pairs with a designated flawed turn and failure category, then checks them through automated verification. A Bradley–Terry/Rasch model jointly estimates judge ability and item difficulty, supporting both ranking and benchmark curation. The paper evaluates 21 judges across three domains. Further experiments examine generator dependence, scoring criteria, incomplete coverage, prompting, pointwise evaluation, and fine-tuning on items of different difficulty.

Critical strengths
  1. The paper addresses the practical need to keep multi-turn judge benchmarks current.

  2. The additional experiments give a fuller picture than a single leaderboard.

  3. The method is tested in different domains and the experiments reveals substantial differences in judge ability.

Critical weaknesses
  1. The audit used to justify difficulty-based filtering depends on the judges being evaluated. A blinded audit with overlapping annotations is needed to establish this relationship and assess the quality of the released set.

  2. The fine-tuning study does not separate label noise from genuine difficulty or differences in failure categories. Each difficulty band contributes one 100-pair training set, while both difficulty and retention vary substantially by category.

  3. The connection between synthetic diagnostic accuracy and broader judge quality needs independent validation.

  4. The incomplete-coverage experiment needs clearer uncertainty estimates and a comparison with simpler estimators.

Questions for authors

Please refer to the Weakness section.

1 comment

We thank the reviewer for the feedback. All new results below are in the revised PDF.

W1. Blinded audit with overlapping annotations. We ran this audit. To break the dependence on the judges, we repeated the audit without use of judge outputs, per-judge accuracy, or Elo ratings. The same 120 pairs were relabeled by an annotator who is not an author and took no part in building the benchmark, so each pair now has two independent labels (new Appendix C.4).

  • Agreement: the blinded labels agree with the first audit on 113 of the 120 pairs (Cohen's kappa = 0.75).
  • Relation to difficulty: the blinded labels flag 11 of the 30 highest-Elo pairs, against 5 of the remaining 90 (Fisher's exact test, p = 8e-5). The first audit found 12 and 5.
  • Released set: of the 96 audited pairs that remain in the release, the blinded labels flag only 2 (2.1%, 95% CI 0.3% to 7.3%).

The blinded audit therefore shows that the dependence on the judges is weak, and the relation between flagged pairs and Elo holds without access to judge outputs.

W2. Fine-tuning study. We reran the study with the failure categories matched across bands. Each training set has 40 pairs, with the same number of pairs per category in every band (the original used 100 pairs per band without matching). We drew 5 training sets per band, fine-tuned one model on each (15 models in total), and evaluated them on the held-out clean pairs of the same categories.

Training bandJoint accuracy (mean +/- sd)
Top 20%20.6 +/- 2.4
20 to 40%26.1 +/- 2.9
Bottom 60%30.8 +/- 2.7

The top band is the lowest in all 5 draws, so the gap is not explained by category composition or sampling variation. If the top band held only valid hard pairs, its labels would be correct, and training on it would not lower accuracy. The drop therefore indicates that part of the top band is label noise, which agrees with the audit in W1.

W3. Independent validation of the judge ranking. The revision adds two independent checks.

  • External: judge Elo agrees with the LMArena text leaderboard, which is fit only on human preference votes from real user conversations and uses none of our data, at Spearman rho = 0.82 over the 20 of our judges listed there (95% bootstrap CI 0.55 to 0.96, new Appendix A.14).
  • Human: in the first stage of the blinded audit of W1, the annotator read each pair with no label shown. The annotator chose the labeled worse conversation on 114 of the 120 pairs, and also the labeled turn on 107 of the pairs (new Appendix C.4).

Judge Elo therefore tracks model quality as rated by humans, and the labeled failures are ones that a human finds independently.

W4. Elo and simpler estimators under incomplete coverage. We now report the mean and 95% confidence interval over 200 random subsamples per setting, and compare Elo with two simpler estimators. Raw accuracy is the fraction correct on the pairs each judge was observed on. Centered accuracy subtracts, on each pair, the mean accuracy of the other judges observed on it. Entries are Spearman rho with the full-data ranking.

ObservedElo, uniformElo, unevenRaw accuracy, unevenCentered accuracy, uneven
10%0.922 [0.855, 0.971]0.905 [0.817, 0.966]0.747 [0.565, 0.884]0.891 [0.799, 0.956]
20%0.969 [0.939, 0.992]0.961 [0.917, 0.988]0.779 [0.595, 0.910]0.942 [0.891, 0.977]
50%0.991 [0.977, 0.999]0.989 [0.971, 0.996]0.852 [0.759, 0.930]0.974 [0.943, 0.992]
80%0.997 [0.988, 1.000]0.996 [0.990, 1.000]0.947 [0.913, 0.979]0.991 [0.979, 0.999]

Under uniform coverage, the three estimators are within 0.02 of each other. They separate under uneven coverage, where each judge is observed more often on easy or on hard pairs. This is the setting of a new judge that is run on part of the pool. Raw accuracy then falls to 0.75 at 10% coverage, while Elo stays at 0.91, and centered accuracy is below Elo at every fraction (new Appendix A.13). Figure 5(c) and Figure 12 now show the 95% confidence intervals.

Reviewer_DpuLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an automatic benchmark for LLM-as-a-Judge evaluation instruments. The benchmark is based on simulation in which a model automatically generates two conversations one of which contains a flaw (based on a preset automatically extracted). These conversations are then put through a single prompt, single model LLMaJ process with the goal of recognising which conversation at which turn has which flaw type. These results are put through a scoring model which at the end generates an ELO score for each model (and each item).

Critical strengths

Good idea of applying an item based ELO calculation. The context grounded multi-turn evaluation still has a gaps which need research.

Critical weaknesses

lack of human validation -- how accurate are the categories for flaws and user behaviours lack of prompt and model variability investigation -- would a different prompt re-rank differently, would a different model for extracting flaws create different rankings authors themselves annotated the data in a human validation study introducing possible bias same model used for judging and being judged -- the ablation study is only 150 pairs difficulty is relative but presented as absolute -- how difficult are difficult cases for humans post hoc choices for restricting the data -- why wasn't the 5% taken from each category and not in total writing needs clarification -- for example phrases like "We therefore generate the branches separately while staging comparable conversational pressures in both." (what are conversational pressures?) or “Confirming that the Elo ranking is robust under residual noise” (what is residual noise) or BT model is introduced without a reference or an explanation (which only follows )

Questions for authors

Why was the sepcific model chosen to extract the flaw types, and would other models extract a different taxonomy? What is the evidence that the proposed taxonomy correctly covers flaw types in real conversations? How were prompts optimised, and why were the specific prompts chosen? Would different prompts reverse the ranking of the models? Is the difficulty checked agains real life task difficulty? Why is 5% removed, and does the distribution remain stable over all flaw types after this removal?

6 comments

We thank the reviewer for the detailed comments. The review does not number its points, so we number them in order and quote each one before our answer. Several weaknesses and questions concern the same part of the paper, and we answer these together. Additions are marked in blue in the revised PDF.

W1. Validation of the failure-type labels

W1: lack of human validation -- how accurate are the categories for flaws and user behaviours

Each generated pair is checked twice against its label. The adherence check rejects a pair when the worse conversation lacks the target failure at the designated turn, when another flaw is present, or when the user departs from the assigned behavior (Section 3.2). The human audit then checks the full label by hand. Annotators confirmed the label, including the failure type, for 103 of 120 sampled pairs, and the other 17 were removed (Section 3.4). A second, independent audit by an annotator who is not an author agrees with the first on 113 of the 120 pairs (see W3). The flagged pairs concentrate at the top of the Elo range, so the release also removes that range among the pairs that were not audited (see W6). User behaviors serve to diversify the conversations and are not part of the label that judges are scored on.

Q1, Q2. Source and coverage of the taxonomy

Q1: Why was the sepcific model chosen to extract the flaw types, and would other models extract a different taxonomy?

Q2: What is the evidence that the proposed taxonomy correctly covers flaw types in real conversations?

The seven failure types start from failures documented in prior studies of multi-turn assistants (Section 3.1). We then refined them with gemini-3.1-pro on the full MT-Bench and MT-Bench 101 datasets over several rounds, and stopped when no new category emerged. In each round the model only proposes candidate categories (prompt in Appendix B.3), and the authors decide which to keep. The choice of model therefore has little effect on the taxonomy. We also ran gemini-3-flash on 100 dialogues from MT-Bench 101, and the categories it proposed match those from gemini-3.1-pro, with small differences in wording (Appendix B.3).

Two further results support the coverage. In the human audit, annotators also checked both conversations for flaws other than the labeled one, and none falls outside the seven types (Section 3.4). The benchmark built on these types also agrees with real use. Judge Elo agrees with the LMArena leaderboard, which is fit only on human preference votes from real user conversations, at Spearman rho = 0.82 over 20 judges (new Appendix A.14).

W2, Q3, Q4. Judge ranking under different prompts

W2: lack of prompt and model variability investigation -- would a different prompt re-rank differently, would a different model for extracting flaws create different rankings

Q3: How were prompts optimised, and why were the specific prompts chosen?

Q4: Would different prompts reverse the ranking of the models?

All 21 judges use the same default prompt, which states the task, the definitions of the seven types, and the output format (Appendix B.8). It was written once and was not tuned for any judge. To test other prompts, we wrote four revised prompts for opus-4.7 from an analysis of its errors: category hints, explicit comparison of the two conversations, decision rules, and chain of thought. These prompts target this judge directly, so this is the case where a prompt change should move the rank most. With the other judges on the default prompt, its rank stays between 11 and 13 of 21, against 11 under the default prompt (Section 4, Appendix B.8).

W3. Independent relabel of the audited pairs

W3: authors themselves annotated the data in a human validation study introducing possible bias

The revision adds a second, independent audit of the same 120 pairs by an annotator who is not an author and took no part in building the benchmark. This audit was done without use of judge outputs, per-judge accuracy, or Elo ratings. Its annotations agree with the first audit on 113 of the 120 pairs (Cohen's kappa = 0.75), and flag 11 of the 30 highest-Elo pairs against 5 of the remaining 90 (new Appendix C.4). The audit also does not produce the labels, which come from the generator and the automated checks. It affects the release only through the 17 pairs it removes and the top-Elo trim it motivates. Without both, the judge ranking agrees with the published one at Spearman rho >= 0.997 (Appendix A.10).

W4. Judge ranking under a different generator

W4: same model used for judging and being judged -- the ablation study is only 150 pairs

The paper tests this by repeating generation with opus-4.7 in place of gpt-5.5 (Appendix A.3). The shifts concentrate on the two generator models. The ranking of the other 19 judges agrees across the two generators at Spearman rho = 0.979, and gemini-3.1-pro is first under both. On the size of this study, the rerun has 107 informative pairs, each judged by all 21 judges. Rankings are stable at this scale. With 10% and 20% of the published judgments (about 65 and 130 per judge), the ranking agrees with the full one at rho = 0.92 and 0.97 (Section 4).

W5, Q5. Pair difficulty and the judge pool

W5: difficulty is relative but presented as absolute -- how difficult are difficult cases for humans

Q5: Is the difficulty checked agains real life task difficulty?

Pair difficulty in the paper is difficulty for the fitted judge pool, as Section 3.3 states ("Ratings are relative to the fitted population"). The revision repeats this where difficulty is first used for curation in Section 3.4. This is the quantity that curation needs, since its purpose is to keep pairs that separate current judges. Difficulty for humans is a different quantity, and the paper makes no claim about it. For the relation to real use, we compare at the level of judges. Judge Elo agrees with the LMArena leaderboard, which is fit only on human preference votes from real user conversations, at Spearman rho = 0.82 over 20 judges (new Appendix A.14).

W6, Q6. Top-Elo trim and failure-type distribution

W6: post hoc choices for restricting the data -- why wasn't the 5% taken from each category and not in total

Q6: Why is 5% removed, and does the distribution remain stable over all flaw types after this removal?

The trim is meant to remove label noise, and two results place this noise in the pairs with the highest Elo over the whole pool (Section 3.4). In the human audit, 12 of the 30 highest-Elo audited pairs are flagged, against 5 of the other 90. In the fine-tuning study, training on the top Elo band gives the lowest held-out accuracy. A 5% cut within each type would not serve this purpose. It would remove pairs from the easier types whose Elo is well below this range, and it would keep part of the high-Elo tail in the harder types. The value of 5% is a tradeoff between removing label noise and keeping hard pairs with correct labels. After the trim, all seven types remain, and the share of each type changes by at most 2.6 percentage points (Table 5, Appendix A.4).

W7. Wording and the Bradley-Terry reference

W7: writing needs clarification -- for example phrases like "We therefore generate the branches separately while staging comparable conversational pressures in both." (what are conversational pressures?) or “Confirming that the Elo ranking is robust under residual noise” (what is residual noise) or BT model is introduced without a reference or an explanation (which only follows )

Thank you for pointing these out. By conversational pressure we mean a situation where the target failure could surface, such as an out-of-scope question for fabricated_answer. Both conversations of a pair contain one. By residual noise we mean the label errors that remain after the automated checks. The revision replaces both phrases with these descriptions. It also adds the reference and a one-sentence description of the Bradley-Terry model at its first mention in the introduction.

The reviewer appreciates the clarifications the authors made regarding number of utterances per conversation, and the details about annotators. However, the main concerns are still not fully addressed.

REDACTED BY THE REVIEWER DUE TO THE ERROR IN POSTING.

Possibly the main issue with the paper is the use of single model and author verification for taxonomy extrction,. The cross-model check (App. B.3) is one sentence: a same-family model, 100 dialogues from one dataset, judged by the authors. Please report the two category sets side by side, quantify overlap, and add a model from another family) The “no flaw outside the taxonomy” finding comes from an audit by the authors, with the seven types visible, on benchmark conversations generated from those types. This tests consistency, not coverage. Please validate on real, unsynthesized failures with open-ended labeling by external annotators. The LMArena correlation (ρ = 0.82, CI [0.55, 0.96]) reflects general model quality and doesn’t speak to taxonomy validity. On prompts, only opus-4.7 was re-prompted. Please report rank stability for all 21 judges across several prompt variants.

Reviewer_dTsFOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors focus on a current blindspot of LLM judge evaluation, multi-turn benchmarking, i.e. judge evaluations on interactive generations. The paper introduces RankJudge, a synthetic generation process for multi-turn judge benchmarks that depend on a reference document and current instantiation of the generator on 21 LLM judges and a wide variety of conversation pairs, spanning different failure types specific to multi-turn generations and user behavioral archetypes. The RankJudge generation process is presented as highly scalable due to its fully synthetic nature, and uses multiple dimensions of automated checks to ensure accuracy/quality without human-in-the-loop. Finally, the paper presents results from its instantiation of RankJudge to validate its utility with respect to discriminative judge evaluation.

Critical strengths
  • Setting and problem identification is extremely strong, on two levels. The paper identifies fairly significant issues with current judge benchmarks — largely single-turn and static. The paper presents a reasonable taxonomy to failures that are specific to multi-turn settings (which are common in many settings) and presents a benchmark generator process that is thoroughly thought out as opposed to a static benchmark and thus can be re-used when (somewhat inevitably), the instantiation presented in the paper becomes, perhaps, ecologically irrelevant.
  • The current framework as it stands seems very flexible and scalable, both in terms of change in taxonomies of failure modes, different user behavior types, and the fact that the current build does not require human-in-the-loop or human labeling in order to generate a benchmark, outside of identification of failure types. While the accuracy and utility of solely automated and synthetic generation can be debated, the framework is indeed highly scalable and cheap to create benchmarks. Additionally the framework appears to be usable for pair-wise or rank-based preferences and seemingly could easily be extended to work with other preference schema, as stated in the paper.
  • Decomposition and interaction in the judge-versus-problem estimation seems reasonable and well-grounded in current literature on evaluation in the IRT vein. To some extent, this ensures an understanding of coverage of problem difficulty, given that the fit procedure is reasonable, and allows the benchmark process to be dynamic
  • Conversation pair construction and taxonomy is well thought-out and clear, also human verification for the current instantiation is extremely useful
  • The paper is extremely thorough in terms of examining assumptions, and running ablative experiments
Critical weaknesses

In my estimation, most of the weaknesses are very minor. Nonetheless:

  • The human audits for the conversation pairs is extremely useful. A very minor weakness is that I also think some additional human labeling is missing just for this initial instantiation (though this feels like a minor weakness), including
    • Human preference data across conversation pairs to validate the constructed failure taxonomy and utility
    • Human identification of any failure modes of the generated conversation pairs outside of the failure taxonomy
  • The paper does not offer too much in the way of analysis for when the IRT decomposition / model may not be well-suited to the judge-problem decomposition and what that means for the benchmark evaluation slice selection beyond the random-walk alternative (this is great!). Primarily, the paper claims that the problem difficulty estimation can guide benchmark renewal and understanding of evaluation coverage, but does not discuss how or when these estimations could be misleading, confounded or collapse. Additionally, as problem difficulty was found to relate to ambiguous / noisy conversation pairs, the authors opted to remove the top 5% of pairs by Elo. However, they do not mention as a limitation that such a choice could eliminate truly difficult (not noisy) pairs that are more discriminative than other pairs for judge evaluation, nor give some analysis of how to determine whether the tail represents noise or a true measure of difficulty without human intervention. This seems like a limitation to be discussed at least a little bit but does not need to necessarily be mathematically analyzed.
  • The constraints on each conversation pair only representing a single failure mode or even multiple identifiable failure modes is useful in the sense that it helps with synthetic data generation and automated checking. However, it doesn’t necessarily represent a comprehensive reflection of multi-turn judge evaluation. This is largely an artifact of the fully automated process requiring that the pairs are interpretable according to (better, worse), and is simply a tradeoff.
Questions for authors

No additional questions beyond weaknesses.

1 comment

We thank the reviewer for the careful reading and the supportive assessment, and appreciate the recognition of the problem setting, the judge-versus-problem decomposition, and the ablations. We address the three points below. The corresponding additions are marked in blue in the revised PDF.

W1. Agreement with human preference and coverage of the taxonomy. On human preference, the labels in RankJudge are deliberately not preference labels. Each worse conversation contains one failure that is fixed in advance and verified against the reference document and the conversation, so which conversation is worse is a matter of fact, and the audit checks it as such. To relate this to human preference and utility, the revision compares judge Elo with the LMArena leaderboard, which is fit only on human preference votes. The two rankings agree at Spearman rho = 0.82 over 20 judges (new Appendix A.14), so judges that detect these failures well tend to be the models that human users prefer. The rankings are also not identical. Among these judges, opus-4.7 ranks first on LMArena and 11th on RankJudge, so the benchmark also captures behavior specific to grounded multi-turn judging that a preference leaderboard does not reveal.

On failures outside the taxonomy, the seven types start from failures documented in prior studies of multi-turn assistants. We refined them on MT-Bench and MT-Bench 101 over several rounds. In each round, a model labeled these conversations with the current taxonomy and listed the cases that did not fit (prompt in Appendix B.3), and the authors merged or dropped categories. We stopped when no new category emerged. For the generated pairs, the audit annotators also checked both conversations for any flaw other than the labeled one. Every flaw they named is one of the seven types (now stated in Section 3.4).

W2. Top-Elo trim and ranking stability. We agree, and the revision adds this discussion to Section 3.4. Pair difficulty is estimated from judge responses, so a pair with a wrong label is also scored as hard. Separating valid hard pairs from mislabeled ones without human input is an open problem that is not specific to our model. Recent methods that flag benchmark items from response patterns still rely on human review to confirm them [1, 2]. The estimates are also relative to the judge pool, and pairs that all judges answer the same way carry no information, so we remove them before fitting.

The purpose of the top-Elo trim is the label quality of the released pairs. The audit covers 120 pairs and the pairs it flags concentrate at the top of the Elo range, so the trim removes that range among the pairs that were not audited. The cost is the valid hard pairs that it may also remove, and this cost is small for judge evaluation. Without the trim and the audit exclusions, the judge ranking agrees with the published one at Spearman rho >= 0.997, with the top 10 preserved (Appendix A.10).

For a check without human input, the fine-tuning study in Section 3.4 tests a difficulty band as a whole. If the top band held only valid hard pairs, training on it would teach the same decision rule as the lower bands. Instead it gives the lowest accuracy on clean pairs, which indicates wrong labels in that band.

W3. One verified failure per pair. We agree that this is a tradeoff, and it is a deliberate design choice. One verified failure per pair gives a label with a known turn and type. The automated checks can then verify the label, and each judge error can be attributed to the verdict, the turn, or the type. Within this constraint, the pairs still vary over seven failure types, seven user behaviors, and three domains. Pairs with several failures are a natural extension of the generator.

References

[1] Truong et al. Fantastic Bugs and Where to Find Them in AI Benchmarks. NeurIPS 2025 Datasets and Benchmarks Track.

[2] Land and Bikel. Auditing LLM Benchmarks with Item Response Theory. arXiv:2605.30504, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.