acceptodds
Withdrawn from ICLR 2027

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (n=500 sentences per category), as well as a binary accuracy classification of question-answer pairs. For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are not impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, meta-evaluation, benchmarking, prompt engineering, human annotation, replication

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jLGyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper systematically examines how design choices, including the evaluator model, prompt detail, rating scale, and reasoning effort, affect LLM-as-a-judge evaluations. It finds that LLM judges are generally reliable, but not interchangeable: judges with similar accuracy can differ substantially in bias, leniency, and item-level decisions. Model identity is the largest source of variation, while sensitivity to design choices increases for subjective or ambiguous tasks. The paper therefore recommends evaluating metrics beyond accuracy and combining multiple judge models to improve robustness and reproducibility.

Critical strengths

The paper conducts a well-controlled comparison across 10 judge models, multiple prompt designs, rating scales, and reasoning-effort settings, allowing the effect of each design choice to be examined separately.

It evaluates judges against human annotations and reports bias, leniency, and decision-switch rates in addition to accuracy, revealing important behavioral differences that conventional accuracy metrics would miss.

Critical weaknesses

The paper provides a useful and carefully controlled synthesis, but its novelty is limited because most headline findings (model-dependent variability, prompt and scale sensitivity, reduced robustness on subjective tasks, and the benefits of judge aggregation) have already been reported separately. The main contribution is therefore the unified factorial comparison rather than a fundamentally new methodological or conceptual insight.

The experiments cover only sentiment, toxicity, and question-answer correctness, so it is unclear whether the findings generalize to common LLM-judge tasks such as dialogue, summarization, coding, and long-form generation.

The rating sentences are synthetically generated by Grok, while the evaluated answers are generated by Gemini. These data may contain generator-specific patterns and may not represent real-world model outputs.

Because the classification answers come from a Gemini model, the unusually high leniency of the Gemini judge may reflect same-provider preference. This makes it difficult to separate genuine model differences from answer-source effects.

The paper recommends cheaper models, shorter prompts, and lower reasoning effort, but it does not directly analyze cost, latency, or robustness across model updates.

Questions for authors

How do the authors control for same-provider preference, since the evaluated answers are generated by Gemini and Gemini is also included as a judge?

Were judgments repeated across multiple sampling runs or random seeds? If not, how can design effects be distinguished from stochastic model variation?

Given the substantial overlap with prior work on prompt sensitivity, grading scales, judge leniency, and model aggregation, what result do the authors consider uniquely established by this study?

Reviewer_Am16OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how LLM-as-a-Judge evaluations change with the choice of model, prompt detail, rating scale, and reasoning effort. It considers sentiment and toxicity rating, as well as reference-based correctness classification. Beyond aggregate performance, the paper examines systematic over- or under-rating, the composition of classification errors, and verdict changes across configurations.

Critical strengths
  1. The paper compares a wide range of judge configurations, varying model choice, prompt design, rating scales, and reasoning effort. This provides a useful overview of how different design choices affect evaluation results.

  2. For the sentiment and toxicity tasks, the authors collect multiple human ratings per item, allowing them to compare LLM judgments against human evaluations rather than relying solely on automated labels.

Critical weaknesses
  1. Prior studies have already examined how prompt detail and rating scales affect judge consistency [1], how prompt variations affect judgments across models and tasks [2], and how judge reliability varies across models and evaluation tasks [3]. While this paper extends these analyses to additional configurations, it is unclear which findings provide substantially new insights beyond these existing results.

  2. The experiments focus on sentiment, toxicity, and accuracy classification. It is unclear whether the findings generalize to other common LLM-as-a-Judge tasks, such as pairwise preference evaluation or open-ended response quality assessment.


[1] Lee, Noah, Jiwoo Hong, and James Thorne. "Evaluating the consistency of LLM evaluators." Proceedings of the 31st International Conference on Computational Linguistics. 2025.

[2] Bhat, Savita, and Vasudeva Varma. "All Prompts Are Created Equal? Evaluating Robustness of LLM Judges Against Non-Adversarial Prompt Variations." Findings of the Association for Computational Linguistics: ACL 2026. 2026.

[3] Bavaresco, Anna, et al. "Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2025.

Questions for authors
  1. What are the main findings or practical insights that distinguish this work from previous studies [1–3] (references in weakness section) ?

  2. Why were sentiment, toxicity, and accuracy classification selected as the evaluation tasks? What evidence supports the applicability of the paper’s practical recommendations to other LLM-as-a-Judge settings, such as pairwise preference evaluation?

Reviewer_dfU4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how judge model, prompt specificity, rating granularity, and reasoning budget shape LLM-as-a-judge behavior. Ten models are evaluated across 120 settings per numeric-scoring tasks (sentiment and toxicity) varying model, prompt specificity, and rating granularity over 500 evaluation sentences per task with separately gathered human scorings. A further 60 settings vary model, prompt specificity, and reasoning effort to assess 600 HLE-derived question–answer pairs. Judges are given reference answers, while the classification labels used for assessment are produced through judge consensus and manual arbitration. The authors report generally small summarized scoring bias and high classification correctness, alongside setting-dependent shifts in bias, judge orderings, and error direction. They recommend looking beyond correctness metrics and combining several judges. The intended contribution is a systematic empirical side-by-side comparison to inform the design of LLM-based evaluation.

Critical strengths
  1. Broad and practically relevant empirical comparison. The study systematically varies evaluator models, prompts, and rating scales, providing useful descriptive evidence of sensitivity to evaluation design. The centered-scale toxicity results and configuration-dependent judge rankings illustrate the limitations of relying on a single configuration. Reporting error direction and verdict-switch rates alongside accuracy also reveals differences that aggregate performance can conceal. The label-free verdict-switch rate remains a useful sensitivity measure even without independent correctness labels. 2. Independently annotated scalar-rating benchmarks. The sentiment and toxicity tasks include ten and fifteen human ratings per item, respectively, with reported inter-annotator agreement and separate support and evaluation splits. These annotations, together with the detailed experimental results, could provide a useful resource for subsequent research. 3. Attention to effect magnitude and practical guidance. The authors report effect sizes and score differences in interpretable units instead of relying only on significance tests, and they examine ensembling the judges’ outputs as a practical mitigation. The study also tests a practically relevant factor, reasoning effort. These elements make the work easier to interpret and apply, although the claims built on them need stronger support, as noted below.
Critical weaknesses
  1. The classification benchmark does not independently establish correctness. Section 3.3.2 derives 553 of the 600 reference labels from the same 60 judge configurations being evaluated: 429 items receive unanimous judgments, and 124 receive at least 90% agreement. Under this procedure, pooled accuracy is at least 90.1% even if every judgment on the remaining 47 items is incorrect, so the reported 96.5% largely reflects agreement with a reference built from the evaluated systems themselves. Table 13 reports 64.4% accuracy on the 47 manually resolved items. However, the supplied manuscript does not describe the adjudicators, decision criteria, or disagreement-resolution procedure. Please provide these details or identify where they are documented in the supplementary materials. Because these items were selected for disagreement among judges, their accuracy cannot be extrapolated to the full benchmark. Neither figure independently establishes correctness across the full benchmark. Because all judges could share systematic errors, unanimity does not guarantee correctness. Although Section 5.1 acknowledges the consensus-label limitation, the central reliability claim remains stronger than the independent validation supports. Independent human validation of a random sample of the agreement-labeled items is needed to support the central reliability claim.

  2. Several headline results rest on aggregate bias rather than per-item accuracy. The 0.11-point deviation is the mean absolute value of each configuration’s signed bias, whereas Table 8 reports per-item MAE of 0.37 for sentiment and 0.61 for toxicity. The paper explicitly distinguishes these metrics in Section 3.3.1, but the abstract’s reliability claim emphasizes aggregate bias, which does not establish low per-item error. A leave-one-annotator-out baseline (each annotator versus the mean of the remaining raters) would help interpret these errors. The abstract’s claim that toxicity judges “even outperform standard classifiers” relies on comparing human toxicity ratings with classifier probabilities rescaled onto a 1–7 scale. Rescaling alone does not establish that these outputs measure the same target. The paper notes that the Base prompt, which has no examples, also outperforms the classifiers, but this does not resolve target comparability. A suitable score mapping should be justified and validated on held-out items. The separate support split could help develop that mapping, although its selection of high-agreement, anchor-like examples should be considered. Rank correlation with human means would provide complementary evidence, but would not by itself establish absolute agreement or equivalent targets. Finally, the ensemble bias reductions in Appendix D.3 may reflect cancellation of positive and negative errors; per-item MAE or agreement results for the ensembles are needed to establish improved judgments.

  3. Leniency should be distinguished from acceptance behavior and class-conditional error rates. The statistic FP/(FP+FN) describes the composition of errors, not the overall acceptance rate. However, Table 14 does support reduced acceptance under the Long configuration: relative to Base, True→False switches occur on 2.5% of judgments and False→True switches on 0.6%. Thus, stricter rejection behavior is supported for this configuration comparison, although it should not be inferred from leniency alone or attributed solely to prompt detail. Please report confusion matrices, class-conditional false-positive and false-negative rates, and overall acceptance rates directly to make these trade-offs easier to interpret. Error rates remain relative to the partly consensus-derived labels, whereas acceptance rates do not require correctness labels.

  4. Two reported figures appear inconsistent with the stated definitions. First, Table 1 defines Direction as the fraction of design pairs whose bias sign reverses, but reports 83% for model changes. Table 12 indicates n = 540 model pairs, consistent with twelve prompt–scale settings containing 45 unordered model pairs each. Within each setting, at most 25 of 45 pairs (55.6%) can have strictly opposite bias signs, so 83% cannot arise under this all-pairs definition. Second, Tables 13–14 report 386 short-prompt and 475 long-prompt errors over 12,000 judgments each, implying an accuracy difference of about 0.74 percentage points, which exceeds the 0.6-point maximum prompt effect in Table 17. Different denominators or aggregation rules may explain these discrepancies, but the stated definitions do not make this clear, and the authors should clarify.

  5. The comparisons do not isolate prompt detail or scale granularity as individual mechanisms, so the recommendations need narrower scope. Section 3.2 describes prompt variants that jointly change instructions, examples, explicit grading principles, and reasoning procedures. The Long classification prompt explicitly requires checking reasoning validity and rejecting responses containing a critical error (Appendix C, Table 6). This requirement is a plausible contributor to its lower acceptance rate. However, the complete Base and Short classification prompts are not printed in the supplied manuscript, so it is unclear whether they impose equivalent requirements. The tested scales also differ in numerical range and output precision. Lower MAE for finer scales may partly reflect reduced quantization error against averaged human ratings. A common-grid analysis, such as rounding rescaled fine-scale outputs to the coarse grid before recalculating MAE, would help assess this possibility; no such analysis is reported in the supplied manuscript. These results support comparisons of the tested configurations, but broader recommendations about minimal prompt detail or scale granularity require additional controls or narrower wording.

  6. The incremental contribution is insufficiently distinguished from close prior work. Section 2 motivates the study partly by stating that judge-design choices are rarely varied and compared within a single study, but omits relevant predecessors. Lee, Hong, and Thorne (COLING 2025; 2024 preprint, Section 2.2) jointly vary five models, three levels of criterion detail, and four scoring formats. Salinas, Swelam, and Hutter (ICML 2025) systematically analyze and tune judge-design choices, including accuracy–cost trade-offs. Murugadoss et al. (2024 preprint, Table 1) compare correlations with human annotations across prompt setups; their reported values also exhibit descriptive reversals in relative model performance, although these do not by themselves establish statistically significant ranking changes.

These studies address different tasks and outcomes and do not duplicate the submission. Nevertheless, these studies establish that joint investigation of judge-design choices is already present in the literature. The new human annotations, scalar-task comparisons, and reasoning-effort experiments may constitute a useful empirical extension. The paper should explicitly distinguish these contributions from the closest precedents and identify which new findings are supported by controlled comparisons and uncertainty estimates.

Close prior work not found in the supplied bibliography:

Questions for authors
  1. Independent verification of classification labels. Were any of the 553 agreement-labeled items confirmed independently of the judges under evaluation? If so, please describe the selection procedure, scoring rubric, adjudicators’ expertise, blinding, and estimated mistake rate with uncertainty. How were the 47 ambiguous items adjudicated? Section 3.3.2 describes the supplied reference answers as verified; please explain the verification procedure, including who performed it and how questionable answers were handled. Existing separate verification could materially change the assessment of the classification results. 2. Interpretation of scalar deviations and comparative performance. Can the existing labels support a leave-one-annotator-out baseline for MAE and bias, with uncertainty estimates?Does the toxicity-classifier advantage persist after validating a suitable score mapping using separate support data and held-out evaluation items? Please also provide complementary rank-correlation results, while distinguishing ranking performance from absolute score agreement. Do the ensembles improve item-level MAE or agreement as well as summarized bias? Please also provide confusion matrices or class-conditional error rates to clarify what the reported changes in leniency entail. 3. Attribution and reach of design impacts. Do the classification prompts enforce equivalent requirements for final-answer correctness, reasoning validity, and completeness? Please also clarify the setups and measures for which the recommendation to use minimal prompts is meant to apply. 4. Reporting consistency. How is Direction in Table 1 computed? An 83% sign-reversal rate for model changes appears incompatible with 45 model pairs per prompt–scale setting (Table 12, n = 540). Please also reconcile the roughly 0.74-point Short-versus-Long accuracy gap implied by Tables 13–14 with the 0.6-point maximum prompt effect in Table 17. 5. Variability and statistical treatment. Were judge configurations run repeatedly, and what decoding settings were used? The reported temperature of 0.7 concerns candidate-answer generation rather than judging. The small point estimates and nonsignificant tests for reasoning effort are suggestive, but do not establish practical equivalence. Please provide confidence intervals accounting for paired judgments on the same items, justify a practically negligible difference, and assess equivalence through an appropriate interval-based analysis or equivalence test.

Comments on the paper

1 comment
Area_Chair_Z61rOpenReview ↗

Discussion Period - No Author Response Received

Dear Authors,

We have not received a response from you during this discussion period. Please respond soon to facilitate the discussion of the paper.

Best Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.