acceptodds
Under review as a conference paper at ICLR 2027

Prompt Lottery in LLM-As-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing

Abstract

LLM-as-a-Judge (LLMaJ) has become a standard paradigm for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument. Prior work has primarily explored deliberate prompt modifications such as scoring rubrics, role descriptions, or chain-of-thought reasoning. In contrast, we investigate whether validated prompt variants intended to preserve evaluation semantics while differing in surface realisation (paraphrastic rephrasing) yield consistent judgments under a fixed evaluation task. We introduce Prompt Lottery, a phenomenon in which prompt reformulations intended to preserve semantic content produce different LLMaJ judgments. Across a large-scale study spanning diverse evaluation tasks, judge families, and model backbones, we find that paraphrastic variation can alter evaluation outcomes and, in some settings, reverse relative model rankings. We conducted a user study and show that variability of LLMaJ exceeds the observed variability in out study, as well as generally accepted variability in human subject studies raising concerns about the stability of single-prompt evaluation. Finally, we show that multi-prompt aggregation substantially improves robustness by emphasising judgments that remain consistent across prompt variants.

Primary area:
datasets and benchmarks
Keywords:
llm as a judge, evaluation, human study, prompt sensitivit

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GLe4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies “Prompt Lottery”: variation in LLM-as-a-Judge evaluations under prompt rephrasings intended to preserve task meaning. Across 12 datasets, six judge families, and five backbone models, it examines judgment consistency, agreement with human ratings, and changes in evaluated models’ rankings. Human validation and annotation studies provide additional context for interpreting the observed sensitivity. The paper also evaluates multi-prompt aggregation and proposes a classifier-based gating strategy to selectively invoke additional judgments, aiming to improve robustness while reducing calls relative to full aggregation.

Critical strengths

A practically consequential evaluation problem. The paper connects sensitivity to judge-prompt wording with changes in evaluated models’ rankings, moving beyond differences in individual scores. This makes the study relevant to the reliability of comparative model evaluations and supports its value as an empirical contribution.

Broad empirical coverage with complementary validation. The study spans 12 datasets, six judge families, and five backbone models, and includes human annotation studies and fixed-prompt repeatability controls. Its task-dependent findings—including settings where rankings remain stable—provide a more informative picture than isolated failure examples. This breadth is a central reason for my positive assessment.

Critical weaknesses

The experiments do not fully isolate wording sensitivity. Some prompt variants change the evaluation criteria or provide additional reference information. Consequently, the observed disagreement may partly reflect substantive differences in the evaluation setup rather than sensitivity to equivalent rephrasing. This weakens the causal interpretation of the central findings.

Questions for authors

Clarification and supporting analysis: Some variants in Appendix H appear to change more than wording: G5 emphasizes different evaluation dimensions across variants, and G11/G12 P4 includes reference information absent from other variants. Were these differences present in the prompts actually used for the reported experiments? If so, do the main consistency and ranking findings remain when comparisons are restricted to variants with identical evaluation criteria, reference information, and scoring rules? This would materially affect my confidence that the observed effects can be attributed to paraphrastic rephrasing.

1 comment

We wish to thank the reviewer for their constructive review. We truly appreciate to see a review which has a main goal to improve the paper it is reviewing. The proposed improvements are spot-on, and we present additional information including some of our ongoing work with the same goal as the ones the reviewer suggested.

We agree that some variants differ in more than surface wording, and we will describe them accurately as task-equivalent prompt variants. This reflects how evaluation prompts vary in practice: there is no canonical way to write a rubric, and evaluators routinely change criteria, score definitions and emphasis. Our goal in this paper was to establish whether an LLM judge is a stable evaluation instrument when its prompt has not been specifically optimised or checked for task equivalence, which is the situation of most practitioners. The human study supports the equivalence of the variants: annotators given the same variants remained consistent (Fleiss’ κ ≈ 0.80), while LLM judges did not. We pursued this before exploring which individual factors influence stability, as the reviewer suggests. Isolating the contribution of individual factors, such as score definitions, emphasized dimensions or reference information, is a distinct question that we consider out of scope here and are pursuing in follow-up work. As a robustness check, we additionally report our main consistency and ranking results restricted to variants with identical criteria, reference information and scoring rules, as well as without P4 in the SummEval groups.

We present the reviwer with our initial research on evaluation prompt factors influence on stability, but consider this a sparate stand-alone work. We use the prompts used in this paper where they included a Likert scale in the original (default) prompt Recent work shows that rubric text alone, without the evaluated response, predicts LLM-judge outputs better than chance (Bagaria et al., 2026 (https://arxiv.org/abs/2609.02942)), so rubric choice is not neutral and no single rubric can be treated as universal; we do not attempt to isolate or optimize these factors, but to show that a single, unoptimized, varied rubric, which is how researchers and practitioners typically work, is an unstable evaluation instrument.

Celln fullΔκ fulln matchedΔκ matched
hotpot_qa__correctness150−0.12195−0.152
hotpot_qa__groundedness150−0.06388−0.176
nq_open__correctness150−0.06539−0.016
nq_open__groundedness150+0.06243+0.124
squad_v2__correctness150−0.082102−0.078

This shows that the delta \kappa is smaller than what we observed with fully varied prompt (which may be the effect of discriminatory power of rubrics themselves as described in the above cited work by Bagaria et all). This is still research in early

Reviewer_hGr1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the “Prompt Lottery”: changes in LLM judgments under evaluation prompts intended to be semantically equivalent. Across 825 experimental conditions and a human annotation study, it reports substantial judgment instability and task-dependent model-ranking reversals. It evaluates multi-prompt aggregation and proposes a DeBERTa-based gate that predicts when additional judgments are needed to reduce evaluation costs.

Critical strengths
  1. Addresses an important reliability problem in LLM-based evaluation.
  2. Covers diverse datasets, judge families, scoring scales, and backbone models.
  3. Includes repeated-prompt controls and a human study to contextualize observed instability.
  4. Provides verbatim prompt templates, enabling inspection of the experimental manipulations.
  5. Explores a practically motivated approach to reducing the cost of multi-prompt evaluation.
Critical weaknesses
  1. Prompt variations introduce substantive confounds. Some variants change evaluation criteria, score anchors, or available reference information. For example, MT-Bench variants specify different quality dimensions, while SummEval G11–G13 introduce a reference summary only in P4. These differences weaken attribution of disagreement specifically to syntactic rephrasing.

  2. The human-relative degradation claim is insufficiently supported. The cited Benikova and Wiebe studies do not establish the attributed paraphrase-induced decreases in agreement. The manuscript’s own fixed-instruction human baseline is also insufficiently specified. Consequently, the headline 6–9× comparison requires stronger evidence.

  3. The cost derivation does not establish population-level savings. Equation (2) treats conditional TNR as the overall fraction skipping aggregation. The correct expectation also requires class prevalence, false-negative frequency, and aggregation costs conditional on routing. The reported approximately two-call average and 60% saving therefore need empirical verification.

  4. Aggregation quality and coverage are not adequately separated. Many non-binary examples lack consensus, but Table 5 does not clearly report retained coverage or matched-subset comparisons. Table 7’s recovery and no-consensus metrics also lack sufficiently clear definitions. Recovering ensemble decisions does not itself establish better human alignment or ranking reliability.

  5. The contribution needs clearer positioning against prior work. Mizrahi et al. (2024) study validated paraphrases and ranking instability, rather than merely formatting changes. Bhat and Varma (2026) already investigate judge paraphrases, a preliminary human baseline, and prompt aggregation. JudgeSense v3 directly compares within-judge paraphrase agreement with repeated-prompt agreement.

  6. Important conclusions lack decisive controls or uncertainty analysis. Missing budget-matched same-prompt repetition prevents isolating the benefit of prompt diversity. Ranking reversals require score margins, confidence intervals, and ranking-level repetition controls. Additionally, Figure 2’s labeled axes appear inconsistent with Section 5.1’s claim favoring inter-prompt aggregation.

Questions for authors
  1. Can you reproduce the main findings using prompts with identical criteria, input information, scale anchors, and output requirements, varying only wording?

  2. What matched fixed-instruction human baseline supports the reported degradation? Which passages in the cited studies support the claimed human paraphrasing effects?

  3. What are the empirical held-out call counts, class prevalences, and routing-conditional costs? Do these support the claimed overall savings?

  4. How are no-consensus examples handled in Tables 5 and 7? What are the exact denominators, retained coverage, and evaluation targets?

  5. Does paraphrase aggregation outperform repeated-identical-prompt judging at matched cost and coverage, measured against human judgments and ranking stability?

  6. Can you report uncertainty in system-score differences and winner identity, reconcile Figure 2 with its interpretation, and clarify the distinct contributions beyond the closest prior studies?

7 comments

We thank the reviewer for detailed feedback. This feedback is on-point, and address the points we might have missed to clearly describe in our initial submission. We also feel that most of these can be addressed by adding details of the work we have already conducted. With the goal of providing enough details, we will answer the questions in separate comments.

Q*: Prompts concerns

A: We agree that isolating how individual prompt factors (for example, whether intermediate score levels are defined) contribute to the Prompt Lottery is valuable. It is not what this paper set out to do. Our aim is to establish that semantically equivalent prompts can yield different judgments and rankings. Annotators in our human validation study judged all prompt pairs, including the original, as requiring the same evaluation task (average equivalence 76/100, κ = 0.78, α = 0.81), despite the differences in how the prompts are described. We acknowledge that equivalence is established by human and model judgments, not by construction, and will revise the wording accordingly. MT-Bench is the task in which prompts emphasize different quality dimensions, yet it is the most stable task in our study (τ = 1.00 on DailyDialog and 0.80 on Persona, with 0% and 10% top-1 flips; Table 4), compared with τ = 0.18 on SQuAD v2 and 40% reversals on SummEval. If those differences were driving the instability, we would expect MT-Bench to be less stable, not more. The reference summary in P4 of the SummEval groups (G11–G13) is a reporting error left over from an earlier version of the experiments that included a reference. Removing P4 does not change our findings, and we will correct the prompts in the appendix and report the analysis without it.

We present the reviwer with our initial research on evaluation prompt factors influence on stability, but consider this a sparate stand-alone work. We use the prompts used in this paper where they included a Likert scale in the original (default) prompt Recent work shows that rubric text alone, without the evaluated response, predicts LLM-judge outputs better than chance (Bagaria et al., 2026 (https://arxiv.org/abs/2609.02942)), so rubric choice is not neutral and no single rubric can be treated as universal; we do not attempt to isolate or optimize these factors, but to show that a single, unoptimized, varied rubric, which is how researchers and practitioners typically work, is an unstable evaluation instrument.

Celln fullΔκ fulln matchedΔκ matched
hotpot_qa__correctness150−0.12195−0.152
hotpot_qa__groundedness150−0.06388−0.176
nq_open__correctness150−0.06539−0.016
nq_open__groundedness150+0.06243+0.124
squad_v2__correctness150−0.082102−0.078

This shows that the delta \kappa is smaller than what we observed with fully varied prompt (which may be the effect of discriminatory power of rubrics themselves as described in the above cited work by Bagaria et all).

AuthorsOpenReview ↗

Baseline concerns

Q: Baseline derivation concerns

A: We understand that in shortening our paper we have reduced the description of the degradation calculation up to a point of unclarity and made it very difficult to undersatnd. Let's try to make it a bit more verbose here, and we will work it back in the paper. First study Benikova et al. (2014) reports pairwise κ of 0.63 vs. 0.74 for an initial and a refined guideline version (Δκ = 0.11, on different but comparable annotator groups). Second study Wiebe et al. (2005) agreement 8 points higher (κ = 0.77) than an earlier, less detailed scheme (Bruce and Wiebe, 1999; https://dl.acm.org/doi/abs/10.1017/S1351324999002181), concluding that "adding detail to the annotation task can help annotators perform more reliably" (Δκ ~ 0.08) Both of these studies, even as they did not paraphrase, keep the same overall semantic task, it is reasonable to treat the instructions changes larger change than meaning-preserving paraphrasing. Thus we would expect that parphrasing instructions would yeild higher levels of disagreement. We used these two studies to set the context, and we treat them as indicative of scale rather than as a strict bound. Our main comparison is the user study we run on the same prompts as reported in the paper -- in this study the agreement drops by 0.07. To get to the 6-9x degradation we use the following calcuation. When LLM is run repeaditly on a small temperature (T ~0) we observe the agreement at 0.89. Obseved agreement when used paraphrased prompts is 0.24, which leads to a difference of 0.65. The table bellow compares this difference to the two studies from literature and our own user study:

SourceWhat changed (same annotation goal)Human ΔκLLM Δκ ÷ human Δκ
Benikova et al. (2014)Initial vs. refined guidelines (added examples, clarified rules)0.11 (0.63 → 0.74)0.65 / 0.11 ≈ 5.9×
Wiebe et al. (2005) vs. Bruce & Wiebe (1999)Coarse vs. detailed annotation scheme≈0.08 (≈0.69 → 0.77)0.65 / 0.08 ≈ 8.1×
Our user studyParaphrased prompts, same annotators0.07 (0.76 → 0.69)0.65 / 0.07 ≈ 9.3×

Q: Sources of overall savings

A: We thank the reviewer for pointing out the flaw in our reporting on savings assumption. Eq. 2 treats TNR as the fraction of all examples routed to a single call. TNR is conditional on stable examples, so the equation is exact only for a workload in which every example is stable (the default prompt is treated as correct). It omits class prevalence and the false-negative rate, and it prices aggregated cases at the average cost over all examples, which includes the non-converging ones. It therefore does not describe a realistic deployment, and the 60% figure should not be read as the savings on a mixed workload. The abstract and conclusion state the 1.99-call figure for stable instances, which matches this conditional reading. We recognize, however, that this qualifier is easy to miss, that connecting it to the derivation in Section 5.2.1 is not obvious, and that the summary table lists “~60%” without it.

We therefore recalculate expected cost under three explicit assumptions about which examples are stable. The corrected expectation is P(single)x1 + (1 − P(single)) x t_agg, where P(single) = p_0 x TNR + p_1x(1−TPR). It uses class prevalence, the false-negative rate (unstable examples that the gate misses and that receive a single call), and aggregation cost conditional on the routing decision. We use TNR = 69.42% and TPR = 85.06%, and report each setting with all aggregated cases at 5 calls and with the observed convergence split (plurality at three calls: 37.68%, at five: 21.05%, none: 41.27%):

Stable-only: every example is stable. This is the conditional case the original equation describes, with the aggregation cost corrected for stable examples. Observed over all prompts: an example is stable when a plurality exists (58.7%). This is the most favorable mixed case. Observed default-prompt stability: an example is stable when the default prompt matches the plurality (44%). This is the most realistic case.

Group & ScenarioAggregation Cost AssumptionP(single)P(\text{single})Expected CallsSavings vs. 5 CallsSavings vs. Early Stop (4.25)
Group 1: Stable-Only (\p0=1.0\p_0 = 1.0)
A. Published Eq. 2 specificationFixed mean =4.247= 4.247 calls0.6941.9960.1%53.1%
B. Conservative boundFixed upper =5.000= 5.000 calls0.6942.2255.6%47.7%
C. Conditional aggregationStable-specific split (3.30–4.00 calls)0.6941.70–1.9261.7–66.0%54.8–59.9%
Group 2: Plurality Exists (\p0=58.7%\p_0 = 58.7\%)
D. Conservative boundFixed upper =5.000= 5.000 calls0.4693.1237.5%26.5%
E. Observed convergence splitUnstable =5.0= 5.0; stable plurality split0.4692.8942.2%31.9%
Group 3: Default prompt plurality observation (\p0=44.0%\p_0 = 44.0\%)
F. Conservative boundFixed upper =5.000= 5.000 calls0.3893.4431.1%19.0%
G. Observed convergence splitUnmeasured joint range (conservative)0.3893.05–3.2135.8–39.0%24.4–28.2%

On stable examples the gate reduces calls by about 56–66%. On a mixed non-binary workload with the observed 44% default-prompt agreement, the expected reduction is about 31–39% against the five-prompt baseline, and about 24–28% against full aggregation with early stopping (4.247 calls). The mixed-workload cases are more informative for a practitioner, because they account for unstable examples and the false-negative rate, so we will report them as the main results and present the stable-only figure as a conditional best case. Empirical verification. All inputs above are measured on our data, but the expected cost is computed from marginal statistics. Changes to the paper we propose are to replace Eq. 2 with the corrected derivation and add the table with different scenarions, update the abstract and conclusions to report the mixed-workload figures, state the stable-only result as a conditional bound.

Reviewer_MBupOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how paraphrased evaluation prompts affect LLM-as-a-Judge reliability. Across 825 conditions covering 12 datasets, six judge families, and five backbone models, it reports substantial disagreement between prompt variants and task-dependent reversals in model rankings. A study with 10 human annotators provides a comparison with human robustness. Multi-prompt aggregation improves reported human alignment, while a learned gate aims to reduce its inference cost.

Critical strengths
  1. Coverage across multiple tasks, scoring scales, and model families makes prompt sensitivity difficult to dismiss as an isolated benchmark artifact. The fixed-prompt repetition baseline also helps establish that rephrasing introduces additional instability.
  2. Measuring downstream ranking changes makes the findings consequential for evaluation practice. The contrast between unstable SQuAD v2 rankings and stable DailyDialog rankings usefully qualifies the broader claim.
Critical weaknesses
  1. The claimed inference savings rely on an incorrect weighting in Equation 2. TNR is conditional on truly stable examples; it isn't the overall fraction routed to single-prompt evaluation. Expected cost requires class prevalence, both error rates, and aggregation cost conditional on routing. Consequently, the approximately two-call estimate and 60% savings aren't established by this derivation.
  2. Table 5’s strongest aggregation method discards examples without consensus. Its correlation therefore appears to be measured on a selected subset, whereas averaging retains all examples. Without coverage and comparisons on identical subsets, improved alignment cannot be separated from excluding difficult cases. Table 7’s “worst-case” percentages also lack a clear evaluation target.
  3. Some variants change rubric specificity rather than wording alone. For example, G1-P2 defines intermediate scores that G1-P1 leaves unspecified. Such changes can alter scoring behavior even when annotators recognize the same general task, weakening the claim that the measured differences isolate syntactic rephrasing.
  4. The headline human-versus-model degradation comparison lacks a clearly reported matched human baseline under unchanged instructions. Cross-annotator disagreement and within-model prompt sensitivity aren't directly interchangeable, so the reported 6–9× difference needs a more explicit, comparable calculation.
Questions for authors
  1. After correcting Equation 2, what are the measured average call count and decision agreement on the held-out test set?
  2. What fraction of examples does each Table 5 method retain, and do its gains persist when all methods are evaluated on the same examples?
  3. Does substantial degradation remain when rubric content, including intermediate score definitions, is held fixed?
  4. How is the 6–9× human comparison calculated, and does it hold with matched examples and unchanged-instruction baselines for both humans and models?
5 comments
AuthorsOpenReview ↗

Clarification of baselines and numbers

We wish to start thanking the reviewers for their comments and careful reading of our paper. The comments presented are all in place, and rightfully point to the details we might have not explained clearly enough. We hope that the answers bellow provide enough details to present the valuable contributions of our work. To provide a detailed answer, we will break each concern in a separate comment

Q: EQ.concerns 2:

A: We thank the reviewer for pointing out the flaw in our stability assumption. teq. 2 treats TNR as the fraction of all examples routed to a single call. TNR is conditional on stable examples, so the equation is exact only for a workload in which every example is stable (the default prompt is treated as correct). It omits class prevalence and the false-negative rate, and it prices aggregated cases at the average cost over all examples, which includes the non-converging ones. It therefore does not describe a realistic deployment, and the approx. 60% figure should not be read as the savings on a mixed workload. The abstract and conclusion state the 1.99-call figure for stable instances, which matches this conditional reading. We recognize, however, that this qualifier is easy to miss, that connecting it to the derivation in Section 5.2.1 is not obvious, and that the summary table lists “~60%” without it.

We therefore recalculate expected cost under three explicit assumptions about which examples are stable. The corrected expectation is P(single)x1 + (1 − P(single)) x t_agg, where P(single) = p_0 x TNR + p_1x(1−TPR). It uses class prevalence, the false-negative rate (unstable examples that the gate misses and that receive a single call), and aggregation cost conditional on the routing decision. We use TNR = 69.42% and TPR = 85.06%, and report each setting with all aggregated cases at 5 calls and with the observed convergence split (plurality at three calls: 37.68%, at five: 21.05%, none: 41.27%):

Stable-only: every example is stable. This is the conditional case the original equation describes, with the aggregation cost corrected for stable examples. Observed over all prompts: an example is stable when a plurality exists (58.7%). This is the most favorable mixed case. Observed default-prompt stability: an example is stable when the default prompt matches the plurality (44%). This is the most realistic case.

Group & ScenarioAggregation Cost AssumptionP(single)P(\text{single})Expected CallsSavings vs. 5 CallsSavings vs. Early Stop (4.25)
Group 1: Stable-Only (\p0=1.0\p_0 = 1.0)
A. Published Eq. 2 specificationFixed mean =4.247= 4.247 calls0.6941.9960.1%53.1%
B. Conservative boundFixed upper =5.000= 5.000 calls0.6942.2255.6%47.7%
C. Conditional aggregationStable-specific split (3.30–4.00 calls)0.6941.70–1.9261.7–66.0%54.8–59.9%
Group 2: Plurality Exists (\p0=58.7%\p_0 = 58.7\%)
D. Conservative boundFixed upper =5.000= 5.000 calls0.4693.1237.5%26.5%
E. Observed convergence splitUnstable =5.0= 5.0; stable plurality split0.4692.8942.2%31.9%
Group 3: Default prompt plurality observation (\p0=44.0%\p_0 = 44.0\%)
F. Conservative boundFixed upper =5.000= 5.000 calls0.3893.4431.1%19.0%
G. Observed convergence splitUnmeasured joint range (conservative)0.3893.05–3.2135.8–39.0%24.4–28.2%

On stable examples the gate reduces calls by about 56–66%. On a mixed non-binary workload with the observed 44% default-prompt agreement, the expected reduction is about 31–39% against the five-prompt baseline, and about 24–28% against full aggregation with early stopping (4.247 calls). The mixed-workload cases are more informative for a practitioner, because they account for unstable examples and the false-negative rate, so we will report them as the main results and present the stable-only figure as a conditional best case. All inputs above are measured on our data, but the expected cost is computed from marginal statistics. Changes to the paper we propose are to replace Eq. 2 with the corrected derivation and add the table with different scenarios, update the abstract and conclusions to report the mixed-workload figures, state the stable-only result as a conditional bound.

AuthorsOpenReview ↗

T5 clarrification

--

Q: Table 5 subsetting clarification?

A: The table does present the evaluation on the same subset (only plurality reaching data points), and on second reading we understand this is not stated, and that introducing the plurality+average on a full dataset introduces further unclearity. The reason we introduced it was to show how the correlation degrades when the average is included. Our intent is not to claim that Plurality is a better aggregator on all examples, but that prompt agreement identifies examples an LLM judge can evaluate reliably, in line with the practice in human annotation studies of discarding items below an agreement threshold. The drop in correlation from Plurality to PA, which adds back the no-consensus examples, shows that these examples are where LLM judgments align least with humans, and supports excluding them. We will report the correlation on the excluded examples directly, together with coverage per dataset.

Q: Rubric equivalence considirations

A: We agree that isolating how individual prompt factors (for example, whether intermediate score levels are defined) contribute to the Prompt Lottery is valuable. It is not what this paper set out to do. Our aim is to establish that semantically equivalent prompts can yield different judgments and rankings, and to show how aggregation mitigates this. Attributing the effect to specific factors is a different study, and one we are pursuing in our current research. Our paper treats deliberate changes to evaluation criteria as out of scope, as stated in the Limitations. We describe our paraphrasing process in Section 3.1 and validated equivalence with three LLM raters, sBERT similarity, and a human study in which annotators judged the prompt pairs to require the same task (κ = 0.78). We acknowledge that this study is limited and that some prompts may not be considered fully equivalent by all readers.

Recent work shows that rubric text alone, without the evaluated response, predicts LLM-judge outputs better than chance (Bagaria et al., 2026 (https://arxiv.org/abs/2609.02942)), so rubric choice is not neutral and no single rubric can be treated as universal; we do not attempt to isolate or optimize these factors, but to show that a single, unoptimized, varied rubric, which is how researchers and practitioners typically work, is an unstable evaluation instrument. We present our initial research on evaluation prompt factors influence on stability, but consider this a sparate stand-alone work. We use the prompts used in this paper where they included a Likert scale in the original (default) prompt

Celln fullΔκ fulln matchedΔκ matched
hotpot_qa__correctness150−0.12195−0.152
hotpot_qa__groundedness150−0.06388−0.176
nq_open__correctness150−0.06539−0.016
nq_open__groundedness150+0.06243+0.124
squad_v2__correctness150−0.082102−0.078

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.