- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies “Prompt Lottery”: variation in LLM-as-a-Judge evaluations under prompt rephrasings intended to preserve task meaning. Across 12 datasets, six judge families, and five backbone models, it examines judgment consistency, agreement with human ratings, and changes in evaluated models’ rankings. Human validation and annotation studies provide additional context for interpreting the observed sensitivity. The paper also evaluates multi-prompt aggregation and proposes a classifier-based gating strategy to selectively invoke additional judgments, aiming to improve robustness while reducing calls relative to full aggregation.
Critical strengths
A practically consequential evaluation problem. The paper connects sensitivity to judge-prompt wording with changes in evaluated models’ rankings, moving beyond differences in individual scores. This makes the study relevant to the reliability of comparative model evaluations and supports its value as an empirical contribution.
Broad empirical coverage with complementary validation. The study spans 12 datasets, six judge families, and five backbone models, and includes human annotation studies and fixed-prompt repeatability controls. Its task-dependent findings—including settings where rankings remain stable—provide a more informative picture than isolated failure examples. This breadth is a central reason for my positive assessment.
Critical weaknesses
The experiments do not fully isolate wording sensitivity. Some prompt variants change the evaluation criteria or provide additional reference information. Consequently, the observed disagreement may partly reflect substantive differences in the evaluation setup rather than sensitivity to equivalent rephrasing. This weakens the causal interpretation of the central findings.
Questions for authors
Clarification and supporting analysis: Some variants in Appendix H appear to change more than wording: G5 emphasizes different evaluation dimensions across variants, and G11/G12 P4 includes reference information absent from other variants. Were these differences present in the prompts actually used for the reported experiments? If so, do the main consistency and ranking findings remain when comparisons are restricted to variants with identical evaluation criteria, reference information, and scoring rules? This would materially affect my confidence that the observed effects can be attributed to paraphrastic rephrasing.
We wish to thank the reviewer for their constructive review. We truly appreciate to see a review which has a main goal to improve the paper it is reviewing. The proposed improvements are spot-on, and we present additional information including some of our ongoing work with the same goal as the ones the reviewer suggested.
We agree that some variants differ in more than surface wording, and we will describe them accurately as task-equivalent prompt variants. This reflects how evaluation prompts vary in practice: there is no canonical way to write a rubric, and evaluators routinely change criteria, score definitions and emphasis. Our goal in this paper was to establish whether an LLM judge is a stable evaluation instrument when its prompt has not been specifically optimised or checked for task equivalence, which is the situation of most practitioners. The human study supports the equivalence of the variants: annotators given the same variants remained consistent (Fleiss’ κ ≈ 0.80), while LLM judges did not. We pursued this before exploring which individual factors influence stability, as the reviewer suggests. Isolating the contribution of individual factors, such as score definitions, emphasized dimensions or reference information, is a distinct question that we consider out of scope here and are pursuing in follow-up work. As a robustness check, we additionally report our main consistency and ranking results restricted to variants with identical criteria, reference information and scoring rules, as well as without P4 in the SummEval groups.
We present the reviwer with our initial research on evaluation prompt factors influence on stability, but consider this a sparate stand-alone work. We use the prompts used in this paper where they included a Likert scale in the original (default) prompt Recent work shows that rubric text alone, without the evaluated response, predicts LLM-judge outputs better than chance (Bagaria et al., 2026 (https://arxiv.org/abs/2609.02942)), so rubric choice is not neutral and no single rubric can be treated as universal; we do not attempt to isolate or optimize these factors, but to show that a single, unoptimized, varied rubric, which is how researchers and practitioners typically work, is an unstable evaluation instrument.
This shows that the delta \kappa is smaller than what we observed with fully varied prompt (which may be the effect of discriminatory power of rubrics themselves as described in the above cited work by Bagaria et all). This is still research in early