- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper systematically examines how design choices, including the evaluator model, prompt detail, rating scale, and reasoning effort, affect LLM-as-a-judge evaluations. It finds that LLM judges are generally reliable, but not interchangeable: judges with similar accuracy can differ substantially in bias, leniency, and item-level decisions. Model identity is the largest source of variation, while sensitivity to design choices increases for subjective or ambiguous tasks. The paper therefore recommends evaluating metrics beyond accuracy and combining multiple judge models to improve robustness and reproducibility.
Critical strengths
The paper conducts a well-controlled comparison across 10 judge models, multiple prompt designs, rating scales, and reasoning-effort settings, allowing the effect of each design choice to be examined separately.
It evaluates judges against human annotations and reports bias, leniency, and decision-switch rates in addition to accuracy, revealing important behavioral differences that conventional accuracy metrics would miss.
Critical weaknesses
The paper provides a useful and carefully controlled synthesis, but its novelty is limited because most headline findings (model-dependent variability, prompt and scale sensitivity, reduced robustness on subjective tasks, and the benefits of judge aggregation) have already been reported separately. The main contribution is therefore the unified factorial comparison rather than a fundamentally new methodological or conceptual insight.
The experiments cover only sentiment, toxicity, and question-answer correctness, so it is unclear whether the findings generalize to common LLM-judge tasks such as dialogue, summarization, coding, and long-form generation.
The rating sentences are synthetically generated by Grok, while the evaluated answers are generated by Gemini. These data may contain generator-specific patterns and may not represent real-world model outputs.
Because the classification answers come from a Gemini model, the unusually high leniency of the Gemini judge may reflect same-provider preference. This makes it difficult to separate genuine model differences from answer-source effects.
The paper recommends cheaper models, shorter prompts, and lower reasoning effort, but it does not directly analyze cost, latency, or robustness across model updates.
Questions for authors
How do the authors control for same-provider preference, since the evaluated answers are generated by Gemini and Gemini is also included as a judge?
Were judgments repeated across multiple sampling runs or random seeds? If not, how can design effects be distinguished from stochastic model variation?
Given the substantial overlap with prior work on prompt sensitivity, grading scales, judge leniency, and model aggregation, what result do the authors consider uniquely established by this study?
Discussion Period - No Author Response Received
Dear Authors,
We have not received a response from you during this discussion period. Please respond soon to facilitate the discussion of the paper.
Best Regards,
AC