- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper shows that changing the order of fields in a structured LLM judge's output can alter its criterion-level evaluations, even when the final verdict remains unchanged, potentially affecting error analysis and model rankings.
Critical strengths
The paper shows that changing the output field order can affect criterion-level evaluations, even when the final verdict stays the same. Experiments across multiple models and tasks show that this effect is not simply due to normal variation between runs.
Critical weaknesses
-
Previous studies have already shown that output order, prompt formatting, and other changes can affect LLM judge reliability. While this paper focuses on field order when the final verdict stays the same, it is unclear what new insights it provides beyond these studies.
-
Although the paper shows large changes in criterion-level evaluations, the differences in correctness against human labels are relatively small. Also, 85.5% of the effect in the main Llama experiment comes from just three instances. It is unclear how much these changes actually affect the reliability of LLM judges in practice.
Questions for authors
-
What are the main findings or practical insights that distinguish this work from previous studies on output order and prompt formatting in LLM judges?
-
Given the relatively small changes in correctness against human labels on MCJudgeBench, how do the authors expect these findings to affect the reliability of LLM judges in real-world evaluation settings?