acceptodds
Under review as a conference paper at ICLR 2027

Same Verdict, Different Check Values: What Field Order Changes in a Structured LLM Judge

Abstract

Structured LLM judges return a verdict together with per-criterion checks, and auditors rely on those checks to diagnose errors. We show that these checks can depend on the order in which the fields are declared in the output schema, a choice that leaves the content of the schema unchanged. The verdict can remain unchanged, and correct, while the checks differ. Holding the instances, rubrics and all other content fields fixed, we move the rating and confidence fields before or after the checks and rerun the judge. Ten pre-specified randomised designs cover eight (model, task) configurations. On claim verification, 28% of instances retain their verdict while changing their checks when the two runs use different field orders, against 5% when both runs use the same order: an excess of 23 pp. Two further judge models give +15.8 and +7.8 pp on the same task. The effect persists under every remedy and decoding change we tested. Reordering other keys of the schema changes the checks as well. Aggregate accuracy conceals the change: on MCJudgeBench, where every constraint carries a human label, mean constraint accuracy differs by at most 0.34 pp between the two orders. Beneath that, which constraints the judge scores correctly changes, and so does the set of constraints flagged for repair. Used as an evaluator, the same judge ranks a different model first on every cross-order pair of runs, against one pair in five within a single field order. These results show that verdict-level agreement does not certify a structured judge's audit record: error analyses, repair lists and rankings built on the judge can change with the field order alone. We recommend that the field order be reported, and that check-level evidence be read against a replicate in the same field order.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, automatic evaluation, prompt sensitivity, structured output, reproducibility, robustness, randomized experiment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zy6LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that changing the order of fields in a structured LLM judge's output can alter its criterion-level evaluations, even when the final verdict remains unchanged, potentially affecting error analysis and model rankings.

Critical strengths

The paper shows that changing the output field order can affect criterion-level evaluations, even when the final verdict stays the same. Experiments across multiple models and tasks show that this effect is not simply due to normal variation between runs.

Critical weaknesses
  1. Previous studies have already shown that output order, prompt formatting, and other changes can affect LLM judge reliability. While this paper focuses on field order when the final verdict stays the same, it is unclear what new insights it provides beyond these studies.

  2. Although the paper shows large changes in criterion-level evaluations, the differences in correctness against human labels are relatively small. Also, 85.5% of the effect in the main Llama experiment comes from just three instances. It is unclear how much these changes actually affect the reliability of LLM judges in practice.

Questions for authors
  1. What are the main findings or practical insights that distinguish this work from previous studies on output order and prompt formatting in LLM judges?

  2. Given the relatively small changes in correctness against human labels on MCJudgeBench, how do the authors expect these findings to affect the reliability of LLM judges in real-world evaluation settings?

Reviewer_drtqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how field order in a structured LLM judge’s output schema affects its per-criterion checks. Changing the order can change those checks even when the overall verdict stays the same, beyond ordinary run-to-run variation. The experiments also show changes in which errors are flagged and how models are ranked, despite little change in average constraint accuracy.

Critical strengths

The same-order replicates are a clear strength. They help separate field-order effects from normal run-to-run variation. The randomized experiments and fixed-generation-order control also strengthen the evidence.

Critical weaknesses
  1. The novelty and practical significance need clearer justification. Prior work already studies several closely related forms of order sensitivity. Here, excess gold-scored correctness migration is around 0.8–1.3 percentage points. Actual repair outcomes are not evaluated. The ranking study uses one judge and one benchmark, and the winner is uncertain under task resampling.
  2. My main concern is the incremental novelty. Prior work already documents several closely related forms of order sensitivity. This paper provides careful additional evidence for structured checks, but I am not yet convinced that it establishes a sufficiently new insight. The downstream experiments help, though their scope and practical implications remain limited. A theoretical account or effective mitigation is not essential, but without either, the empirical findings themselves need to offer a stronger contribution.
  3. The paper is very difficult to follow. It introduces many overlapping terms and frequently switches units and denominators. Important details are scattered across appendices. This makes it hard to connect each claim to its supporting evidence. A single experimental-design table and clearer metric definitions would help.
Questions for authors
  1. What is the main new insight beyond prior work on field-order sensitivity and unstable diagnostic outputs?
  2. How much of the ranking change comes specifically from checks changing while verdicts stay fixed?
  3. Could you simplify the presentation of the experimental design and metrics? The overlapping terminology and changing units and denominators make the results hard to follow. A consolidated design table and a worked example of the main metrics would help.
Reviewer_ZhB5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

1.The paper studies the effect of field order on structured LLM judges, focusing on changes in checks under unchanged verdicts. This is a relevant problem for understanding the reliability of LLM judges.

2.The experiments are extensive, including randomised replications across judge models, a gold-scored audit on MCJudgeBench, a model-ranking study, and several control and remedy experiments.

3.The related work is presented in detail. It covers several relevant lines of research, including prompt-format sensitivity, position bias and output-order effects in LLM judges, constraint-level judge evaluation, rationalisation in judge explanations, and the statistical estimation of prompt instability.

Critical strengths

1.The paper studies the effect of field order on structured LLM judges, focusing on changes in checks under unchanged verdicts. This is a relevant problem for understanding the reliability of LLM judges.

2.The experiments are extensive, including randomised replications across judge models, a gold-scored audit on MCJudgeBench, a model-ranking study, and several control and remedy experiments.

3.The related work is presented in detail. It covers several relevant lines of research, including prompt-format sensitivity, position bias and output-order effects in LLM judges, constraint-level judge evaluation, rationalisation in judge explanations, and the statistical estimation of prompt instability.

Critical weaknesses

W1. Limited novelty. Prior studies have shown that output order can change LLM judge scores [1] and that field order in structured outputs affects model judgments [2]. The paper motivates its study by an assumption that is “rarely stated”: that “the checks would be the same if the fields of the schema were arranged differently” (§1). However, it provides no direct literature or survey evidence that this assumption is widely held, and the effect of field order has already been studied [1,2]. Moreover, the finding that aggregate accuracy remains stable while per-constraint judgments migrate under input-side perturbations has already been reported in [3]. The contribution is therefore mainly a measurement refinement of a known phenomenon via Tstable. Overall, the novelty of this work is insufficient.

W2. The claims are not fully consistent with the experimental results. The title and abstract attribute the effect to field order, stating that audit results “can change with the field order alone”. However, the authors’ own controls show that edits changing no order also produce the effect. Merely doubling the indentation yields intervals excluding zero on all three judges (Table 14). On gpt-5.4, a length-matched rewording even exceeds the field-order effect (−5.05 pp, Appendix M.3). The phenomenon thus appears to reflect a general sensitivity to prompt edits rather than an effect specific to field order.

W3. Insufficient task design. The authors’ main rubric is ambiguous: evidence_sufficient, whose clause “admits two readings”, accounts for 42% of the check changes (Appendix I), and disambiguating it reduces the primary effect by 31–38% (Appendix J). However, this test covers only this one clause and lacks more detailed validation, such as additional rewordings or randomised designs. The paper therefore cannot verify whether the effect reflects field order or rubric ambiguity. Thus, the task design is insufficient.

[1] Yi-Pei Chen, KuanChao Chu, and Hideki Nakayama. LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation. arXiv:2406.02863, 2024.

[2] Tom van Nuenen and Pratik S. Sachdeva. The Fragility of Moral Judgment in Large Language Models. arXiv:2603.05651, 2026.

[3] Jaeyun Lee, Junyoung Koh, Zeynel Tok, Hunar Batra, and Ronald Clark. MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following. GEM, 2026.

Questions for authors

Q1.Why is correctness migration more concentrated in a few instances in the primary MCJudgeBench configuration than in the other configurations? Could the authors provide a blind human re-review of the migrating constraints and a replication on additional benchmarks with per-criterion labels, to determine whether the migrations reflect judge instability or ambiguous gold labels?

Q2.Finding 2 states that “Accuracy stays stable while the audit record changes,” yet gold accuracy shifts from 42.90 to 55.87 on safety rating and from 68.59 to 62.62 on gpt-5.4 claim verification (Table 5), and Appendix H reports changes from +13.43 to −12.29 pp. Since the claim appears to hold only on MCJudgeBench, how do the authors reconcile it with these results?

Q3.What new insight does this work provide beyond existing studies of prompt formatting and output-order effects?

Q4.How much do the changes in repair lists and model rankings affect practical LLM evaluation?

Q5.Could the authors provide indentation, whitespace, and rewording experiments under the same exact randomised design as Table 2, and repeat the main experiment on a locally pinned deployment? This would show whether the field-order effect exceeds that of edits that change no order.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.