acceptodds
Withdrawn from ICLR 2027

Rules Move Verdicts Mote Than Paraphrases: Controlled Evaluation of LLM-Judge Panels

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agreement does not reveal whether language-model judges follow a stated rule, express an interpretation, or have evidence about an answer. We separate these questions through controlled comparisons. On 160 authored cases and four judges, reversing an explicit rule changes 462/469 complete ambiguous judge-case verdicts; paraphrasing the same rule changes 4/469. The assigned-profile contrast is 90.04 percentage points, with completion bounds [83.59, 97.46]. A prospectively fixed extension reuses all cases with concurrent explicit controls. With neither rule supplied, an absence notice raises undetermined responses by 33.59 points over filler and turns the 41 binary-unanimous filler panels, among 128 ambiguous cases, into mixed responses. This does not make every compatible binary verdict wrong. A 96-case wording follow-up finds positive aggregate undetermined differences for every notice against both filler and convention mention. However, convention mention also eliminates binary unanimity, notice magnitudes vary, and one wording induces undetermined responses on 23/128 determinate controls. Exploratory vote analysis shows that broken unanimity need not remove a binary majority. A separate matched-call comparison retains its observed accuracy tie, not equivalence; older elicitation and checking studies retain their registered nulls, uncertainty and harms. These findings concern fixed cases and judge rosters. Replay code and data are available in the https://anonymous.4open.science/r/llm-judge-replay-iclr2027-DE13/README.md.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, evaluation validity, task underspecification, judge ensembles, prompt sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_J48qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how explicit rules, paraphrases, and missing requirements affect LLM-judge decisions. Through several controlled experiments, the authors examine verdict changes, panel agreement, abstention behavior, and the effects of additional verification procedures.

Critical strengths

The paper includes several controlled comparisons and reports detailed experimental results.

Critical weaknesses
  1. Very limited novelty and significance.

The central finding that changing an explicit evaluation rule changes the judge's verdict, while paraphrasing the same rule has little effect, is largely expected from instruction-following LLMs. The paper does not establish why this observation constitutes a meaningful new insight into LLM-judge reliability.

  1. Weak experimental implications.

The reported increase in abstention after explicitly mentioning missing rules does not necessarily indicate better evaluation quality. In fact, Table 2 shows that merely mentioning conventions also eliminates binary unanimity, while one notice causes 23/128 determinate controls to abstain, raising concerns about the practical value of these findings.

  1. Fragmented experimental design and unclear contributions.

The paper combines numerous experiments with different datasets, judge panels, metrics, and objectives, making it difficult to identify a coherent central contribution. Several additional experiments report null or negative results, and the matched-call comparison shows no accuracy improvement (94.34% for both methods).

  1. Limited generalizability.

The main controlled experiments rely on only eight manually authored mechanisms and four judges from two providers. The omission and wording experiments also reuse the same cases, leaving the generalizability of the findings to realistic evaluation tasks largely untested.

Questions for authors

see weaknesses

1 comment

Thank you. We take the four points in turn.

  1. Novelty. We agree that rule-following alone is expected. It is our control, not our claim. The finding is the omission result: with no rule stated and UNDETERMINED available, four judges from two providers return the same binary verdict on 41/128 ambiguous cases, (post-hoc) always on the same default convention within a mechanism, and on 37 of those cases the same judges follow either explicit rule correctly. Agreement there reflects a shared default. Details are in our reply to Reviewer a4UK.

  2. Implications. We agree that more abstention is not better evaluation, and the paper does not propose notices as a remedy. The two facts you cite are our own controls and we read them as you do: unanimity disappears when conventions are merely mentioned, and one wording over-abstains on determinate cases. The implication is a caution. Panel unanimity and abstention rates both move with cues that carry no rule, so neither should be taken as evidence that a task was well specified. The dependable intervention in our data is stating the rule: 98.5% of verdicts follow a reversed rule, and 0.9% change under paraphrase.

  3. Fragmentation. This is fair, and it is the main cause of the presentation problem. The paper carries earlier studies (eleven judges; SQL, regex and FOLIO cohorts) beside the four-judge experiments, because the project moved through them on the way to the matched design (we describe that path in our reply to the Area Chair). The core is three linked experiments on one case set: the explicit rule, the omitted rule and the wording controls. The matched-call tie is reported as a negative result. We kept the nulls and the tie because dropping them would overstate the evidence.

  4. Generality. We agree this is a limitation. There are eight authored mechanisms and four judges from two providers, and the omission and wording studies reuse the same cases. The claims are about this fixed benchmark and we state them that way. Natural tasks, more judge families and new mechanisms are needed before any claim about how often this happens in practice.

Reviewer_iy5FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

ai slop.

Critical strengths

nothing.

Critical weaknesses

ai slop signs.

Questions for authors

nothing

Reviewer_a4UKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper shows how explicit task specifications, omitted conventions, and additional verification processes affect LLM-as-judge decisions. The authors accomplish this by conducting several controlled experiments that involve 1) rule reversal, 2) paraphrasing, 3) missing-rule notices, 4) panel aggregation, 5) ambiguity elicitation, and 6) execution-based verification. The central results are that reserving an explicit rule changes 462/469 judge-case verdicts, whereas paraphrasing the same rule changes only 4 cases.

While this topic is relevant, I am not convinced that the work makes a sufficient novel contribution. Much of the experimental evidence establishes expected sensitivity to explicit instructions, while the more interesting claims, such as ambiguity, remains insufficiently supported.

Critical strengths
  1. The paper explores an important issue in LLM-based eval: agreement among judges should not automatically be interpreted as evidence of correctness.
Critical weaknesses
  1. The main finding lacks novelty: of course LLMs follow explicit rules and give similar answers when those rules are paraphrased. The experiments use simple tasks such as rounding and sorting. The reported ~90 percentage difference largely reflects the experimental setup rather than a new insight into judge reliability.

  2. The results are tautological and mainly driven by wording. For example, telling models that a result is unspecified naturally increase undetermined responses. The results largely suggest sensitivity to prompting rather than any sort of reliable ambiguity detection.

Questions for authors
  1. What is the primary scientific insight beyond the observation that instruction-following models respond differently to explicitly different rules? Which result would the authors consider genuinely new and unexpected?

  2. Can the authors identify a practical evaluation setting in which the proposed distinctions would propose better decisions, under matched coverage and cost, than existing abstention, clarification, or calibrated-routing approaches?

1 comment

Thank you. Your two questions are the right ones, and the paper did not answer them clearly.

W1 / Q1 — novelty. We agree that judges following an explicit rule is expected, and we should not have led with it. In our design that test is the control. It shows that the same judges can apply either convention on these cases (both stated rules followed on 466/512 judge–case profiles) and that their verdicts follow the rule's meaning and not its wording (4/469 change under paraphrase).

The result we consider new is what happens when the rule is left out. UNDETERMINED is offered in every arm, and each ambiguous case has two admissible conventions that give opposite verdicts. With a neutral filler note:

  • judges still return a binary verdict in 306 of 509 valid replies (60%);
  • all four judges, from two providers, return the same binary verdict on 41 of 128 ambiguous cases;
  • these verdicts are not scattered. Within a mechanism, 91% of binary votes fall on the same convention (inclusive interval endpoints 61/61; date as written rather than UTC 62/62), and all 41 unanimous panels side with it. In 12 cases, the panel unanimously marks INCORRECT a candidate that is correct under the other convention;
  • in 37 of these 41 cases, the same four judges are correct under both explicit rules in the same run.

So this agreement is a default the judges share. It is not evidence about the candidate, and nothing in the panel's output separates it from agreement on a fully specified task.

W2 — "tautological", driven by wording. We agree that an absence notice partly acts as a cue, and the paper does not claim reliable ambiguity detection. That is why the follow-up adds a control that only mentions conventions (it also removes all binary unanimity, 19/64 to 0/64) and why we report that the strongest notice gives UNDETERMINED on 23/128 determinate controls. What those controls show is the point: both unanimity and abstention move with twelve words that carry no rule, so neither is a dependable sign that a task was well specified.

The tasks are simple on purpose. Each needs an executable reference under both conventions, so that "the rule changed" and "the answer changed" can be separated exactly. We agree this limits how far the results transfer.

Q2 — a practical setting. We do not have one in the sense you ask, and we do not claim that. We did not compare against abstention, clarification or calibrated routing at matched coverage and cost. Our one matched-cost comparison is an exact tie (94.34% for both procedures), and in the earlier SQL study routing on an elicited flag loses to deferring everything. The practical content is narrower. First, a convention stated in the judge prompt is followed almost perfectly and survives rewording, so the cheap fix is in the rubric. Second, when the rubric is silent, unanimity should not be read as confidence, and reports should give the number of non-decisions beside conditional accuracy. A procedure that beats existing routing methods is future work.

Comments on the paper

2 comments
Area_Chair_HrSyOpenReview ↗

Discussion Period

Dear Authors,

We have not received a response from you regarding this paper, nor has it been withdrawn. Please clarify to what extent AI was used in this paper and assure us that you are taking full responsibility for the paper as written. Reviewers have raised serious concernd with respect to the paper's contents and the extent to which AI was used. Please address these in your response.

Sincerely,

AC

Dear Area Chair,

We apologise for the late reply. Having read the reviews, we have decided to withdraw the submission. Your questions deserve an answer first, and we give it here.

AI use. AI was used in the experiments, code, and writing, under our direction. To be concrete about the extent: we worked with LLM coding agents. We set the questions and the direction. For each experiment, we decided what it should test, the kind of design it should have, and what we expected to see. The agents then worked out the detailed protocol (arms, exact wording, sample sizes, analysis plan), sometimes as several candidate versions, and at some points suggested follow-up experiments. We then analyze each version, edit the details, and decide the final versions to be carried out. We then ask the coding agents to write the code, run the model calls, compute the results, and make visualizations of the results. We read and analyze the results and the visualizations to decide the next step of the research. We also used panels of LLMs to simulate critiques and reviews of our experimental design, methodology, results analysis, and proposed next steps. When we finally reached the writing stage, we made the paper outline and storyline, and the bullet points of paragraphs development, then wrote the first draft. We then used AI to edit the rough first draft based on our outline and improve the writing in iterations.

The project. We used panels of LLMs from different providers as a judge panel in an agentic workflow. The question was when agreement among LLM judges from different providers is evidence about the work they are judging. The project changed direction more than once. We dropped an early line on the effective number of independent judges when Kohli (2026, arXiv:2605.29800) appeared with the same result. Our next finding, that judges from the same provider share more errors, turned out to be very close to Kuai et al. (2026, arXiv:2604.07650) and related to Kim et al. (2025, arXiv:2506.07962), so we changed direction a second time. Later experiments pointed to tasks whose instructions leave a rule open. Late in the project, we realized that our main comparison changed too much text at once. We then built the matched design the paper leads with. Its first run was mixed; the four studies in the paper followed.

Checking the results. The four studies had their cases, prompts, parser and analysis fixed before the first model call (Appendix H). Verdicts are scored by executable reference functions. The anonymous repository replays all 8,960 judge calls with one command and reproduces the reported numbers.

We accept that the paper is hard to follow and did not make clear what is new. We have replied to reviewers on their specific points, and we thank all three reviewers and you for the time spent on it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.