acceptodds
Under review as a conference paper at ICLR 2027

When Position Predicts Reward in LLM Judges

Abstract

LLM judges can learn a position rule instead of comparing response quality when preference data always place the preferred response in the same slot. We study this training-interface confound as an identification problem: a content-sensitive judge should reverse its verdict when the two responses are exchanged, whereas a position shortcut should not. We introduce a paired-order audit and train matched judges with either a fixed preferred-first interface or both physical arrangements. In a controlled four-arm study with Qwen2.5-7B, four seeds, and 200 update steps, two-order pointwise training raises order-averaged accuracy from 65.14% to 87.05% on RewardBench and from 55.51% to 75.31% on an UltraFeedback clean split. First-position selection falls from 83.30% to 49.73% and from 93.12% to 56.18%, respectively. A separate dose study shows that increasing the preferred-first share monotonically reduces paired robustness. The results establish a simple audit and training intervention for testing whether an apparent judge gain survives a change to the feature that predicts its reward.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, position bias, pairwise preference evaluation, evaluation robustness, preference learning, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dNkMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how LLM judges can be trained to select the first response (when asked for preference based judgements) when the training data always put the preferred answer there. The authors compare this setup with training on different types of preference orders. They evaluate accuracy, consistency and position preference using paired judgements. To this end, they provide a controlled study of training time position-label correlation and a dose experiment where they vary the dosage of the first-position preference ratio. Experimental results show that two-order training improves order-averaged accuracy and demonstrates recovery performance even from confounded training setups.

Critical strengths
  1. The paper is well written+structured and relatively easy to follow along. It tackles an important (and well documented) problem in LLMs-as-a-judge and provides a clear problem structure to analyze this.
  2. The main intervention produces improvements under a reasonably matched control.
  3. The dose study strengthens the paper's argument that increasing the preferred-first proportion produces worse judge performance.
Critical weaknesses
  1. I'm not sure what the novelty/main advance of this paper is when compared to existing literature. JudgeLM[1] already introduced training-time swap augmentation and also measured its effect on consistency. Also, J1[2] compared random single-order/both-order training alongside a consistency reward. Although this paper adds useful evidence via the controlled imbalance experiments, neither the intervention itself nor separating layout coverage from consistency rewards is particularly novel. I recommend the authors take a look at these particularly relevant literature and add it to their related work section.

  2. I don't think the experimental specification and interpretation are reliable enough for independent assessment. The paper omits the training data sources, selection rules for the evaluation subsets, exact prompts, numerical training settings, and formulas/weights for the additional objectives. I looked through the appendix, but it only reproduces statistics rather than supplying these missing experimental details.

  3. Figure 1 describes three experiments SFT/DPO/GRPO that are absent from the reported study.

  4. Section 4.1 claims that both orientations improve, but Tables 5 and 6 (from the appendix, when going from fixed-order to two-order) show original order accuracy falling on both RewardBench and UF-clean. Can you clarify what you mean in this section?

[1] JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR, 2025).

[2] J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (ICLR, 2026).

Questions for authors

See weaknesses first


  1. What exact datasets, evaluation-subset selection rules, prompts, and training configurations produced these results? Please specify the objectives for all four arms, including whether cross-entropy supervises rationale tokens or only the verdict.
  2. Concerning W3, do you have plans to run experiments detailed in Figure 1 or relevant results?
Reviewer_WH3XOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors work on a well known issue of position bias in LLM as a judge. They formulate a training set up where they try and isolate the shortcut learning from content learning. They train four set ups of preference learning: (a) ARM A: fixed order: where the preferred responses is always in position A, (b) ARM B: mixed order where the preferred response occurs in positions A and B and two baselines (c) ARM C: its mixed order data but consistency is also enforced from the objective (d) ARM D: mixed order but there is some advantage reweighting.

The authors evaluate the above set up with an auditing protocol that measures the following (i) accuracy single order (ii) accuracy swapped order (iii) rate of model predicting A as the winner (iv) rate of model switching the response when the order of prompting is switched

The authors train Qwen-2.5-7B for the four settings, with the same number of samples and on four seeds and evaluate on RewardBench and UltraFeedback as the test sets.

The results show that the fixed order arm drops in performance compared to the mixed order arm (Reward Bench w/ both orders achieves 87.05 but with fixed order achieves 65.14) . The mixed order (also referred to as pointwise in the results table) also remains on par performance wise with arms C and D. The authors also run an additional experiment where they progressively make the training data more position biased, which results in the performance decreasing and metric (iii) i.e. number of times the model predicts A increasing. The results also discuss the potential explanations why this happens.

Critical strengths
  • The authors have picked an important issue to work with.
  • the scoping is honest
  • takeaways are practical
  • the different training arms are matched in setup
Critical weaknesses
  1. The problem is well studied in our domain and researchers are very well aware of how to mitigate the position bias issue (J1[1] and JudgeLM[2]) . The authors seem to have taken a part of this; I am not sure about the novelty of the proposed approach
  2. The details around the training data split, examples, and how the authors choose subsets of RewardBench or UltraFeedback (or how UltraFeedback Clean was created) is missing. Details around the 10k training instances for the dose study is also missing
  3. The paper is missing details around what the trained algorithm is. At some points in the paper they say they optimize for cross-entropy loss (lines #207) but then they say that they add a penalty if samples are inconsistent so it is unclear and also doesn’t tie in with what they show in Figure 1 which talks about
  4. The paper motivates the work by saying that a biased judge harms downstream policy training but doesn’t show any downstream results
  5. The authors talk about five possible reasons for this but they do not really substantiate their explanation with quantification (other than the first one maybe)
  6. Authors do not discuss other possible issues of bias eg: length bias, verbosity, assertiveness etc when it comes to LLM as a judge issues

Missing citations:

  1. RewardBench,UltraFeedback, Qwen models are missing citations
  2. The second half of the related work section is missing citations

Some things that would need to be fixed in the draft

  1. Interchangeably using mixed order, equal mixes, two-order and pointwise
  2. Citations are not clicking
  3. For 3.2 the authors can point to the equations lines #115-126
  4. Line 237 and 240 are repeated
  5. No reference to figure 2

I would also recommend a major restructuring of the paper as it was hard to follow

[1] J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning

[2] JudgeLM: Fine-tuned Large Language Models are Scalable Judges

Questions for authors
  1. What dataset do you train with? how did you choose the 449 instances for evaluation? Can you give examples?
  2. What is the performance of an untrained baseline for the main evaluation set?
  3. What kind of problems are you training on? Is it indistribution for the evaluation datasets? Do you try different prompting formats?
  4. What is the “preferred response A” portion of the two datasets that you are evaluating on? Wondering if that can explain the high (greater than 50%) accuracy for the test sets?
  5. What is the training algorithm you are working with?
  6. How is that the arms B, C, and D perform similarly?
  7. What is the truncation rate? how many instances do you drop after truncation?
  8. Did you also try different ways of prompting?
Reviewer_x26JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript argues that if preference data always put the preferred response in a specified slot A, a judge LLM model can obtain the maximum reward with a position specific shortcut i.e. always select A. Cannonical order accuracy therefore is insufficient in seggragating between content based judgement and position based judgement. To this end, the authors propose (a) a paired order audit with four metrics and (b) a four-arm study on the publicly available Qwen2.7-7B LLM model across 4 randoms seeds. They also show that their order averaged accuracy metric is consistently better in separating the behaviour of the LLM judge across various experiments.

Critical strengths
  1. The paired metrics idea is an useful practice that can be adopted.
  2. The manuscript benefits from clear statistical reporting, specifically the per-seed analysis for each arm, leave-one-out seed analysis, positive seed contrasts, etc.
  3. Results are overall exhaustive and well presented. The null result on the objectives is interesting and the dose study allows the reader to gauge the size of the harm concretely to these judge models.
Critical weaknesses
  1. Potentially ignored related work, questioning novelty. Swap augmentation with label flipping is already used in judgeLM, Auto-J, EvalPlanner, etc. Section 2 mentions this line of work but none of them are cited for better readability.
  2. Fig 1 is very poorly designed and has significant inconsistencies with the text. It is overall very hard to follow. It shows “Fixed GRPO” deltas (+14.5 / −3.0 / −22.8 / 68.8%) and “Mirrored GRPO” results (OA 82.6, Con 84.0, First 50.0%). It also lists “SFT / DPO / GRPO”, “3 backbones”, “Size-matched” and “Online random” variants. None of these appear in the text.
  3. Most experiments are performed on a single LLM model with 200 steps. This narrows the scope significantly and must be tested with different backbones.
  4. The main effect appears to be true as default, due to lack of control nuance. The paper lacks evidence that fixed-order conversion happens in practice, and is only described as something that a "silent conversion choice can" cause. I would however, defer here to more knowledgeable referees in the field, for a better perspective.
  5. Some training details are underspecified. Line 149-150 claims that the exact prompt schema, filtering rules and aggregation pseudocode appear in appendix but they cannot be found. The paper does not state the learning rate, batch size, full fine-tuning vs. LoRA, base vs. Instruct checkpoint, where the rationales come from, or the consistency and advantage weights.
Questions for authors
  1. Can the authors show generalization of the proposed metrics across multiple backbones of varying sizes, longer training and different cue types, so that thos claimed "audit" can be enriched beyond a relatively narrow scope?
  2. Can the authors report the untrained Qwen2.5-7B on both four-arm evaluation sets.
  3. Is training SFT/cross-entropy or RL (GRPO)? Please provide full hyperparameters, the checkpoint (base or Instruct), where the rationale supervision comes from, and the weights for the C and D terms. Were these weights tuned? As a rule of thumb, the overall manuscript needs an inspection and explanations/intuitions can be made more lucid.
  4. Figure 1 can be significantly improved. At the moment, it reads like a chart of operations and mathematical jargons, ill-described (at times undescribed even) and poorly presented.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.