- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how LLM judges can be trained to select the first response (when asked for preference based judgements) when the training data always put the preferred answer there. The authors compare this setup with training on different types of preference orders. They evaluate accuracy, consistency and position preference using paired judgements. To this end, they provide a controlled study of training time position-label correlation and a dose experiment where they vary the dosage of the first-position preference ratio. Experimental results show that two-order training improves order-averaged accuracy and demonstrates recovery performance even from confounded training setups.
Critical strengths
- The paper is well written+structured and relatively easy to follow along. It tackles an important (and well documented) problem in LLMs-as-a-judge and provides a clear problem structure to analyze this.
- The main intervention produces improvements under a reasonably matched control.
- The dose study strengthens the paper's argument that increasing the preferred-first proportion produces worse judge performance.
Critical weaknesses
-
I'm not sure what the novelty/main advance of this paper is when compared to existing literature. JudgeLM[1] already introduced training-time swap augmentation and also measured its effect on consistency. Also, J1[2] compared random single-order/both-order training alongside a consistency reward. Although this paper adds useful evidence via the controlled imbalance experiments, neither the intervention itself nor separating layout coverage from consistency rewards is particularly novel. I recommend the authors take a look at these particularly relevant literature and add it to their related work section.
-
I don't think the experimental specification and interpretation are reliable enough for independent assessment. The paper omits the training data sources, selection rules for the evaluation subsets, exact prompts, numerical training settings, and formulas/weights for the additional objectives. I looked through the appendix, but it only reproduces statistics rather than supplying these missing experimental details.
-
Figure 1 describes three experiments SFT/DPO/GRPO that are absent from the reported study.
-
Section 4.1 claims that both orientations improve, but Tables 5 and 6 (from the appendix, when going from fixed-order to two-order) show original order accuracy falling on both RewardBench and UF-clean. Can you clarify what you mean in this section?
[1] JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR, 2025).
[2] J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (ICLR, 2026).
Questions for authors
See weaknesses first
- What exact datasets, evaluation-subset selection rules, prompts, and training configurations produced these results? Please specify the objectives for all four arms, including whether cross-entropy supervises rationale tokens or only the verdict.
- Concerning W3, do you have plans to run experiments detailed in Figure 1 or relevant results?