- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a self-supervised uncertainty estimation method for LLM-as-a-Judge systems by using preference reversals under swapped response orders as uncertainty labels. The judge is fine-tuned to predict an additional uncertainty token (U), enabling single-pass uncertainty estimation without human annotations or auxiliary models. Experiments demonstrate improved flip prediction, calibration, and judgment consistency, along with more efficient inference through uncertainty-based routing.
Critical strengths
(1. Novel use of preference flips as self-supervised signals) The paper presents an interesting approach that leverages preference reversals under response-order swaps as self-supervised uncertainty labels. Incorporating these signals into fine-tuning enables single-pass estimation of judgment instability without requiring ground-truth correctness labels. This offers a simple and practical approach to improving the reliability of LLM judges.
(2. Comprehensive empirical evaluation) The paper provides extensive experiments, including comparisons with various uncertainty estimation baselines, calibration analysis, cross-domain and cross-model evaluations, and ablation studies.
Critical weaknesses
(1.Insufficient justification of the scope and significance of preference-flip uncertainty) The Related Work section primarily discusses conventional uncertainty estimation methods, many of which focus on prediction correctness or predictive uncertainty. However, the proposed method specifically targets order-induced preference instability, which does not necessarily reflect conventional uncertainty. For instance, a judge may consistently make incorrect predictions without exhibiting preference flips, while position bias may cause flips even when the model is highly confident in its predictions. Although the paper explicitly defines preference flips as its supervision target, it frequently uses the broader term "uncertainty estimation" without sufficiently clarifying this distinction, potentially misleading readers about the scope of its claims.
I believe the authors should more consistently distinguish preference-flip uncertainty from conventional uncertainty, particularly when interpreting their calibration results. If the authors intend to claim that the proposed method also captures broader judgment uncertainty, they should provide a more convincing conceptual justification or direct empirical evidence demonstrating the relationship between preference flips and judgment correctness. Although the improvements in judging accuracy and the error detection experiments provide indirect evidence, they do not establish whether the proposed uncertainty scores are calibrated with respect to judgment correctness.
Alternatively, if the intended contribution is limited to preference-flip uncertainty, the paper should more clearly justify its significance as a standalone uncertainty measure and articulate what aspects of judgment reliability it captures beyond existing indicators of order inconsistency.
Questions for authors
(1. Clarification of the scope of uncertainty estimation.) Is the proposed (P(U)) intended to estimate only order-induced preference instability, or is it also expected to reflect broader judgment uncertainty, such as the probability of making an incorrect judgment? Since the calibration experiments evaluate preference flips rather than judgment correctness, I would appreciate clarification on how the authors intend their uncertainty estimation claims to be interpreted.
(2. Significance of preference-flip uncertainty.) If the intended scope is limited to preference-flip uncertainty, could the authors further justify its significance as a standalone uncertainty measure? While the paper demonstrates improvements in judgment consistency and downstream applications, I would like to better understand why estimating preference-flip uncertainty is important in its own right and what practical value it provides beyond conventional uncertainty measures or existing approaches to handling order inconsistency.
(3. Connection to broader judgment uncertainty.) If the authors intend the proposed method to capture broader judgment uncertainty beyond preference flips, could they provide more direct empirical evidence and theoretical insights supporting this connection? In particular, I would be interested in a principled explanation of why preference-flip uncertainty should reflect judgment correctness, including any assumptions under which such a relationship might hold. Additionally, evaluating correctness-based uncertainty estimation on RewardBench, including error-detection performance and calibration where appropriate, would help substantiate the broader claims.
3 comments
Q3 (Connection to broader judgment uncertainty)
[...] I would be interested in a principled explanation of why preference-flip uncertainty should reflect judgment correctness, including any assumptions under which such a relationship might hold. Additionally, evaluating correctness-based uncertainty estimation on RewardBench, including error-detection performance and calibration where appropriate, would help substantiate the broader claims.
Ans: Both are now in the paper.
Principled explanation (Sec. 4.4, Appendix E). Assume exactly one response in each pair is correct. A flipped pair then holds exactly one incorrect verdict, and a consistent pair zero or two. If either order is equally likely to hold the incorrect verdict of a flipped pair, a call is incorrect with probability e = ½ρ + β. Here ρ is its flip probability and β the probability that its pair is wrong in both orders. Ranking verdicts by flip probability therefore ranks them by error probability, as long as consistent errors do not become rarer as flips become likelier. We check this on RewardBench. Across ten bins of P(U), the error rate is close to a line in the flip rate (96% of variance; slopes 0.90 for Qwen2.5-7B and 1.29 for 32B), and consistent errors rise with the flip rate rather than fall. The equal-chance assumption also departs in our favor, as noted under Q1. Verifying the condition needs gold labels, so flip data alone cannot certify it for a new judge, and our paper now says so.
Error detection on RewardBench (Sec. 4.3, App. B.2). P(U)'s correctness-AUROC exceeds the untrained judge's A/B margin on all four judges tested: Qwen2.5-7B 0.777 vs 0.737, Llama-3.1-8B 0.705 vs 0.650, Qwen2.5-14B 0.785 vs 0.747, and Qwen2.5-32B 0.833 vs 0.809. On 7B it ties correctness-trained Self-REF and beats five other correctness-trained estimators. On 14B and 32B, Self-REF leads by 0.02.
Calibration. P(U) is calibrated as a flip probability (ECE 0.018), not as an error probability. Because the error rate is close to linear in the flip rate across P(U) bins, a calibrated error probability needs only a monotone remapping, but fitting it requires correctness labels.
Thank you for the detailed response and revisions. I appreciate that the authors have substantially addressed my original concerns by clarifying the distinction between preference-flip uncertainty and judgment correctness, providing additional correctnessbased evaluations, and demonstrating the practical utility of flip uncertainty.
These additions strengthen the paper’s empirical contributions. I have decided to raise my rating from 2 (Weak Rejection) to 3 (Weak Acceptance). However, I want to emphasize that my assessment remains firmly borderline. This change does not indicate a clear endorsement of acceptance, but rather a slight preference for Weak Acceptance over Weak Rejection.
I still have two reservations:
1. Theoretical justification: The analysis in Section 4.4 relies on the assumption that consistent errors do not become less frequent as flip probability increases. While empirically supported on RewardBench, it remains unclear why this relationship should hold across different distributions or judges.
2. Methodological novelty: The main difference from prior work such as Self-REF is the use of self-supervised preference-flip labels instead of correctness labels. Despite the compelling empirical results, I remain uncertain whether this constitutes a sufficiently substantial conceptual contribution.
Overall, I view this paper as a close borderline case rather than a clear acceptance recommendation, and leave the final assessment of its novelty and significance to the Area Chair.
Thank you for an exceptionally careful review. You laid out the choice before us clearly: either justify a broader claim about correctness, with direct evidence and a principled explanation, or justify flip uncertainty on its own terms. That framing shaped most of the revision. We ended up doing both, and the first held up better than we expected. The new Sec. 4.3 and 4.4, and Appendices B.2 and E, follow your suggestions closely, including the RewardBench evaluation you proposed. CW1 and Q1 (Scope of the uncertainty estimate)
Ans: Both, now kept clearly apart. P(U) is trained only to predict order-induced flips. The revised PDF (changes in blue) is now consistent: we report "flip-AUROC" for that target and the new "correctness-AUROC" for predicting incorrect verdicts. The link to correctness is now empirical with an explanation, rather than implied (see Q3).
The revision now measures both of your examples. For Qwen2.5-7B, 53% of RewardBench errors come from pairs that never flip, and for Qwen2.5-32B that was 73% (Appendix E). Confident flips also turn up often. Even when Qwen2.5-7B puts more than 95% probability on its chosen answer, it flips 34% of the time, barely below its overall rate of 37% (Appendix L.4). We would argue these confident flips still carry information. When one response in each pair is correct, every flipped pair contains exactly one incorrect verdict, so in one of the two orders the judge was confidently wrong. Trained P(U) tends to catch this: within flipped pairs, it is higher on the incorrect verdict about two thirds of the time (Appendix E).
Q2 (Significance of preference-flip uncertainty)
Ans: Three reasons, beyond its link to correctness: