acceptodds
Under review as a conference paper at ICLR 2027

If You Know, You Know: Self-Supervised Uncertainty Estimation for LLM Judges Over Binary Choices

Abstract

'LLM-as-a-judge' systems are increasingly relied upon for numerous tasks, such as labeling training data for AI models, and evaluating LLM generations in industry applications. However, a pervasive problem of these systems is that LLMs exhibit an alarming tendency to reverse their preference over a binary choice, when the presentation order of the choice is inverted. This behavior is patently irrational, and calls into question whether it makes sense for the field to rely on LLM judges, absent a reliable measure of uncertainty over their verdicts. To that end, we propose a simple, scalable, and self-supervised method of training LLM judges to measure their own uncertainty in preferences over binary choices. We show that our method is more accurate than any of our tested baselines, and generalizes surprisingly well across data domains. We also show that our method scales gracefully across model-sizes, and transfers across model families. Finally, we show that our method can enable downstream use-cases, such as adaptive inference compute for more accurate LLM-as-a-judge pipelines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, Uncertainty Quantification, Self-Supervised Learning, Metacognition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vmuFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised uncertainty estimation method for LLM-as-a-Judge systems by using preference reversals under swapped response orders as uncertainty labels. The judge is fine-tuned to predict an additional uncertainty token (U), enabling single-pass uncertainty estimation without human annotations or auxiliary models. Experiments demonstrate improved flip prediction, calibration, and judgment consistency, along with more efficient inference through uncertainty-based routing.

Critical strengths

(1. Novel use of preference flips as self-supervised signals) The paper presents an interesting approach that leverages preference reversals under response-order swaps as self-supervised uncertainty labels. Incorporating these signals into fine-tuning enables single-pass estimation of judgment instability without requiring ground-truth correctness labels. This offers a simple and practical approach to improving the reliability of LLM judges.

(2. Comprehensive empirical evaluation) The paper provides extensive experiments, including comparisons with various uncertainty estimation baselines, calibration analysis, cross-domain and cross-model evaluations, and ablation studies.

Critical weaknesses

(1.Insufficient justification of the scope and significance of preference-flip uncertainty) The Related Work section primarily discusses conventional uncertainty estimation methods, many of which focus on prediction correctness or predictive uncertainty. However, the proposed method specifically targets order-induced preference instability, which does not necessarily reflect conventional uncertainty. For instance, a judge may consistently make incorrect predictions without exhibiting preference flips, while position bias may cause flips even when the model is highly confident in its predictions. Although the paper explicitly defines preference flips as its supervision target, it frequently uses the broader term "uncertainty estimation" without sufficiently clarifying this distinction, potentially misleading readers about the scope of its claims.

I believe the authors should more consistently distinguish preference-flip uncertainty from conventional uncertainty, particularly when interpreting their calibration results. If the authors intend to claim that the proposed method also captures broader judgment uncertainty, they should provide a more convincing conceptual justification or direct empirical evidence demonstrating the relationship between preference flips and judgment correctness. Although the improvements in judging accuracy and the error detection experiments provide indirect evidence, they do not establish whether the proposed uncertainty scores are calibrated with respect to judgment correctness.

Alternatively, if the intended contribution is limited to preference-flip uncertainty, the paper should more clearly justify its significance as a standalone uncertainty measure and articulate what aspects of judgment reliability it captures beyond existing indicators of order inconsistency.

Questions for authors

(1. Clarification of the scope of uncertainty estimation.) Is the proposed (P(U)) intended to estimate only order-induced preference instability, or is it also expected to reflect broader judgment uncertainty, such as the probability of making an incorrect judgment? Since the calibration experiments evaluate preference flips rather than judgment correctness, I would appreciate clarification on how the authors intend their uncertainty estimation claims to be interpreted.

(2. Significance of preference-flip uncertainty.) If the intended scope is limited to preference-flip uncertainty, could the authors further justify its significance as a standalone uncertainty measure? While the paper demonstrates improvements in judgment consistency and downstream applications, I would like to better understand why estimating preference-flip uncertainty is important in its own right and what practical value it provides beyond conventional uncertainty measures or existing approaches to handling order inconsistency.

(3. Connection to broader judgment uncertainty.) If the authors intend the proposed method to capture broader judgment uncertainty beyond preference flips, could they provide more direct empirical evidence and theoretical insights supporting this connection? In particular, I would be interested in a principled explanation of why preference-flip uncertainty should reflect judgment correctness, including any assumptions under which such a relationship might hold. Additionally, evaluating correctness-based uncertainty estimation on RewardBench, including error-detection performance and calibration where appropriate, would help substantiate the broader claims.

3 comments

Thank you for an exceptionally careful review. You laid out the choice before us clearly: either justify a broader claim about correctness, with direct evidence and a principled explanation, or justify flip uncertainty on its own terms. That framing shaped most of the revision. We ended up doing both, and the first held up better than we expected. The new Sec. 4.3 and 4.4, and Appendices B.2 and E, follow your suggestions closely, including the RewardBench evaluation you proposed. CW1 and Q1 (Scope of the uncertainty estimate)

[...] the proposed method specifically targets order-induced preference instability, which does not necessarily reflect conventional uncertainty. For instance, a judge may consistently make incorrect predictions without exhibiting preference flips, while position bias may cause flips even when the model is highly confident in its predictions. [...] Is the proposed (P(U)) intended to estimate only order-induced preference instability, or is it also expected to reflect broader judgment uncertainty, such as the probability of making an incorrect judgment?

Ans: Both, now kept clearly apart. P(U) is trained only to predict order-induced flips. The revised PDF (changes in blue) is now consistent: we report "flip-AUROC" for that target and the new "correctness-AUROC" for predicting incorrect verdicts. The link to correctness is now empirical with an explanation, rather than implied (see Q3).

The revision now measures both of your examples. For Qwen2.5-7B, 53% of RewardBench errors come from pairs that never flip, and for Qwen2.5-32B that was 73% (Appendix E). Confident flips also turn up often. Even when Qwen2.5-7B puts more than 95% probability on its chosen answer, it flips 34% of the time, barely below its overall rate of 37% (Appendix L.4). We would argue these confident flips still carry information. When one response in each pair is correct, every flipped pair contains exactly one incorrect verdict, so in one of the two orders the judge was confidently wrong. Trained P(U) tends to catch this: within flipped pairs, it is higher on the incorrect verdict about two thirds of the time (Appendix E).

Q2 (Significance of preference-flip uncertainty)

If the intended scope is limited to preference-flip uncertainty, could the authors further justify its significance as a standalone uncertainty measure? [...] what practical value it provides beyond conventional uncertainty measures or existing approaches to handling order inconsistency.

Ans: Three reasons, beyond its link to correctness:

  1. Self-labeled preference data. When a judge labels its own preference pairs (RLAIF), flipped pairs are noisy labels. Filtering by P(U) improves DPO outcomes over unfiltered self-labeling under all six reward-model evaluators (Appendix O).
  2. Domains without a correct answer. For preferences over art or writing style, correctness is undefined, but response to presentation order is still measurable, and P(U) estimates it without labels (Appendix D). Flip labels also cover pairs where both responses are poor, which correctness labels cannot.
  3. Versus checking both orders. The standard remedy for order inconsistency is to query both orders. Even with prefix caching this costs 1.8 to 1.9 times a single call (see our reply to Reviewer emJV), and it yields only a binary flip. P(U) gives a graded, calibrated estimate (ECE 0.018) in one call, which supports routing at a chosen compute budget (Sec. 4.10). Training also makes the judge itself more consistent (flip rate 37.4% to 23.1%, Sec. 4.9), and P(U) still predicts the trained judge's remaining flips (0.816 flip-AUROC, App. L.2).

Q3 (Connection to broader judgment uncertainty)

[...] I would be interested in a principled explanation of why preference-flip uncertainty should reflect judgment correctness, including any assumptions under which such a relationship might hold. Additionally, evaluating correctness-based uncertainty estimation on RewardBench, including error-detection performance and calibration where appropriate, would help substantiate the broader claims.

Ans: Both are now in the paper.

Principled explanation (Sec. 4.4, Appendix E). Assume exactly one response in each pair is correct. A flipped pair then holds exactly one incorrect verdict, and a consistent pair zero or two. If either order is equally likely to hold the incorrect verdict of a flipped pair, a call is incorrect with probability e = ½ρ + β. Here ρ is its flip probability and β the probability that its pair is wrong in both orders. Ranking verdicts by flip probability therefore ranks them by error probability, as long as consistent errors do not become rarer as flips become likelier. We check this on RewardBench. Across ten bins of P(U), the error rate is close to a line in the flip rate (96% of variance; slopes 0.90 for Qwen2.5-7B and 1.29 for 32B), and consistent errors rise with the flip rate rather than fall. The equal-chance assumption also departs in our favor, as noted under Q1. Verifying the condition needs gold labels, so flip data alone cannot certify it for a new judge, and our paper now says so.

Error detection on RewardBench (Sec. 4.3, App. B.2). P(U)'s correctness-AUROC exceeds the untrained judge's A/B margin on all four judges tested: Qwen2.5-7B 0.777 vs 0.737, Llama-3.1-8B 0.705 vs 0.650, Qwen2.5-14B 0.785 vs 0.747, and Qwen2.5-32B 0.833 vs 0.809. On 7B it ties correctness-trained Self-REF and beats five other correctness-trained estimators. On 14B and 32B, Self-REF leads by 0.02.

Calibration. P(U) is calibrated as a flip probability (ECE 0.018), not as an error probability. Because the error rate is close to linear in the flip rate across P(U) bins, a calibrated error probability needs only a monotone remapping, but fitting it requires correctness labels.

Thank you for the detailed response and revisions. I appreciate that the authors have substantially addressed my original concerns by clarifying the distinction between preference-flip uncertainty and judgment correctness, providing additional correctnessbased evaluations, and demonstrating the practical utility of flip uncertainty.

These additions strengthen the paper’s empirical contributions. I have decided to raise my rating from 2 (Weak Rejection) to 3 (Weak Acceptance). However, I want to emphasize that my assessment remains firmly borderline. This change does not indicate a clear endorsement of acceptance, but rather a slight preference for Weak Acceptance over Weak Rejection.

I still have two reservations:

1. Theoretical justification: The analysis in Section 4.4 relies on the assumption that consistent errors do not become less frequent as flip probability increases. While empirically supported on RewardBench, it remains unclear why this relationship should hold across different distributions or judges.

2. Methodological novelty: The main difference from prior work such as Self-REF is the use of self-supervised preference-flip labels instead of correctness labels. Despite the compelling empirical results, I remain uncertain whether this constitutes a sufficiently substantial conceptual contribution.

Overall, I view this paper as a close borderline case rather than a clear acceptance recommendation, and leave the final assessment of its novelty and significance to the Area Chair.

Reviewer_dGuLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for estimating the uncertainty of judges-LLM in the binary verdict case. (Lack of) confidence is equated with the probability that the judge will flip its decision if the alternatives A and B are swapped in the prompt. A LoRA adaptor is trained on self-supervised data obtained from a base judge model (stable vs. flipping pairs) using a specific token (‘U’) as the correct verdict to output if the model is uncertain. Experiments confirm the effectiveness of the training, its OOD generalization, and a neutral to positive effect on general capabilities.

Critical strengths
  • The proposed method seems highly effective for predicting when a judge would flip its decision if presented with alternatives in the inverted order, which may be a suitable proxy for uncertainty estimation in many cases.
  • Several experiments address legitimate questions about the method, with experiments covering multiple model families and sizes.
Critical weaknesses
  • There is a mismatch between what is claimed in the title (a method for estimating uncertainty on a decision) and what is delivered (a method that predicts whether labels would flip if the alternatives were reversed). These are different questions. A judge could well give the same answer irrespective of the order of alternatives, but have varying degrees of uncertainty about the correctness of the answer. This mismatch is not clearly discussed in the main body of the paper.
  • There is no measurement of the calibration between the proposed uncertainty metric p(U) and the error rate in the binary A/B decision, which is ultimately what a confidence/uncertainty estimation metric is expected to deliver.
  • The method is claimed not to require training an additional model, but LoRA fine-tuning an adapter is not much different from training a probe, in practice.
Questions for authors
  • The uncertainty measure in Eq. 4 is independent of the relative magnitude of p(A) and p(B). Is this desirable? For the same value of p(U), shouldn’t a verdict with P(A) = P(B) not be more uncertain than one with P(A) >> P(B)?
2 comments

Thank you for the great suggestions. Individually:

CW1

There is a mismatch between what is claimed in the title (a method for estimating uncertainty on a decision) and what is delivered (a method that predicts whether labels would flip if the alternatives were reversed). These are different questions. A judge could well give the same answer irrespective of the order of alternatives, but have varying degrees of uncertainty about the correctness of the answer. This mismatch is not clearly discussed in the main body of the paper.

Ans: Agreed, and thank you; this was the central gap. The updated PDF (changes in blue) now separates the two notions throughout. We report "flip-AUROC" for predicting order reversals and the new "correctness-AUROC" for predicting wrong verdicts, and the Introduction now names both. New Sec. 4.3 measures correctness directly on RewardBench. New Sec. 4.4 addresses your exact case, a judge that is consistent but wrong. Such consistent errors are common (53% of Qwen2.5-7B's errors, 73% of 32B's). P(U) still ranks errors well because consistent errors also become more frequent as flip probability rises (Appendix E). Given these results, we believe "uncertainty estimation" in the title now holds in both senses, but we are happy to adjust the title if the reviewers or AC prefer.

CW2

There is no measurement of the calibration between the proposed uncertainty metric p(U) and the error rate in the binary A/B decision, which is ultimately what a confidence/uncertainty estimation metric is expected to deliver.

Ans: We now measure this (new Sec. 4.3, Appendix B.2, Appendix E). On RewardBench, P(U) ranks Qwen2.5-7B's incorrect verdicts above correct ones with correctness-AUROC 0.777. That ties Self-REF, which is trained on correctness labels (0.775), and is above five other correctness-trained estimators. Fig. 3 shows the accuracy of the verdicts kept, as coverage shrinks: the most confident half is 94.8% accurate, against 90.9% for the untrained judge. On calibration specifically, P(U) is calibrated as a flip probability (ECE 0.018, Sec. 4.2), not as an error probability. Across ten P(U) bins, the error rate is close to linear in the flip rate (96% of variance explained, Appendix E, Fig. 10). A calibrated error probability therefore needs only a simple monotone remapping, but fitting it requires correctness labels, which our method deliberately avoids.

CW3

The method is claimed not to require training an additional model, but LoRA fine-tuning an adapter is not much different from training a probe, in practice.

Ans: A fair point about training: fitting a LoRA adapter costs about as much as fitting a probe, and we have revised the wording accordingly. The conclusion now says the method "requires only inexpensive fine tuning using a LoRA adapter". Our claim concerns inference. The adapter can be merged into the judge's weights, so deployment is one model and one forward pass that returns the verdict and P(U) together, with no second network or head. The adapter also improves the judge itself, which a probe on frozen states cannot: RewardBench accuracy rises from 79.7% to 83.0% (Sec. 4.7), and the flip rate falls from 37.4% to 23.1% (Sec. 4.9). It also outperforms probes trained on the same flip labels: 0.842 flip-AUROC, against 0.798 for the Radharapu et al. probe and 0.768 for a linear residual probe (Fig. 1).

Question

The uncertainty measure in Eq. 4 is independent of the relative magnitude of p(A) and p(B). Is this desirable? For the same value of p(U), shouldn't a verdict with P(A) = P(B) not be more uncertain than one with P(A) >> P(B)?

Ans: A natural intuition, and we tested it (new Appendix G). On its own, a small A/B gap does flag the judge's errors: on RewardBench with Qwen2.5-7B, the relative gap |P(A) − P(B)| / (P(A) + P(B)) ranks incorrect verdicts with AUROC 0.749, against 0.777 for P(U). Once P(U) is known, however, the gap adds nothing. Within each decile of P(U), it ranks incorrect verdicts at or slightly below chance (AUROC 0.48 over three adapter seeds: 0.472, 0.480, 0.477). In other words, P(U) already carries the information in the A/B split.

We also tested seven scores that combine P(U) with the A/B split, such as dividing P(U) by the gap, or adding the smaller of P(A) and P(B) (App. G, Table 8). The best improve error AUROC by at most 0.001, with intervals containing zero. The rest do worse: multiplying P(U) by (1 − gap) loses 0.015, and entropy over A, B and U loses 0.021. Consistent with this, training already reshapes the A/B margins: even with U removed from the prompt, the margin rises on pairs the judge handles consistently and falls on pairs where it flips (Sec. 4.9). We therefore keep Eq. 4 as defined.

Thanks to the authors for performing the additional experiments showing the strong alignment between flip uncertainty and correctness uncertainty. Since this was my main object of concern, I am increasing my score to a weak accept.

Reviewer_emJVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes a self-supervised method for estimating the uncertainty of LLM judges on pairwise preference judgments. A pair is considered uncertain when the judge’s preference flips after the two responses are presented in reversed order. These flip labels are used to LoRA fine-tune the judge on an extended A/B/U prompt, so that a single forward pass yields both the verdict and an uncertainty score, P(U), without requiring an auxiliary model or human labels. On held-out UltraFeedback comparisons, P(U) predicts flips with 0.842 AUROC, outperforming training-free and trained baselines, and is well calibrated. The authors further claim that it transfers zero-shot to GSM8K, TriviaQA and HH-RLHF, works across model sizes (1.5B–32B), families (Llama, Mistral) and modalities, and preserves judging accuracy and general benchmark performance.

Critical strengths
  • S1: According to the proposed evaluation, the method performs very well across different model families and sizes. It outperforms the baselines and generalizes reliably beyond the training domain (Table 2 and Figure 3). In addition, it does not degrade accuracy on standard benchmarks compared to the original model (no catastrophic forgetting).
Critical weaknesses
  • W1: Missing correctness-based evaluation: While the evaluation appears comprehensive, it does not directly assess performance with respect to correctness. The AUROC and calibration evaluations focus solely on the binary flip target, which is introduced in this work. However, this target does not necessarily align with downstream performance. Evaluating how well P(U) detects whether the LLM’s preference is correct (correctness-AUROC), and comparing it to other methods on the same task, is a crucial missing component. I understand that correctness labels are expensive to obtain for training, but they remain essential for evaluation to establish the practical utility of the method, which is still an open question.
  • W2: Limited novelty and significance:
    • The motivation for the training-based approach is not convincing: The authors insist on avoiding multiple LLM calls. However, directly observing the flip target via two inference calls adds negligible overhead compared to the trained P(U) at inference time.
      • The first forward pass is needed in both cases. It is actually cheaper when observing the flip directly, because the prompt does not contain the extra instruction about outputting the token U.
      • The second forward pass is a single forward pass with no autoregressive generation, since only the first generated token is needed.
      • The KV cache can also be reused for the tokens preceding the two options in the prompt, further reducing the cost of this second pass.
      • Grouping this approach with methods such as paraphrase consistency under “more than one model call” understates the large difference in inference cost between them. Directly observing the target at inference time is therefore much cheaper in practice than the paper suggests. It also avoids the training overhead and trivially achieves perfect AUROC and calibration on the flip target across all tasks in a zero-shot manner, which again highlights that the flip-target evaluation alone is incomplete. As a result, training a model to predict this target is only weakly motivated.
    • Lack of novelty: Fine-tuning dedicated confidence tokens with LoRA adapters and using them for query routing is already established in the literature [1]. The main difference from this prior work (Self-REF) is the use of the flip label instead of correctness as the fine-tuning target. Moreover, the results are largely expected and offer few new insights or conclusions for the community. Overall, this work is incremental.

[1] Chuang, Y., Sarma, P.K., Gopalan, P., Boccio, J., Bolouki, S., Hu, X. & Zhou, H. (2025). Learning to Route LLMs with Confidence Tokens. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:10859–10878. https://proceedings.mlr.press/v267/chuang25b.html

Questions for authors

Which model was used in the main experiments (Figure 1, Figure 3 and Table 2)? Based on the Figure 2 caption and Appendix C (line 975), it seems safe to infer that it is Qwen2.5-7B-Instruct. However, this should be stated explicitly in the corresponding figures and table, as they present the main results of the paper, establishing superior performance over the baselines and generalization to other domains.

3 comments

Thank you for the detailed review. Individually:

W2 (Lack of novelty)

Fine-tuning dedicated confidence tokens with LoRA adapters and using them for query routing is already established in the literature [1]. The main difference from this prior work (Self-REF) is the use of the flip label instead of correctness as the fine-tuning target. Moreover, the results are largely expected and offer few new insights or conclusions for the community. Overall, this work is incremental.

Ans: Thank you for pointing us to Self-REF. We agree it is the closest prior work, and that fine-tuning confidence tokens with LoRA and routing on them is established. We do not claim that mechanism. The revision now cites Self-REF (Sec. 2.2) and compares against it directly (Sec. 4.3, 4.10; Appendix B.2, M).

We also agree that the key difference is getting away from needing labels, and that difference is our contribution. Flip labels are self-supervised. This matters most for preference judging, where correctness only comes from human annotations. It also lets our method apply in open-ended domains with no single correct answer.

We would argue the outcome is not expected. Most of the judge's errors are consistent errors that never flip (53% on Qwen2.5-7B, 73% on 32B), yet a score trained only on flips predicts correctness about as well as one trained on correctness. We are grateful to you for prompting this finding! On RewardBench, P(U) matches [Self-REF] on Qwen2.5-7B (whose hyperparameters were tuned on that model), while our judge is significantly more accurate (83.0% vs 82.1%, 95% CI [0.5, 1.3]). Ours comes within 0.02 on 14B and 32B, and beats five other correctness-trained estimators. It does this without labels, using under 20% of Self-REF's GPU-hours once its hyperparameter search is counted. As a router, it is at least as accurate as Self-REF over most of the compute range (Fig. 9). Sec. 4.4 states the condition under which flips predict errors and checks it on RewardBench.

One design difference may also interest the community: our U token competes with the verdict at the same position, rather than following a generated answer. This is the placement raised as an open question in Self-REF's own reviews, and our results show it works for binary judgments.

W1 (Missing correctness-based evaluation)

Evaluating how well P(U) detects whether the LLM's preference is correct (correctness-AUROC), and comparing it to other methods on the same task, is a crucial missing component. I understand that correctness labels are expensive to obtain for training, but they remain essential for evaluation to establish the practical utility of the method, which is still an open question.

Ans: Agreed. New Sec. 4.3 and 4.4 now evaluate correctness directly, with the results summarized in our answer above. Beyond correctness, flip prediction has its own practical use. When a judge labels its own preference data (RLAIF), flipped pairs are noisy labels. In Appendix O (previously M), filtering self-labeled DPO data by P(U) beats unfiltered self-labeling under all six reward-model evaluators, and the P(U) filter ranks first by mean rank.

W2 (Motivation for the training-based approach)

The authors insist on avoiding multiple LLM calls. However, directly observing the flip target via two inference calls adds negligible overhead compared to the trained P(U) at inference time. [...] The KV cache can also be reused for the tokens preceding the two options in the prompt, further reducing the cost of this second pass.

Ans: The KV cache is helpful, but not that helpful. On 2,000 RewardBench and 2,000 UltraFeedback calls, that is 30% of the prompt tokens. The other 70% must be computed again (ASCII Figure below). So querying twice computes 1.7 times as many prompt tokens as querying once. Needing only one call is attractive now, and holds promise for tasks with more than two multiple-choice answers.

RewardBench
  first order    ###################################                    418
  swapped order  ..........#########################                    123 + 294  (30% fewer)

UltraFeedback
  first order    ######################################################  644
  swapped order  ................######################################  197 + 446  (31% fewer)

                 . reused   # computed
                 Prompt tokens per call (mean over 2,000 calls)

Question

Which model was used in the main experiments (Figure 1, Figure 3 and Table 2)? [...] this should be stated explicitly in the corresponding figures and table, as they present the main results of the paper.

Ans: Qwen2.5-7B-Instruct. Thank you; we now state this explicitly in the captions of Fig. 1, Fig. 4 (previously Fig. 3) and Table 2.

I thank the authors for their effort in the rebuttal. In particular, I appreciate them adressing W1 by including an evaluation against correctness which is cruicial for the soundness of the paper. I also appreciate their reactiveness to the proposed baseline. Including comparison to Self-REF especially definitely makes the experimental part more honest. As a result, I will raise the soundness score. However, I am still not convinced by the motivation of the training-based approach. A 1.7 speedup at inference time (where the unit here is 1 non-autoregresive LLM forward pass to generate a single token) at the cost of extra finetuning is only incremental. The novelty compared to Self-REF is also limited. The paper is more of a nice-to-have observation than a novel and significant finding. Since W2 was the main reason behind my rating, I will stand by my clear rejection recommendation.

AuthorsOpenReview ↗

Clarification

Thank you to Reviewer emJV for continuing to put care into our submission/rebuttal. We appreciate the time you took to go through our revised paper, and for your response.

We do think a few points in particular are worth clarifying here, for the sake of posterity, and for the convenience of the Area Chair:

The reviewer sees a 1.7x speedup at inference time as “only incremental”. Very respectfully, we would push back on this. A 70% speedup over any baseline is significant if the training overhead can be amortized.. The reviewer states that “novelty compared to Self-REF is also limited”. We would also push back against this, again with the greatest respect. Self-REF requires ground-truth labels of correctness. Our method requires none so works on more tasks, as we train in a completely self-supervised manner. Our aims may be the same, but our method is different both conceptually and practically. Further, our method requires 80% less GPU hours than Self-REF to train, and beats Self-REF in a number of settings, including Self-REF’s flagship setting of routing based on confidence.

Given the clear differences in method, the 80% compute saving, the 70% speed-up at inference time, and the superior performance of our method versus the competition, we are curious that they settle on viewing our method as “more of a nice-to-have observation than a novel and significant finding”. Objectively though, this reviewer's suggestion has helped us massively to validate our approach on a second and adjacent task, so we are sincerely grateful to them for the specific push.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author response summary

Beyond the typical gratitude for the reviewers' time and suggestions, we are genuinely excited to report how their specific pointers have doubled the impact of our core approach. What was originally a gulf between two forms of uncertainty, is now reconciled in the updated pdf, with a much-needed experiment, [Self-REF] as the missing link, and an explanation of why it works! We hope the reviewers can spare one more look, because they have nudged us to make a very nice discovery. If you have limited time, please just see at least the new Sec. 4.3 and 4.4.

The crux is 2 types of uncertainty: All 3 reviewers conveyed clearly that conventional "uncertainty" is the norm, and they prefer that the term should be used for prediction correctness: probability of the LLM's judgment agreeing with the (one) correct label. We came from the self-supervised and human-preference setting, and introduced preference-flip uncertainty. The two are not the same, and the reviewers correctly called us out for that and for missing [Self-REF], which is an excellent bridge here.

A concrete example for the Area Chairs: What color is the dress's lace? A) Gold or B) Black? We introduced the notion of flip-uncertainty, to answer the question, "What's the probability that the LLM would change its mind and choose the other color if we reverse (flip) which color option is (A) versus (B)?" But we neglected that conventional uncertainty sticks to one ordering, so asks "What's the probability of the LLM choosing Black? (Given one phrasing of the options and given that the correct answer is black lace.)

The exciting thing we learned by now adding a correctness experiment, with [Self-REF] included, is that the two uncertainties are linked in practice! Preference-flip uncertainty is a great indicator of correctness uncertainty! And correctness uncertainty derived our way matches Self-REF on Qwen2.5-7B, and comes within 0.02 AUROC on 14B and 32B, without needing correctness labels and at under 20% of its GPU-hours (counting its hyperparameter search).

We have included the two needed experiments and made the critical text revisions in blue, in the updated pdf.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.