acceptodds
Under review as a conference paper at ICLR 2027

Certifying No-Winner Decisions in Large Language Model Judging

Abstract

Large language models (LLMs) are increasingly used as automatic judges of response quality and preference alignment. Existing guarantee-based evaluation frameworks typically assume that each instance has a unique human-preferred response, an assumption that often fails in open-ended settings where candidates may be similarly poor leaving no meaningful winner. We study reliable LLM judgment under this more general setting. Allowing no-winner outcomes introduces a key challenge: confidence scores can conflate evidence about absolute response quality, relative preference, and candidate ambiguity, complicating the identification of reliable no-winner decisions. Moreover, controlling overall selective risk does not ensure reliability separately for no-winner and response-selection decisions. We address this challenge with a two-stage certified selection framework. First, conformal prediction constructs a localized label set over candidate responses and the no-winner label, retaining the reference label with a distribution-free, finite-sample marginal coverage guarantee under exchangeability. Second, conditioned on the localized set, separate calibrated hypothesis tests certify either a no-winner or response-selection decision. The resulting framework provides high-probability selective correctness guarantees while abstaining or escalating when evidence is insufficient. Extensive experiments demonstrate the benefits of our two stage framework for no-winner recognition and characterize its effectiveness across controlled and human-rated evaluation settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Alignment, Judge

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QL8wOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies certified LLM judging when no candidate response may be acceptable, adding a “no-winner” label alongside response selection. It proposes a two-stage method that first localizes plausible labels via conformal prediction and then performs decision-specific certification. Experiments show gains over a one-stage baseline, but mainly on the synthetic data.

Critical strengths

1, The paper addresses a practically relevant problem by explicitly considering no-winner outcomes in LLM judging and separately controlling the reliability of winner and no-winner decisions.

2, The paper is clearly written and well-structured, with theoretical guarantees and extensive experiments across multiple datasets and LLM judges.

Critical weaknesses

1, The theoretical contribution appears somewhat incremental, see question 1 below.

2, Assumption 2 excludes ties among acceptable responses, even though such cases are common in open-ended evaluation and are mentioned in Figure 1(a). The conformal prediction set in Stage I represents uncertainty over a single correct label rather than multiple valid winners. Moreover, GOOD–GOOD pairs are excluded from the experiments (Appendix C), raising concerns about generalizability to real-world settings. The authors should justify this restriction and discuss how the framework could accommodate acceptable ties.

Questions for authors

1, Proposition 1 follows from standard conformal prediction, while Propositions 2–3 rely on existing statistical techniques, including Clopper–Pearson bounds and fixed-sequence testing. Moreover, the branch-specific guarantees do not depend on Proposition 1, suggesting that conformal localization is not essential for risk control. The authors should better clarify the theoretical novelty of the proposed two-stage framework.

2, Could the authors provide ablations that separately evaluate shortlist pruning, no-winner-first routing, and localized scoring?

Reviewer_JCcEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a guarantee-based evaluation framework for using large language models as automatic judges. It suggests that no-winner outcomes should be allowed, because sometimes all candidates are of bad quality and forcing a winner in that case can create artificial preference labels and overstate evaluation reliability. The proposed framework includes two stages. First, a conformal localization stage builds a shortlist of candidate outcomes, which may contain every candidate responses and the no-winner outcome. Second, a branch-specific fixed-sequence testing stage certifies a winner as the human-best response or a no-winner outcome depending on evidence strength. This work demonstrates how high error in no-winner outcomes can be hidden by a pooled risk control. Evaluations are done on simulations, two human-rated benchmarks, and a synthetic benchmark to show the effectiveness of this framework for no-winner recognition.

Critical strengths
  • The problem is real and the proposed branch-wise protocol is clearly motivated. The no-winner cases do exist, especially in open-ended settings. The given example of hidden error on no-winner outcomes is simple and useful.
Critical weaknesses
  • The claimed benefit on no-winner recall is not well supported by the evaluation results, especially on human-rated benchmarks. The main method, which uses branch-risk budgets, only shows good results on the synthetic Arena-Controlled dataset. In separate branch-risk evaluation, both methods have zero accepted no-winner recall on TL;DR, while HelpSteer2 is either not evaluated (with Flash judge) or not reported (with open judges).

  • More experiments are needed to cover different judge cases. Fig. 1 and surrounding texts mention challenges that also include weak margins and human ambiguity, and in the problem setup m refers to the amount of candidate responses for comparison. However, in the evaluation, m is set as 2 across all experiments, and only BAD–BAD (no-winner) and GOOD–BAD pairs are considered, representing a simplified special case of the problem. More tests on different harder cases would help justify the effectiveness of the proposal.

  • Whether the guarantees can hold on human-rated data is unclear. In Table 2, for example, with Gemini Flash on TL;DR with all methods, the guarantee success rates are very low. There are also low GSRs for HelpSteer2. It would help to explain why these numbers are low and their implications on the effectiveness of the proposal.

  • The evaluation lack ablation study on the effectiveness of Stage I. The main baseline method in this paper is One-stage H. While the paper proposes a two-stage framework, the branch-wise protocol can be applied to the raw scores without Stage I's shortlisting, and no abalation study is conducted to assess the benefits of Stage I.

Questions for authors
  1. Can you provide more results and analysis that explain the low GSRs, such as 0% GSR for Gemini on TL;DR?

  2. Can you provide results for the branch-wise protocol for HelpSteer2 with open judges?

  3. How does separate branch-risk evaluation, directly applied on raw scores without Stage I's shortlisting, compare with the proposed full two-stage method?

  4. How do results change with α, m, and other parameters?

Reviewer_PbJJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper extends selective LLM-as-a-judge evaluation with probabilistic guarantees to a label space that includes an “all-bad” (i.e., no-winner) outcome, apart from abstention.

The method works in two stages. First, the judge’s scores are turned into a short list of plausible labels. Second, using held-out labeled data, a cutoff is picked to determine the system's output (e.g., the preferred output, "all-bad" or obstain), so that a finite-sample guarantee holds.

Experiments use response pairs, six open-weight judges and Gemini-2.5-Flash. The main gain seems to be on the synthetic data benchmark, where accepted all-bad recall rises from 15.5% to 46.4% at a 30% error target.

Critical strengths

It is an interesting idea. The authors are also candid about which results are retrospective or synthetic, which I appreciate.

Critical weaknesses

Conceptually/philosophically, I see limited novelty of the "all-bad" concept beyond the already-well-studied "tied-response" concept. (That is, the judge will say that both responses are "equally" good/bad.) Not to mention there is a separate "abstain" concept that also captures part of subtlety of "no clear winner".

On the technical/engineering side, this paper only deals with evaluation itself. It would be nice to see how this can be actually applied to the fine-tuning/alignment pipeline.

Empirically, it would also be better to see what is the real benefit on human-generated data, where the motivation of "evaluation on open-ended problems" can be tested more directly?

Questions for authors

What is the true material advantage of this method on human-generated data, especially where the questions are intrinsically open-ended (and hence the concept of "all-bad" may not even be well defined)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.