- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies certified LLM judging when no candidate response may be acceptable, adding a “no-winner” label alongside response selection. It proposes a two-stage method that first localizes plausible labels via conformal prediction and then performs decision-specific certification. Experiments show gains over a one-stage baseline, but mainly on the synthetic data.
Critical strengths
1, The paper addresses a practically relevant problem by explicitly considering no-winner outcomes in LLM judging and separately controlling the reliability of winner and no-winner decisions.
2, The paper is clearly written and well-structured, with theoretical guarantees and extensive experiments across multiple datasets and LLM judges.
Critical weaknesses
1, The theoretical contribution appears somewhat incremental, see question 1 below.
2, Assumption 2 excludes ties among acceptable responses, even though such cases are common in open-ended evaluation and are mentioned in Figure 1(a). The conformal prediction set in Stage I represents uncertainty over a single correct label rather than multiple valid winners. Moreover, GOOD–GOOD pairs are excluded from the experiments (Appendix C), raising concerns about generalizability to real-world settings. The authors should justify this restriction and discuss how the framework could accommodate acceptable ties.
Questions for authors
1, Proposition 1 follows from standard conformal prediction, while Propositions 2–3 rely on existing statistical techniques, including Clopper–Pearson bounds and fixed-sequence testing. Moreover, the branch-specific guarantees do not depend on Proposition 1, suggesting that conformal localization is not essential for risk control. The authors should better clarify the theoretical novelty of the proposed two-stage framework.
2, Could the authors provide ablations that separately evaluate shortlist pruning, no-winner-first routing, and localized scoring?