- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to solve the problem of how to reduce the computational cost of repeated LLMs judgements while preserving the collective decision of the LLM judgements when run on the full suite. To this end, the authors introduce paired unanimity stopping, which allows the judge to stop evaluations early when two order-balanced judgements agree unanimously. The paper also shows theoretical guarantees that allow paired unanimity stopping to allow decision preservation and variance. 8 LLM judges were used across a suite of experiments, showing up to a 46% reduction in evaluation LLM calls, while changing at most 1.7% of reference decisions compared to a full evaluation.
Critical strengths
- Strong theoretical analysis The paper tackles a relevant problem in efficient LLM judgements and also provides rigorous proofs of their approach. In particular, it makes an important distinction between preserving average evaluation scores and preserving individual LLM decisions.
- Strong experimental coverage The paper provides extensive experimentation. The authors evaluate their approach across several LLM judges, benchmarks and reasoning configurations. They show that their method is superior compared to existing methods and measure realistic serving-time improvements over these previous methods. The results provide evidence that the approach can reduce evaluation costs while largely preserving original decisions.
Critical weaknesses
- Limited novelty The proposed stopping rule is fairly simple and is equivalent to existing adaptations of ESC[1] and Adaptive-Consistency[2]. From what I can see, the main novelty is just from the theoretical analysis of the existing methods.
- Limited generalizability and practical significance Most of the theoretical guarantees focus exclusively on a special case (4 call evaluations) making it unclear whether these findings can scale to larger budgets. More importantly, the paper focuses on reproducing decisions from a four-call reference which by itself could be noisy, which leads to doubts on whether this can improve (or at least retain) evaluation quality.
[1] ESC-Judge: A Framework for Comparing Emotional Support Conversational Agents (EMNLP '25) [2] Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs (EMNLP '23)
Questions for authors
- In practical scenarios, when would repeated judgements be preferred over evaluating more questions? Wouldn't evaluating a larger number of questions (even just once) provide more reliable aggregate scores or model rankings than repeatedly judging the same questions? I'm not sure whether this is the best use of limited computational budget. Can you provide practical use-cases for when this would be useful?
- Why should we preserve potentially incorrect reference decisions? If reference decisions can be noisy or incorrect, why should matching them to the primary objective be preferred instead of improving agreement with the ground truth?
- What is fundamentally new compared with existing stopping methods beyond the theoretical characterization of a special case with four calls? If not, I feel like these methods should be explicitly mentioned in the main text and that this is a theoretical analysis of the existing methods, rather than a fundamentally new method.
- Is this method generalizable (empirically or theoretically) to larger evaluation budgets for call judgements?
5 comments
3/ What is new beyond ESC and Adaptive-Consistency (Question 3 & Weakness 1). Standard self-consistency stopping methods (Early-Stopping Self-Consistency (ESC, Li et al., ICLR 2024) and Adaptive-Consistency (AC, [2] Aggarwal et al., EMNLP 2023)) were designed for repeated sampling on a fixed input. They do not account for the two defining structural properties of pairwise judging: (1) display-order asymmetry and (2) 0.5-credited neutrality. Applying existing stopping rules without our structural constraints introduces clear failure modes.
-
Failure of naive adaptations: Consider individual-call AC(0.80) as an example. The criterion would trigger stops at a 2:0 state of decisive votes, which can occur on two calls from the same display order. On a purely position-biased neutral reference, AC(0.80) returns a decisive label with 1/3 probability under uniform call ordering and the individual non-strict boundary does so on every ordering, but paired unanimity flips none of such reference panels. Conversely, if stopping takes place on any cross-order pair with a nonzero net vote (without requiring two decisive calls), then expected-score preservation is violated on cases such as (A, TIE) and (B, B) by returning A early when full execution chooses B.
-
Two-stage formulation and exact guarantees: Existing stopping methods take a single sampling distribution as given. Our work first resolves reference selection prior to execution, which establishes the exact display-order composition law for decisive call outcomes under equal marginal exposure (Proposition 1). On the execution level, our work formalizes score vs. decision preservation and proves the exact variance-disagreement identity for arbitrary reference sizes (Section 2.3 & Equation 8), alongside our sharp four-call guarantee that paired unanimity has at most half the disagreement probability of the individual non-strict boundary on every balanced reference (Proposition 4). With cross-order pairing, decisive-only unanimity, and uniform random matching and ordering, four-call ESC and AC adaptations coincide with paired unanimity and inherit this guarantee.
Following your suggestion, we will point out this equivalence directly alongside Algorithm 1 in Section 4.1 and clarify what cross-order adaptations preserve. We will also spell out ESC at first mention so it is not confused with ESC-Judge [1].
4/ Scaling beyond four calls (Question 4 & Weakness 2). Four calls constitute the minimal order-balanced reference panel that incorporates within-order seed replication. Based on this, we establish our closed-form composition law and sharp half-disagreement bound. Beyond , our preservation hierarchy (Section 2.3), variance-disagreement identity (Equation 8), and individual-call boundaries (Proposition 3) remain applicable for arbitrary call budgets . Empirically, across the same reasoning configurations on 6- and 8-call references (Appendix D.5), the non-strict boundary preserves expected scores at median costs of 3.60 and 4.87 calls (~40% call savings), while the individual strict boundary preserves all reference decisions at 4.66 and 5.94 calls (~22%-26% savings). We also evaluated its paired strict counterpart on the same recorded calls by checking the condition only after each completed cross-order pair (), which likewise preserves all reference decisions at median costs of 4.90 and 6.59 calls (~18% savings). Waiting to complete each cross-order pair uses slightly more calls than stopping on odd steps, but keeps display orders balanced at every checkpoint and requires at most 3-4 parallel pair rounds (vs. up to 6-8 rounds under serial individual-call dispatch).
We will incorporate these multi-budget paired results and clarifications into the revision.
Update: Manuscript revised
Dear Reviewer A2ut,
We have uploaded the revised PDF to address your comments. Section 6 now spells out Early-Stopping Self-Consistency and states which of its cross-order adaptations coincide with paired unanimity. For larger budgets, Section 4.1 and Appendix D.4 now include paired strict stopping, which preserves every reference decision and saves a median of 18% of calls at six and eight calls. On why preserving reference decisions matters, Section 5.4 now includes downstream results: in HelpSteer2 at matched judge calls, paired unanimity needs 36%-42% fewer human reviews for the same number of wrong binary labels, and in accept/defer curation it accepts fewer wrong labels than supervised mixtures in all 17 configurations.
New content is marked in blue, which we summarized in our top-level note too. Please let us know if any questions remain. We would greatly appreciate your feedback and reconsideration. Thank you very much for your effort!
Thank you for recognizing both our theoretical analysis and experimental coverage. We also used bold and italic fonts to facilitate your skimming of our response.
Stepping back, we found a common question that your four questions point to, which is "Once judge calls vary across display orders and permit neutral outcomes credited as 0.5, how is per-item decision making different from leaderboard score estimation?". In short, equal expected scores can still conceal per-item decision flips and excess variance. Addressing this connects all four questions: repeating order-balanced calls per item (Q1), retaining neutral verdicts instead of guessing (Q2), pairing cross-order calls to control decision changes (Q3), and extending score- and decision-preserving boundaries to larger references (Q4).
1/ Why repeat calls per item instead of evaluating more items once (Question 1). These two are for different purposes. If the goal is to estimate a dataset-wide leaderboard average, evaluating more prompts once is often the right use of budget because it reduces prompt-sampling variance. However, our target setting is per-item decision making (such as curating RLHF/DPO preference pairs or routing ambiguous outputs to human review), where each candidate pair requires a reliable verdict (Accept A, Accept B, or Defer). Evaluating another prompt usually doesn't help resolve the current pair, nor can one single call detect whether the verdict flips when display order is swapped. Moving from one balanced pair to a four-call balanced reference improves benchmark score by a median 3.24 pp across all reasoning configurations (Figure 2), which motivated us to use it as our target reference panel.
2/ Why preserve a noisy reference? (Question 2 & Weakness 2) TLDR: Because fixed-reference score-preserving stopping never really fixes wrong verdicts but only turns neutral references into coin-flip labels. We argue two facts to address the concern about "preserving incorrect reference decisions".
First, the decisive verdict never changes. At labeling time, the ground truth is unobserved and stopping rules only rely on judge calls. Whenever the four-call reference is decisive (whether right or wrong against ground truth), both the individual non-strict boundary and paired unanimity return the same verdict on every execution path. In fact, because preserving an expected score under both possible binary targets implies a probability of 1 on every decisive reference outcome, no fixed-reference score-preserving rule can ever alter a wrong decisive reference verdict (Section 2.3 & Section 4.2).
Second, preserving neutrality prevents coin-flip noise. Score-preserving rules return different labels only when the four-call reference is neutral (e.g., split 2-vs-2 under position bias or true ambiguity). Because a random guess on binary labels also scores 0.5 in expectation, the non-strict boundary preserves the average score while converting 12.28% of items on Nemotron-49B/JudgeBench from neutral into 50/50 coin-flip hard labels.
Even under matched cost and score constraints, an adaptive mixture optimized on training folds changes 3.59% of held-out decisions, more than double paired unanimity (1.68%, a 1.91 pp [1.49, 2.36] reduction). Across benchmarks, under both call and output-token budgets, paired unanimity achieves lower disagreement than the tuned mixture in 15 out of 17 configurations (and lower than the non-strict boundary in all configurations).
Importantly, in an accept/defer workflow (accepting decisive verdicts, deferring neutral ones), this improves the accuracy of accepted labels against benchmark ground truth. On Nemotron-49B/JudgeBench, non-strict stopping adds 6.14 wrong accepted labels per 100 items above full execution vs. just 0.84 under paired unanimity. This means paired unanimity removes 86% of excess stopping errors by deferring 10.60 pp more ambiguous items instead of guessing.