acceptodds
Under review as a conference paper at ICLR 2027

Equal Exposure, Different Decisions: Reference Selection and Early Stopping for Pairwise LLM Judges

Abstract

Repeatedly querying pairwise LLM judges across seeds and display orders can reduce sampling noise and position bias, but increases evaluation cost. We show that neither equal marginal inclusion probabilities across call conditions in reference selection nor preserving the score of a fixed reference in expectation during execution guarantees the same aggregate decisions. In reference selection, equal marginal inclusion probabilities fix expected additive recorded cost but do not determine the joint selection distribution. With four distinct seeds, independent display order randomization captures exactly of the difference in expected agreement between a common display order and exact balance when the corresponding call outcomes are decisive. In reference execution, because neutral decisions receive a score of 0.5, a stopping rule can preserve the reference score in expectation while turning a neutral reference decision into a decisive one. For hard labels, the per-instance probabilities of such changes exactly determine the additional variance in the aggregate score. We introduce paired unanimity stopping, which stops early only when both calls in an order-balanced pair return decisive outputs that agree, a condition that a consistent preference for one display position cannot satisfy. Under uniform matching, its rate of disagreement with reference decisions is at most that of the non-strict boundary under uniform call ordering, which stops when the vote margin is at least the number of remaining calls. Across 8 judges and 17 reasoning configurations on JudgeBench and RM-Bench math/code, it saves 12% to 46% of expected calls relative to full four-call execution while changing at most 1.68% of reference decisions. Matching its agreement requires the best nonadaptive mixture to use 9% to 60% more expected calls. These results establish decision preservation as a primary objective alongside score and cost in repeated LLM judging.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM as a Judge, LLM evaluation, pairwise evaluation, position bias, test time compute, adaptive stopping, ensemble aggregation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_A2utOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve the problem of how to reduce the computational cost of repeated LLMs judgements while preserving the collective decision of the LLM judgements when run on the full suite. To this end, the authors introduce paired unanimity stopping, which allows the judge to stop evaluations early when two order-balanced judgements agree unanimously. The paper also shows theoretical guarantees that allow paired unanimity stopping to allow decision preservation and variance. 8 LLM judges were used across a suite of experiments, showing up to a 46% reduction in evaluation LLM calls, while changing at most 1.7% of reference decisions compared to a full evaluation.

Critical strengths
  1. Strong theoretical analysis The paper tackles a relevant problem in efficient LLM judgements and also provides rigorous proofs of their approach. In particular, it makes an important distinction between preserving average evaluation scores and preserving individual LLM decisions.
  2. Strong experimental coverage The paper provides extensive experimentation. The authors evaluate their approach across several LLM judges, benchmarks and reasoning configurations. They show that their method is superior compared to existing methods and measure realistic serving-time improvements over these previous methods. The results provide evidence that the approach can reduce evaluation costs while largely preserving original decisions.
Critical weaknesses
  1. Limited novelty The proposed stopping rule is fairly simple and is equivalent to existing adaptations of ESC[1] and Adaptive-Consistency[2]. From what I can see, the main novelty is just from the theoretical analysis of the existing methods.
  2. Limited generalizability and practical significance Most of the theoretical guarantees focus exclusively on a special case (4 call evaluations) making it unclear whether these findings can scale to larger budgets. More importantly, the paper focuses on reproducing decisions from a four-call reference which by itself could be noisy, which leads to doubts on whether this can improve (or at least retain) evaluation quality.

[1] ESC-Judge: A Framework for Comparing Emotional Support Conversational Agents (EMNLP '25) [2] Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs (EMNLP '23)

Questions for authors
  1. In practical scenarios, when would repeated judgements be preferred over evaluating more questions? Wouldn't evaluating a larger number of questions (even just once) provide more reliable aggregate scores or model rankings than repeatedly judging the same questions? I'm not sure whether this is the best use of limited computational budget. Can you provide practical use-cases for when this would be useful?
  2. Why should we preserve potentially incorrect reference decisions? If reference decisions can be noisy or incorrect, why should matching them to the primary objective be preferred instead of improving agreement with the ground truth?
  3. What is fundamentally new compared with existing stopping methods beyond the theoretical characterization of a special case with four calls? If not, I feel like these methods should be explicitly mentioned in the main text and that this is a theoretical analysis of the existing methods, rather than a fundamentally new method.
  4. Is this method generalizable (empirically or theoretically) to larger evaluation budgets for >4>4 call judgements?
5 comments

Thank you for recognizing both our theoretical analysis and experimental coverage. We also used bold and italic fonts to facilitate your skimming of our response.

Stepping back, we found a common question that your four questions point to, which is "Once judge calls vary across display orders and permit neutral outcomes credited as 0.5, how is per-item decision making different from leaderboard score estimation?". In short, equal expected scores can still conceal per-item decision flips and excess variance. Addressing this connects all four questions: repeating order-balanced calls per item (Q1), retaining neutral verdicts instead of guessing (Q2), pairing cross-order calls to control decision changes (Q3), and extending score- and decision-preserving boundaries to larger references (Q4).

1/ Why repeat calls per item instead of evaluating more items once (Question 1). These two are for different purposes. If the goal is to estimate a dataset-wide leaderboard average, evaluating more prompts once is often the right use of budget because it reduces prompt-sampling variance. However, our target setting is per-item decision making (such as curating RLHF/DPO preference pairs or routing ambiguous outputs to human review), where each candidate pair requires a reliable verdict (Accept A, Accept B, or Defer). Evaluating another prompt usually doesn't help resolve the current pair, nor can one single call detect whether the verdict flips when display order is swapped. Moving from one balanced pair to a four-call balanced reference improves benchmark score by a median 3.24 pp across all reasoning configurations (Figure 2), which motivated us to use it as our target reference panel.

2/ Why preserve a noisy reference? (Question 2 & Weakness 2) TLDR: Because fixed-reference score-preserving stopping never really fixes wrong verdicts but only turns neutral references into coin-flip labels. We argue two facts to address the concern about "preserving incorrect reference decisions".

  • First, the decisive verdict never changes. At labeling time, the ground truth is unobserved and stopping rules only rely on judge calls. Whenever the four-call reference is decisive (whether right or wrong against ground truth), both the individual non-strict boundary and paired unanimity return the same verdict on every execution path. In fact, because preserving an expected score under both possible binary targets implies a probability of 1 on every decisive reference outcome, no fixed-reference score-preserving rule can ever alter a wrong decisive reference verdict (Section 2.3 & Section 4.2).

  • Second, preserving neutrality prevents coin-flip noise. Score-preserving rules return different labels only when the four-call reference is neutral (e.g., split 2-vs-2 under position bias or true ambiguity). Because a random guess on binary labels also scores 0.5 in expectation, the non-strict boundary preserves the average score while converting 12.28% of items on Nemotron-49B/JudgeBench from neutral into 50/50 coin-flip hard labels.

ProcedureExpected callsBenchmark score (%)Reference disagreement (%)
Non-strict boundary2.3381.4512.28
Paired unanimity2.6481.451.68
Held-out adaptive mixture (AC + 3-call)2.6481.453.59
Exact-preserving three-call check3.3781.450.00

Even under matched cost and score constraints, an adaptive mixture optimized on training folds changes 3.59% of held-out decisions, more than double paired unanimity (1.68%, a 1.91 pp [1.49, 2.36] reduction). Across benchmarks, under both call and output-token budgets, paired unanimity achieves lower disagreement than the tuned mixture in 15 out of 17 configurations (and lower than the non-strict boundary in all configurations).

Importantly, in an accept/defer workflow (accepting decisive verdicts, deferring neutral ones), this improves the accuracy of accepted labels against benchmark ground truth. On Nemotron-49B/JudgeBench, non-strict stopping adds 6.14 wrong accepted labels per 100 items above full execution vs. just 0.84 under paired unanimity. This means paired unanimity removes 86% of excess stopping errors by deferring 10.60 pp more ambiguous items instead of guessing.

3/ What is new beyond ESC and Adaptive-Consistency (Question 3 & Weakness 1). Standard self-consistency stopping methods (Early-Stopping Self-Consistency (ESC, Li et al., ICLR 2024) and Adaptive-Consistency (AC, [2] Aggarwal et al., EMNLP 2023)) were designed for repeated sampling on a fixed input. They do not account for the two defining structural properties of pairwise judging: (1) display-order asymmetry and (2) 0.5-credited neutrality. Applying existing stopping rules without our structural constraints introduces clear failure modes.

  • Failure of naive adaptations: Consider individual-call AC(0.80) as an example. The criterion would trigger stops at a 2:0 state of decisive votes, which can occur on two calls from the same display order. On a purely position-biased neutral reference, AC(0.80) returns a decisive label with 1/3 probability under uniform call ordering and the individual non-strict boundary does so on every ordering, but paired unanimity flips none of such reference panels. Conversely, if stopping takes place on any cross-order pair with a nonzero net vote (without requiring two decisive calls), then expected-score preservation is violated on cases such as (A, TIE) and (B, B) by returning A early when full execution chooses B.

  • Two-stage formulation and exact guarantees: Existing stopping methods take a single sampling distribution as given. Our work first resolves reference selection prior to execution, which establishes the exact 34\tfrac{3}{4} display-order composition law for decisive call outcomes under equal marginal exposure (Proposition 1). On the execution level, our work formalizes score vs. decision preservation and proves the exact variance-disagreement identity for arbitrary reference sizes (Section 2.3 & Equation 8), alongside our sharp four-call guarantee that paired unanimity has at most half the disagreement probability of the individual non-strict boundary on every balanced reference (Proposition 4). With cross-order pairing, decisive-only unanimity, and uniform random matching and ordering, four-call ESC and AC adaptations coincide with paired unanimity and inherit this guarantee.

Following your suggestion, we will point out this equivalence directly alongside Algorithm 1 in Section 4.1 and clarify what cross-order adaptations preserve. We will also spell out ESC at first mention so it is not confused with ESC-Judge [1].

4/ Scaling beyond four calls (Question 4 & Weakness 2). Four calls constitute the minimal order-balanced reference panel that incorporates within-order seed replication. Based on this, we establish our closed-form composition law and sharp half-disagreement bound. Beyond n=4n=4, our preservation hierarchy (Section 2.3), variance-disagreement identity (Equation 8), and individual-call boundaries (Proposition 3) remain applicable for arbitrary call budgets nn. Empirically, across the same reasoning configurations on 6- and 8-call references (Appendix D.5), the non-strict boundary preserves expected scores at median costs of 3.60 and 4.87 calls (~40% call savings), while the individual strict boundary preserves all reference decisions at 4.66 and 5.94 calls (~22%-26% savings). We also evaluated its paired strict counterpart on the same recorded calls by checking the condition ∣S2k∣>n−2k\lvert S_{2k}\rvert > n - 2k only after each completed cross-order pair (t=2kt=2k), which likewise preserves all reference decisions at median costs of 4.90 and 6.59 calls (~18% savings). Waiting to complete each cross-order pair uses slightly more calls than stopping on odd steps, but keeps display orders balanced at every checkpoint and requires at most 3-4 parallel pair rounds (vs. up to 6-8 rounds under serial individual-call dispatch).

We will incorporate these multi-budget paired results and clarifications into the revision.

AuthorsOpenReview ↗

Update: Manuscript revised

Dear Reviewer A2ut,

We have uploaded the revised PDF to address your comments. Section 6 now spells out Early-Stopping Self-Consistency and states which of its cross-order adaptations coincide with paired unanimity. For larger budgets, Section 4.1 and Appendix D.4 now include paired strict stopping, which preserves every reference decision and saves a median of 18% of calls at six and eight calls. On why preserving reference decisions matters, Section 5.4 now includes downstream results: in HelpSteer2 at matched judge calls, paired unanimity needs 36%-42% fewer human reviews for the same number of wrong binary labels, and in accept/defer curation it accepts fewer wrong labels than supervised mixtures in all 17 configurations.

New content is marked in blue, which we summarized in our top-level note too. Please let us know if any questions remain. We would greatly appreciate your feedback and reconsideration. Thank you very much for your effort!

Reviewer_151gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reference selection and early stopping for repeated pairwise LLM judgments. The authors analyze how display-order assignments and stopping rules affect aggregate decisions and propose Paired Unanimity Stopping to reduce evaluation costs while largely preserving full-reference decisions.

Critical strengths
  • The paper provides a systematic analysis of repeated LLM judging under different display-order and stopping strategies.

  • Experiments across multiple judge models demonstrate potential reductions in evaluation costs.

Critical weaknesses
  • The methodological novelty is very limited. The proposed Paired Unanimity Stopping essentially stops when two order-balanced judgments agree, a straightforward strategy closely related to existing early-stopping methods. More importantly, the authors explicitly acknowledge in Appendices D.1 and H.7 that its four-call implementation is equivalent to existing ESC and Adaptive-Consistency variants, making the actual algorithmic contribution unclear.

  • The main theoretical findings offer limited new insights. The observation that equal marginal inclusion probabilities do not guarantee identical aggregate decisions is a general consequence of different joint sampling distributions under nonlinear aggregation. Similarly, preserving expected scores does not imply preserving individual decisions, particularly when neutral outcomes receive partial credit. Although these properties are formally characterized, their broader significance is not sufficiently established.

  • The practical importance of the proposed decision-preservation objective remains unclear. The method primarily aims to reproduce the decisions of a fixed four-call reference, but preserving these decisions does not necessarily improve judgment correctness or reliability. The paper does not convincingly demonstrate why this objective should be prioritized over existing accuracy- and cost-oriented evaluation strategies.

  • The empirical results do not establish a compelling advantage over existing adaptive approaches. While the proposed method outperforms nonadaptive mixtures, expanded mixtures incorporating adaptive strategies achieve higher agreement point estimates across all reported configurations. Given the equivalence to existing early-stopping methods and the narrow four-call setting, the experiments provide limited evidence of a substantial new contribution.

Questions for authors

See weaknesses.

7 comments

Thank you for noting the clarity of our presentation and our systematic evaluation across multiple judge models. We also used bold and italic fonts to facilitate your skimming of our response.

Below, we address what the conditions and guarantees in our paper contribute beyond qualitative intuition about nonlinear aggregation and 0.5 neutral credit, as well as existing early-stopping adaptations. In particular, qualitative intuition alone does not quantify the expected-score gap between independent and balanced display-order selection, identify when ESC and Adaptive-Consistency adaptations lose score or decision preservation (under position bias, call-level ties, or larger budgets), or establish how label-free reference disagreement determines the added score variance and eventually the excess wrong accepted labels.

1/ Requirements beyond what ESC and Adaptive-Consistency specify (Weakness 1). ESC/AC assume exchangeable draws on a single input, but the direct adaptations can convert the position-biased neutral references into decisive verdicts or violate the expected score preservation with a fixed reference under ties. Also, the two-call window stopping does not have expected score guarantee for n≥6n \ge 6.

  • Failure of direct adaptations. Original Early-Stopping Self-Consistency ([ESC, Li et al., ICLR 2024]) and Adaptive-Consistency ([AC, Aggarwal et al., EMNLP 2023]) do not specify cross-order pairing or how ties are treated. Under uniform individual-call ordering, AC(0.80) stops at a 2:0 state of decisive votes, which can occur on two calls in the same order. On a purely position-biased neutral reference, individual-call AC(0.80) returns an equiprobable decisive verdict with probability 1/3 and the individual non-strict boundary does so with probability 1. In contrast, paired unanimity retains the neutral decision with probability 1. Even with cross-order pairs, stopping whenever the first pair has a nonzero signed sum without requiring both calls to be decisive (the decisive-pair ablation) still undermines expected score preservation with a fixed reference under ties, such as returning A half the time on cross-order pairs (A, TIE) and (B, B) when full execution chooses B (Section 4.1 and Appendix D.3).
  • Four-call equivalence and why two-call window stopping does not have the expected score guarantee at n≥6n \ge 6. With cross-order pairing and agreement between two decisive canonical votes, two-window ESC and paired AC(0.80)/AC(0.85) are functionally equivalent to paired unanimity at n=4n=4 (Section 6 and Appendices D.1 and H.7). Under uniform random matching and pair order, they inherit Propositions 2 and 4. Among the even reference sizes with within-order seed replication, n=4n=4 is the only one where a unanimous first pair (+2) can at worst be offset by the remaining pair (-2), which produces a neutral aggregate (0) with no overturning. For n≥6n \ge 6 (Appendix D.5), stopping after one unanimous two-call window violates expected score preservation when later pairs were to reverse the initial +2 lead, such as one (+1, +1) pair and two (-1, -1) pairs. Following your comment, we will move this four-call equivalence and n≥6n \ge 6 note beside Algorithm 1 in Section 4.1 to clarify that our contribution in Section 4 lies in characterizing these preservation conditions and proving Propositions 2 and 4.

2/ Quantitative guarantees beyond qualitative intuition (Weakness 2). With all eight call outcomes being decisive, independent display-order randomization captures 75% of the difference in expected score between common order and exact balance at equal cost. Also, our execution analysis shows that label-free reference disagreement determines both added score variance and excess wrong accepted labels.

  • Composition identity in reference selection. Under equal marginal inclusion probabilities across call conditions with all eight call outcomes decisive, Proposition 1 proves that independent display-order randomization satisfies A(x)(μind,4)=14A(x)(μsame,4)+34A(x)(μbal,4D)A^{(x)}(\mu_{\mathrm{ind},4}) = \tfrac{1}{4}A^{(x)}(\mu_{\mathrm{same},4}) + \tfrac{3}{4}A^{(x)}(\mu_{\mathrm{bal},4}^{\mathrm D}). When call-level ties or invalid outputs occur, the departure from this identity is bounded by 332\tfrac{3}{32} (Equation 6). Under decisive call outcomes, when exact balance improves expected benchmark score over common-order selection, independent randomization captures 75% of the gain and exact balance obtains the other 25% at the same four-call cost. Empirically, exact balance improves expected benchmark score in 15 of 17 reasoning configurations compared to independent randomization (with 10 paired 95% intervals above zero, Section 3.3) and lowers ground-truth error rate among decisive verdicts at matched coverage in 15 of 17 configurations (Appendix C.5).
  • Label-free reference disagreement determines added variance and excess wrong accepted labels. Under binary benchmark targets yi∈{0,1}y_i \in \lbrace 0, 1\rbrace, fixed-reference expected score preservation leaves every decisive reference verdict unchanged and restricts reference disagreements (dp,id_{p,i}) to neutral references. Conditional on the conversion from neutral to decisive, ground-truth correctness is 50% based on execution randomization. Thus, the per-instance rates dp,id_{p,i} (with weighted average dpd_p, which can be measured without ground truth labels) determine both added aggregate score variance (Var⁡(ΔHˉp)=14∑iwi2dp,i\operatorname{Var}(\Delta\bar H_p) = \tfrac{1}{4}\sum_i w_i^2 d_{p,i} in Equation 8 for arbitrary nn) and excess wrong accepted labels (12dp\tfrac{1}{2}d_p). Proposition 4 guarantees that, for every instance ii, dunanim,i≤12dweak,id_{\mathrm{unanim},i} \le \tfrac{1}{2}d_{\mathrm{weak},i}, which also holds conditional on every selected balanced four-call reference. On Nemotron-49B/JudgeBench, paired unanimity is shown to reduce added aggregate score variance by 86% (Section 5.3 and Table 1).

3/ Practical impact on judgment correctness (Weakness 3). When evaluated against binary benchmark targets in accept/defer curation, paired unanimity reduces excess wrong accepted labels by 73% relative to a matched non-strict mixture and even outperforms supervised adaptive mixtures fitted on ground truth labels.

In labeling preference data and routing outputs to human review (Section 1), benchmark targets are unobserved at labeling time, so decisive verdicts (A or B) are accepted as hard labels while neutral verdicts are deferred. Under Equation (1), retaining a neutral reference decision scores 0.5 and matches the 0.5 expected score of returning either decisive verdict with probability 1/2. For binary benchmark targets, let Accept(p)\mathrm{Accept}(p) and WrongAccept(p)\mathrm{WrongAccept}(p) be the expected fractions of evaluation instances receiving a decisive label and an incorrect decisive label. Under fixed-reference expected score preservation, any procedure pp keeps every decisive reference decision unchanged and satisfies Accept(p)=Accept(full)+dp\mathrm{Accept}(p) = \mathrm{Accept}(\mathrm{full}) + d_p and WrongAccept(p)=WrongAccept(full)+12dp\mathrm{WrongAccept}(p) = \mathrm{WrongAccept}(\mathrm{full}) + \tfrac{1}{2}d_p. On Nemotron-49B/JudgeBench in reasoning mode, all rules (full execution, the individual non-strict boundary, and paired unanimity) share the same 81.45% expected benchmark score, but they differ significantly on error rates among accepted labels (Equation 9 in Appendix C.5).

Procedure on Nemotron-49B / JudgeBench (Reasoning)Expected callsAccepted items (%)Wrong accepted labels (per 100 items)Excess wrong labels w.r.t. full 4-callError rate among accepted (%)
Full four-call reference execution4.0086.4011.75+0.0013.60
Individual non-strict boundary (raw)2.3398.6817.89+6.1418.13
Non-strict/full/deferral mixture (matched calls & acceptance)2.6488.0814.84+3.1016.85
Paired unanimity stopping2.6488.0812.59+0.8414.29
  • Matched cost and acceptance against non-strict stopping. When used alone, the individual non-strict boundary converts 90.3% of neutral reference decisions (12.28% / 13.60%) into decisive verdicts with 50% correctness, raising the error rate among accepted labels from 13.60% (under full execution) to 18.13%. Even when a non-strict/full/deferral mixture reallocates the entire call difference to match paired unanimity (2.64 expected calls and 88.08% accepted items), paired unanimity still outperforms by reducing the remaining excess wrong accepted labels by 73% (+0.84 versus +3.10). Conversely, for the mixture to match paired unanimity in both acceptance rate and error rate among accepted labels, it requires 3.77 expected calls, which is 43% higher cost.
  • Held-out improvement over supervised adaptive mixtures. Across five question-group-level folds, we fitted accuracy- and cost-oriented mixtures of existing baselines (full four-call execution, the three-call decision-preserving check, the individual non-strict boundary, AC, and pre-query deferral) on the four training folds of ground truth labels to minimize wrong accepted labels while remaining subject to the training-fold cost upper bound and acceptance rate of paired unanimity (with held-out differences below 0.25% in cost and 0.01 pp in acceptance). Even without label-based calibration or threshold tuning, paired unanimity reduces excess wrong accepted labels by about 41% (under the call budget, 38% under the token budget) relative to the fitted mixture on JudgeBench, and also still consistently outperforms the ground-truth fitted mixture across both JudgeBench and RM-Bench math/code under both call budgets and token budgets (all 95% intervals strictly above zero). Compared to a mixture that assigns instances to pre-query deferral, individual-call stopping, or three- and four-call execution before observing any outputs, paired unanimity executes an initial balanced pair on every instance and allocates follow-up calls only when that pair does not agree decisively.

4/ Section 5.1's expanded mixtures, same-reference adaptive baselines, and the cases for n=n= 6, 8 (Weakness 4). Section 5.1's expanded mixtures exceed the four-call expected benchmark score because their candidate pool includes an eight-call component, not from adaptive stopping on a fixed four-call reference. When the four-call reference is held fixed (Section 5.2), paired unanimity achieves lower held-out reference disagreement than fitted adaptive mixtures in 15 of 17 configurations. Also, Proposition 3 extends to n=n= 6 and 8.

  • Section 5.1 versus same-reference adaptive mixtures (Section 5.2). No score-preserving stopping rule on a fixed four-call reference can exceed that reference's expected benchmark score ("expected agreement" in Section 5.1, 81.45% on Nemotron-49B/JudgeBench). In Section 5.1 (Figure 3c), the expanded mixtures achieve higher point estimates across configurations because their candidate pool jointly varies reference selection and execution and includes an eight-call component executing all eight recorded call conditions (Appendix D.4, which is required to exceed the four-call score on Nemotron and 13 of 17 configurations), and fits weights on benchmark labels without constraining reference disagreement. To evaluate adaptive baselines on the same balanced four-call reference (extending Section 5.2's same-reference comparison to held-out question groups without pre-query deferral or the eight-call component), we fitted mixtures across the same five question-group folds over individual-call AC(0.80), AC(0.90), the individual non-strict boundary, and exact-preserving checks to minimize reference disagreement subject to the cost upper bound and benchmark score lower bound of paired unanimity. On Nemotron-49B/JudgeBench, the fitted mixture gives 3.59% pooled out-of-fold reference disagreement, which is more than double of 1.68% by paired unanimity at matched cost and score (41%~56% reduction, 2.64 expected calls with difference below 1e-3). Without label-based calibration or threshold tuning, paired unanimity achieves lower held-out reference disagreement in 15 of 17 configurations under both call budget and output token budgets.
  • n=6n=6 and n=8n=8. While n=4n=4 is the unique replicated even size where single-pair unanimity preserves expected benchmark scores, our preservation hierarchy (Section 2.3), variance-disagreement identity (Equation 8), and remaining-vote boundaries (Proposition 3) all hold for arbitrary reference sizes nn. We did an additiona analysis and found that, across all 17 configurations on balanced 6- and 8-call references (Appendix D.5), preserving all reference decisions still saves 22%~26% of calls under the individual strict boundary (∣St∣>n−t\lvert S_t\rvert \gt n - t where StS_t is the running signed vote sum after tt calls, median 4.66 and 5.94 calls), or 18% under the paired strict boundary (∣S2k∣>n−2k\lvert S_{2k}\rvert \gt n - 2k, median 4.90 and 6.59 calls), which checks stopping only after completed cross-order pairs so executed prefixes stay balanced across display orders. In contrast, although the individual non-strict boundary (∣St∣≥n−t\lvert S_t\rvert \ge n - t) uses fewer calls (median 3.60 and 4.87, ~39% savings), those additional savings come from stopping at ∣St∣=n−t\lvert S_t\rvert = n - t, which again costs converting neutral references into equiprobable decisive verdicts. Also, checking the non-strict condition after cross-order pairs can violate expected score preservation under call-level ties (Appendix H.7).
Reviewer_iad7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how to choose and run repeated calls to a pairwise LLM judge across seeds and display orders without changing the evaluator. For reference selection with four calls, it compares a common display order, independently randomized orders and exact order balance, which all include each call condition with equal probability, and proves that when all outcomes are decisive, independent randomization captures exactly three quarters of the agreement difference between the other two (Proposition 1), with a sharp bound otherwise. For executing a fixed four-call reference, it notes that because a neutral decision scores 0.5, a stopping rule can keep expected agreement while turning neutral reference decisions into decisive ones, and shows that these changes exactly determine the added score variance (Equation 8). It proposes paired unanimity stopping, which stops after the first order-balanced pair only when both calls are decisive and agree; this preserves expected agreement and has at most half the disagreement rate of the non-strict boundary (Proposition 4). Experiments with 8 judges on JudgeBench and RM-Bench math/code (17 reasoning configurations), computed exactly from 8 recorded calls per item, report 12% to 46% fewer calls than full execution with at most 1.68% of reference decisions changed, and compare against Adaptive-Consistency, a decision-preserving three-call check and fitted mixtures of baselines.

Critical strengths

Paired unanimity changes far fewer reference decisions than the non-strict boundary, at the same expected accuracy and with a guarantee. Across all 17 reasoning configurations it changes 0.98 to 18.97 pp fewer decisions (lines 398–401); on Nemotron-49B/JudgeBench, 1.68% vs 12.28%, which cuts the added score standard deviation from 0.70 to 0.26 pp (Table 1). Proposition 4 guarantees at most half the non-strict disagreement rate on every instance. This lower variance could matter in sensitive use cases, or when the verdicts become training labels.

Critical weaknesses
  1. No accuracy or cost advantage over the non-strict boundary. Both rules preserve the reference's expected agreement by construction (Propositions 2 and 3), so accuracy is identical (81.45% for both on Nemotron-49B/JudgeBench, line 402), while paired unanimity uses more expected calls in every configuration, up to 0.67 more (lines 398–401, Figure 13; 2.64 vs 2.33 on Nemotron). Its case therefore rests entirely on lower variance. The trade-off is built into the rule, so new results cannot fix it.
  2. The variance benefit is not shown to matter in practice. The paper motivates decision preservation with uses such as labeling preference data (lines 40–41) and concludes that the results "establish decision preservation as a primary objective alongside score and cost" (lines 33–35), but no experiment shows the rule helping a downstream task, such as training on its labels, compared with the non-strict boundary or other baselines. Without such an example, it is unclear when the extra calls are worth paying for. A small downstream example during the discussion would partly address this.
Questions for authors
  1. Downstream example [Requested analysis]: Can you show, even at small scale, a downstream task where paired unanimity's lower variance helps compared with the non-strict boundary and other baselines at matched cost, for example training a reward model or DPO on the labels each rule produces? A clear gain would raise my rating; no measurable difference would confirm my main concern.
  2. Target use cases [Clarification]: Which concrete use cases justify reproducing the four-call verdict at 13% more calls than the non-strict boundary (2.64 vs 2.33 on Nemotron-49B/JudgeBench), such as sensitive applications or building training data? A concrete case where the decisions the non-strict boundary changes would matter would strengthen the significance and could raise my rating; if none applies, my assessment is unchanged.

Score criteria: I would raise my rating if the authors show a downstream use where the rule's lower variance gives a measurable benefit over the non-strict boundary at matched cost. Without such evidence the rule trades more calls for equal accuracy, and my rating stays; if a downstream test showed no benefit at all, I would lower it.

7 comments

Thank you for your insightful assessment and for stating an empirical criterion of demonstrating a downstream benefit over the non-strict boundary at matched cost. We also used bold and italic fonts to facilitate your skimming of our response.

We develop our response below around the question your concerns seem to point to, which is "When expected benchmark scores are identical (81.45%), why pay 13% more calls to reduce neutral-to-decisive flips and score variance?"

The accepted-label purity is different. With Equation (1), neutral receives 0.5 credit, so an 81.45% expected benchmark score treats two opposite behaviors identically on neutral references.

  1. Abstaining (neutral) as to retain a balanced split (e.g., 2 votes for A, 2 for B) so the item can be deferred receives 0.5 credit.
  2. Premature coin-flipping (A or B) as to stop early before a split four-call reference resolves to neutral (e.g., at 2:0 or 2:1 when remaining votes tie the reference). This outputs a 50/50 hard label whose expected credit is also 12(1)+12(0)=0.5\tfrac{1}{2}(1)+\tfrac{1}{2}(0)=0.5.

In preference data curation and the task of routing for human review (motivated in Section 1), decisive verdicts (A or B) are accepted as hard labels while neutral verdicts are deferred. Converting neutral references into 50/50 guesses injects hard labels with 50% error into the accepted pool. Below, we present downstream accept/defer curation and review routing at matched cost and matched acceptance, followed by learner and judge pilot checks and an ongoing (queued) reward model training study at a larger scale.

1/ Mechanism and matched-cost comparison against non-strict stopping (Questions 1-2 & Weakness 1) TLDR: The per-item disagreement probabilities dp,id_{p,i} determine the excess variance of Equation (8) and excess wrong accepted labels. Paired unanimity removes 73% of the excess label error left even after mixing non-strict stopping with full four-call execution and deferral at matched expected calls and acceptance. Conversely, matching both acceptance and label purity of paired unanimity requires the mixture to use 43% more calls.

For stopping rule pp, let Accept(p)\mathrm{Accept}(p) and WrongAccept(p)\mathrm{WrongAccept}(p) be the expected fractions of items receiving a decisive label and an incorrect decisive label. Under binary benchmark targets, any rule with fixed-reference expected score preservation leaves every decisive four-call reference verdict unchanged. Reference disagreements (dp,id_{p,i} with weighted average dpd_p) only occur on neutral references and resolve to either correct or incorrect hard labels with probability 12\tfrac{1}{2} (Propositions 2-3 and Section 4.2). Compared to the full four-call execution (full\mathrm{full}),

\\ \\ \mathrm{WrongAccept}(p)=\mathrm{WrongAccept}(\mathrm{full})+\tfrac12 d_p.$$ When neutral receives 0.5, the flips contribute to the added aggregate score variance in Equation (8) as a second order statistic ($\operatorname{Var}(\Delta\bar H_p)=\tfrac14\sum_i w_i^2 d_{p,i}$), but under selective accept/defer routing, the same flips contribute to an increase ($\tfrac12 d_p$) in expected fraction of wrong accepted labels as a first order statistic. Because Proposition 4 guarantees $d_{\mathrm{unanim},i} \le \tfrac{1}{2} d_{\mathrm{weak},i}$ on every item, paired unanimity reduces excess wrong accepted labels by **at least 50%** relative to raw non-strict stopping across all configurations. On the Nemotron-49B/JudgeBench (reasoning) setting highlighted in your review, we ask whether non-strict stopping can recover this purity if we reallocate the entire 13.4% call difference. In the table below, full four-call execution, raw non-strict stopping, and paired unanimity all share an 81.45% expected benchmark score (where the 11.75 wrong accepted labels per 100 inputs under full execution serve as the full-reference baseline and 13.60% of balanced four-call references are neutral). Row 3 uses pooled recorded means to form a non-strict/full/deferral mixture that matches both paired unanimity's expected calls and acceptance rate. | Procedure on Nemotron-49B / JudgeBench (Reasoning) | Expected calls | Accepted items (%) | Wrong accepted labels (per 100 items) | Excess wrong labels w.r.t. full 4-call | Error rate among accepted (%) | |---|---:|---:|---:|---:|---:| | Full four-call reference execution | 4.00 | 86.40 | 11.75 | +0.00 | 13.60 | | Individual non-strict boundary (raw) | 2.33 | 98.68 | 17.89 | +6.14 | 18.13 | | Non-strict/full/deferral mixture (matched calls & acceptance) | 2.64 | 88.08 | 14.84 | +3.10 | 16.85 | | **Paired unanimity** | 2.64 | 88.08 | **12.59** | **+0.84** | **14.29** |

Among neutral four-call references (13.60%, including 2-vs-2 splits due to position bias or ambiguity), where the rules for preserving scores can diverge, the raw non-strict stopping prematurely accepts 90.3% of them (12.28 / 13.60) as 50/50 coin-flip labels, but paired unanimity successfully defers 87.6% of them (11.92 / 13.60), which reduces 86% of excess stopping errors (+0.84 vs. +6.14 above the full-reference baseline) and lowers the accepted-label error rate from 18.13% to 14.29%. Even after reallocating the call difference entirely into the matched non-strict/full/deferral mixture, paired unanimity still reduces 73% of the remaining excess label errors (+0.84 vs. +3.10). Conversely, for the mixture to match both the acceptance rate and label purity of paired unanimity, it must run full four-call execution on 86.3% of items (3.77 expected calls, 43% more costly). Compared to a pre-query mixture that selects its component before seeing any outputs, paired unanimity stops at two calls only when both cross-order votes are decisive and agree, reserving follow-up calls specifically for pairs that disagree, tie, or produce an invalid output.

2/ Held-out curation against supervised adaptive mixtures (Question 1 & Weakness 2) TLDR: Across 5-fold held-out evaluations, calibration-free paired unanimity significantly lowers excess stopping errors even against the mixtures of existing adaptive baselines fitted on training-fold ground truth labels under both call and output token budgets.

To test whether combining existing adaptive rules can match paired unanimity out of sample, we fitted supervised mixtures across five question-group-level folds (combining full four-call execution, the three-call decision-preserving check, individual non-strict stopping, Adaptive-Consistency, and pre-query deferral). Even when this baseline mixture is optimized with training-fold ground truth labels to minimize wrong accepted labels at paired unanimity's cost cap and acceptance rate (with held-out cost gaps below 0.25% and acceptance gaps below 0.01 pp), which is unfair against paired unanimity, calibration-free paired unanimity still reduces excess stopping-induced errors by ~41% on JudgeBench under the call budget (held-out reduction of 0.58 pp [0.47, 0.71] under call budget and 0.51 pp [0.39, 0.63] under token budget) and consistently outperforms the supervised mixture on RM-Bench math/code (0.49 pp [0.33, 0.66] under call budget and 0.46 pp [0.31, 0.61] under token budget).

3/ Downstream reward model training (Question 1) In addition to the matched-cost curation evaluation above, we initiated a larger-scale matched-budget reward model training comparison on HelpSteer2 using Qwen 2.5 3B. Prior to the full runs, we completed learner and judge pilot checks to verify the setup: (i) a pilot reward model trained on human preference labels reached 69.3% accuracy on a 256-pair held-out development split, and (ii) a 64-pair judge verification check confirmed that the decision-preservation gap also occurs on HelpSteer2 (1.56% reference decision changes under paired unanimity vs. 8.59% under individual non-strict stopping, reduced by 5.5x). Our full scale matched-budget labeling and reward model training experiments are currently queued on our compute cluster. Due to the extremely high job loads, the waiting time has been much longer than expected. We will post these results in a follow-up response as soon as they are complete. Updated: please see our response below.

We will add the accepted/defer wrong-label error identity alongside Equation (8) in Section 4.2 and include the matched-cost/acceptance curation evaluation and held-out adaptive mixture comparisons in Section 5 and Appendix D.

AuthorsOpenReview ↗

Follow-Up Experiment Update

Our follow-up experiments on HelpSteer2 are now completed. We used 20000 individual judge calls with 2500 balanced four-call panels for each judge on Nemotron-49B and Qwen3-14B. Due to limited computational resources we have access to for long context reasoning generation within the discussion window, we chose these two models to span two model families, medium vs. small parameter scales, and high- versus moderate-disagreement cases. We complement this by replaying the reviews under expected budget across all 17 configurations in the paper. We also audited the human-review routing and trained reward models based on Qwen2.5-3B.

A1/ Human review routing: 36%-42% savings on review count & wins under all 17 configurations. We extend the accept/defer evaluation in the first response and tested routing deferred Neutral items to human review across 5 paired seeds on all 2500 HelpSteer2 panels. Within each seed, both paired unanimity (PU) and non-strict stopping receive random completion to Full to spend the same number of judge calls (6875) and receive the same review quota. We let the observed Neutral verdicts be prioritized before uniform random review (simulated with recorded human labels, including human ties).

With almost identical final hard-label coverage (96.91% vs. 97.07% on Nemotron, 96.26% vs. 96.23% on Qwen), PU improves final A/B/Tie human agreement on both judges across 5 seeds (+1.94 pp on Nemotron, +1.12 pp on Qwen) and leaves fewer binary hard errors (361.8 vs. 406.4 on Nemotron, 379.2 vs. 407.8 on Qwen). To reach the same remaining error count as PU, non-strict stopping requires additional uniform random reviews. We find that PU provides considerable total review savings at matched judge-call cost:

Judge modelMean review quotaExtra random reviews needed by non-strict stoppingPU expected savings on review count (binary parity, 95% prompt/seed CI)PU expected review-count savings (full A/B/Tie parity, 95% prompt CI)
Nemotron-49B328.4+237.9 (+72.4%)41.9% ([33.6, 48.0]% / [38.5, 45.3]%)26.6% ([18.3, 33.3]%)
Qwen3-14B280.4+155.5 (+55.5%)35.6% ([24.5, 42.8]% / [32.4, 38.8]%)20.4% ([10.8, 27.7]%)

Note that binary parity includes wrong decisive labels on binary human targets as errors. Full A/B/Tie parity further includes decisive labels on human ties as errors.

Applying the expected budget review replay across all 17 configurations in the paper (by randomly completing non-strict to match PU's expected calls and matching expected review quotas, which achieved the complete final coverage on binary benchmark targets), we found PU is favored in all configurations. Specifically, PU reduced the final wrong labels by ranging from 0.41 to 3.35 pp with all CIs above 0, including 2.71 pp [2.35, 3.05] on Nemotron-49B/JudgeBench (12.59 vs. 15.30 wrong labels per 100 inputs), 2.09 pp [1.62, 2.59] on Nemotron-49B/RM-Bench, and 3.35 pp [2.98, 3.73] on EXAONE-32B/RM-Bench.

A2/ Downstream reward model results & analysis. We trained the Qwen2.5-3B scalar Bradley-Terry reward model on the HelpSteer2 panels without human review and found that it gives a mild positive improvement in binary accuracy on untouched test pairs (+0.40 pp for Nemotron, +0.28 pp for Qwen, and +0.79 pp in the separate Nemotron comparison with training set size matched).

From the perspective of mechanism, this is indeed different from curation and review routing. Note that, under a fixed call cap without manual review, dropping Neutral label means trading the cleaner accepted labels against extra non-strict training pairs. When Neutral is retained as q=0.5q=0.5 with equal item weights, because the Bradley-Terry NLL ℓ(m,q)=log⁡(1+em)−qm\ell(m, q) = \log(1+e^m) - qm is affine in qq, randomly assigning q=0q=0 or q=1q=1 with equal probability gives the same conditional expected loss and gradient as retaining q=0.5q=0.5 at any given mm.

For Questions 1 and 2 in your comment, PU removes 73% of excess wrong accepted labels at matched calls and acceptance on Nemotron/JudgeBench, lowers final wrong labels in all 17 paper configurations at matched expected call and review budgets, and reduces the total expected reviews needed for binary error parity by 36%-42% at matched judge-call cost on HelpSteer2.

We hope these results address the criterion in your review as the measurable downstream benefits over the non-strict boundary at matched cost. Should any questions remain as you revisit your assessment, we would be happy to clarify them. Thank you again for your constructive suggestions!

Reviewer_DhYcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

The paper introduces paired unanimity stopping, a stopping rule for repeated pairwise LLM judging. Evaluation stops and returns the final decision as soon as an “execution pair” produces a decisive, unanimous winner. The paper focuses mainly on four-call selection, which extends the balanced pair design. It compares three four-call designs (independent, balanced, and common) and develops theoretical results on expected agreement and stopping boundaries. The authors test their methods with 8 judge models and 17 reasoning configurations on two benchmarks: JudgeBench and the RM-Bench math/code set.

Critical strengths

The paper shows promising results for the unanimity stopping in its experiments in reducing reference disagreement and expected number of calls compared to other stopping rules in the case of four-call selection.

Critical weaknesses

Throughout the paper, terms are often introduced unclearly, which makes it hard to follow. In general, the paper lacks clear definitions for the terms it introduces, and it does not explain the intuition or the technical steps behind its propositions and equations.

For example, Proposition 1 and the derivation of the three-quarters factor are unclear. The argument does not connect cleanly from the scores to the polynomials and expectations. Most of the reasoning is given in words rather than equations, so the argument needs a more formal, step-by-step derivation to be convincing.

Furthermore, the meaning of “agreement” is confusing and changes between sections. It is often left to the reader to decide whether it means a match between paired decisions or a match between a call and the ground-truth answer. Similar problems arise with other terms, such as “reference decision” and “disagreement.”

Despite the promising results reported for unanimity stopping, I question how much it adds. Compared with unanimity stopping, three-call decision-preserving achieves better agreement with the reference (as in 0 difference), at a cost of only about one additional call on average. Given the four-call limit and the definitions of the rules, the expected call counts were largely predictable: between 2.17 and 3.53 for unanimity stopping and around 3.3-3.4 for the three-call decision-preserving rule. In other words, an extra call appears to buy better agreement with the reference. It seems unanimity stopping’s lower expected call count follows from its structure: it runs either 2 or 4 calls, so its average falls below that of the three-call rule. Thus, given how small this difference in computation is, I question whether the savings from unanimity stopping justify the loss in agreement.

Questions for authors

Is there a more robust justification for preferring the paired unanimity rule over the three-call decision-preserving rule?

Why restrict the evaluation to four-call selection? Wouldn't it be more informative to increase the call budget to at least six and compare the rules from there? As I noted in my previous comment, the four-call restriction makes the expected number of calls for the paired unanimity rule highly predictable, so I question the novelty of these results.

6 comments

Thank you for your constructive feedback and for acknowledging our empirical results regarding the reduction of expected call counts and reference disagreement. We also used bold and italic fonts to facilitate your skimming of our response.

1/ Why prefer paired unanimity over the three-call check (Addressing Weakness 2 and Question 1)

On a four-call reference, without imposing prior constraints on unseen votes, then at least three calls are necessary to preserve the reference decision. Even if the first two observed votes (+1, +1) support canonical response A, the remaining two unseen votes could be (-1, -1) for canonical response B, eventually resolving to a neutral decision (0). Consequently, any rule that guarantees exact decision preservation requires that the average number of calls cannot fall below 3. While any rule based on {2,4}\lbrace 2, 4\rbrace calls will naturally yield an average call count in between, crossing below 3 calls without triggering large decision flips is nontrivial as shown by the balanced four-call reference on Nemotron-49B/JudgeBench (reasoning mode).

ProcedureExpected CallsReducible Calls Saved (of 2.0)Expected Benchmark Score (%)Reference Disagreement (%)Stopping Error SD (pp, Score diff. SD, Table 1)
Three-call decision-preserving check3.370.63 (31.6%)81.450.000.00
Held-out adaptive mixture (AC(0.80) & 3-call check)2.641.36 (67.9%)81.453.59-
Paired unanimity stopping2.641.36 (67.9%)81.451.680.26
Individual non-strict boundary2.331.67 (83.5%)81.4512.280.70
  1. It saves 67.9% compared to 31.6% of reducible calls (plus optional pathwise savings when optimizing purely for call count). Any order-balanced evaluation (observing each of the two presentation orders at least once) needs at least 2 irreducible calls (one for A first in id\mathrm{id} order and one for B first in swap\mathrm{swap} order). This leaves only 2 replication calls reducible, within which the three-call check saves only 0.63 (31.6%) but paired unanimity saves 1.36 (67.9%) calls and hence eliminates 2.15 times as many reducible calls (21.6% fewer total calls). The three-call check never achieves an average of <3<3 calls per instance, but paired unanimity crosses below 3 calls in 9/17 configurations. Also, in the cases with a call budget where sequential latency is secondary, a third-call check can be triggered whenever the first pair is non-unanimous (third-call continuation, Appendix H.9), which yields the same decisions yet using no more calls than either rule on every execution path (averaging <3<3 calls in 13/17 configurations and saving a median 0.75 calls relative to the three-call check).

  2. Under paired scheduling, the process completes in a single parallel round if the first pair reaches decisive unanimity, which avoids a mandatory second round. With our paired scheduling scheme (Section 4.1), both rules start the first pair of cross-order evaluations in parallel. When the first pair reaches decisive unanimity, paired unanimity can conclude immediately, which accounts for 68% of the instances on Nemotron-49B/JudgeBench and reaches up to 91% across all configurations (Equation 7). In contrast, the three-call check requires a second round of sequential generation on all of the instances, because running three calls from the outset would not only forfeit all two-call savings but also make the display orders within the prefix unbalanced. This causes a substantial latency penalty when reasoning judges generate up to 65K output tokens. In end-to-end serving on NVIDIA H200 GPUs (Appendix E), this ability to finish after a single concurrent pair helps pipelined paired unanimity reduce total serving time for GPT-OSS-20B by 32%-36% compared to full four-call execution.

  3. The expected loss in score is zero and the added variance of the aggregate score is 86% lower than non-strict stopping. Paired unanimity preserves all decisive reference decisions as well as all neutral reference decisions exhibiting pure position bias (cases where the model selects A when A appears first and selects B when B appears first; Proposition 2). The 1.68% of cases involving disagreement occur only when each display order contains one vote for A and one for B, and matching across orders forms opposing unanimous pairs (i.e., (+1, +1) versus (-1, -1)). Under uniform random pair ordering, where the probability of selecting either side is 12\tfrac{1}{2}, the expected benchmark score remains unchanged. Changing neutral ties to decisive verdicts induces a stopping error relative to full execution with SD 0.26 pp across the evaluation set, compared to 0.70 pp under the non-strict boundary (reduced added score variance by 86%, Equation 8 and Table 1).

  1. The rate of reference disagreement of paired unanimity is less than half that of an adaptive mixture operating under an equivalent budget. To test whether another rule can match this fidelity at the same cost, we fit an adaptive mixture (combining Adaptive-Consistency at 0.80 with the three-call check from the adaptive baseline pool) on training folds to match paired unanimity's call budget and benchmark score. On the held-out question groups, this mixture changes 3.59% of reference decisions, which is consistent with the 3.59% full-dataset fit in Section 5.2. This is more than double the 1.68% reference disagreement rate of paired unanimity (a reduction of 1.91 pp [1.49, 2.36]). Moreover, the adaptive mixture must fit mixture weights, but paired unanimity achieves this lower disagreement without any threshold tuning.

2/ Why n=4n=4 is structurally significant and how rules compare for n=6n=6 and n=8n=8 (Question 2)

Focusing our single-pair stopping theory on n=4n=4 is not an arbitrary upper limit but reflects a structural boundary. As noted in Section 1 and Section 3.1, four calls constitute the minimal order-balanced reference panel that incorporates replication with 2 seeds per order. In this replicated setting, n=4n=4 is the only even reference size where a decisively unanimous first pair can at worst be tied (00) by the remaining pair and can never be reversed to favor the opposite candidate. Under uniform pair ordering, this guarantees the full reference's expected benchmark score to be preserved with paired unanimity (Proposition 2 and Appendix H.6). For larger reference sizes (n≥6n \ge 6), this guarantee regarding a single pair no longer holds (Appendix D.5). When n=6n=6 (three pairs of votes), even if the first pair reaches decisive unanimity (+2), the subsequent two pairs can both be -2 and thus the initial unanimous decision is reversed.

For larger budgets (n≥6n \ge 6), Proposition 3 (Section 4.1) compares the running signed vote sum StS_t after tt executed calls against the n−tn - t remaining calls. Under uniform ordering of individual calls, the individual non-strict boundary (∣St∣≥n−t\lvert S_t\rvert \ge n - t) preserves expected benchmark scores, while the individual strict boundary (∣St∣>n−t\lvert S_t\rvert > n - t) preserves all reference decisions.

When these conditions are checked only after complete cross-order pairs (t=2kt = 2k) so that all prefixes stay balanced across display orders, the non-strict condition can bias the expected scores under call-level ties, but the paired strict condition (∣S2k∣>n−2k\lvert S_{2k}\rvert > n - 2k, e.g., stopping at t=4t=4 under n=6n=6 once ∣S4∣≥3\lvert S_4\rvert \ge 3 so the final pair cannot tie or reverse the lead) still preserves all reference decisions (Appendix H.7). Across all reasoning configurations (Appendix D.5), the median expected calls on balanced references with n=6n=6 and n=8n=8 (3 and 4 seeds per order) are

Reference BudgetIndividual Non-Strict (Preserves Expected Scores, App D.5)Individual Strict (Preserves Decisions, App D.5)Paired Strict (Preserves Decisions) (new evaluation)
n=6n=63.60 (~40% savings)4.66 (~22% savings)4.90 (~18% savings)
n=8n=84.87 (~39% savings)5.94 (~26% savings)6.59 (~18% savings)

Overall, early stopping still achieves 18%-40% call savings under n=6n=6 and n=8n=8. Although the paired strict condition uses slightly more calls than the individual strict condition because it waits to finish each cross-order pair instead of stopping on odd-numbered calls, it preserves all of reference decisions while keeping display orders balanced after every step and finishes in at most 3-4 parallel pair rounds instead of up to 6-8 rounds under serial individual-call dispatch.

3/ Clarification of Terminology (Weakness 1) While in Section 2 and Table 2 we used distinct symbols for these quantities, we will standardize four distinct terms in the revision so that the word "agreement" isn't overloaded.

  • Expected benchmark score (ApA_p): The expected score of procedure pp against the ground-truth benchmark target yy under Equation (1) (11 for a match, 00 for a mismatch, and 0.50.5 for a neutral decision on binary targets), replacing "expected agreement".
  • Reference decision (Gfull∈{−1,0,+1}G_{\mathrm{full}} \in \lbrace -1, 0, +1\rbrace): The aggregate verdict from executing all calls in a selected reference call set F\mathcal F (+1+1 for A, −1-1 for B, and 00 for neutral).
  • Reference disagreement (dpd_p): The probability by averaging over the evaluation instances that the execution procedure pp returns a different categorical verdict from GfullG_{\mathrm{full}} on the same selected reference F\mathcal F (Equation 4).
  • Paired unanimity: The early-stopping condition in Algorithm 1 in which two decisive calls in opposite display orders (id\mathrm{id} and swap\mathrm{swap}) select the same canonical winner.

4/ Derivation of Proposition 1 (Weakness 1) In Section 3.2, we summarized this derivation in words for space and reading flow. We provide a brief proof outline here to link from scores to polynomials and expectation for your reference. We will add key equations to Section 3.2 for connecting to App H.4 and provide a full proof in Appendix H.4 in the revision.


For a fixed instance x=xix=x_i with signed target y~=2y−1∈{−1,+1}\tilde y = 2y - 1 \in \lbrace -1, +1\rbrace, let sj∈{+1,−1}s_j \in \lbrace +1, -1\rbrace be the display-order bit assigned to seed j∈{1,2,3,4}j \in \lbrace 1, 2, 3, 4\rbrace (+1+1 for id\mathrm{id}, −1-1 for swap\mathrm{swap}), vj(sj)∈{−1,+1}v_j(s_j) \in \lbrace -1, +1\rbrace be its decisive vote, and zj(sj)=y~vj(sj)∈{−1,+1}z_j(s_j) = \tilde y v_j(s_j) \in \lbrace -1, +1\rbrace be its correctness sign.For 0,1,2,3,0, 1, 2, 3, or 44 benchmark matches (zj=+1z_j = +1), the benchmark score in Equation (1) is 0,0,12,1,0, 0, \tfrac{1}{2}, 1, and 11. Over sign states of z=(z1,z2,z3,z4)∈{−1,+1}4\mathbf z = (z_1, z_2, z_3, z_4) \in \lbrace -1, +1\rbrace^4, this step function equals the multilinear polynomial (Appendix H.4, Equation 13) a4(z)=12+316∑j=14zj−116∑1≤j1<j2<j3≤4zj1zj2zj3.a_4(\mathbf z) = \frac{1}{2} + \frac{3}{16}\sum_{j=1}^{4} z_j - \frac{1}{16}\sum_{1 \le j_1 \lt j_2 \lt j_3 \le 4} z_{j_1} z_{j_2} z_{j_3}. Because sj∈{+1,−1}s_j \in \lbrace +1, -1\rbrace is binary, each seed's correctness sign is affine in sjs_j zj(sj)=αj+βjsj,αj=zj(+1)+zj(−1)2,βj=zj(+1)−zj(−1)2.z_j(s_j) = \alpha_j + \beta_j s_j, \\ \\ \alpha_j = \frac{z_j(+1) + z_j(-1)}{2}, \\ \\ \beta_j = \frac{z_j(+1) - z_j(-1)}{2}. We plug zj=αj+βjsjz_j = \alpha_j + \beta_j s_j into zj1zj2zj3z_{j_1} z_{j_2} z_{j_3} and get the order-bit polynomial of at most third degree with quadratic terms αj1βj2βj3sj2sj3+βj1αj2βj3sj1sj3+βj1βj2αj3sj1sj2\alpha_{j_1}\beta_{j_2}\beta_{j_3} s_{j_2}s_{j_3} + \beta_{j_1}\alpha_{j_2}\beta_{j_3} s_{j_1}s_{j_3} + \beta_{j_1}\beta_{j_2}\alpha_{j_3} s_{j_1}s_{j_2}.

For μsame,4\mu_{\mathrm{same},4}, μind,4\mu_{\mathrm{ind},4}, and μbal,4D\mu_{\mathrm{bal},4}^{\mathrm D}, flipping all order bits (s↦−s\mathbf s \mapsto -\mathbf s) leaves the distribution unchanged, so all odd moments E[sj]=E[sj1sj2sj3]=0\mathbb E[s_j] = \mathbb E[s_{j_1} s_{j_2} s_{j_3}] = 0. By symmetry, every distinct pair shares the same pairwise correlation ρ(μ)=Eμ[sj1sj2]\rho(\mu) = \mathbb E_\mu[s_{j_1} s_{j_2}]. Thus Eμ[zj]=αj\mathbb E_\mu[z_j] = \alpha_j, Eμ[zj1zj2zj3]=αj1αj2αj3+ρ(μ)(αj1βj2βj3+βj1αj2βj3+βj1βj2αj3)\mathbb E_\mu[z_{j_1}z_{j_2}z_{j_3}] = \alpha_{j_1}\alpha_{j_2}\alpha_{j_3} + \rho(\mu)(\alpha_{j_1}\beta_{j_2}\beta_{j_3} + \beta_{j_1}\alpha_{j_2}\beta_{j_3} + \beta_{j_1}\beta_{j_2}\alpha_{j_3}). We also reduce the expected score into an affine function of ρ(μ)\rho(\mu) A(x)(μ)=Eμ[a4(z)]=c0+c2,ρ(μ),A^{(x)}(\mu) = \mathbb E_\mu[a_4(\mathbf z)] = c_0 + c_2\\,\rho(\mu), where c0,c2c_0, c_2 depend only on the fixed call outcomes. Finally, ρ(μsame,4)=+1\rho(\mu_{\mathrm{same},4}) = +1 (since s1=⋯=s4s_1=\dots=s_4), ρ(μind,4)=0\rho(\mu_{\mathrm{ind},4}) = 0 (independent bits), and ρ(μbal,4D)=−13\rho(\mu_{\mathrm{bal},4}^{\mathrm D}) = -\tfrac{1}{3} (since ∑j=14sj=0  ⟹  E[(∑j=14sj)2]=4+12ρ=0\sum_{j=1}^4 s_j = 0 \implies \mathbb E[(\sum_{j=1}^4 s_j)^2] = 4 + 12\rho = 0). Because 0=14(+1)+34(−13)0 = \tfrac{1}{4}(+1) + \tfrac{3}{4}(-\tfrac{1}{3}), with affine in ρ\rho, we reach Proposition 1 (Equation 5) A(x)(μind,4)=14A(x)(μsame,4)+34A(x)(μbal,4D).A^{(x)}(\mu_{\mathrm{ind},4}) = \frac{1}{4}A^{(x)}(\mu_{\mathrm{same},4}) + \frac{3}{4}A^{(x)}(\mu_{\mathrm{bal},4}^{\mathrm D}).


We will revise the manuscript as follows: (1) add a summary of terminology in Section 2 and use the terms expected benchmark score, reference decision, and reference disagreement throughout the paper, (2) include the key equations into Section 3.2 and update Appendix H.4 with the full proof we outlined above, and (3) summarize the findings for both the 6-call and 8-call scenarios in the discussion of the n=4n=4 structure in Section 4

Comments on the paper

1 comment

Summary of revisions

Dear Area Chair and Reviewers,

Thank you very much again for your constructive feedback and engagement. We have uploaded the revised manuscript (with new content marked in blue) with both the requested clarifications and new analyses and downstream experiments across the main text and appendices:

  • Terminology and the composition identity (DhYc). Section 2 defines expected benchmark score (replacing "expected agreement"), reference decision, and reference disagreement. Section 3.2 presents the key equations behind Proposition 1, with the full proof in Appendix H.4.
  • Larger references (DhYc, 151g, A2ut). Section 4.1 explains why a unanimous first pair no longer preserves expected score beyond four calls and introduces paired strict stopping, which preserves every reference decision while saving a median of 18% of calls at six and eight calls (Appendix D.4).
  • Accept/defer curation (iad7, 151g). Section 4.2 derives the identity to link reference disagreement to wrong accepted labels. On Nemotron-49B/JudgeBench at matched calls and acceptance, paired unanimity removes 73% of the excess wrong accepted labels of a pre-query mixture, and with no fitted parameters it accepts fewer wrong labels on held-out data than supervised mixtures in all 17 configurations (Section 5.4, Appendix D.7).
  • Held-out reference disagreement (DhYc, 151g). Supervised baseline mixtures fitted to minimize disagreement at the cost and score of paired unanimity still exhibit higher held-out disagreement (e.g. 3.59% versus 1.68% for paired unanimity on Nemotron-49B/JudgeBench) (Section 5.2, Appendix D.6).
  • Human review and reward models on HelpSteer2 (iad7). At matched judge calls, paired unanimity requires 36%-42% fewer human reviews to leave the same number of wrong binary labels, and tie-aware reward models trained on its labels achieve $\sim$7% lower ordinal error in every seed for both judges, with no significant loss in forced-choice binary accuracy (Section 5.4, Appendix I).
  • Relation to ESC and Adaptive-Consistency (151g, A2ut) and expanded mixtures (151g). Section 6 names Early-Stopping Self-Consistency in full and clarifies that the four-call cross-order adaptations of both methods coincide with paired unanimity and inherit Propositions 2 and 4. Section 5.1 clarifies that the expanded mixture can allocate saved calls to run all eight calls on other inputs, which changes the evaluator.

Additionally, for the Appendix, the revision adds Appendices D.6, D.7, and I, and merges several short subsections for clarity. Relative to citations in our earlier responses, now C.5 → C.3, D.4 → D.3, D.5 → D.4, and D.6 → D.5. The decisive-pair counterexample formerly in D.3 is now moved to Section 4.1 and Appendix D.1. All other cited section and equation numbers remain unchanged.


Please let us know if any questions remain or if anything can be clarified further before the discussion period concludes. We hope the revised manuscript and additional experiments fully address your concerns, and we would be grateful if you would consider reflecting these updates in your final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.