Equal Exposure, Different Decisions: Reference Selection and Early Stopping for Pairwise LLM Judges

Repeatedly querying pairwise LLM judges across seeds and display orders can reduce sampling noise and position bias, but increases evaluation cost. We show that neither equal marginal inclusion probabilities across call conditions in reference selection nor preserving the score of a fixed reference in expectation during execution guarantees the same aggregate decisions. In reference selection, equal marginal inclusion probabilities fix expected additive recorded cost but do not determine the joint selection distribution. With four distinct seeds, independent display order randomization captures exactly $\frac{3}{4}$ of the difference in expected agreement between a common display order and exact balance when the corresponding call outcomes are decisive. In reference execution, because neutral decisions receive a score of 0.5, a stopping rule can preserve the reference score in expectation while turning a neutral reference decision into a decisive one. For hard labels, the per-instance probabilities of such changes exactly determine the additional variance in the aggregate score. We introduce paired unanimity stopping, which stops early only when both calls in an order-balanced pair return decisive outputs that agree, a condition that a consistent preference for one display position cannot satisfy. Under uniform matching, its rate of disagreement with reference decisions is at most $\frac{1}{2}$ that of the non-strict boundary under uniform call ordering, which stops when the vote margin is at least the number of remaining calls. Across 8 judges and 17 reasoning configurations on JudgeBench and RM-Bench math/code, it saves 12% to 46% of expected calls relative to full four-call execution while changing at most 1.68% of reference decisions. Matching its agreement requires the best nonadaptive mixture to use 9% to 60% more expected calls. These results establish decision preservation as a primary objective alongside score and cost in repeated LLM judging.