acceptodds
Under review as a conference paper at ICLR 2027

Consensus Need Not Be Evidence: Information Limits and Adaptive Auditing for Multi-LLM Evaluation

Abstract

Multi-LLM evaluation is often justified by agreement: independent judges should cancel idiosyncratic errors. The premise fails when evaluators inherit a common hallucination, retrieval artifact, or semantic convention. We separate the external fact from each evaluator's complete private state and from the transcript generated by peer interaction. This yields three limits. First, we derive a phase boundary above which a less accurate model family receives strictly higher agreement, determinant mutual information, and Shannon mutual information. Second, every peer-only transcript is a garbling of the complete private-state experiment, so its full-coverage factual error cannot beat that experiment's Bayes risk. Third, unlimited unlabeled peer observations cannot identify semantic orientation. Over a truth-swap-closed class, the minimax error is one half, even with selective prediction. We then introduce Gavel, which fits and orients an aggregation policy with verified labels, freezes it, and returns a risk certificate or no certificate using a fresh sequential audit. For heterogeneous task strata, we characterize the information rate of certifying a safe frozen policy by a Chernoff game. An adaptive auditor is anytime-valid and attains the characteristic almost-sure evidence rate. A scalar beta-mixture benchmark admits an exact finite-budget recursion. Synthetic and PEG-policy semi-synthetic experiments expose the reversal and evaluate policy construction and heterogeneous audits. With stratified evidence held fixed, GAVEL reduces mean capped audit draws by 10.6% and 21.2% relative to prevalence sampling on two heterogeneous laws. Frozen-policy comparisons on MMLU and HellaSwag improve verification accuracy over matched single evaluators by 1.07 and 1.14 percentage points. The results separate limits of the peer information state from the evidence needed to orient and audit a frozen policy.

Primary area:
learning theory
Keywords:
LLM evaluation, peer prediction, correlated errors, factuality, statistical identifiability, selective prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yAf1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies multi‑LLM evaluation: high consensus among models does not guarantee correct judgements, as models can share common errors. It analyzes the capability boundary of relying only on internal model information, and derives theoretical findings including consensus reversal, Bayesian error lower bound and unidentifiability of semantic orientation. It proposes GAVEL: train and freeze an aggregation policy using ground‑truth labels, then validate risk via independent sequential auditing, and provide the optimal sampling scheme for stratified auditing. Experiments are conducted on synthetic, semi‑synthetic data and four real‑world benchmarks. Performance gains are mixed on real‑world tasks, and labeling‑cost savings are mainly observed under heterogeneous‑risk scenarios.

Critical strengths

Overall, this paper is clear and rigorous, and presents a closed‑loop investigation. It points out that LLMs can make the same mistakes even when they agree with each other. The paper explains this problem with theory and experiments, and suggests using independent ground-truth labels to check how reliable the system is. It covers problem finding, cause analysis and solution design, and also states when this method works, so it is helpful for us to understand and improve multi‑model evaluation.

Critical weaknesses

While the theoretical analysis in this paper is thorough and rigorous, its incremental contributions compared to prior work are not sufficiently established. The early results on information limits and semantic orientation draw heavily on existing statistical ideas. The more novel part lies in the specific solution for optimal audit allocation. However, the paper’s own control experiments show that the simpler Local allocation achieves most of the gains and performs closely to GAVEL.

As a result, the paper does not fully demonstrate how substantial an advance these theoretical results and methods represent over existing approaches. This shapes my assessment of its novelty and overall significance.

Questions for authors
  1. Could the authors clarify what the semantic‑orientation result adds beyond the sign ambiguity already discussed in unlabeled ensemble learning, such as Jaffe et al. (2015: Estimating the Accuracies of Multiple Classifiers Without Labeled Data, AISTATS)?

  2. Could the authors clarify which parts of the auditing theory are new contributions and which follow from existing anytime-valid risk-control methods?

  3. How do the theoretical results on the limitations of peer‑only evaluation inform the specific design choices in GAVEL?

  4. Given the competitive performance of Local allocation, under what conditions does exact Chernoff allocation provide meaningful additional benefits? A small extension of the existing synthetic experiments, varying the number of strata or their prevalences, would help clarify this.

7 comments
AuthorsOpenReview ↗

Response to yAf1 — Part 1/6: Q1–Q2: Scientific question and information limits

Response to Reviewer yAf1

Thank you for reviewing our paper and for your comments. Below, we respond to each of your concerns point by point.

Q1–Q2 — Scope of the information limits and the auditing contribution

Weakness / Q1–Q2. What does the semantic-orientation result add beyond Jaffe et al.’s sign ambiguity, and “which parts of the auditing theory are new contributions” rather than established anytime-valid risk-control methods?

Our scientific question is which evidence can establish factual reliability under shared evaluator errors, and how many verified labels are needed to certify it. A panel can become more consistent without becoming more correct. This matters when models are used to judge or supervise other models: more peer observations can estimate a misleading agreement ranking more precisely, while verified labels remain costly. The paper separates three questions that require different answers: whether the complete peer state contains the fact, whether unlabeled observations orient that state toward truth, and how fresh labels certify the risk of a chosen policy.

The related work already establishes important parts of this motivation. Kim et al. document correlated LLM errors across a large model collection and examine downstream effects. Goel et al. show how model similarity affects judging and weak-to-strong supervision. Li et al. explicitly model worker dependence through EBCC for label aggregation. Our additional target is the relationship between observable peer evidence, factual identifiability, and the verified-label cost of a risk certificate. The exact reversal boundary, decision-level orientation limit, and aggregate-risk allocation solution address these different parts of that target.

The comparison with Shi et al., Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching, also clarifies the scientific gap. They characterize payoff conditions for desirable preference-based equilibrium properties and establish a preference-matching impossibility within their specified smooth ratio-based payoff class. Our question concerns factual correctness when peer observations themselves may share an error. Preference consistency does not supply a factual orientation assumption. In particular, if a game uses only peer observables that remain unchanged under a truth swap, its output is still subject to our orientation limit, even if it satisfies a preference-consistency criterion. This applies our information model to game outputs built from peer observations. Together, the two perspectives motivate methods that state both what their interaction objective preserves and what external evidence grounds its factual interpretation.

What the orientation result adds. Jaffe et al. explicitly identify sign ambiguity. Their estimation results use conditional independence, and their setup fixes orientation by assuming that a majority of classifiers have balanced accuracy above one half. Theorem 3 — Exact semantic-orientation minimax limit states a decision-level identity for the same learner in two observationally identical worlds:

RiskP(h)+RiskP↔(h)=1.\mathrm{Risk}_{P}(h)+\mathrm{Risk}_{P^{\leftrightarrow}}(h)=1.

It allows dependent peer states, arbitrary peer-only interaction, and arbitrarily many unlabeled training samples. The same identity holds conditional on a peer-observable acceptance event with common positive coverage. Consequently, the minimax risk on any nonempty truth-swap-closed class is exactly 1/21/2. Confidence-based abstention cannot orient the accepted predictions. An assumption that a designated evaluator is reliably better than chance can break the symmetry; verified labels provide another source of orientation. The contribution is the scope and decision consequence of the ambiguity, including selective prediction.

Theorem 2 compares a transcript with the evaluators' complete private state, not only their initial answers. Interaction may improve computation or reveal information already present in that state. The bound identifies the residual uncertainty that such interaction cannot remove without additional evidence. Its proof builds on classical comparison-of-experiments and data-processing arguments.

AuthorsOpenReview ↗

Response to yAf1 — Part 2/6: Q1–Q3: Auditing contributions and theory-guided design

What the auditing theory adds. Ville's inequality, predictable betting, and sequential change-of-measure lower bounds are established. The problem-specific step is solving the allocation game for a composite aggregate-risk null:

\sum_k w_k\\,\mathrm{kl}(r_k,q_k).$$ The closest **unsafe profile** depends on the allocation, and it can move risk between strata. A variance rule sees only the local geometry. `Proposition 2 — One-dimensional Chernoff characterization` solves the full game for interior safe risks through a single equal-KL equation. It gives a unique positive optimal allocation and the characteristic rate $C^{\ast}$. The decomposition $C^{\ast}\geq C_\pi\geq\mathrm{kl}(\pi^\top r,\alpha)$ distinguishes the gain from allocating labels from the gain from retaining stratum identities. The fixed-allocation likelihood-ratio calculation connects the same unsafe profile to the log-optimal evidence factor. `Theorem 4` constructs anytime-valid evidence and, under the stated convergent-design assumptions, attains the characteristic almost-sure evidence and stopping-time rates. `Theorem 5` gives the expected-label lower bound with the same constant. **The adaptive result is an almost-sure rate matched to an expected-time lower bound. Expected-time attainment is established only for the scalar benchmark.** The derivation combines convex saddle-point/KKT geometry with predictable evidence and change of measure. The new `Proposition 3` and `Theorem 6`, detailed in Q4, address why this exact solution can matter despite Local's strong performance. The former supplies an exact loss identity and computable bounds. The latter identifies when Local's rate collapses and gives a sharp separation rate. A nearly error-free stratum can still require substantial auditing because an unsafe alternative can place enough errors there. This provides a concrete direction for future work: estimate whether such alternatives remain plausible, and choose audit initialization or allocation accordingly. **Revision.** Sections 2–4 and Appendix A.1 now distinguish the contributions from the results of Kim, Goel, Li, Jaffe, and Shi, and clarify the complete-state and orientation assumptions. Section 6 separates the explicit aggregate-risk solution from the classical validity tools and states the scope of the rate guarantees. It also contains the new `Proposition 3` and `Theorem 6`; their full proofs are in Appendices D.10 and D.11. ### Q3 — How the limits determine GAVEL's design > **Q3. “How do the theoretical results on the limitations of peer‑only evaluation inform the specific design choices in GAVEL?”** Each stage addresses a specific obstruction. The results justify verified orientation, the separation of policy fitting from auditing, and the allocation of fresh evidence. **`Theorem 1 — Exact agreement reversal`** In the stated shared-error family model, crossing an explicit dependence threshold makes the less accurate family score higher under agreement, determinant mutual information, and Shannon mutual information. GAVEL therefore fits against verified truth rather than peer agreement. The synthetic study checks the phase boundary; the semi-synthetic MMLU intervention reverses the ranking at 0.85 on its grid. In the **new real-panel experiment** added for this revision in response to Reviewer `Lrwg`'s Q1, the natural same-family panel has `3.92 percentage points more agreement but 3.84 points less accuracy` than the cross-family panel, without injected errors. **`Theorem 2 — Complete-state Bayes barrier` and `Proposition 1 — Exact shared-error floor`** A peer-only transcript cannot outperform the Bayes decision based on the complete private state. In the shared-flip model, the exact ensemble risk is $R_n^{\ast}=\beta+(1-2\beta)\delta_n$: increasing the number of judges removes the idiosyncratic term but leaves the common-error floor. These results allow interaction to improve access to existing information and computation. They identify when adding judges or discussion still leaves factual uncertainty. GAVEL acquires verified evidence rather than treating a larger panel as a certificate. The shared-flip experiment checks the exact residual floor and the decay of the idiosyncratic term.
AuthorsOpenReview ↗

Response to yAf1 — Part 3/6: Q3: Verified fitting and the fit–freeze–audit pipeline

Theorem 3 — Exact semantic-orientation minimax limit

Unlabeled peer observations cannot select between truth-swapped worlds, even with dependence, interaction, or peer-observable selective acceptance. A truth-swap-closed class has minimax risk 1/21/2.

Verified fitting labels orient the builder before it is frozen. Unlabeled data can still estimate useful report structure; they do not generally determine its alignment with truth. The anchor study in Appendix E.2, based on Appendix B.7, covers four signal strengths, nine label budgets, and 100,000 repetitions per direction. It checks how external labels resolve the ambiguity, while preserving the distinction between orientation and residual deployment-time uncertainty.

Proposition 5 — Oracle correlated-error certificate

The population certificate for a weighted report rule depends on both its alignment with truth and the covariance of its truth-aligned errors. Unlabeled reports identify M=E[AA⊤]M=\mathbb E[AA^\top]; they do not generally identify μ=E[YA]\mu=\mathbb E[YA].

The regularized builder combines an unlabeled estimate of MM with verified fitting estimates of μ\mu, then freezes its coefficients and threshold. The moment identities and Cantelli bound motivate this choice. Experiments compare nine builders at three synthetic fitting budgets and ten on the real panel. The four cached tasks retain mixed outcomes: +1.07, +1.14, 0.00, and -1.85 percentage points versus matched fitted single evaluators on MMLU, HellaSwag, ARC-Challenge, and WinoGrande. Ridge is one eligible builder; the auditor can also assess a frozen EBCC predictor or single judge.

Proposition 2 — One-dimensional Chernoff characterization

For interior safe risks, an equal-KL unsafe profile determines the unique positive allocation that maximizes the worst-case information rate.

Fresh audit labels are directed toward the unsafe alternatives that are hardest to exclude. This determines the exact target used by GAVEL's adaptive allocation. The 192-setting known-risk grid and the matched-evidence allocation experiments test that consequence while holding the evidence construction fixed. The new Proposition 3 quantifies the loss from using Local, and Theorem 6 identifies its boundary failure; Q4 gives their conditions and the complete new mechanism experiment.

Theorem 4 — Anytime validity and characteristic evidence rate and Theorem 5 — Information-theoretic lower bound

After the policy and audit population are frozen, predictable sampling and valid evidence factors permit adaptive allocation and optional stopping while controlling false certification. Under the stated convergence assumptions, the evidence attains the characteristic rate; every qualifying valid procedure faces the corresponding expected-label lower bound.

This yields fit–freeze–audit separation: fitting labels choose the policy, fresh labels assess it, and stopping requires the evidence threshold. The real-panel replay tests this pipeline at four preset targets. At the primary target, GAVEL gains 2.71 accuracy points over the development-selected native single judge and uses 34.5% fewer audit labels; both certify on all replay paths. Independent Bernoulli null checks assess the implementation. These savings concern the full fitted policy and audit, not Exact allocation alone.

The counterintuitive implication is that agreement, confidence, and low observed error can each be insufficient for a different reason. Agreement may reward shared mistakes; confidence may not resolve semantic orientation; low variance may hide an unsafe alternative from Local. The design uses external labels at the stages where they resolve the relevant uncertainty. Future extensions can study richer sources of external evidence, noisy anchors, and estimation-aware allocation while retaining a separate audit of the final policy.

Revision. Sections 4–6 now state these links between the limits, builder, and auditor explicitly. Section 7 reports the corresponding mechanism and empirical checks. Natural reversal, real-panel aggregation, and the full audit replay appear in Appendices E.12, E.13, and E.14, respectively. Appendix E.10 retains the complete initialization and audit-length comparison; Appendix E.15 adds the Exact/Local mechanism checks.

Reviewer_JLqcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the limitations of consensus-based multi-LLM evaluation, particularly when evaluators share correlated errors. The authors establish several theoretical results showing that higher agreement does not necessarily imply greater factual accuracy, peer-only interactions cannot overcome information limitations, and unlabeled observations cannot resolve semantic ambiguity. To address these challenges, they propose GAVEL, a framework that uses verified labels to construct and freeze an aggregation policy, followed by adaptive sequential auditing to certify its risk

Critical strengths
  1. The paper addresses a meaningful limitation of multi-LLM evaluation: consensus can be misleading when evaluators share correlated errors. The distinction between peer agreement and factual correctness provides a useful perspective on the reliability of multi-agent evaluation.

  2. The paper provides a systematic theoretical treatment of consensus limitations and risk certification, including information-theoretic bounds, an explicit characterization of optimal audit allocation, and anytime-valid statistical guarantees. The analysis is supported by controlled experiments and evaluations on real LLM reports.

Critical weaknesses
  1. The main findings on consensus failure and adaptive auditing are primarily demonstrated through synthetic or semi-synthetic experiments, with limited validation in realistic multi-LLM evaluation settings. Moreover, the results on real LLM reports show modest and inconsistent improvements. GAVEL achieves only marginal accuracy gains on MMLU and HellaSwag, no improvement on ARC-Challenge, and worse performance on WinoGrande. These results raise concerns about the practical effectiveness and generalizability of the proposed framework.

  2. Several closely related studies, including Correlated Errors in Large Language Models, Great Models Think Alike and This Undermines AI Oversight, and Exploiting Worker Correlation for Label Aggregation in Crowdsourcing, are not adequately discussed. These works have already investigated correlated errors among language models and correlation-aware aggregation, which overlap substantially with the paper's motivation and methodology. Without a clearer comparison and positioning against these studies, it is difficult to assess the novelty and distinct contributions of the proposed approach.

Questions for authors
  1. Can the authors provide stronger empirical evidence of the proposed framework's effectiveness in realistic multi-LLM evaluation settings, particularly with naturally occurring correlated errors beyond synthetic or artificially introduced dependencies?

  2. Could the authors clarify how their contributions differ from closely related studies, such as Correlated Errors in Large Language Models, Great Models Think Alike and This Undermines AI Oversight, and Exploiting Worker Correlation for Label Aggregation in Crowdsourcing?

4 comments
AuthorsOpenReview ↗

Response to JLqc — Part 1/4: Q2: Scientific question, related work, and reversal

Response to Reviewer JLqc

Thank you for reviewing our paper and for your comments. Below, we respond to each of your concerns point by point.

W2 / Q2 — Scientific question and distinction from prior work

W2 / Q2. The discussion of closely related work is incomplete. How do this paper's scientific question, mathematical results, and algorithmic consequences differ from work on correlated LLM errors and dependence-aware aggregation?

Our scientific question is: when judges share errors, what evidence establishes factual reliability, and how many verified labels does that evidence require? This matters because a panel can become more consistent without becoming more correct. A more accurate estimate of consensus may then make a wrong evaluator ranking more certain. We ask both what peer reports can identify and what additional evidence can certify a frozen evaluator's risk below a chosen target.

The cited studies establish the motivation and several useful prediction methods. Kim et al. document correlated LLM errors; Goel et al. connect mistake similarity to judge affinity and oversight; Li et al. exploit worker dependence to infer labels. These studies do not formulate our aggregate-risk certification problem or determine the allocation of fresh verified labels that solves it. Jaffe et al. estimate classifier accuracies under conditional-independence and orientation assumptions. We characterize the decision risk when such orientation is not supplied, allowing dependent reports and interaction. The gap is the connection from peer observations to factual orientation, and from orientation to an information-efficient certificate.

Shi, Liu, Long, Su, and Xiao, Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching, provide a particularly useful comparison. They study how pairwise-preference payoffs affect Nash alignment, characterize Smith consistency, and establish a preference-matching impossibility within their specified smooth ratio-based payoff class. Their obstruction concerns the design of a preference objective. Our truth-swap obstruction concerns observational equivalence: the same peer data can correspond to opposite factual labels. A payoff satisfying a preference-consistency property does not by itself identify that factual orientation. Our subsequent question is also different: once a policy is frozen, how should verified labels be allocated to exclude every unsafe risk profile? Their payoff characterization does not answer this audit-design question. The connection is constructive: both analyses turn a mismatch between an internal objective and its intended target into explicit design conditions.

Related workMain object and resultContribution addressed by our paper
Kim et al., Correlated Errors in Large Language ModelsShared errors across LLMs and their consequences for judging and hiringAn exact reversal threshold for three peer scores in a specified channel, followed by risk certification
Goel et al., Great Models Think Alike and this Undermines AI OversightMistake similarity, CAPA, judge affinity, and weak-to-strong supervisionLimits on factual information and orientation available to peer interaction
Li et al., Exploiting Worker Correlation for Label Aggregation in CrowdsourcingEBCC models dependence through latent subtypes to infer labelsVerified-risk certification of any frozen builder, including EBCC, without requiring its latent model to be correct
Jaffe et al., Estimating the Accuracies of Multiple Classifiers without Labeled DataAccuracy estimation under conditional-independence and better-than-chance assumptionsAn exact orientation risk identity allowing dependent reports, interaction, and selective acceptance
Shi et al., Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference MatchingPayoff conditions and limits for preference alignment through Nash policiesFactual orientation and the audit-label cost of excluding unsafe aggregate risk

The central difficulty is that dependence can carry information about other judges without identifying truth. After orientation, a second difficulty appears: the unsafe alternative that is hardest to reject changes with the sampling allocation. We make these questions tractable by combining Blackwell's comparison of experiments and truth-swap symmetry with controlled sensing and predictable betting. The first pair separates information availability from semantic orientation. The second turns certification into a coupled max–min problem over sampling designs and unsafe risk profiles.

Theorem 1 — Exact agreement reversal gives the family-channel boundary where agreement, determinant mutual information, and Shannon mutual information all favor the less accurate family.

AuthorsOpenReview ↗

Response to JLqc — Part 2/4: Q2: New theory, design implications, and validation

Theorem 3 — Exact semantic-orientation minimax limit gives the risk-sum identity for indistinguishable truth-swapped worlds, including peer-based selective acceptance. Thus more precise consensus and higher confidence need not resolve factual orientation. Jaffe et al. supply orientation through a better-than-chance assumption; GAVEL uses verified fitting labels. Theorem 2 permits debate to improve computation within the available information state. A verified observation can enlarge that state and change its Bayes limit.

For frozen stratum risks rr, prevalences π\pi, and target α\alpha, the certification rate is

C∗(r)=max⁡w∈ΔKmin⁡q:π⊤q≥α∑kwk,kl(rk,qk).C^{\ast}(r)=\max_{w\in\Delta_K}\min_{q:\pi^\top q\geq\alpha}\sum_k w_k\\,\mathrm{kl}(r_k,q_k).

Proposition 2 — One-dimensional Chernoff characterization solves this allocation game through one equal-KL equation. Theorems 4–5 connect its rate to attainable almost-sure evidence growth and an expected-label lower bound. The audit remains valid under predictable sampling and stopping, conditional on the frozen setup and the paper's fresh-sampling assumptions. Correlation among judges on the same item is allowed. Building on classical sequential-testing principles, we solve the aggregate-risk allocation problem and derive its label-cost consequences.

The revision adds a quantitative loss identity and a sharp boundary result:

For this comparison, Local uses wkL=πkrk(1−rk)/∑jπjrj(1−rj)w_k^L=\pi_k\sqrt{r_k(1-r_k)}/\sum_j\pi_j\sqrt{r_j(1-r_j)}, and CLC_L denotes its worst-case information rate.

Proposition 3 — Exact loss and optimality criterion for Local.

For positive prevalences and interior safe risks, let q∗q^{\ast} and qLq^L be the least-favorable unsafe alternatives under Exact and Local. With ϕk(q)=kl(rk,q)\phi_k(q)=\mathrm{kl}(r_k,q) and BϕkB_{\phi_k} its Bregman divergence,

C∗−CL=∑kwkLBϕk(qk∗,qkL).\boxed{C^{\ast}-C_L=\sum_k w_k^L B_{\phi_k}(q_k^{\ast},q_k^L)}.

This loss has explicit lower and upper curvature bounds proportional to the squared separation of the alternatives. It is zero if and only if rk(1−rk)(qk∗−rk)/[qk∗(1−qk∗)]\sqrt{r_k(1-r_k)}(q_k^{\ast}-r_k)/[q_k^{\ast}(1-q_k^{\ast})] is constant across strata. Thus the result determines both when Local is optimal and how much information it loses otherwise.

The added loss identity and bounds quantify Local's inefficiency at any fixed safety margin, beyond the optimality criterion derived from the original Chernoff solution.

Theorem 6 — Boundary separation of Exact and Local.

Partition the strata into two nonempty sets: ZZ has interior risks tending to zero, while BB has fixed interior risks bkb_k. With fixed positive prevalences, write pZ=∑Zπkp_Z=\sum_Z\pi_k, RB=∑BπkbkR_B=\sum_B\pi_kb_k, and fix RB<α<1R_B < \alpha < 1. Then Local's rate tends to zero if and only if α−RB≤pZ\alpha-R_B\leq p_Z, while Exact's rate tends to a positive constant. If every risk on ZZ equals ε\varepsilon and a=(α−RB)/pZ∈(0,1)a=(\alpha-R_B)/p_Z\in(0,1), the sharp general-KK rate is

&\boxed{C_L\sim A\sqrt{\varepsilon}},\\\\ &A=\frac{p_Z[-\log(1-a)]}{\sum_{k\in B}\pi_k\sqrt{b_k(1-b_k)}}. \end{aligned}$$ Exact retains positive limiting sampling mass on each low-risk stratum; Local's mass there vanishes. This **boundary result** goes beyond the fixed-interior, small-margin expansion: it identifies when Local becomes arbitrarily inefficient and gives the sharp rate under the stated common-risk and strict-capacity conditions. **Low error variance can coexist with high certification value.** A nearly error-free stratum can still hold enough hypothetical error mass to violate the aggregate target. It must be sampled to exclude that unsafe explanation. This is the design implication that a variance-only allocation misses near the boundary. It suggests risk-sensitive allocation diagnostics and better use of independent pre-audit estimates, while keeping the original valid GAVEL evidence process. The new numerical study covers **`1,248 grid configurations plus eight reference cases`**, with up to 128 strata. The exact loss identity has maximum absolute residual **$2.68\times10^{-11}$**, with no curvature-bound violation. At $\varepsilon=10^{-8}$, all **104** strict-capacity settings match the general-$K$ sharp rate within **`0.5%`**; all **52** above-capacity controls retain a positive limiting Local rate. These known-risk calculations test the mathematical mechanism. The finite-budget study below separately measures its operational effect. Other frozen builders, including EBCC or a debate policy, can enter the same audit; the framework therefore supports extensions beyond the particular aggregation rule. **Revision.** Section 2 and Appendix A.1 now cite and compare Kim et al., Goel et al., Li et al., Jaffe et al., and Shi et al., including the distinction between preference-payoff design and factual-risk certification.
AuthorsOpenReview ↗

Response to JLqc — Part 3/4: Q2–Q1: Revision details and real-panel experiments

Sections 3–4 clarify the information boundary, computational benefits of debate, and the comparison with Jaffe et al. after Theorem 3. Section 6 adds Proposition 3 and Theorem 6; Appendices D.10–D.11 give their complete proofs, and Appendix E.15 reports the new quantitative checks and complete numerical grid.

W1 / Q1 — Natural LLM errors and the complete pipeline

W1 / Q1. Does the framework work on real multi-LLM outputs with naturally occurring correlated errors? Please show the reversal, the complete fit–freeze–audit pipeline, and its accuracy and label-cost effects.

The experiment covers five real judges, ten aggregation methods, two data sources, and four audit targets. Its 909 eligible questions (318 development, 591 evaluation) and two answer orders yield 9,090 judgments. Questions are independent evaluation units; orders are averaged within each question. No shared errors are injected.

Evaluation dataQuestionsSame-family accuracy (%)Cross-family accuracy (%)Same-family agreement (%)Cross-family agreement (%)
Overall59154.6858.5281.4477.52
MATH43056.2460.5885.1979.73
GPQA16150.5253.0071.4371.64

The same-family panel is Qwen2.5-32B/14B/7B; the cross-family panel replaces the two smaller judges with Mistral-Small-24B-Instruct-2501 and Llama-3.3-70B-Instruct. Same-family agreement is 3.92 pp higher [1.75, 6.03] overall; cross-family accuracy is 3.84 pp higher [2.57, 5.13]. GPQA does not establish this joint reversal. Pointwise intervals use 8,192 source-stratified paired question-group bootstrap draws; this panel comparison was selected after the initial evaluation. Both answer orders and the original-format parser support the overall reversal.

The real-panel figure shows agreement and accuracy contrasts between the same-family and cross-family panels.

GAVEL improves over native single and the original EBCC baseline on the combined set. All methods share the 318-question development split; supervised builders use its verified labels. Policies are frozen before evaluation. Native single is development-selected; EBCC uses a frozen inductive adaptation. Accuracy averages each question's two answer orders (%). Added pointwise 95% intervals use 8,192 source-stratified question-group bootstrap draws shared across all ten frozen methods. Overall intervals retain the observed mix of 430 MATH and 161 GPQA questions and condition on frozen policies, excluding refitting uncertainty.

MethodOverall (%)95% CIMATH (%)95% CIGPQA (%)95% CI
Native single59.90[57.36, 62.35]63.60[60.70, 66.51]50.00[45.34, 54.66]
Calibrated single60.24[57.70, 62.69]64.07[61.16, 66.98]50.00[45.34, 54.66]
Equal vote55.84[53.81, 57.95]57.44[55.35, 59.65]51.55[46.89, 56.21]
Same-family vote55.25[53.30, 57.28]56.86[54.88, 58.95]50.93[45.96, 55.90]
Cross-family vote58.12[55.75, 60.49]60.93[58.26, 63.60]50.62[45.65, 55.59]
Family-balanced vote56.51[54.31, 58.71]58.60[56.16, 61.05]50.93[45.96, 55.90]
Dawid–Skene56.43[54.31, 58.54]58.37[56.16, 60.58]51.24[45.96, 56.28]
EBCC56.43[54.31, 58.54]58.37[56.16, 60.58]51.24[45.96, 56.28]
Moment-ridge60.58[58.04, 63.11]64.42[61.40, 67.44]50.31[45.65, 55.28]
GAVEL62.61[59.90, 65.23]66.51[63.37, 69.53]52.17[47.20, 57.45]

GAVEL improves over native single by 2.71 pp [1.27, 4.23]. This paired-gain interval retains the original ordinary question-group bootstrap; the table adds source-stratified absolute-accuracy intervals. The GPQA gain alone remains statistically unresolved. All ten methods are retained, including identical Dawid–Skene and EBCC outcomes on this cache.

The same frozen GAVEL and native-single policies are audited with equal development-search budgets. Complete costs at δ=0.05\delta=0.05 are:

Target error (%)GAVEL audit labelsSingle audit labelsAudit saving (%)GAVEL certifiedSingle certifiedGAVEL total labelsSingle total labels
48.68136.62208.4834.5256/256256/256454.62526.48
43.68389.57544.1128.4231/25625/256707.57862.11
41.68511.51575.1111.150/2568/256829.51893.11
40.68553.51584.825.422/2563/256871.51902.82

Means use 256 paired cache replays; failures incur the 591-label cap. Targets are set before evaluation from development out-of-fold risk plus fixed margins.

AuthorsOpenReview ↗

Response to JLqc — Part 4/4: Q1 / W1: Total label costs and practical effectiveness

Totals include all 318 development labels. The primary target saves 34.5% of audit labels and 13.65% of total labels. At 43.68% risk, certification rises from 9.77% to 90.23%. These differences compare frozen policies with their auditors; the matched Exact/Local allocation comparison appears separately below.

The real-audit figure visualizes audit and total-label savings and certification rates at all four preset risk targets.

Revision. Section 7 reports the natural reversal and the real-panel fit–freeze–audit results. Appendix E.12 gives the panel and answer-order protocol, Appendix E.13 contains all ten aggregation methods with the added source-specific accuracy intervals, and Appendix E.14 reports all four targets, capped audit costs, total fitting-plus-audit costs, and certification counts.

W1 — Practical effectiveness across tasks

W1 — Practical effectiveness. Gains on MMLU and HellaSwag are modest, ARC-Challenge is unchanged, and WinoGrande is worse. When does aggregation help, what explains the loss, and can certification help when aggregation does not?

GAVEL's builder improves three of five settings, ties on one, and loses on WinoGrande. The direct-panel result complements these four cached tasks. The revised abstract includes the loss.

TaskGAVEL accuracy (%)Fitted-single accuracy (%)Gain (pp)Paired 95% interval (pp)
MMLU77.1676.08+1.07[+0.38, +1.77]
HellaSwag89.5088.36+1.14[+0.65, +1.63]
ARC-Challenge96.5096.500.00[0.00, 0.00]
WinoGrande79.9581.79-1.85[-2.90, -0.92]

MMLU and HellaSwag gains mean 4.5% and 9.8% fewer errors. ARC-Challenge has 41 single-judge errors among 1,170 questions; the selected aggregate makes identical decisions. On WinoGrande, a two-error fitting advantage on 509 rows (88 versus 90) reverses at evaluation (152 versus 138). Of 16 additional acceptances, 15 are incorrect. The narrow fitting advantage was insufficient to select a better policy. Auditing remains separate: it assesses the frozen policy, including a single judge; audit labels cannot be used to repair its predictions.

Proposition 3 and Theorem 6 address a separate design question: when can Exact improve the information efficiency of auditing a fixed policy? An oracle advantage must also be learned early enough to affect finite stopping cost. We test this across 96 synthetic risk settings, two initializations, three confidence levels, and 256 paired paths per setting. The complete heterogeneous portion is summarized below; homogeneous controls are reported separately.

Strata KKPrevalenceC, δ=.05\delta=.05P, δ=.05\delta=.05C, δ=.001\delta=.001P, δ=.001\delta=.001C, δ=10−6\delta=10^{-6}P, δ=10−6\delta=10^{-6}
2U-0.452.950.838.951.0715.58
2G-0.020.150.080.920.091.46
2R-0.612.35-0.444.600.355.84
4U-0.172.170.517.450.7411.93
4G0.141.730.450.990.502.38
4R0.043.48-0.429.250.6112.34
8U0.903.680.517.220.2910.57
8G-0.121.040.483.360.605.15
8R-0.103.220.146.980.3810.81
16U0.014.06-0.026.010.117.87
16G0.292.470.544.990.567.20
16R0.243.730.137.000.179.09

Entries are median Exact-over-Local audit savings (%), each covering three risk patterns and two margins. U/G/R denote uniform, geometric (16:1), and one-5%-stratum prevalences. C uses common mean 3α/43\alpha/4 and strength 64; P uses 50 independent pre-audit labels per stratum and strength 50. Other audit components are shared within columns; C/P changes both initialization mean and strength.

Across 72 heterogeneous settings, P yields median savings 2.43%, 5.82%, and 9.05% as δ\delta decreases; including pre-audit labels gives 0.78%, 2.47%, and 4.65%. Common-initialization medians are -0.09%, 0.23%, and 0.53%. Informative estimates and longer audits can therefore reveal the allocation advantage predicted by the new results.

Revision. The abstract and Section 7 retain all four cached-task outcomes, including the WinoGrande loss. Appendix E.7 explains the decision changes and fitting-margin failure. Appendix E.10 now presents the complete initialization and confidence comparison through the 12-row heterogeneous summary, the homogeneous-control and total-cost summary, and a heatmap containing all 576 cells. The earlier 32-setting close-results control is retained in the text, with its full table archived. Appendix E.15 connects the new numerical mechanism checks to Proposition 3 and Theorem 6.

Reviewer_LrwgOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies when agreement among multiple LLM evaluators constitutes evidence about whether something is true, and how much verified evidence is needed to certify a frozen evaluation policy. The first part is theory: when LLM evaluators are correlated in their errors, the less accurate group can actually agree more and score higher on agreement than the more accurate group. Debate between the evaluators cannot add any more information than they already hold. In the second part, the authors propose GAVEL (grounded aggregation with verified evidence labels): it fits a weighted vote over the judges using labelled data, locks that voting rule, and checks it on fresh labels to get an evidence score, until there is enough evidence to confidently state that the error rate is below a target. In experiments, the theoretical effects show up in synthetic data, and the adaptive checking saves 10–21% of labels in synthetic settings. On real cached judge outputs, the method shows about 1 pp gain over the best single judge on two benchmarks, ties on one and loses on another.

Critical strengths
  1. The theoretical results are stated precisely and the math appears correct. The experiments are also neatly reported, including stress tests.
  2. The problem is relevant. The point that agreement among LLM evaluators is not evidence when errors are shared matters from LLM-as-judge and debate-based evaluation, and the paper applies established practice (held-out sets, auditing) to this setting.
Critical weaknesses
  1. Limited novelty, obscured by the presentation. Most of the theoretical contributions restate established, classical results: theorem 2 is the data-processing inequality applied to debate. This is simplistic and excludes computational effects, which are how debate is usually thought to help. Theorem 1 is just a correlated voter result. The main new result, the exact optimal audit allocation, adds little over a simple local allocation, as the authors themselves note. The dense terminology makes all this hard to see. After careful reading, I couldn’t identify an insight that goes meaningfully beyond existing work.
  2. The experiments don’t test the central claims. The agreement reversal (theorem 1) is never shown in real judge panels, and the certification procedure is tested only on synthetic data. On real model outputs, GAVEL gains about 1pp on two benchmarks, ties on one and is significantly worse on one, and the abstract reports only the gains.
Questions for authors

Q1 - can the authors show the agreement reversal in real cases of LLM judge panels without injected shared errors? For example, compare same family judges with cross family judges.

Q2 - can the authors run the full fit freeze and audit pipeline on a real evaluation task and report the label cost of certifying a realistic target?

9 comments
AuthorsOpenReview ↗

Response to Lrwg — Part 1/6: W1: Scientific contributions and information limits

Response to Reviewer Lrwg

Thank you for reviewing our paper and for your comments. Below, we respond to each of your concerns point by point.

W1 — Scientific contribution and the value of Exact allocation

W1. The novelty is unclear: Theorem 1 appears to restate correlated-voter behavior, Theorem 2 applies data processing without capturing the computational benefits of debate, and Exact seems to add little over Local. Which contributions go beyond these foundations, and can the presentation make them clear?

Our scientific question is: when judges share errors, what evidence establishes factual reliability, and how many verified labels does that evidence require? This matters when an LLM panel is used to approve answers, assess another model, or judge a debate. A panel may become more consistent while remaining wrong. The required output is therefore a statement about the factual risk of its decisions, not agreement alone.

The prior work establishes important parts of this problem. Kim et al., Correlated Errors in Large Language Models measure shared LLM errors and their consequences for judging and hiring. Goel et al., Great Models Think Alike and this Undermines AI Oversight introduce CAPA and relate model similarity to judge affinity and complementary knowledge. Li et al., Exploiting Worker Correlation for Label Aggregation in Crowdsourcing use latent within-class subtypes to model dependent annotations and infer labels. These works motivate or improve prediction from dependent reports. They do not formulate and solve the certification problem studied here: testing every unsafe aggregate-risk profile of a frozen policy, with fresh verified outcomes and a quantified label requirement.

Shi et al., Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching provide a useful related comparison. They study how transformations of pairwise human preferences determine Nash-alignment policies, characterize Smith consistency, and prove a preference-matching impossibility for their specified payoff class. That impossibility concerns constant diagonal payoffs and a common smooth off-diagonal function of policy-probability ratios, independent of response count. It is a restriction on objective design. Our obstruction is observational: two worlds can produce exactly the same peer reports while reversing factual correctness. If a payoff and its solution are functions of those reports, the truth swap preserves both. Preference consistency alone then cannot identify external factual risk. Our additional task is to acquire verified outcomes and allocate them efficiently to certify that risk.

The central difficulty is that dependence can reveal what other judges will say without identifying which answer is true. Theorem 1 — Exact agreement reversal gives the precise threshold, in the stated family channel, at which agreement, determinant mutual information, and Shannon mutual information all rank the less accurate family higher. The contribution is the joint, exact reversal boundary for all three peer scores. A counterintuitive consequence is that estimating those peer scores more accurately can make the wrong ranking more certain. The real-panel experiment in Q1 below exhibits the predicted qualitative reversal: the same-family panel agrees more while being less accurate overall.

Theorem 3 — Semantic orientation addresses a different ambiguity. The same learner has risks summing to one in truth-swapped worlds with identical peer observations, including dependent reports, interaction, and peer-based selective acceptance with positive coverage. Jaffe et al. already recognized sign ambiguity and resolved it under conditional-independence and orientation assumptions. Our addition is this exact decision-risk identity beyond that model. A valid orientation assumption can enable label-free recovery; our identity characterizes the decision risk when that orientation is unavailable.

Debate can improve reasoning. Theorem 2 applies Blackwell's comparison of experiments to the complete information available to the protocol. Its benchmark is the best decision based on the complete information available to the protocol, rather than the quality of its initial answers. Debate can approach that benchmark through computation. If the answer is determined by the problem statement, the Bayes error is zero and the theorem leaves computational gains unrestricted. A positive barrier concerns missing factual information; retrieval or a verified observation changes the available information and can change the bound. This separates computational improvement from the acquisition of new evidence.

AuthorsOpenReview ↗

Response to Lrwg — Part 2/6: W1: Exact audit allocation and new theoretical results

The constructive contribution is an explicit solution for the resulting aggregate-risk audit. Verified fitting labels orient a policy; freezing makes its risk a fixed target; fresh audit labels test that target. For risks rr, prevalences π\pi, and target α\alpha, the optimal information rate is

C∗(r)=max⁡w∈ΔK;min⁡q:π⊤q≥α∑kwk,kl(rk,qk).C^{\ast}(r)=\max_{w\in\Delta_K}\\;\min_{q:\pi^\top q\geq\alpha}\sum_k w_k\\,\mathrm{kl}(r_k,q_k).

The hardest unsafe profile changes with the allocation, so optimizing each stratum separately does not solve the problem. Proposition 2 reduces this coupled optimization to one scalar equal-KL equation and gives the unique positive allocation for interior safe risks. Theorem 5 places the same C∗C^{\ast} in the expected-label lower bound. Theorem 4 constructs anytime-valid evidence with the almost-sure characteristic evidence and stopping-time rates under its convergence conditions. The decomposition C∗≥Cπ≥kl(π⊤r,α)C^{\ast}\geq C_\pi\geq\mathrm{kl}(\pi^\top r,\alpha) separates gains from allocation and from retaining stratum identities. Using classical controlled-sensing and predictable-betting tools and Ville's inequality, we obtain the explicit aggregate-risk solution, its almost-sure attainment, and the same characteristic constant in the expected-label lower bound.

We have added two results to the main text to identify precisely when Local is sufficient and when it can fail. Their full proofs and quantitative details are in the appendix. The fixed-interior small-margin expansion in Proposition 6 explains Local's strong baseline; the new results describe its loss at arbitrary margins and near the risk boundary.

For this comparison, Local uses wkL=πkrk(1−rk)/∑jπjrj(1−rj)w_k^L=\pi_k\sqrt{r_k(1-r_k)}/\sum_j\pi_j\sqrt{r_j(1-r_j)}, and CLC_L denotes its worst-case information rate.

Proposition 3 — Exact loss and optimality criterion for Local.

For positive prevalences and interior safe risks, let q∗q^{\ast} and qLq^L be the least-favorable unsafe alternatives under Exact and Local. Define ϕk(q)=kl(rk,q)\phi_k(q)=\mathrm{kl}(r_k,q) and Bk(x,y)=ϕk(x)−ϕk(y)−ϕk′(y)(x−y)B_k(x,y)=\phi_k(x)-\phi_k(y)-\phi_k'(y)(x-y). Then Local's exact information-rate loss is

GL:=C∗−CL=∑kwkLBk(qk∗,qkL).\boxed{G_L:=C^{\ast}-C_L=\sum_k w_k^L B_k(q_k^{\ast},q_k^L)}.

If mkm_k and MkM_k are the minimum and maximum of ϕk′′\phi_k'' on the interval joining qkLq_k^L and qk∗q_k^{\ast}, then

\frac12\sum_k w_k^L m_k(q_k^{\ast}-q_k^L)^2 &\leq G_L\\\\ &\leq \frac12\sum_k w_k^L M_k(q_k^{\ast}-q_k^L)^2. \end{aligned}$$ The loss is zero if and only if $\sqrt{r_k(1-r_k)}(q_k^{\ast}-r_k)/[q_k^{\ast}(1-q_k^{\ast})]$ is constant across strata; equivalently, the KL distances at $q^L$ are equal. Otherwise Exact has a strictly larger oracle rate. **Proof sketch.** Strict convexity gives a unique interior Local projection with active constraint $\pi^\top q^L=\alpha$. Its KKT gradient is proportional to $\pi$. Since $q^{\ast}$ lies on the same hyperplane, the linear term in $F_{w^L}(q^{\ast})-F_{w^L}(q^L)$ vanishes, leaving exactly the weighted Bregman divergences. Taylor's integral remainder gives the curvature bounds. Their zero case and the explicit optimal allocation yield the equality criterion. Building on the Chernoff optimality criterion, `Proposition 3` adds an exact loss **identity** and computable bounds that quantify Local's inefficiency at any fixed safety margin. The separation between the two hardest unsafe alternatives and the KL curvature determine the rate loss, distinguishing a merely strict advantage from a large one. **`Theorem 6 — Boundary separation of Exact and Local.`** Partition the strata into nonempty sets $Z,B$, with positive fixed prevalences. Risks on $Z$ remain in $(0,1)$ and tend to zero, while risks on $B$ equal fixed $b_k\in(0,1)$. Let $p_Z=\sum_Z\pi_k$, $R_B=\sum_B\pi_kb_k$, and fix $R_B < \alpha < 1$. Local's rate tends to zero **if and only if** $\alpha-R_B\leq p_Z$, whereas Exact's rate tends to a positive constant $z_0$. If every risk on $Z$ equals $\epsilon$ and $a=(\alpha-R_B)/p_Z\in(0,1)$, then, for any fixed number of strata, $$\begin{aligned} &\boxed{C_L\sim A\sqrt{\epsilon}},\\\\ &A=\frac{p_Z[-\log(1-a)]}{\sum_B\pi_k\sqrt{b_k(1-b_k)}},\\\\ &\frac{C^{\ast}}{C_L}\sim\frac{z_0}{A\sqrt{\epsilon}}. \end{aligned}$$ Exact retains positive limiting sampling mass on every stratum in $Z$; Local's total mass on $Z$ vanishes.
AuthorsOpenReview ↗

Response to Lrwg — Part 3/6: W1: Boundary proof and finite-budget experiments

Proof sketch. Local's limiting rate is the weighted KL minimum over BB subject to ∑Bπkqk≥α−pZ\sum_B\pi_kq_k\geq\alpha-p_Z. This minimum vanishes exactly when the unchanged risks qB=bq_B=b are feasible; strictly feasible approximations also handle equality at the threshold. Continuity of the equal-KL equation gives Exact's positive limiting root. Under common risk ϵ\epsilon, the unsafe alternative qZ=a,qB=bq_Z=a,q_B=b gives an upper bound. The active aggregate constraint and Jensen's inequality give the matching lower bound and its explicit constant.

This result gives a concrete design lesson: a nearly error-free stratum can still require sustained sampling because an unsafe alternative can place errors there. Local allocates by observed variance and can suppress those observations. Exact allocates to exclude unsafe alternatives. The fixed-interior approximation is therefore not uniform as a risk approaches zero. The capacity threshold concerns rate collapse, not whether any gap exists: above capacity, Local retains a positive limiting rate and may still be substantially worse than Exact.

The new theory specifies two complementary experimental checks. First, verify the gap identity and the predicted boundary rate with known risks. Second, test whether that oracle advantage reduces actual audit labels when risk estimation and stopping overhead are present.

Deterministic checkComplete evidence retainedResult
Exact gap and curvature boundsAll 1,248 boundary-grid cases and 8 reference casesAll 1,256 identities pass; maximum absolute residual 2.676×10−112.676\times10^{-11} nats per label; no curvature-bound violations
Sharp general-KK boundary constantAll 104 strict-capacity settings at ϵ=10−8\epsilon=10^{-8}, with KK through 128CL/(Aϵ)∈[0.995031,0.999640]C_L/(A\sqrt{\epsilon})\in[0.995031,0.999640], median 0.999014
Insufficient-capacity controlsAll 52 settings with capacity ratio 1.25 at the same endpointEvery reduced Local limit is strictly positive, as predicted

The full boundary grid contains 1,248 configurations: seven stratum counts, four low-stratum prevalences, two remaining-risk shapes where distinct, three capacity ratios, and eight values of ϵ\epsilon. Ratios 0.25 and 0.75 satisfy strict capacity; 1.25 is retained as a control. In the reference case, π=(0.5,0.5)\pi=(0.5,0.5), r=(ϵ,0.30)r=(\epsilon,0.30), and α=0.2505\alpha=0.2505, the predicted limit CL/ϵ→0.48966858C_L/\sqrt{\epsilon}\to0.48966858 compares with 0.48948305 at ϵ=10−8\epsilon=10^{-8}. These are known-risk information rates, not measured audit-cost savings.

Finite-budget follow-up. The allocation experiment holds the estimator, evidence process, exploration, and stopping rule fixed within each comparison. It covers 96 risk settings, two initializations, three confidence levels, and 256 paired paths per setting. Of these settings, 72 are heterogeneous and 24 are homogeneous controls. The following table covers all 72 heterogeneous settings, including negative outcomes.

Strata KKPrevalenceC, δ=.05\delta=.05P, δ=.05\delta=.05C, δ=.001\delta=.001P, δ=.001\delta=.001C, δ=10−6\delta=10^{-6}P, δ=10−6\delta=10^{-6}
2U-0.452.950.838.951.0715.58
2G-0.020.150.080.920.091.46
2R-0.612.35-0.444.600.355.84
4U-0.172.170.517.450.7411.93
4G0.141.730.450.990.502.38
4R0.043.48-0.429.250.6112.34
8U0.903.680.517.220.2910.57
8G-0.121.040.483.360.605.15
8R-0.103.220.146.980.3810.81
16U0.014.06-0.026.010.117.87
16G0.292.470.544.990.567.20
16R0.243.730.137.000.179.09

Each entry is median audit-label saving of Exact over Local (%), over three risk patterns and two margins. A positive median does not mean that every setting wins. KK is the number of strata. U is uniform prevalence; G is geometric prevalence with a 16:1 largest-to-smallest ratio; R assigns 5% to the final stratum and splits the remainder equally. C initializes every risk at 3α/43\alpha/4 with strength 64. P uses 50 independent pre-audit labels per stratum with strength 50. The C/P comparison changes both the initial mean and strength; the Exact/Local comparison within each column is matched. All paths certify in this safe grid. Pointwise paired intervals use 4,000 bootstrap draws.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to AC and Reviewers — Summary of Revisions

A Kind General Response to All Reviewers and Dear AC

We sincerely thank the reviewers for their positive assessments (clear and rigorous [yAf1], systematic theoretical treatment [JLqc], the math appears correct [Lrwg]) and valuable suggestions. In the revised manuscript, we have addressed the following points, with changes marked in blue:

  1. Clarification of Core Contributions and Presentation:

    • Clarifying the scientific question, the contributions beyond prior work, and the technical challenges in aggregate-risk certification (yAf1–Q1/Q2, JLqc–W2/Q2, Lrwg–W1).
    • Explaining how the theory guides fit–freeze–audit, adding a plain-language reading guide, and simplifying the presentation of complete experimental results (yAf1–Q3, Lrwg–W1).
  2. New Theory and Validation for Exact Allocation:

    • Adding Proposition 3, an exact loss identity with computable bounds, and Theorem 6, a sharp boundary separation identifying when Local's information rate collapses while Exact retains a positive rate, with full proofs (yAf1–Q4, Lrwg–W1).
    • Adding 1,248 theoretical validation configurations plus eight reference cases, and reporting the complete 96-setting finite-budget study across stratum counts, prevalences, initializations, and confidence levels (yAf1–Q4, JLqc–W1, Lrwg–W1).
  3. Addition of Real-World Experiments:

    • Demonstrating natural agreement reversal with five LLM judges on MATH/GPQA: the same-family panel has 3.92 percentage points higher agreement but 3.84 points lower accuracy overall (JLqc–W1/Q1, Lrwg–W2/Q1).
    • Comparing ten aggregation methods and evaluating the full fit–freeze–audit pipeline at four targets. GAVEL gains 2.71 accuracy points overall over native single; at the primary target, it saves 34.5% of audit labels and 13.65% of total labels. We report complete costs and certification outcomes, and update the abstract to reflect all benchmark outcomes (JLqc–W1/Q1, Lrwg–W2/Q2).

These revisions are included in the blue-marked manuscript and detailed in our point-by-point responses.

Thank you again for the time and care invested in these reviews. We welcome further questions and would be happy to discuss any aspect of the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.