acceptodds
Under review as a conference paper at ICLR 2027

No One Wins Forever: Rethinking Voting, Self-refinement, and Debate of LLM Agents

Abstract

Multi-agent debate (MAD) has been established as a paradigm for improving reasoning capabilities through communication among multiple large language models (LLMs). Despite its empirical appeal, the mechanisms underlying multi-agent debate remain underexplored. Existing work often draws largely binary conclusions, characterizing MAD as either effective or ineffective under narrow or even unfair experimental settings. Such analyses fail to capture the rich dynamics of debate, resulting in limited conclusions. In this paper, we develop a simple mathematical model that captures two properties of model behavior: generation quality and answer stability. Our theory characterizes the complete ordering among three methods: majority voting (MV), self-refinement (SR), and MAD. Instead of drawing a binary conclusion, our analysis suggests that when independent generation favors the correct answer, voting eventually dominates by amplifying this advantage. Otherwise, debate outperforms self-refinement when the correct answer is the most prevalent in the long-run group distribution. We further show that answer stability shapes this collective behavior: stronger interaction increasingly favors answers that the model is more likely to retain under revision. For a fixed number of agents, we also derive stationary accuracy limits and exact finite-budget evaluation formulas. Through extensive experiments across six benchmarks and six LLMs, we find that the proposed theory closely matches empirical results. Guided by these insights, we further propose gated debate, which decides whether to use debate or self-refinement at each turn based on a training-free statistic. Results show it outperforms both self-refine and debate baselines at larger model-call budgets. Code and data will be released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent LLMs, Large Language Model, Multi-agent Debate, Self-refinement, Majority Voting, Agent Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_tEy9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors develop a mathematical model explaining when multi-agent debate, majority voting, and self-refinement performs best under a fixed model-call budget. Accordingly, the authors propose Gated Debate, which selectively shares the peer answers based on the support for the model's base favorite, and evaluate this across various models and benchmarks.

Critical strengths
  1. I like how the authors set a budget constraint to ensure fair comparison across the three modes. To my knowledge, it hasn't been considered in previous work to match the budget across voting, debate, and self-refinement.
  2. Strong theoretical motivation and demonstration.
  3. The approach to gate the three different methods is new and is a plausible direction to consider.
Critical weaknesses
  1. The biggest issue is the significance of the reported performance. In Table 2, out of 18 settings, only 4 show the advantage of Gated Debate compared to the baselines, and the performance gap is not large enough even in those several cases. While the authors claim that Gated Debate often outperforms "Always debate", this does not justify that the method is useful.

  2. I appreciate the theoretical explanations in the methods section, but to me, it seems to be overly complicated. I think it would be better if the authors could explain the method in plain text, rather than having to set a formal definition (e.g., Definition 5.1). In its current form, some additional cognitive load is required to understand the method.

  3. To my understanding, the theoretical guarantee concerns a binary oracle gate that knows the correct answer. However, the implemented algorithm substitutes it with the empirical mean. But wouldn't this be a problem when the model doesn't know the correct answer? That is, to my understanding, the proposed method can actually reinforce wrong answers, given that the ground-truth solution is not known. If this is the case, the not-so-good experimental results make sense. To show that this method works, please provide results on a difficult benchmark whose single-agent accuracy is below 1/N, where N is the number of agents involved in debate.

  4. This is relatively minor, but the motivation stated in the second paragraph of the Intro section is questionable. As far as I know, Du et al. (2023) proposed "multi-agent debate", but didn't compare with majority voting; Xu et al. (2026) did not emphasize the multi-agent debate protocol as its key results, and thus it is difficult to say it was positive about MAD; Choi et al., 2026 and Estornell & Liu, 2024 were some of the works that identified problems of multi-agent debate. The motivating works that the authors pose are not necessarily pointing to opposite directions, but rather are a natural flow of research on MAD. Hence, deriving the motivation to mix the three protocols (debate, voting, self-refinement) from this background can be a bit misleading.

Overall, I am not convinced about the method itself, and the experiments also show questionable gains compared to baselines. Thus, I lean towards rejection at this point.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Responses to Weakness 1-3

We thank you for the careful reading and thoughtful comments. We provide detailed responses below.

W1: The biggest issue is the significance of the reported performance. In Table 2, out of 18 settings, only 4 show the advantage of Gated Debate compared to the baselines, and the performance gap is not large enough even in those several cases. While the authors claim that Gated Debate often outperforms "Always debate", this does not justify that the method is useful.

A1: We would like to clarify that the theory in Section 5 (Theorem 5.3) considers a special case, a binary oracle gate, and is used as a heuristic to inspire the design of Gated Debate; it is not meant to analyze whether Gated Debate itself performs well. We therefore do not claim that Gated Debate beats every baseline. The main contribution of the paper is the theory in Section 4, which separates generation quality (dpd_p) from revision stability (dρ∗d^*_\rho) and explains why the ranking of MV, SR and MAD changes across settings (General Response §1).

The count in Table 2 is correct: Gated Debate is best in 4 of the 18 cells and tied in 1, and it is above Matched No-Group in 13. Paired 95% intervals over 27,189 questions at 40 calls show a significant gain over SR (+1.36 pp [+0.99, +1.71]) and differences within noise for MV (−0.20 [−0.49, +0.09]), Always Debate (+0.16 [−0.14, +0.47]) and Matched No-Group (+0.15 [−0.15, +0.43]).

The gains are modest because the label-free reference is sometimes wrong, not because the gating rule is ineffective. With the same prompts and budget, giving the gold label only to the gate controller improves accuracy by +3.93 pp on MMLU-Pro and +4.05 pp on MMMU (General Response §2). We will revise the abstract and Sections 1, 5 and 6 accordingly and remove the remark that attributes the 10-call results to noise.

W2: I appreciate the theoretical explanations in the methods section, but to me, it seems to be overly complicated. I think it would be better if the authors could explain the method in plain text, rather than having to set a formal definition (e.g., Definition 5.1). In its current form, some additional cognitive load is required to understand the method.

A2: We will add a plain-language description before Definitions 5.1–5.4:

(1) Before the debate, make 20 independent calls and take the most frequent answer as the reference, with frequency p^ref\hat p_{\text{ref}}. (2) Before each revision round, check whether the share of agents holding the reference exceeds p^ref\hat p_{\text{ref}} by more than τ. (3) If so, each agent sees the other four answers when revising; if not, each agent revises alone. Each round costs 5 calls either way. Since the reference may be wrong, opening can reinforce either a correct answer or a shared error (A3).

W3: To my understanding, the theoretical guarantee concerns a binary oracle gate that knows the correct answer. However, the implemented algorithm substitutes it with the empirical mean. But wouldn't this be a problem when the model doesn't know the correct answer? That is, to my understanding, the proposed method can actually reinforce wrong answers, given that the ground-truth solution is not known. If this is the case, the not-so-good experimental results make sense. To show that this method works, please provide results on a difficult benchmark whose single-agent accuracy is below 1/N, where N is the number of agents involved in debate.

A3: We would like to clarify that the theory in Section 5 (Theorem 5.3) considers a special case, a binary oracle gate, and is used as a heuristic to inspire the design of Gated Debate; it is not meant to analyze whether Gated Debate itself performs well. The practical gate replaces the correct answer with the baseline mode, and when the baseline mode is wrong it can reinforce the wrong answer, as the reviewer notes. This is consistent with the failure mechanism suggested by Eq. (15) in the binary setting, and our results confirm it (General Response §2). With a wrong reference, Gated Debate reaches 7.37% vs. 10.94% for Matched No-Group on MMLU-Pro (361 questions) and 10.34% vs. 13.97% on MMMU (358 questions); on the 27 such GPQA-Diamond questions it reaches 22.22%, vs. 25.93% for Always Debate and 18.52% for Matched No-Group. This also explains the modest overall gains in Table 2.

Regarding SA < 1/N, none of our 34 runs falls in this regime. We computed 1-call single-agent accuracy from the first answer in each question's MV pool, counting invalid answers as wrong. The lowest value is 27.6% for Qwen3.5-4B on SuperGPQA (27.0% over all 60 independent samples per question), followed by Qwen3.5-9B (32.8%) and Qwen3.6-35B-A3B (39.5%) on the same benchmark. Keeping only the questions a model gets wrong would not substitute for a benchmark chosen in advance, so we make no empirical claim about this regime; testing it requires a new evaluation on such a benchmark.

AuthorsOpenReview ↗

Response to Weakness 4

W4: This is relatively minor, but the motivation stated in the second paragraph of the Intro section is questionable. As far as I know, Du et al. (2023) proposed "multi-agent debate", but didn't compare with majority voting; Xu et al. (2026) did not emphasize the multi-agent debate protocol as its key results, and thus it is difficult to say it was positive about MAD; Choi et al., 2026 and Estornell & Liu, 2024 were some of the works that identified problems of multi-agent debate. The motivating works that the authors pose are not necessarily pointing to opposite directions, but rather are a natural flow of research on MAD. Hence, deriving the motivation to mix the three protocols (debate, voting, self-refinement) from this background can be a bit misleading.

A4: Our intention was to motivate a matched-budget comparison, not to suggest that prior studies contradict each other. To avoid this reading, we will revise Sections 1 and 2 to describe the literature as a progression across tasks, protocols and assumptions:

  • Du et al. (2023; ICML 2024) included majority-voting baselines, compared debate at matched call counts (6 independent calls vs. 3 agents × 2 rounds) and analyzed tokens (§3.1, Table 1, App. A.3–A.4). We will cite this accurately.
  • We will remove Xu et al. (2026), which studies long-context chunking rather than standard MAD, and correct the year of Choi et al. to NeurIPS 2025.
  • We will credit prior theory on debate dynamics and interventions (Estornell & Liu, NeurIPS 2024; Choi et al., NeurIPS 2025; Okawa et al., ICML 2026) and contrast gating during the debate with supervised overturning after it (e.g., Minority Sentinel, arXiv:2606.29270).
Reviewer_WnQzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a mathematical model at the answer level to compare three inference protocols under matched budgets for model calls: majority voting, self critique, and multiagent debate. The model distinguishes baseline generation quality from answer stability, derives the stationary distribution of self critique, and characterizes the equilibrium of multiagent debate in large populations through a scalar equation.

Critical strengths
  1. The paper is well motivated, directly addressing the tension between prior findings that debate is effective and those that debate fails to outperform voting.

  2. The claim that interaction amplifies stable answers rather than correct ones is intriguing and offers an explanation for the error propagation observed in prior work.

  3. Gated Debate is simple and requires no training, and oracle gating achieves strong results at K = 2.

  4. The experiments demonstrate substantial effort, and the authors commit to publicly releasing the code and data.

Critical weaknesses
  1. The theory reduces debate to exchanging answer labels, and the experiments deliberately withhold rationales. Since practical MAD often derives its value from exchanging arguments, it remains unclear whether “answer stability” is intrinsic to each question or an artifact of suppressing explanations.

  2. The theorems require both the number of agents and calls per agent to approach infinity, whereas the experiments use five agents with only two, four, or eight calls each. Although ranking agreement is well above chance in some settings, the predicted asymptotic behavior remains unclear. The theoretical analysis for a fixed number of agents also lacks experimental validation.

  3. The paper claims that Gated Debate outperforms Self Refine and debate baselines. However, the budget matched baseline without group interaction is often strongest in Table 2 and beats Gated Debate in several settings. This leaves the practical value of gating unclear: much of the improvement may come from revision without group interaction rather than the gating mechanism itself.

Questions for authors
  1. Could you provide a sensitivity analysis of the hyperparameters?

  2. Gated Debate underperforms the matched baseline without group interaction on several benchmarks, including MedQA and HellaSwag. Could this be due to the choice of threshold? Please provide accuracy curves across thresholds and analyze how frequently the gate is activated on each benchmark.

  3. Do the fitted parameters transfer across models or benchmarks, or must calibration be repeated for every combination of model and benchmark?

  4. How would the model change if agents exchanged supporting arguments rather than just answer option letters?

3 comments
AuthorsOpenReview ↗

Responses to Weakness 1-2

We thank you for the careful reading and thoughtful comments. We provide detailed responses below.

W1: The theory reduces debate to exchanging answer labels, and the experiments deliberately withhold rationales. Since practical MAD often derives its value from exchanging arguments, it remains unclear whether “answer stability” is intrinsic to each question or an artifact of suppressing explanations.

A1: Answer stability in our model is not an intrinsic property of a question. It is defined for a given model, prompt and protocol, so under rationale exchange α\alpha and ρ\rho would be estimated for that protocol, and the ordering results apply to any protocol whose revisions follow the answer-level model.

We use label-only exchange on purpose (Appendix G.1), because it isolates answer persistence and the effect of peer answer frequencies. We have not run a rationale-exchange ablation, so we make no claim about that setting, and we will state this scope in Sections 3 and 7.

On Q4: arguments could enter the model through state-dependent parameters, for example P(a→b∣μ,z)=(1−αa(z))1{a=b}+αa(z)[(1−ρ(z))pb(z)+ρ(z)μb]P(a \to b \mid \mu, z) = (1-\alpha_a(z))\mathbb{1}\{a=b\} + \alpha_a(z)[(1-\rho(z))p_b(z) + \rho(z)\mu_b], where zz summarizes the visible rationales. If zz depends on the history, label counts alone are no longer a time-homogeneous Markov state, so our stationary and ordering results would not carry over directly. We leave this extension to future work.

W2: The theorems require both the number of agents and calls per agent to approach infinity, whereas the experiments use five agents with only two, four, or eight calls each. Although ranking agreement is well above chance in some settings, the predicted asymptotic behavior remains unclear. The theoretical analysis for a fixed number of agents also lacks experimental validation.

A2: At N=5N=5, the experiments already follow the orderings that the theory predicts in the limit, and this check does not reuse the evaluated outcomes. We also validate the fixed-NN model on held-out rounds and held-out questions.

The exact finite-NN chain in Appendix B.4 shows why this is possible. For example, with N=5N=5, 40 calls, pg=0.4p_g=0.4, α=(0.2,0.8)\alpha=(0.2, 0.8) and ρ=0.5\rho=0.5 (dp<0<d∗d_p < 0 < d^*), the exact accuracies are MAD 92.79% > NG 86.43% > SR 72.73% > MV 10.21%, which matches Table 1, and MAD overtakes NG between 20 and 25 calls.

The experiments show the same pattern. Pooled over the 27,189 test questions at 40 calls, the accuracies of MV, SR and MAD follow the Table 1 ordering in all four (dp,d∗)(d_p, d^*) quadrants (λ = 0.1; General Response §1, Table A). Most questions fall in the two same-sign quadrants: MV 98.05% > MAD 95.11% > SR 92.74% when both margins are positive (19,292 questions), and SR 13.41% > MAD 11.38% > MV 5.22% when both are negative (7,471 questions).

None of the evaluated outcomes is used to assign the quadrants. dpd_p is computed from 20 calibration calls that no protocol reuses, the MV and SR accuracies come from separate runs, and for MAD, d∗d^* is fitted only on the calibration answers and debate rounds R1–R3, while accuracy is measured at R7.

The fixed-NN model is validated with the same separation. With parameters fitted on R1–R3, running the exact N=5N=5 model from the observed R3 state to R7 gives a pooled observed − predicted accuracy between +0.05 and +0.24 pp for the three group arms (MAD, NG and Gated Debate) and all λ, and every 95% CI includes zero. Across questions, parameters fitted on micro-train questions predict 40-call test accuracy with MAE 1.32 pp for MAD and 0.73 pp for MV (Figure 3).

AuthorsOpenReview ↗

Responses to Weakness 3 and Question 1

W3: The paper claims that Gated Debate outperforms Self Refine and debate baselines. However, the budget matched baseline without group interaction is often strongest in Table 2 and beats Gated Debate in several settings. This leaves the practical value of gating unclear: much of the improvement may come from revision without group interaction rather than the gating mechanism itself.

A3: We would like to clarify that the theory in Section 5 (Theorem 5.3) considers a special case, a binary oracle gate, and is used as a heuristic to inspire the design of Gated Debate; it is not meant to analyze whether Gated Debate itself performs well. The gating itself does change outcomes beyond revision without peers: it helps when the reference is correct and hurts when it is wrong, so its net effect in Table 2 is small.

In Table 2 (4 models on MMMU and 6 elsewhere), Gated Debate is above Matched No-Group in 13 of 18 cells and below it in 5 (MedQA at all three budgets and SuperGPQA at 10 and 40 calls); on HellaSwag it is above at all budgets. General Response §2 shows where the difference comes from. With a wrong baseline-mode reference, opening the gate amplifies the wrong answer, and Gated Debate falls below NG (7.37% vs. 10.94% on MMLU-Pro and 10.34% vs. 13.97% on MMMU). With a correct reference, it is above NG (96.62% vs. 92.98% and 93.80% vs. 89.67%). Over all 33 runs, Gated Debate − NG is +0.92 pp with a correct reference and −1.81 pp with a wrong one. This is what Eq. (15) predicts once the correct label is replaced by an estimate. We will state this in Sections 5 and 6.

Q1: Could you provide a sensitivity analysis of the hyperparameters?

A-Q1: The prediction results change little across the three regularization strengths λ ∈ {0.1, 1, 10}, which we report side by side without selecting one. Only the number of opposite-sign cases depends on λ, because stronger shrinkage pulls the question-level fits toward the shared fit. The settings, taken from the code and checked against the saved fits, are:

ComponentSetting
p^\hat p in the live runs (gate reference and score)20 independent calls per question with the R0 prompt; add-one smoothing on valid replies, p^(b)=(cb+1)/(nvalid+K)\hat p(b) = (c_b+1)/(n_{\text{valid}}+K); questions with fewer than 16/20 valid replies are excluded. Checked on all 36,390 saved distributions.
p^\hat p in the Section 6 fitsPseudocount chosen from {0.1, 0.5, 1} by held-out NLL on the independent replies only (seeded 80/20 split per question, no gold labels). 0.1 was chosen in 32 of 33 runs and 0.5 for Qwen3.5-4B on GPQA-Diamond. The reference (argmax) does not depend on the pseudocount.
Redraw probability αD\alpha^Dclip(σ(β₀ + β₁·(−log p^a\hat p_a)), 0.001, 1 − 10⁻⁶)
Optimizerβ₀ ∈ [−15, 15], β₁ ∈ [−10, 10] (no sign constraint), ρ ∈ [0, 0.999]; L-BFGS-B from 4 fixed starts
Estimator selectionConstant, shared, partial (λ ∈ {0.1, 1, 10}) and free estimators compared by micro-dev question-mean NLL among estimators that can predict every state; shared was selected in 31 runs and constant in 2
Within-question fitsQuestion-specific redraw rates are fitted on rounds R1–R3 and evaluated on R4–R7, with logit shrinkage toward the shared fit of strength λ ∈ {0.1, 1, 10} (all three reported); ρ is shared within each run and kept fixed
Gate thresholdτ grid {0, 0.05, 0.10, 0.15, 0.20} on micro-dev states; highest classification accuracy for the target μ(g)−p^(g)>0\mu(g) - \hat p(g) > 0, ties broken toward the smallest τ (see A-Q2)

The sensitivity results are:

  • λ: the opposite-sign counts are 56/94, 11/13 and 0/0 at λ = 0.1, 1 and 10. One-step R4–R7 NLL is 0.229, 0.232 and 0.234 (0.235 for the shared fit), and the R3-to-R7 observed − predicted accuracy is between +0.05 and +0.24 pp (General Response §1). λ cannot affect estimator selection, because dev questions have no within-question transitions.
  • Pseudocount: dev NLL is logged for every candidate, for example 0.526, 0.650 and 0.789 for 0.1, 0.5 and 1 on Qwen3.6-35B-A3B MMLU-Pro.
  • τ: see A-Q2.
AuthorsOpenReview ↗

Responses to Question 2-4

Q2: Gated Debate underperforms the matched baseline without group interaction on several benchmarks, including MedQA and HellaSwag. Could this be due to the choice of threshold? Please provide accuracy curves across thresholds and analyze how frequently the gate is activated on each benchmark.

A-Q2: The threshold does not explain these losses. τ = 0 in five of the six MedQA runs (0.05 for Qwen3.6-35B-A3B), the same value used on benchmarks where Gated Debate beats NG, and on HellaSwag Gated Debate is in fact above NG at all three budgets (Table 2). A3 discusses the role of the reference.

τ is selected on micro-dev only and frozen before any test call. For each dev question we build 5 controlled states, with s = 0, …, 4 peers sharing the focal agent's answer, and from {0, 0.05, 0.10, 0.15, 0.20} we pick the τ whose rule μ(g^)−p^(g^)>τ\mu(\hat g) - \hat p(\hat g) > \tau best classifies the target μ(g)−p^(g)>0\mu(g) - \hat p(g) > 0, breaking ties toward the smallest τ.

The frozen τ and the share of rounds in which the gate opens are:

ModelMMLU-ProMMMUMedQAGPQA-DHellaSwagSuperGPQA
Qwen3.5-4B0 / 77.90 / 69.10 / 67.50.10 / 53.80 / 93.90.20 / 50.1
Qwen3.5-9B0 / 82.00 / 73.10 / 75.70.10 / 44.40 / 94.20.20 / 55.9
Qwen3.6-35B0 / 89.00 / 78.10.05 / 87.80.05 / 67.60 / 97.40 / 84.2
Kimi-K2.60 / 94.80 / 91.80 / 93.20 / 83.30 / 99.00 / 91.9
GLM-5.30 / 94.4—0 / 95.50 / 88.60 / 98.60 / 90.5
GPT-5.4-mini0 / 94.6—0 / 96.20 / 83.90 / 97.10 / 90.2

τ = 0 in 28 of the 34 runs. The activation rate varies widely across models and benchmarks, from 44.4% to 99.0% of rounds (85.55% of all 196,749 round decisions). GLM-5.3 and GPT-5.4-mini have no complete MMMU runs (Reviewer ZMUk, A-Q3).

Each run used one frozen τ. Accuracy curves across τ would need new live runs, since early gate decisions change later states, and we could not complete them within the rebuttal period. On the logged states, the share of decisions (%) that would open with threshold τ + Δ is:

Δτ−0.20−0.10−0.050+0.05+0.10+0.15+0.20
All 34 runs93.090.487.685.583.876.837.933.5
MMLU-Pro94.993.290.988.787.685.879.475.8
MMMU90.785.581.378.074.365.530.219.5
MedQA93.590.787.285.984.569.09.06.2
GPQA-D86.478.474.770.059.749.223.218.9
HellaSwag98.597.797.196.796.295.57.65.6
SuperGPQA87.183.780.476.974.965.962.056.9

Closed rounds still cost 5 calls, so the budget is the same for every τ.

Q3: Do the fitted parameters transfer across models or benchmarks, or must calibration be repeated for every combination of model and benchmark?

A-Q3: Parameters are calibrated separately for each model and benchmark, and we do not claim that they transfer. The micro-train fits, the estimator and shrinkage selection, τ and p^\hat p are all estimated within each run, and no run received parameters from another run; we will list this as a limitation. What we do evaluate is held-out prediction within a run, both within each question (fit on R1–R3, scored on R4–R7 and on R3-to-R7 accuracy) and across questions (micro-train fits predicting test-question MAD accuracy from R0 and MV accuracy from p^\hat p; Figure 3).

Q4: How would the model change if agents exchanged supporting arguments rather than just answer option letters?

A-Q4: Please see A1.

Reviewer_KsQmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper is on comparing the three reasoning paradigms: Multi-agent Debate, Self-refinement, and Majority Voting. The paper argues existing work often considers ralatively narrow experiemntal settings, or fail to control for inference budget. The paper establishes a strict ordering of MV, SR, and MAD as both the number of agents and rounds grow; The paper also provides a gated debate design based on the established theoretical modeling.

Critical strengths
  • The paper provides a detailed theoretical discussion on pros and cons of the three reasoning paradigms. The theory part is clear and rigorous.
  • The experiments controls the reasoning budget and covers six different benchmarks.
Critical weaknesses
  • The theoretical inspired gated debate methods requires the correct label of the answers. At test time, the correct label is replaced by the most likely independent answer as a reference. This would introduce additional overhead cost on finding this reference. Also, based on the results in Table 2, the benefits from the proposed gated debate is not obvious to me.
  • The theoretical analysis introduces the correct answer’s margins in formalization. This choice makes the theoretical conclusion requires the correct labels to make decisions (and one need to estimate the correct label to use the proposed method). It would be more interesting to relax this setting to avoid the knowledge of ground truth answers.
Questions for authors

How could one estimate the correct answer margins in practice? The paper suggests to use majority voting to estimate the correct answer, then it goes back to the original point: when should one use MV, SR, or MAD to estimate the correct answer for the proposed answer?

4 comments
AuthorsOpenReview ↗

Responses to Weakness 1-2

We thank you for the careful reading and thoughtful comments. We provide detailed responses below.

W1: The theoretical inspired gated debate methods requires the correct label of the answers. At test time, the correct label is replaced by the most likely independent answer as a reference. This would introduce additional overhead cost on finding this reference. Also, based on the results in Table 2, the benefits from the proposed gated debate is not obvious to me.

A1: We would like to clarify that the theory in Section 5 (Theorem 5.3) considers a special case, a binary oracle gate, and is used as a heuristic to inspire the design of Gated Debate; it is not meant to analyze whether Gated Debate itself performs well. Gated Debate itself does not need the correct label: it applies the rule from this special case, which shows peer answers only when the reference answer's group share exceeds its baseline probability by more than a threshold, with the baseline mode g^\hat g as the reference. This adds 20 calibration calls per question, and its gains in Table 2 are small because the reference is sometimes wrong. We explain both points below.

Cost. p^\hat p and g^\hat g are estimated from 20 independent calls per test question (add-one smoothing, ties broken alphabetically). These calls are not shared with other methods: R0 (5 calls, shared by MAD, NG and Gated Debate), MV and SR make their own calls. τ also needs a one-time labelled dev set but no further LLM calls. The standalone cost per test question, logged over 34 runs (USD is logged only for GPT-5.4-mini), is:

MethodCalls/qTinker prompt / sampled tokensGPT-5.4-mini in / out tokensGPT-5.4-mini USD per 1k q
MV (40)40.0016,798 / 41514,431 / 5,77936.79
SR (40)39.9217,644 / 37415,345 / 6,13238.65
MAD (5 R0 + 35)39.9417,757 / 27615,366 / 5,17234.73
NG (5 R0 + 35)39.9417,561 / 37715,261 / 6,19139.25
Gated Debate (5 R0 + 35) + 20 calibration59.9426,123 / 53622,585 / 8,21653.83
One-time τ dev set7,520 per 1,000-q benchmark8,318 / 270 per dev q7,154 / 2,891 per dev q18.36 per 1k dev q

The 20 calibration calls are a fixed cost per question that does not grow with the budget. At 40 calls, Gated Debate uses 1.5× the calls of each baseline (1.46× the dollar cost of MV and 1.55× that of MAD for GPT-5.4-mini), and at 100 calls the ratio would be 1.2×. Since Tables 2–8 count only its 40 in-protocol calls, we will describe them as matched in-protocol budgets rather than matched total cost.

Table 2 gains. The rule itself works when the reference is accurate (General Response §2). Giving the gold label only to the gate controller, with everything else unchanged, raises 40-call accuracy by +3.93 pp on MMLU-Pro and +4.05 pp on MMMU. With a correct g^\hat g, Gated Debate beats Matched No-Group (96.62% vs. 92.98% on MMLU-Pro and 93.80% vs. 89.67% on MMMU); with a wrong g^\hat g, it falls below (7.37% vs. 10.94% and 10.34% vs. 13.97%). These two cases offset each other in Table 2, and the remaining bottleneck is estimating the reference.

The main contribution of the paper is the matched-budget ordering theory in Section 4, which separates generation quality (dpd_p) from revision stability (dρ∗d^*_\rho). We will revise the abstract and Sections 1, 5 and 6 to make the role of Theorem 5.3 clear.

W2: The theoretical analysis introduces the correct answer’s margins in formalization. This choice makes the theoretical conclusion requires the correct labels to make decisions (and one need to estimate the correct label to use the proposed method). It would be more interesting to relax this setting to avoid the knowledge of ground truth answers.

A2: The theory does not require ground-truth labels at test time. The margins dpd_p and dρ∗d^*_\rho state the conditions under which each protocol eventually wins, in the same way that accuracy and risk are defined with respect to true labels.

In our experiments, test-question labels are used only for evaluation and for the post-hoc stratification that tests the theory. They are never used to fit transition parameters or to run a deployable protocol. The only uses of labels outside evaluation are the selection of τ on a labelled dev set and the Gold Gate in General Response §2, which is a diagnostic in which the agents never see gg.

Part of the theory also yields a label-free check. Given (p,α,ρ)(p, \alpha, \rho), if the baseline leader arg⁡max⁡bpb\arg\max_b p_b differs from the equilibrium leader arg⁡max⁡buρ,b∗\arg\max_b u^*_{\rho,b}, then MV and MAD eventually disagree, so one of them must be wrong. Deciding which one is right requires information about correctness, which we discuss in A-Q1.

AuthorsOpenReview ↗

Response to the Question

Q1: How could one estimate the correct answer margins in practice? The paper suggests to use majority voting to estimate the correct answer, then it goes back to the original point: when should one use MV, SR, or MAD to estimate the correct answer for the proposed answer?

A-Q1: The margins depend on the correct answer, so they cannot be computed from unlabeled data alone, but their ingredients can. The parameters (p,α,ρ)(p, \alpha, \rho) are estimated from independent answers and revisions without labels. Given them, MV converges to the baseline leader arg⁡max⁡bpb\arg\max_{b} p_b and MAD to the equilibrium leader arg⁡max⁡bu∗_ρ,b\arg\max_{b} u^*\_{\rho,b}, so the signs of dpd_p and d∗d^* only ask whether each leader is correct. In practice, this can be estimated at the task level from a small labelled dev set, and the protocol is then chosen for the whole task. Labels are not needed at test time, so the procedure is not circular.

Gated Debate is a separate use of the theory. It uses the baseline mode only to decide when peer answers are shown, and its final answer comes from the debate. Following the special case in Theorem 5.3, the mode is the natural label-free stand-in for gg, since it equals gg whenever dp>0d_p > 0.

A possible extension is to replace the single reference with calibrated weights wb=Pr⁡(G=b∣observable state)w_b = \Pr(G = b \mid \text{observable state}) learned on dev data. Under a shared transition kernel, the one-step expected change in the correct share relative to no peer exposure is ρ(∑aμaαaD)∑bwb(μb−pb)\rho(\sum_a \mu_a \alpha^D_a)\sum_b w_b(\mu_b - p_b), so the gate would open when ∑bwb(μb−p^b)>0\sum_b w_b(\mu_b - \hat p_b) > 0. This requires calibration data, and we have not tested it.

Thanks for the detailed response. So, my best understanding is:

The theoretical part of the paper says:

  • When the current group has already shown a trend toward reinforcing the ground truth, the next round of debate should be allowed.

However, the actual rule in practice is:

  • When the current group has already shown a trend toward reinforcing the MV answer, the next round of debate is allowed.

Therefore, my main concern remains that the practical rule does not necessarily reinforce the correct answer, and may instead reinforce an incorrect MV answer. The theoretical guarantee does not directly apply to the practical method.

Thank you for the quick follow-up. We agree with your understanding of the practical gate and its limitation.

We also hope the reviewer could take our main contribution into consideration: understanding when and why MV, SR, and MAD succeed or fail. Our theory identifies how generation quality and answer stability jointly determine their relative performance, and we validate these predictions empirically.

Gated Debate is one attempt to use this theoretical insight in practice. Importantly, our experiments show that interaction can help when the reference is correct (+4.14 pp), but hurt when it is wrong (−3.34 pp). This illustrates the mechanism our analysis seeks to understand: interaction can reinforce both correct and incorrect answers.

While improving the practical gate remains an open direction, we believe understanding these underlying dynamics is itself an important contribution, independent of the performance of a particular gating strategy.

We sincerely appreciate the reviewer's careful consideration and constructive feedback. We will make this distinction and the central role of our theoretical contribution clearer in the revised manuscript. Thank you again for your time and effort in reviewing our work.

Reviewer_ZMUkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors introduce a probabilistic framework for comparing majority voting (MV), self-refinement (SR), and multi-agent debate (MAD) under matched model-call budgets. The framework models generation quality, answer stability, and group interaction through answer-level transition probabilities. The authors derive stationary and equilibrium distributions, establish theoretical conditions for the eventual ordering of the three methods, and propose a training-free gated debate strategy. Experiments across six LLMs and six benchmarks evaluate the theoretical predictions and compare the methods empirically. The results show task-dependent performance rankings, close agreement between theoretical predictions and empirical observations, and improvements of gated debate over SR and MAD at larger inference budgets.

Critical strengths
  1. Unified Framework for Inference-Time Strategies. The paper provides a coherent probabilistic framework for comparing MV, SR, and MAD under matched model-call budgets. Theorem 4.1 and Table 1 establish four eventual orderings based on the baseline and debate margins (Eq. 7). I believe a concrete example of strategy selection using estimated parameters would further demonstrate its practical utility.
  2. Rigorous Theoretical Analysis and Transparency. The paper derives stationary distributions (Eqs. 4–6), convergence bounds (Lemma 4.2), and exact finite-budget accuracy formulas (Appendix B.4). The authors also explicitly disclose generative AI usage for proof checking and implementation. I think quantitative prediction errors for Figure 3 and numerical verification would further strengthen these contributions.
Critical weaknesses
  1. Gap Between Theoretical Guarantees and Practical Gating. Theorem 5.3 guarantees optimality only for a binary oracle gate, while the practical gate (Definition 5.4) uses the baseline-mode answer and operates on multiclass tasks. Equation (15) implies that activating the practical gate with an incorrect baseline mode in a binary task reduces the next-round expected correct-answer count relative to no-group revision. I believe the authors should characterize these failure conditions and evaluate performance separately for correct and incorrect baseline modes.
  2. Limited Model Diversity and Reasoning Controls. Three of the six models belong to the Qwen family (Appendix D), and reasoning configurations are inconsistently documented. For instance, GLM-5.3 explicitly uses low reasoning, while comparable settings for other models remain unclear, limiting generalizability. I think the authors should report all reasoning configurations and evaluate robustness across reasoning efforts and model capabilities.
  3. Insufficient Reproducibility and Statistical Validation. Key settings, including decoding parameters, calibration sample sizes, gate thresholds, and fitted parameters, are insufficiently specified. Appendix G (Table 9) identifies prompt templates without providing their complete text, while Tables 2–8 report accuracy without uncertainty estimates. It is also unclear whether baseline calibration calls are included in the matched budgets. I believe the authors should provide complete configurations, clarify calibration costs, and report statistical uncertainty.
Questions for authors
  1. Extension to Multiple Valid Answers. The theory assumes a unique correct answer (g\in[K]) (Section 3), limiting its applicability to tasks with multiple valid or semantically equivalent answers. Such settings may change plurality voting and answer stability. I believe the authors should discuss extensions to answer equivalence classes and whether Theorem 4.1 would still hold.
  2. Effectiveness with High-Reasoning Models. The framework uses answer-level revisions (Eqs. 1–2), leaving its applicability to intensive reasoning settings unclear. Stronger answer persistence may reduce redraw rates (\alpha), potentially slowing convergence within finite budgets (Lemma 4.2). I think experiments across reasoning-effort levels, together with estimated stability parameters, would clarify the method's robustness.
  3. Inconsistent Evaluation Pair Counts. Section 6 reports 33 model–benchmark runs and 27,189 model–question pairs, whereas Appendix F (Tables 3–8) totals 34 runs and 28,107 pairs. The difference of 918 questions exactly matches GPT-5.4-mini on SuperGPQA (Table 8), suggesting an unexplained exclusion. I believe the authors should clarify the exclusion criterion and its impact on the fitting results.
4 comments
AuthorsOpenReview ↗

Responses to Weakness 1-2

We thank you for the careful reading and thoughtful comments. We provide detailed responses below.

W1: Gap Between Theoretical Guarantees and Practical Gating. Theorem 5.3 guarantees optimality only for a binary oracle gate, while the practical gate (Definition 5.4) uses the baseline-mode answer and operates on multiclass tasks. Equation (15) implies that activating the practical gate with an incorrect baseline mode in a binary task reduces the next-round expected correct-answer count relative to no-group revision. I believe the authors should characterize these failure conditions and evaluate performance separately for correct and incorrect baseline modes.

A1: We would like to clarify that the theory in Section 5 (Theorem 5.3) considers a special case, a binary oracle gate, and is used as a heuristic to inspire the design of Gated Debate; it is not meant to analyze whether Gated Debate itself performs well. In this special case, Eq. (15) gives an exact rule: expose peer answers only when the correct answer's group share exceeds its baseline probability. Gated Debate (Def. 5.4) applies this rule without labels by replacing gg with the baseline mode g^\hat g, uses a dev-selected threshold τ≥0\tau \ge 0 and handles multiclass answers; it does not inherit the guarantee of Theorem 5.3.

Following the reviewer's suggestion, we will state in Section 5 when the practical gate fails. In a binary task with dp<0d_p < 0, the baseline mode is wrong. If p^=p\hat p = p (or the estimation error is small enough that opening implies μt,g^>pg^\mu_{t,\hat g} > p_{\hat g}) and τ≥0\tau \ge 0, every opening has μt,g<pg\mu_{t,g} < p_g, so by Eq. (15) it lowers the expected number of correct agents in the next round relative to ρt=0\rho_t = 0 whenever ρt∑bnt(b)αbD>0\rho_t \sum_b n_t(b)\alpha^D_b > 0. In multiclass tasks a wrong reference does not by itself fix the sign, since it can also take votes from other wrong answers.

As requested, we split the results by whether g^\hat g is correct (General Response §2). Gated Debate − Matched No-Group is +4.14 pp with a correct reference and −3.34 pp with a wrong one (Qwen3.6-35B-A3B, 40 calls, 1,196 and 746 questions), and +0.92 vs. −1.81 pp over all 33 runs. The two effects largely cancel in Table 2. We will add this split to Section 6.

W2: Limited Model Diversity and Reasoning Controls. Three of the six models belong to the Qwen family (Appendix D), and reasoning configurations are inconsistently documented. For instance, GLM-5.3 explicitly uses low reasoning, while comparable settings for other models remain unclear, limiting generalizability. I think the authors should report all reasoning configurations and evaluate robustness across reasoning efforts and model capabilities.

A2: Our findings are not specific to the Qwen family, and we will report the full reasoning configuration of every model. The six models come from four families (Qwen, Kimi, GLM and GPT) and range from 4B open models to API-served models. For every model, the ranking of MV, SR and MAD changes across benchmarks (Tables 3–8), and Figure 5 reports the ordering agreement of each model separately, including 10/12 for Kimi-K2.6 and 6/8 for GPT-5.4-mini.

The configuration we will add to Appendix D is:

ModelModel ID (provider)Ttop_pMax tokensThinking / reasoningAnswer prefill
Qwen3.5-4BQwen/Qwen3.5-4B (Tinker)0.70.9564Thinking disabledFINAL:
Qwen3.5-9BQwen/Qwen3.5-9B (Tinker)0.70.9564Thinking disabledFINAL:
Qwen3.6-35B-A3BQwen/Qwen3.6-35B-A3B (Tinker)0.70.9564Thinking disabledFINAL:
Kimi-K2.6moonshotai/Kimi-K2.6 (Tinker)0.70.9564Thinking disabledFINAL:
GLM-5.3zai-org/GLM-5.3:peft:262144 (Tinker)0.70.954096Reasoning effort Low; thinking cannot be disabled and shares the 4096-token capNone
GPT-5.4-miniopenai/gpt-5.4-mini (Perplexity Agent API, no tools)0.70.954096reasoning.effort = minimal ("none" is rejected by the API)None
GPT-5.4-mini, SuperGPQA SR continuation onlygpt-5.4-mini-2026-03-17 (OpenAI Responses API), 3,397 SR calls on 91 questionsNot sent (reported 1.0)Not sent (reported 0.98)4096reasoning.effort = lowNone

With these settings, the four non-thinking models output 2.0 tokens per call on average, GLM-5.3 outputs 46.9 and GPT-5.4-mini outputs 143.7, of which 134.7 are reasoning tokens.

We did not vary reasoning effort with all other settings fixed. We will therefore restrict our claims to the tested configurations and note this as a limitation (see also A-Q2).

AuthorsOpenReview ↗

Responses to Weakness 3

W3: Insufficient Reproducibility and Statistical Validation. Key settings, including decoding parameters, calibration sample sizes, gate thresholds, and fitted parameters, are insufficiently specified. Appendix G (Table 9) identifies prompt templates without providing their complete text, while Tables 2–8 report accuracy without uncertainty estimates. It is also unclear whether baseline calibration calls are included in the matched budgets. I believe the authors should provide complete configurations, clarify calibration costs, and report statistical uncertainty.

A3: We will add all of these details to the paper. Calibration calls are counted separately from the 40-call protocol budget, and we report their cost below together with paired confidence intervals for Tables 2–8.

(a) Prompts. Appendix G will give the full system message, the output-format block and the three user templates. Rebuilding all 100,970 logged requests of the six GPQA-Diamond runs from these templates reproduced every request exactly.

=== PROMPT TEMPLATES ===
[System message: identical in every call]
You answer multiple-choice questions. Output exactly FINAL: followed by one space and the chosen option letter. Output no reasoning or explanation, even when asked to reconsider or revise an answer. Do not output option text. This format applies to every response.

[Common output-format block = {answer format}; one "FINAL: <label>" line per legal label, in label order]
Select the correct option and output its LETTER, not its answer text.
Your entire response must be exactly ONE of these lines:

FINAL: A
FINAL: B
...
Keep the literal prefix FINAL: and the single space before the letter. Do not output a bare letter. Do not add the option text, a period, quotes, markdown, or an explanation.

[Independent answer: calibration, R0, all MV calls, SR call 1]
Question: {question}

Choices:
A. {option A}
B. {option B}
...

{answer format}

[Peer-informed revision: micro-train/dev, always-debate, gated rounds when open]
Question: {question}

Choices:
{choices}

Your previous answer: {own label}

Four other agents answered:
{peer label 1}
{peer label 2}
{peer label 3}
{peer label 4}

Reconsider your answer using this information.
{answer format}

[Independent revision: matched no-group, SR calls 2-40, gated rounds when closed]
Question: {question}

Choices:
{choices}

Your previous answer: {own label}


Reconsider your answer independently. You are not shown any other agents' answers.
{answer format}
=== END ===


Peer labels are bare letters in a seeded random order. For MMMU, image placeholders are replaced by the images. Qwen and Kimi replies are prefilled with "FINAL:", and GLM-5.3's template adds "Reasoning Effort: Low".

(b) Decoding, parsing and scoring. The decoding parameters of every model are listed in A2. A reply is valid only if its first line is exactly "FINAL: " followed by a legal letter, and we never retry or repair replies. A question is removed if fewer than 16 of its 20 calibration answers are valid or if any protocol gives an invalid reply. The debate arms and MV take the plurality of their final answers, with credit 1/k when gold is among k tied answers, and SR uses its last answer. At 10, 20 and 40 calls we evaluate debate after rounds 1, 3 and 7, SR at answers 10, 20 and 40, and MV on nested subsets of its 40 samples. We will add these rules to Appendix G.

(c) Calibration, thresholds and fitted parameters. p^\hat p is estimated from 20 independent calls per question with add-one smoothing, and g^\hat g is its most frequent answer (ties broken alphabetically), so no separate reference call is made. τ is selected on micro-dev only, using 5 controlled states for each of 376 dev questions (26 for GPQA-Diamond), as the smallest grid value with the highest accuracy in classifying μ(g)−p^(g)>0\mu(g) - \hat p(g) > 0. This step uses dev labels but no extra LLM calls. The selected τ is 0 in 28 of the 34 runs, and the value for each run is listed in our response to Reviewer WnQz (Q2). The splits are micro-train / micro-dev / test = 1,504 / 376 / 940 (104 / 26 / 65 for GPQA-Diamond), and test questions with incomplete outputs are replaced from a fixed extension pool. Redraw parameters are fitted by maximum likelihood within each run, with the settings given in our response to Reviewer WnQz (Q1), and we will list the fitted values for every run in Appendix E.

(d) Calibration cost. The standalone cost per test question is:

40-call methodCalls/qTinker prompt / sampled tokens per qGPT-5.4-mini in / out tokens per qGPT-5.4-mini USD per 1k q
MV40.0016,798 / 41514,431 / 5,77936.79
SR39.9217,644 / 37415,345 / 6,13238.65
MAD (5 R0 + 35)39.9417,757 / 27615,366 / 5,17234.73
NG (5 R0 + 35)39.9417,561 / 37715,261 / 6,19139.25
Gated Debate (5 R0 + 35 + 20 calibration)59.9426,123 / 53622,585 / 8,21653.83
AuthorsOpenReview ↗

Responses to Weakness 3 and Questions 1-2

The 20 calibration calls are a fixed cost per question that does not grow with the budget. At 40 calls, Gated Debate uses 1.5× the calls of each baseline (1.46× the dollar cost of MV and 1.55× that of MAD for GPT-5.4-mini), and at 100 calls the ratio would be 1.2×. It also needs a one-time labelled dev set of 7,520 calls per 1,000-question benchmark (520 for GPQA-Diamond). We will describe Tables 2–8 as matched in-protocol budgets rather than matched total cost, and give the full cost ledger in Appendix D.

(e) Uncertainty. A new appendix table will report paired differences with 95% bootstrap intervals (2,000 resamples) for every run, model and benchmark. Pooled over the 33 fitted runs (27,189 questions), the differences in pp are:

CallsGated Debate (%)− MV− SR− MAD− NG
1071.31−0.21 [−0.49, +0.10]+1.05 [+0.70, +1.43]0.00 [−0.25, +0.28]+0.17 [−0.13, +0.48]
2071.49−0.17 [−0.47, +0.11]+1.23 [+0.88, +1.59]+0.25 [−0.05, +0.54]+0.23 [−0.07, +0.53]
4071.54−0.20 [−0.49, +0.09]+1.36 [+0.99, +1.71]+0.16 [−0.14, +0.47]+0.15 [−0.15, +0.43]

These intervals do not include the uncertainty from fitting or from selecting τ. Gated Debate is significantly better than SR, and its differences from MV, MAD and NG are within noise. As discussed in A1, Theorem 5.3 motivates the design of the gate but does not guarantee its performance.

Q1: Extension to Multiple Valid Answers. The theory assumes a unique correct answer (g\in[K]) (Section 3), limiting its applicability to tasks with multiple valid or semantically equivalent answers. Such settings may change plurality voting and answer stability. I believe the authors should discuss extensions to answer equivalence classes and whether Theorem 4.1 would still hold.

A-Q1: Theorem 4.1 extends to semantically equivalent answers under a simple condition, while genuinely different valid solutions would require redefining the margins.

For equivalent expressions of one answer, such as "1/2" and "0.5", suppose answers are merged into classes before voting and the redraw rates αD\alpha^D and αS\alpha^S are equal within each class. Then the class-level process keeps the original form, P(C→D)=(1−αC)1{C=D}+αC[(1−ρ)pD+ρμD]P(C \to D) = (1-\alpha_C)\mathbb{1}\{C=D\} + \alpha_C[(1-\rho)p_D + \rho\mu_D], and Theorem 4.1 holds with the correct class in place of gg. If redraw rates differ within a class, the class-level process is in general no longer Markov, and the theorem does not transfer directly.

For genuinely different valid solutions, success means choosing any valid class, so the success event, the aggregation rule and the margins all need to be redefined. We leave this case to future work. We will add this discussion to Section 3. All six benchmarks in our experiments are multiple choice with a single gold option, so neither case arises in our evaluation.

Q2: Effectiveness with High-Reasoning Models. The framework uses answer-level revisions (Eqs. 1–2), leaving its applicability to intensive reasoning settings unclear. Stronger answer persistence may reduce redraw rates (\alpha), potentially slowing convergence within finite budgets (Lemma 4.2). I think experiments across reasoning-effort levels, together with estimated stability parameters, would clarify the method's robustness.

A-Q2: As the reviewer suggests, stronger persistence mainly changes how fast the methods converge, not where they converge, as long as it lowers all redraw rates by the same factor.

Formally, if every redraw rate is multiplied by the same c>0c > 0 while pp and ρ\rho stay fixed, the mean-field map becomes Fc(μ)=μ+c(F(μ)−μ)F_c(\mu) = \mu + c(F(\mu) - \mu) and the SR kernel becomes Pc=(1−c)I+cPP_c = (1-c)I + cP. The debate equilibrium and the SR stationary distribution are therefore unchanged, and for 0<c<10 < c < 1 convergence is slower, so a larger budget is needed to reach them (Lemma 4.2). This invariance does not extend to the stationary distribution of the finite-NN chain or to finite-budget accuracy. In practice, more reasoning may also change pp, the relative α\alpha values and ρ\rho, which can change the margins themselves. We have no runs that vary reasoning effort in a controlled way (A2), so we make no empirical claim here and will list this as a limitation.

AuthorsOpenReview ↗

Responses to Question 3

Q3: Inconsistent Evaluation Pair Counts. Section 6 reports 33 model–benchmark runs and 27,189 model–question pairs, whereas Appendix F (Tables 3–8) totals 34 runs and 28,107 pairs. The difference of 918 questions exactly matches GPT-5.4-mini on SuperGPQA (Table 8), suggesting an unexplained exclusion. I believe the authors should clarify the exclusion criterion and its impact on the fitting results.

A-Q3: Thank you for catching this. The 918-pair difference is the GPT-5.4-mini run on SuperGPQA, which we excluded from fitting because its API provider changed partway through the run; its accuracies remain in Table 8.

In this run, Perplexity (minimal reasoning, T = 0.7, top-p = 0.95) served calibration, R0, all debate arms, MV and the first 33,564 SR calls, and OpenAI (gpt-5.4-mini-2026-03-17, low reasoning, default sampling) served the last 3,397 SR calls. Our fitting script skips any run flagged with a provider change, so this run has no fit and is not part of the 33-run analysis. The rule was added after the switch rather than specified in advance, but it depends only on the provenance flag and not on any outcome. Since no fit exists for this run, its effect on the fitting results cannot be measured. The 34 runs in Tables 3–8 are the 36 model–benchmark pairs minus GLM-5.3 on MMMU, which was not run, and GPT-5.4-mini on MMMU, whose SR chain stopped at 19,659 of about 37,560 calls. We will explain this in Section 6 and Appendix F.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive comments. We first clarify the positioning of the paper and then summarize the evidence that our responses refer to.

Positioning. The main contribution of the paper is the matched-budget theory in Section 4. It separates generation quality, measured by the baseline margin dpd_p, from revision stability, measured by the debate equilibrium margin dρ∗d^*_\rho, and explains why the ranking of MV, SR and MAD changes across settings. We would like to clarify that the theory in Section 5 (Theorem 5.3) considers a special case, a binary oracle gate, and is used as a heuristic to inspire the design of Gated Debate; it is not meant to analyze whether Gated Debate itself performs well. We will revise the paper to make this positioning clear.

§1. The experiments follow the orderings predicted by the theory. Pooled over the 27,189 test questions of the 33 fitted runs, the 40-call accuracies (%) of MV, SR and MAD follow the Table 1 ordering in all four (dp,d∗)(d_p, d^*) quadrants (λ = 0.1; 276 questions with a zero margin are not assigned to a quadrant).

Table A:

Quadrant (dp,d∗)(d_p, d^*)Predicted order (Table 1)nMVSRMAD
(+,+)MV > MAD > SR19,29298.0592.7495.11
(+,−)MV > SR > MAD5660.7133.9325.89
(−,+)MAD > SR > MV9425.0048.9462.23
(−,−)SR > MAD > MV7,4715.2213.4111.38

The two same-sign quadrants contain 26,763 questions from all 33 runs and carry most of the evidence. The two opposite-sign quadrants are small and come from Qwen3.5-4B on MMLU-Pro and SuperGPQA, so we treat them as supporting evidence.

None of the evaluated outcomes is used to assign the quadrants. dpd_p is computed from 20 calibration calls that no protocol reuses, the MV and SR accuracies come from separate runs, and for MAD, d∗d^* is fitted only on the calibration answers and debate rounds R1–R3, while accuracy is measured at R7.

The finite-NN model also predicts the observed accuracies. With parameters fitted on R1–R3, running the exact N=5N=5 model from the observed R3 state to R7 gives a pooled observed − predicted accuracy between +0.05 and +0.24 pp for the three group arms (MAD, NG and Gated Debate) and all λ ∈ {0.1, 1, 10}, and every 95% CI includes zero. Across questions, parameters fitted on micro-train questions predict 40-call test accuracy with MAE 1.32 pp for MAD and 0.73 pp for MV (Figure 3).

§2. Gated Debate. In the special case of Theorem 5.3, Eq. (15) gives an exact one-step rule: showing peer answers raises the expected number of correct agents exactly when the correct answer's group share exceeds its baseline probability. Gated Debate applies this rule without labels, using the baseline mode g^\hat g as the reference and a threshold τ≥0\tau \ge 0 selected on the dev set.

To separate the rule from the choice of reference, we ran a controller-only diagnostic on Qwen3.6-35B-A3B at 40 calls. The Gold Gate uses the gold label as the reference only inside the gate controller; the agents never see it, and the prompts, sampling and τ are the same as for Gated Debate. We also split the results by whether g^\hat g is correct. Accuracy (%):

Benchmark, stratum (n)Gold GateGated DebateAlways DebateMatched NG
MMLU-Pro, all (938)66.2062.2761.8961.41
MMLU-Pro, reference correct (577)96.6296.6294.1992.98
MMLU-Pro, reference wrong (361)17.597.3710.2510.94
MMMU, all (939)66.0361.9861.5060.81
MMMU, reference correct (581)93.8093.8091.1489.67
MMMU, reference wrong (358)20.9510.3413.4113.97
GPQA-Diamond, all (65)63.8561.5459.2353.08
GPQA-Diamond, reference correct (38)88.1689.4782.8977.63
GPQA-Diamond, reference wrong (27)29.6322.2225.9318.52

The rule is effective when the reference is accurate. Gold Gate improves over Gated Debate by +3.93 pp on MMLU-Pro and +4.05 pp on MMMU. When g^\hat g is correct, the two gates make the same decisions (except for two exact ties on GPQA-Diamond), and Gated Debate is above Matched No-Group; when g^\hat g is wrong, opening the gate amplifies the wrong answer. Pooled over the three benchmarks, Gated Debate − Matched No-Group is +4.14 pp with a correct reference (1196 questions) and −3.34 pp with a wrong one (746 questions), and over all 33 runs it is +0.92 and −1.81 pp. This is what Eq. (15) predicts once the correct label is replaced by an estimate, and it explains why the overall gains in Table 2 are modest: the bottleneck is estimating the reference, not the gating rule.

The diagnostic uses the original test splits (938, 939 and 65 questions; two MMLU-Pro questions with invalid updates are excluded), which overlap with the 1,000-question cohort of Table 5 on 897, 939 and 64 questions. This is why its accuracies differ slightly from Table 5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.