- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this work, the authors develop a mathematical model explaining when multi-agent debate, majority voting, and self-refinement performs best under a fixed model-call budget. Accordingly, the authors propose Gated Debate, which selectively shares the peer answers based on the support for the model's base favorite, and evaluate this across various models and benchmarks.
Critical strengths
- I like how the authors set a budget constraint to ensure fair comparison across the three modes. To my knowledge, it hasn't been considered in previous work to match the budget across voting, debate, and self-refinement.
- Strong theoretical motivation and demonstration.
- The approach to gate the three different methods is new and is a plausible direction to consider.
Critical weaknesses
-
The biggest issue is the significance of the reported performance. In Table 2, out of 18 settings, only 4 show the advantage of Gated Debate compared to the baselines, and the performance gap is not large enough even in those several cases. While the authors claim that Gated Debate often outperforms "Always debate", this does not justify that the method is useful.
-
I appreciate the theoretical explanations in the methods section, but to me, it seems to be overly complicated. I think it would be better if the authors could explain the method in plain text, rather than having to set a formal definition (e.g., Definition 5.1). In its current form, some additional cognitive load is required to understand the method.
-
To my understanding, the theoretical guarantee concerns a binary oracle gate that knows the correct answer. However, the implemented algorithm substitutes it with the empirical mean. But wouldn't this be a problem when the model doesn't know the correct answer? That is, to my understanding, the proposed method can actually reinforce wrong answers, given that the ground-truth solution is not known. If this is the case, the not-so-good experimental results make sense. To show that this method works, please provide results on a difficult benchmark whose single-agent accuracy is below 1/N, where N is the number of agents involved in debate.
-
This is relatively minor, but the motivation stated in the second paragraph of the Intro section is questionable. As far as I know, Du et al. (2023) proposed "multi-agent debate", but didn't compare with majority voting; Xu et al. (2026) did not emphasize the multi-agent debate protocol as its key results, and thus it is difficult to say it was positive about MAD; Choi et al., 2026 and Estornell & Liu, 2024 were some of the works that identified problems of multi-agent debate. The motivating works that the authors pose are not necessarily pointing to opposite directions, but rather are a natural flow of research on MAD. Hence, deriving the motivation to mix the three protocols (debate, voting, self-refinement) from this background can be a bit misleading.
Overall, I am not convinced about the method itself, and the experiments also show questionable gains compared to baselines. Thus, I lean towards rejection at this point.
Questions for authors
See Weaknesses.
2 comments
Response to Weakness 4
W4: This is relatively minor, but the motivation stated in the second paragraph of the Intro section is questionable. As far as I know, Du et al. (2023) proposed "multi-agent debate", but didn't compare with majority voting; Xu et al. (2026) did not emphasize the multi-agent debate protocol as its key results, and thus it is difficult to say it was positive about MAD; Choi et al., 2026 and Estornell & Liu, 2024 were some of the works that identified problems of multi-agent debate. The motivating works that the authors pose are not necessarily pointing to opposite directions, but rather are a natural flow of research on MAD. Hence, deriving the motivation to mix the three protocols (debate, voting, self-refinement) from this background can be a bit misleading.
A4: Our intention was to motivate a matched-budget comparison, not to suggest that prior studies contradict each other. To avoid this reading, we will revise Sections 1 and 2 to describe the literature as a progression across tasks, protocols and assumptions:
- Du et al. (2023; ICML 2024) included majority-voting baselines, compared debate at matched call counts (6 independent calls vs. 3 agents × 2 rounds) and analyzed tokens (§3.1, Table 1, App. A.3–A.4). We will cite this accurately.
- We will remove Xu et al. (2026), which studies long-context chunking rather than standard MAD, and correct the year of Choi et al. to NeurIPS 2025.
- We will credit prior theory on debate dynamics and interventions (Estornell & Liu, NeurIPS 2024; Choi et al., NeurIPS 2025; Okawa et al., ICML 2026) and contrast gating during the debate with supervised overturning after it (e.g., Minority Sentinel, arXiv:2606.29270).
Responses to Weakness 1-3
We thank you for the careful reading and thoughtful comments. We provide detailed responses below.
W1: The biggest issue is the significance of the reported performance. In Table 2, out of 18 settings, only 4 show the advantage of Gated Debate compared to the baselines, and the performance gap is not large enough even in those several cases. While the authors claim that Gated Debate often outperforms "Always debate", this does not justify that the method is useful.
A1: We would like to clarify that the theory in Section 5 (Theorem 5.3) considers a special case, a binary oracle gate, and is used as a heuristic to inspire the design of Gated Debate; it is not meant to analyze whether Gated Debate itself performs well. We therefore do not claim that Gated Debate beats every baseline. The main contribution of the paper is the theory in Section 4, which separates generation quality (dp) from revision stability (dρ∗) and explains why the ranking of MV, SR and MAD changes across settings (General Response §1).
The count in Table 2 is correct: Gated Debate is best in 4 of the 18 cells and tied in 1, and it is above Matched No-Group in 13. Paired 95% intervals over 27,189 questions at 40 calls show a significant gain over SR (+1.36 pp [+0.99, +1.71]) and differences within noise for MV (−0.20 [−0.49, +0.09]), Always Debate (+0.16 [−0.14, +0.47]) and Matched No-Group (+0.15 [−0.15, +0.43]).
The gains are modest because the label-free reference is sometimes wrong, not because the gating rule is ineffective. With the same prompts and budget, giving the gold label only to the gate controller improves accuracy by +3.93 pp on MMLU-Pro and +4.05 pp on MMMU (General Response §2). We will revise the abstract and Sections 1, 5 and 6 accordingly and remove the remark that attributes the 10-call results to noise.
W2: I appreciate the theoretical explanations in the methods section, but to me, it seems to be overly complicated. I think it would be better if the authors could explain the method in plain text, rather than having to set a formal definition (e.g., Definition 5.1). In its current form, some additional cognitive load is required to understand the method.
A2: We will add a plain-language description before Definitions 5.1–5.4:
(1) Before the debate, make 20 independent calls and take the most frequent answer as the reference, with frequency p^ref. (2) Before each revision round, check whether the share of agents holding the reference exceeds p^ref by more than τ. (3) If so, each agent sees the other four answers when revising; if not, each agent revises alone. Each round costs 5 calls either way. Since the reference may be wrong, opening can reinforce either a correct answer or a shared error (A3).
W3: To my understanding, the theoretical guarantee concerns a binary oracle gate that knows the correct answer. However, the implemented algorithm substitutes it with the empirical mean. But wouldn't this be a problem when the model doesn't know the correct answer? That is, to my understanding, the proposed method can actually reinforce wrong answers, given that the ground-truth solution is not known. If this is the case, the not-so-good experimental results make sense. To show that this method works, please provide results on a difficult benchmark whose single-agent accuracy is below 1/N, where N is the number of agents involved in debate.
A3: We would like to clarify that the theory in Section 5 (Theorem 5.3) considers a special case, a binary oracle gate, and is used as a heuristic to inspire the design of Gated Debate; it is not meant to analyze whether Gated Debate itself performs well. The practical gate replaces the correct answer with the baseline mode, and when the baseline mode is wrong it can reinforce the wrong answer, as the reviewer notes. This is consistent with the failure mechanism suggested by Eq. (15) in the binary setting, and our results confirm it (General Response §2). With a wrong reference, Gated Debate reaches 7.37% vs. 10.94% for Matched No-Group on MMLU-Pro (361 questions) and 10.34% vs. 13.97% on MMMU (358 questions); on the 27 such GPQA-Diamond questions it reaches 22.22%, vs. 25.93% for Always Debate and 18.52% for Matched No-Group. This also explains the modest overall gains in Table 2.
Regarding SA < 1/N, none of our 34 runs falls in this regime. We computed 1-call single-agent accuracy from the first answer in each question's MV pool, counting invalid answers as wrong. The lowest value is 27.6% for Qwen3.5-4B on SuperGPQA (27.0% over all 60 independent samples per question), followed by Qwen3.5-9B (32.8%) and Qwen3.6-35B-A3B (39.5%) on the same benchmark. Keeping only the questions a model gets wrong would not substitute for a benchmark chosen in advance, so we make no empirical claim about this regime; testing it requires a new evaluation on such a benchmark.