- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies multi-agent debate under a zero-sum token budget, analyzing the tradeoffs between dedicating tokens to reading peers' responses and to an agent's own reasoning. Because both draw on the same context window, adding agents increases the number of samples used for the final majority but leaves each agent less room to reason. The authors model this tradeoff and propose a method that predicts the best number of agents for given context window constraints using only single-agent runs, without needing to run debate.
Critical strengths
The authors provide an interesting framing of context management in a debate setting and provide some theoretical backing to determine how many agents should be deployed for a given context budget to best utilize the debate.
Critical weaknesses
The zero-sum approach to token budgeting would be sensible for debate over long reasoning traces or many rounds that fill the models' context windows. It is unclear how useful it is in practice, however, since deployments typically let models use their full context windows rather than truncating them artificially as the authors do. Windows of 1.2k-6.5k tokens are an unrealistic test setting for models that support 8k-128k tokens, especially given the low cost of running these small LMs. Evidence that debate performance stagnates as more agents fill the actual context window would be more convincing. The evaluated tasks, however, have responses of roughly 250-500 tokens by the authors' own measurements, and do not come close to filling a realistic window. Additionally, in Table 20 (held-out regret by window), TB-MAD has the lowest regret at only one of the seven windows, while simply using all available agents goes from the worst rule to the best as the window grows from 1.2k to 6.5k. Effectively, not using any rule wins at the two largest windows tested, both of which are still well below the context lengths these models would typically be deployed with.
Questions for authors
Could the authors please provide empirical validation of the theory in settings where the context window fills naturally such as with thinking/reasoning models or multi-round debates on difficult problems?
The debate setting studied uses a single round of cross-agent interaction with last-round visibility only; how do the predicted and observed optimal group sizes change when the debate consists of many rounds and optionally when the full chat transcript is retained?
Under the models' native context windows how would the actual debate performance of TB-MAD's recommended group size compare with simply using all available agents for the types of analytical reasoning tasks this paper evaluates? And how does this change as the number of agents increases (e.g 4->8->16)?
5 comments
Table W3. Debate accuracy minus single-agent and self-consistency accuracy (%), with 95% confidence intervals over questions. Positive values favour debate. Bold indicates intervals excluding zero.
| Setting | vs. single agent, | vs. self-consistency, | ||||
|---|---|---|---|---|---|---|
| MATH500 L4-5, full trace | +1.2 [-0.5, +2.8] | -4.5 [-7.3, -1.8] | -30.7 [-36.8, -24.9] | +0.3 [-1.0, +1.7] | -6.3 [-9.2, -3.5] | -33.2 [-39.6, -27.2] |
| MATH500 L4-5, answers only | +1.0 [-0.7, +2.6] | +1.3 [+0.0, +2.8] | +1.0 [-0.5, +2.6] | +0.2 [-1.2, +1.5] | -0.5 [-1.3, +0.3] | -1.5 [-2.8, -0.5] |
| GPQA-Diamond, full trace | -0.9 [-4.4, +2.9] | -10.2 [-14.7, -6.2] | -36.2 [-42.7, -30.0] | -1.3 [-4.7, +2.2] | -12.0 [-16.7, -7.8] | -39.6 [-46.7, -32.4] |
| GPQA-Diamond, answers only | -0.2 [-3.8, +3.6] | +5.1 [+2.0, +8.4] | +4.2 [+0.4, +8.0] | -0.7 [-4.2, +2.9] | +3.3 [+1.1, +6.0] | +0.9 [-1.3, +3.1] |
Insight. Full traces cause accuracy to collapse as group size increases, with 57–76% of agents hitting the context limit at . With answers only, no agents hit the limit and this collapse disappears. The predicted group size matches the measured optimum in three of four settings. For MATH500 with answers only, it predicts instead of , but the regret is only 0.3% because the accuracy curve is flat. Crucially, debate significantly outperforms self-consistency in only one setting: GPQA-Diamond with answers only at . Thus, most gains over a single agent come from multi-sample voting rather than peer communication. This is consistent with our broader analysis, where voting without communication matches or beats debate at 93% of 980 matched operating points and is significantly worse at only 0.5%.
Actions. We will add the native-window results to a new appendix.
Q2. Multiple rounds with the full transcript.
We agree. We ran three rounds (an initial answer and two revisions), retaining the full transcript and full traces, with up to 8. In round 2, each agent reads the other agents' messages from rounds 0 and 1, totalling peer messages. The predictor retains its static accounting but replaces with , the mean tokens contributed by one peer across both earlier rounds. It does not model accuracy recovery in later rounds.
Table W4. The second column reports the mean tokens contributed by each peer from round 1 to round 2. Bold indicates 95% intervals excluding zero. The final column reports the change in the percentage of agents hitting the window at .
| Setting | Tokens per peer, round 1 → 2 | Predicted | Best | Vote at , round 1 (%) | Vote at , round 2 (%) | Round 2 vs. single agent (%) | Round 2 vs. self-consistency (%) | Agents hitting window at , round 1 → 2 (%) |
|---|---|---|---|---|---|---|---|---|
| MATH500 L4–5 | 7607 → 9460 | 2 | 8 | 39 | 73 | +3.5 [+1.3, +5.6] | +1.0 [-1.5, +3.1] | 57 → 5 |
| GPQA-Diamond | 9437 → 9893 | 2 | 2 | 25 | 60 | -0.7 [-4.4, +3.1] | -4.0 [-7.6, -0.7] | 76 → 2 |
Insight. The three-round results reveal a limitation of our static predictor. On MATH500, accuracy at rebounds from 39% in round 1 to 73% in round 2, with the fraction of agents hitting the context limit falling from 57% to 5%. Consequently, the measured optimum is , while the predictor selects . On GPQA, both predict and observe , but debate remains below self-consistency. These results suggest that message-length accounting alone cannot capture round-dependent changes in agent behavior and accuracy.
Actions. We will report these results in the new appendix and acknowledge the inability of our static predictor to capture later-round behavior as a limitation.
Q3. TB-MAD vs. using all agents at the native window.
This comparison is included in the final two columns of Table W2. With full traces, using all agents is substantially worse than following the predicted group size. With answers only, using all agents performs comparably because the shorter messages no longer create context pressure. Thus, the value of TB-MAD is greatest when long peer messages constrain the feasible group size.
Actions. The new appendix will include results for all tested group sizes.
Response to Reviewer Wj7B [3/3]
We uploaded a revised paper (changes in blue). The promised actions now appear in the revision:
- Weakness: scope (shared window) stated in App. E. Native-window results summarised in Sec. 4.3.
- W1: by-window discussion in Sec. 4.2. Table 20's caption now states the shared 60 calibration questions. Per-window intervals and per-cell results are in Tables 21 and 22, and the by-window comparison at matched calibration spend is in Table 23.
- Q1: App. D.11 (Tables 31, 32).
- Q2: App. D.11 (Table 33). The missed recovery after round 2 is listed as a limitation in App. E.
- Q3: App. D.11 (Table 32) reports every group size: 1 to 8 with full traces, 1 to 16 with answers only.
We thank the reviewer for highlighting the gap between controlled experiments and practical deployment settings.
Weakness: unrealistic windows and tasks.
We agree. We also test the theory at a reasoning model's native 32k window, with naturally long reasoning traces and no artificial message cap (Q1, Tables W2-W3). These experiments show that the same context-pressure effects arise at realistic message lengths.
Actions. We will clarify that our analysis targets settings where reading and generation share a context window.
W1. Table 20.
We agree on the reported numbers. TB-MAD achieves the lowest regret only at W=1600, while using all agents performs best at W=4500 and 6500 (Table W1, showing four of the nine rules).
Table W1. Held-out regret (%) across 20 cells. Bold indicates the lowest regret among all nine rules.
Insight. Three observations. (1) All agents outperform TB-MAD by only 0.34 and 0.44% at W=4500 and 6500, respectively. This advantage comes entirely from Llama3.2-3B/MMLU-Pro, where the fitted ability has the wrong sign. Excluding this cell reduces the gaps to 0.03 and 0.00%. (2) All agents perform poorly at W≤3200, incurring up to 58.8% regret. The native-window results in Q1 show the same pattern with full reasoning traces, but not with short messages. (3) Successive Halving or Empirical Argmax outperforms TB-MAD at five windows. However, Table 20 gives all methods the same 60 held-in calibration questions without accounting for calibration-token costs, favouring debate-calibrated rules. Under matched token budgets, TB-MAD performs better up to 8M tokens, while Successive Halving overtakes it at 16M (Table 2, Figure 2).
Actions. We will clarify the calibration setting in the main text and Table 20 caption, identify the windows where competing methods perform best, and include by-window comparison under matched token budgets in the appendix.
Q1. Native context window and reasoning model.
We agree. We tested Qwen3-8B in thinking mode at its native 32k window, without an artificial message cap, on MATH500 levels 4-5 and GPQA-Diamond. We chose these tasks because their reasoning traces naturally reach 5k-10k tokens. Each peer sends either its full response (full trace) or only its final answer (answers only).
Table W2. Native-window results for Qwen3-8B. MATH500 levels 4-5 (202 questions) and GPQA-Diamond (150 questions). All agents means n=8 for full traces and n=16 for answers only. The last column reports the percentage of agents hitting the context limit at n=8. Bold indicates the best measured accuracy and lower regret.