acceptodds
Under review as a conference paper at ICLR 2027

Think More or Read More? A Token-Budget Theory of Reasoning with Shared Context

Abstract

Reasoning systems increasingly improve answers by incorporating additional context into the model's input, from multi-agent debate, where agents read their peers' responses, to self-refine, where a model revisits its own drafts. In these systems, communication consumes context as one agent's output becomes another's input. We study the regime where reading and generation draw on one fixed token window, so every token spent reading is a token that cannot be spent reasoning. We develop a **token-budget theory** that formalizes this trade-off, modeling accuracy as a function of the reasoning budget remaining after reading and aggregating votes with a Condorcet jury model that permits correlated errors. For multi-agent debate, the theory gives a closed-form prediction for the optimal number of agents. This quantity is estimated from single-agent measurements, allowing the optimal group size to be determined without running debate, and the same accounting extends to a heterogeneous set of different models. We empirically verify our theory and show that, at the predicted group size, our method outperforms strong baselines under the tightest calibration spend, where debate-calibrated rules can afford only one or two questions. Self-refine obeys the same accounting, with its own draft in place of a peer message. The theory further predicts that communication pays only when the gain in the accuracy ceiling exceeds the reasoning room spent on reading, with this condition tightening as the window shrinks. Across five open models and four reasoning tasks, self-consistency matches or beats deployed debate at 93% of 980 matched window-and-group-size settings, and a paired bootstrap on the same cells finds self-consistency significantly ahead at 57% and debate at 0.5%. The gap is largest under tight windows and nearly disappears at a loose one, consistent with the theory. Under a shared window, the value of added context depends less on what a model reads than on how much room remains to think.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Debate, Self-Consistency, Self-Refine, Token Budget

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Wj7BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies multi-agent debate under a zero-sum token budget, analyzing the tradeoffs between dedicating tokens to reading peers' responses and to an agent's own reasoning. Because both draw on the same context window, adding agents increases the number of samples used for the final majority but leaves each agent less room to reason. The authors model this tradeoff and propose a method that predicts the best number of agents for given context window constraints using only single-agent runs, without needing to run debate.

Critical strengths

The authors provide an interesting framing of context management in a debate setting and provide some theoretical backing to determine how many agents should be deployed for a given context budget to best utilize the debate.

Critical weaknesses

The zero-sum approach to token budgeting would be sensible for debate over long reasoning traces or many rounds that fill the models' context windows. It is unclear how useful it is in practice, however, since deployments typically let models use their full context windows rather than truncating them artificially as the authors do. Windows of 1.2k-6.5k tokens are an unrealistic test setting for models that support 8k-128k tokens, especially given the low cost of running these small LMs. Evidence that debate performance stagnates as more agents fill the actual context window would be more convincing. The evaluated tasks, however, have responses of roughly 250-500 tokens by the authors' own measurements, and do not come close to filling a realistic window. Additionally, in Table 20 (held-out regret by window), TB-MAD has the lowest regret at only one of the seven windows, while simply using all available agents goes from the worst rule to the best as the window grows from 1.2k to 6.5k. Effectively, not using any rule wins at the two largest windows tested, both of which are still well below the context lengths these models would typically be deployed with.

Questions for authors

Could the authors please provide empirical validation of the theory in settings where the context window fills naturally such as with thinking/reasoning models or multi-round debates on difficult problems?

The debate setting studied uses a single round of cross-agent interaction with last-round visibility only; how do the predicted and observed optimal group sizes change when the debate consists of many rounds and optionally when the full chat transcript is retained?

Under the models' native context windows how would the actual debate performance of TB-MAD's recommended group size compare with simply using all available agents for the types of analytical reasoning tasks this paper evaluates? And how does this change as the number of agents increases (e.g 4->8->16)?

5 comments

We thank the reviewer for highlighting the gap between controlled experiments and practical deployment settings.

Weakness: unrealistic windows and tasks.

Windows of 1.2k-6.5k tokens are an unrealistic test setting for models that support 8k-128k tokens... The evaluated tasks, however, have responses of roughly 250-500 tokens by the authors' own measurements, and do not come close to filling a realistic window.

We agree. We also test the theory at a reasoning model's native 32k window, with naturally long reasoning traces and no artificial message cap (Q1, Tables W2-W3). These experiments show that the same context-pressure effects arise at realistic message lengths.

Actions. We will clarify that our analysis targets settings where reading and generation share a context window.

W1. Table 20.

In Table 20 (held-out regret by window), TB-MAD has the lowest regret at only one of the seven windows, while simply using all available agents goes from the worst rule to the best as the window grows from 1.2k to 6.5k. Effectively, not using any rule wins at the two largest windows tested, both of which are still well below the context lengths these models would typically be deployed with.

We agree on the reported numbers. TB-MAD achieves the lowest regret only at W=1600W=1600, while using all agents performs best at W=4500W=4500 and 65006500 (Table W1, showing four of the nine rules).

Table W1. Held-out regret (%) across 20 cells. Bold indicates the lowest regret among all nine rules.

Window WWSuccessive halving (%)Empirical argmax (%)All agents (nmax⁡n_{\max}) (%)TB-MAD (%)Best of nine
12001.030.8258.821.12Empirical argmax
16001.160.9854.230.96TB-MAD
20001.091.1742.721.49Successive halving
25000.591.0425.571.75Successive halving
32000.621.122.930.87Successive halving
45000.991.560.841.18All agents
65001.271.040.581.02All agents

Insight. Three observations. (1) All agents outperform TB-MAD by only 0.34 and 0.44% at W=4500W=4500 and 65006500, respectively. This advantage comes entirely from Llama3.2-3B/MMLU-Pro, where the fitted ability has the wrong sign. Excluding this cell reduces the gaps to 0.03 and 0.00%. (2) All agents perform poorly at W≤3200W\leq3200, incurring up to 58.8% regret. The native-window results in Q1 show the same pattern with full reasoning traces, but not with short messages. (3) Successive Halving or Empirical Argmax outperforms TB-MAD at five windows. However, Table 20 gives all methods the same 60 held-in calibration questions without accounting for calibration-token costs, favouring debate-calibrated rules. Under matched token budgets, TB-MAD performs better up to 8M tokens, while Successive Halving overtakes it at 16M (Table 2, Figure 2).

Actions. We will clarify the calibration setting in the main text and Table 20 caption, identify the windows where competing methods perform best, and include by-window comparison under matched token budgets in the appendix.

Q1. Native context window and reasoning model.

We agree. We tested Qwen3-8B in thinking mode at its native 32k window, without an artificial message cap, on MATH500 levels 4-5 and GPQA-Diamond. We chose these tasks because their reasoning traces naturally reach 5k-10k tokens. Each peer sends either its full response (full trace) or only its final answer (answers only).

Table W2. Native-window results for Qwen3-8B. MATH500 levels 4-5 (202 questions) and GPQA-Diamond (150 questions). All agents means n=8n=8 for full traces and n=16n=16 for answers only. The last column reports the percentage of agents hitting the context limit at n=8n=8. Bold indicates the best measured accuracy and lower regret.

SettingTokens per peern=1n=1n=2n=2n=4n=4n=8n=8n=16n=16Predicted n∗n^\astBest nnRegret of n∗n^\ast (%)Regret of all agents (%)Agents hitting window at n=8n=8 (%)
MATH500 L4–5, full trace760770716539-220.031.857
MATH500 L4–5, answers only7957071717171840.30.00
GPQA-Diamond, full trace943761605125-110.036.276
GPQA-Diamond, answers only9256161666566440.00.40

Table W3. Debate accuracy minus single-agent and self-consistency accuracy (%), with 95% confidence intervals over questions. Positive values favour debate. Bold indicates intervals excluding zero.

Settingvs. single agent, n=2n=2n=4n=4n=8n=8vs. self-consistency, n=2n=2n=4n=4n=8n=8
MATH500 L4-5, full trace+1.2 [-0.5, +2.8]-4.5 [-7.3, -1.8]-30.7 [-36.8, -24.9]+0.3 [-1.0, +1.7]-6.3 [-9.2, -3.5]-33.2 [-39.6, -27.2]
MATH500 L4-5, answers only+1.0 [-0.7, +2.6]+1.3 [+0.0, +2.8]+1.0 [-0.5, +2.6]+0.2 [-1.2, +1.5]-0.5 [-1.3, +0.3]-1.5 [-2.8, -0.5]
GPQA-Diamond, full trace-0.9 [-4.4, +2.9]-10.2 [-14.7, -6.2]-36.2 [-42.7, -30.0]-1.3 [-4.7, +2.2]-12.0 [-16.7, -7.8]-39.6 [-46.7, -32.4]
GPQA-Diamond, answers only-0.2 [-3.8, +3.6]+5.1 [+2.0, +8.4]+4.2 [+0.4, +8.0]-0.7 [-4.2, +2.9]+3.3 [+1.1, +6.0]+0.9 [-1.3, +3.1]

Insight. Full traces cause accuracy to collapse as group size increases, with 57–76% of agents hitting the context limit at n=8n=8. With answers only, no agents hit the limit and this collapse disappears. The predicted group size matches the measured optimum in three of four settings. For MATH500 with answers only, it predicts n=8n=8 instead of n=4n=4, but the regret is only 0.3% because the accuracy curve is flat. Crucially, debate significantly outperforms self-consistency in only one setting: GPQA-Diamond with answers only at n=4n=4. Thus, most gains over a single agent come from multi-sample voting rather than peer communication. This is consistent with our broader analysis, where voting without communication matches or beats debate at 93% of 980 matched operating points and is significantly worse at only 0.5%.

Actions. We will add the native-window results to a new appendix.

Q2. Multiple rounds with the full transcript.

We agree. We ran three rounds (an initial answer and two revisions), retaining the full transcript and full traces, with nn up to 8. In round 2, each agent reads the other agents' messages from rounds 0 and 1, totalling 2(n−1)2(n-1) peer messages. The predictor retains its static accounting but replaces mm with m3m_3, the mean tokens contributed by one peer across both earlier rounds. It does not model accuracy recovery in later rounds.

Table W4. The second column reports the mean tokens contributed by each peer from round 1 to round 2. Bold indicates 95% intervals excluding zero. The final column reports the change in the percentage of agents hitting the window at n=8n=8.

SettingTokens per peer, round 1 → 2Predicted n∗n^\astBest nnVote at n=8n=8, round 1 (%)Vote at n=8n=8, round 2 (%)Round 2 vs. single agent (%)Round 2 vs. self-consistency (%)Agents hitting window at n=8n=8, round 1 → 2 (%)
MATH500 L4–57607 → 9460283973+3.5 [+1.3, +5.6]+1.0 [-1.5, +3.1]57 → 5
GPQA-Diamond9437 → 9893222560-0.7 [-4.4, +3.1]-4.0 [-7.6, -0.7]76 → 2

Insight. The three-round results reveal a limitation of our static predictor. On MATH500, accuracy at n=8n=8 rebounds from 39% in round 1 to 73% in round 2, with the fraction of agents hitting the context limit falling from 57% to 5%. Consequently, the measured optimum is n=8n=8, while the predictor selects n∗=2n^\ast=2. On GPQA, both predict and observe n=2n=2, but debate remains below self-consistency. These results suggest that message-length accounting alone cannot capture round-dependent changes in agent behavior and accuracy.

Actions. We will report these results in the new appendix and acknowledge the inability of our static predictor to capture later-round behavior as a limitation.

Q3. TB-MAD vs. using all agents at the native window.

This comparison is included in the final two columns of Table W2. With full traces, using all agents is substantially worse than following the predicted group size. With answers only, using all agents performs comparably because the shorter messages no longer create context pressure. Thus, the value of TB-MAD is greatest when long peer messages constrain the feasible group size.

Actions. The new appendix will include results for all tested group sizes.

AuthorsOpenReview ↗

Response to Reviewer Wj7B [3/3]

We uploaded a revised paper (changes in blue). The promised actions now appear in the revision:

  • Weakness: scope (shared window) stated in App. E. Native-window results summarised in Sec. 4.3.
  • W1: by-window discussion in Sec. 4.2. Table 20's caption now states the shared 60 calibration questions. Per-window intervals and per-cell results are in Tables 21 and 22, and the by-window comparison at matched calibration spend is in Table 23.
  • Q1: App. D.11 (Tables 31, 32).
  • Q2: App. D.11 (Table 33). The missed recovery after round 2 is listed as a limitation in App. E.
  • Q3: App. D.11 (Table 32) reports every group size: 1 to 8 with full traces, 1 to 16 with answers only.
Reviewer_3AmfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a very practical design problem in multi-agent debate by introducing a token-budget theory that predicts the optimal debate size without running a debate. The method outperforms strong baselines under the tightest calibration budgets. The theory also explains when debate’s voting benefits outweigh its communication cost relative to self-consistency under the same token budget. Ablations show that reading cost drives performance collapse and the interior optimum, with robustness tests covering longer reasoning and practical token pricing.

Critical strengths
  • This paper studies a practical and important design problem in multi-agent debate: the trade-off between retained context length and reasoning length. The proposed token-budget theory can predict the best-performing group size using single-agent measurements, without running debate for calibration.
  • The experiments cover a comprehensive selection of models and settings. Also, the paper compares several calibration strategies across models, tasks, windows, and calibration budgets.
Critical weaknesses
  • The model mainly captures how reasoning length affects accuracy, while other factors, such as agent sycophancy, can also affect debate outcomes. It is unclear how well the prediction holds when peer communication changes agents’ behavior or error correlations.
  • The aggregation equation does not exactly describe the voting procedure. Section 3.3 treats beating the most probable wrong answer as equivalent to winning the vote. With multiple answer categories, another wrong answer can win a particular sampled vote. Since group-size selection depends on this approximation, the paper should validate its accuracy or clearly limit its claims.
  • Some theoretical claims appear stronger than the results establish. The stated assumptions do not establish uniqueness without additional conditions, and the advertised “closed-form” solution appears to be an optimization problem rather than an explicit formula. These points need clarification because they are central to the theoretical contribution. For example, Algorithm 1 specifies a continuous argmax of (S(n)), followed by rounding, and the proof supplies a stationarity equation. Neither constitutes an explicit closed-form solution for general fitted Hill parameters.
  • The experiments do not clearly enforce the shared-window constraint. The minimum generation allowance can exceed the remaining window, and some configurations nominally contain more peer text than the window permits. The paper should explain how these cases are handled and whether the main results hold when only feasible configurations are included.
Questions for authors
  • How accurately does the surrogate rank group sizes under exact multiclass voting?
  • How are peer messages and generation budgets handled when their combined length exceeds the window?
  • What additional assumptions are needed for uniqueness, and is there an explicit closed-form solution for the optimal group size?
3 comments

We thank the reviewer for their comments. Below we address their concerns and will revise the paper accordingly.

1. Behaviour change under debate.

We agree. We define post-debate measurements as those computed from agents' final-round answers after reading a peer's response, in contrast to the round-0, single-agent measurements used by our predictor. We then perform a diagnostic swap, replacing the round-0 measurements with their post-debate counterparts. The results show that updating correlation alone reduces the peak-location error by less than 2% (with ρ\rho increasing from 0.56 to 0.68). In contrast, updating per-agent accuracy reduces the error by 20% and closes 55% of the regret gap. This suggests that prediction errors arise primarily from changes in agent accuracy after peer interaction, rather than changes in correlation. The largest discrepancies occur on Gemma3-12B/CSQA and Llama3.2-3B/MMLU-Pro.

Actions. We will add an appendix reporting predicted and observed curves for every experimental cell, together with the swap-test results. We will also clarify that this swap test is a retrospective diagnostic using measurements from the same data, not a predictor available at inference time.

2. Surrogate vs. exact voting.

We agree. We estimate exact plurality voting via Monte Carlo sampling from each question's measured answer distribution across all 140 cell-window points. The two methods rank group sizes similarly (Kendall's τ\tau=0.80), agreeing on the optimal size at 56% of points and within one agent at 89%. Exact voting achieves lower regret (0.67% vs. 1.21%), showing a modest advantage over the surrogate. However, it requires question-specific answer distributions, making it a diagnostic rather than a deployable pre-debate predictor. Our surrogate remains deployable and outperforms the difficulty-mixture baseline (2.83% regret).

Actions. We will add this comparison to the appendix and clarify that our score is a surrogate for ranking group sizes in the main text.

3. "Closed form" and uniqueness.

We agree. For general Hill parameters, n∗n^* is defined by a scalar stationarity equation, not a closed form. We will rename it throughout. Uniqueness requires log-concavity of the gain, which we will state explicitly as Assumption 2 and include in Proposition 1. This condition holds by closed-form verification for the deployed Hill form and BE. For the two undeployed candidates (power law and Weibull), we verified it analytically and numerically over the fitted κ\kappa range, a weaker form of evidence (133/133 cases). We will distinguish these results in Appendix A.1. Finally, we will correct Algorithm 1 to reflect the actual implementation, which directly searches over integer group sizes rather than rounding a continuous optimum.

Actions. Replace "closed form" with "scalar stationarity equation" throughout; add Assumption 2 and update Proposition 1; document the verification results in Appendix A.1; and rewrite Algorithm 1 as an exact discrete search.

4. Window feasibility.

We agree. When peers do not fit, our implementation caps each peer message at mm tokens and reserves at least 64 tokens for generation, potentially exceeding the context window WW. This affects 205/980 points (21%). On the remaining 775 feasible points, self-consistency matches or beats debate in 91.6% (93.4% overall), with clear advantages in 46.1% (57.2% overall). Debate is clearly ahead in only 0.6% (0.5% overall). TB-MAD does not select an infeasible group size and maintains 1.20% regret.

Actions. We will document this implementation detail and report results separately for feasible and all points in the appendix.

AuthorsOpenReview ↗

Response to Reviewer 3Amf [2/2]

We uploaded a revised paper (changes in blue). The promised actions now appear in the revision:

  1. Behaviour change under debate: swap test and predicted vs. observed curves in App. D.10 (Table 30, Fig. 11), stated as a diagnostic, not a predictor.
  2. Surrogate vs. exact voting: App. D.8 (Table 27). Sec. 3.3 now calls the score a surrogate for ranking group sizes.
  3. Closed form: renamed to a scalar stationarity condition. Algorithm 1 now states an exact search over integer nn. One change from our response: we state uniqueness for the adopted Hill gain only, where log-concavity follows in closed form (Prop. 1, App. A.1 Step 4). This replaces the proposed Assumption 2 and the Pow/Weibull discussion.
  4. Window feasibility: App. D.9 (Tables 28, 29) and Sec. 4.2.

Dear Reviewer 3Amf, thanks again for your review.

Following your points, we added a swap test showing that prediction errors come mainly from changes in per-agent accuracy after debate (W1), and compared the surrogate with exact voting (W2). We also renamed the "closed form", proved uniqueness for the deployed Hill form, made Algorithm 1 an exact integer search (W3), and reported results on feasible points (W4). Do these address your remaining concerns?

We are happy to clarify anything further.

Reviewer_vUFGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the trade-off between reasoning and communication in multi-agent debate under a shared token window. The authors propose TB-MAD, a theoretical framework that models how reading peer responses reduces the generation budget available to each agent. By combining a single-agent accuracy curve with a correlated-voting model, TB-MAD predicts the group size that maximizes collective accuracy without requiring debate runs for calibration. Experiments across five open-source models and four reasoning benchmarks examine the predicted group-size behavior, calibration efficiency, and comparisons with self-consistency and self-refinement. The results suggest that communication can be counterproductive when the available token window is limited, while its relative cost diminishes as the window increases.

Critical strengths

S1. A meaningful theoretical perspective on the trade-off between reasoning and communication.

The paper provides a useful framework for understanding why adding more agents does not necessarily improve collective reasoning. By explicitly accounting for the generation budget consumed by reading peer responses, it explains how the benefits of majority voting can be offset by reduced individual reasoning capacity. The resulting connection between token allocation and optimal group size is both intuitive and practically relevant.

S2. Strong empirical coverage and calibration efficiency.

The experiments span five open-source models and four reasoning tasks, with additional analyses of heterogeneous groups, self-refinement, and token costs. A particularly appealing aspect is that TB-MAD predicts the optimal group size using single-agent measurements without requiring debate-based calibration. The comparison with self-consistency also provides useful evidence that additional communication is not always the most effective use of a limited inference budget.

Critical weaknesses

W1. The theory does not sufficiently account for the informational value of peer communication.

The proposed framework primarily models communication through the generation budget consumed by reading peer messages. However, communication can also change an agent's ability to solve a problem by providing missing evidence, identifying reasoning errors, or introducing alternative solution strategies. These effects may depend more on the content of the message than on its length.

The authors do examine whether peer reading improves the accuracy ceiling in Appendix D.2 and report small, mixed effects. Nevertheless, this analysis is limited to the evaluated communication settings. It remains unclear whether the same conclusions hold when agents exchange concise, targeted feedback rather than full responses. This matters because the paper's broader interpretation depends on whether communication is genuinely less valuable than additional reasoning, or whether the evaluated protocol simply delivers information inefficiently.

W2. The practical generality of the optimal group-size prediction remains uncertain.

TB-MAD predicts collective accuracy using single-agent generation-cap measurements and a correlated-voting approximation. While the reported regret results are encouraging, they do not fully establish whether the model accurately captures how interaction changes individual correctness and error correlations. A direct comparison between predicted and observed accuracy curves would make the theoretical contribution more convincing.

Moreover, the evaluation primarily assumes a shared token window and last-round visibility, where additional peer messages directly reduce the available generation budget. This is a meaningful controlled setting, but it may not reflect systems that selectively transmit feedback, compress communication, or impose separate input and output limits. The extent to which the predicted optimal group size remains reliable under these alternative configurations is therefore unclear.

Questions for authors

Q1. Separating the informational benefit of communication from its token cost

The central argument is that reading peer responses reduces the reasoning budget available to an agent, and that this reduction can outweigh the benefit of communication. However, the current formulation treats peer messages primarily as a token cost, whereas their informational value may vary substantially depending on what they contain and whether they address an actual reasoning error.

I appreciate that Appendix D.2 directly measures changes in the accuracy ceiling from peer reading and finds small, sign-mixed effects in the evaluated settings. My concern is therefore not that the paper ignores this issue, but that the reported analysis may be insufficient to support the broader interpretation.

Could the authors clarify how the theory should be applied when communication has a measurable positive effect on individual-agent accuracy?

In particular, would it be possible to examine communication settings in which agents receive concise error-specific feedback rather than full peer responses? If the same number of reading tokens is allocated to different types of feedback, does the resulting accuracy improvement remain similar, or does the content of the feedback substantially change the trade-off?

A related question is whether the authors envision extending the per-agent ability function to depend on both the remaining generation budget and the information received. Such an extension may help distinguish cases where communication is merely redundant from cases where it enables an agent to solve a problem it could not solve independently.

This would clarify whether the proposed framework captures a general principle of shared-context reasoning or primarily describes the cost of relatively unstructured peer communication.

Q2. How accurately does the theory predict actual debate behavior?

TB-MAD estimates the individual-agent ability curve from single-agent generation-cap sweeps and combines it with a correlated-voting approximation. This provides an efficient way to predict the optimal group size without running debate during calibration.

However, the resulting prediction depends on the assumption that these measurements provide an adequate description of agents after they have interacted. In practice, debate may alter both individual correctness and the correlation between agents' errors.

Could the authors provide a more direct comparison between the theoretical predictions and the observed debate outcomes across group sizes and window budgets?

Beyond reporting the regret of the selected group size, I would be interested in seeing whether the predicted accuracy curve reproduces the shape of the measured curve, including the location of its maximum and the rate of degradation beyond that maximum.

It would also be useful to identify the settings in which the predicted and observed optimal group sizes disagree most strongly. Are these errors associated with poor fitting of the single-agent ability curve, changes in agent-level accuracy after communication, or inaccuracies in the estimated error correlation?

Such an analysis would help establish which part of the theoretical approximation is responsible for its predictive success and where it is likely to fail.

Q3. Stability of the correlated-voting approximation

The group-accuracy analysis uses a correlated-voting model to account for dependencies among agent predictions. This is an important improvement over treating all votes as independent.

Nevertheless, a single correlation parameter may not fully characterize the dependence structure induced by debate. Agents may converge on the same incorrect reasoning, selectively adopt correct arguments, or form subgroups with different answers. These patterns can produce different majority-vote outcomes even when their aggregate correlation statistics are similar.

Could the authors investigate how the estimated correlation structure changes between independently generated responses and responses produced after peer communication?

For example, do the correlations among incorrect answers increase as the debate progresses? Does the approximation remain well calibrated for larger groups, where correlated errors may become more consequential?

It would be particularly informative to examine whether the prediction errors are systematically larger in settings where debate substantially changes the distribution of agent answers.

This analysis would strengthen the interpretation of the correlated-voting component and clarify whether its assumptions remain appropriate when agents actively influence one another.

Q4. Sensitivity to the communication protocol and effective reading cost

The experiments primarily use last-round visibility, with the retained peer messages consuming part of a shared token window. This is a reasonable way to isolate the communication-versus-generation trade-off, but it also makes the results sensitive to the amount of text agents are required to read.

The authors already provide a zero-reading-cost ablation and a robustness test involving question position. However, neither directly establishes how the optimal group size changes under different practical communication strategies.

Could the authors evaluate a small set of alternative communication protocols, such as shorter peer summaries, selective transmission of conflicting answers, or a fixed communication-token allowance?

The main question is whether the theory continues to predict the optimal group size when the effective reading cost varies independently of the number of agents.

For instance, if additional agents can participate without proportionally increasing the amount of text read by every recipient, does the predicted interior optimum remain accurate? Conversely, if a smaller number of agents exchange longer messages, does the model correctly predict the resulting performance degradation?

These experiments would help determine whether the proposed method can guide the design of communication-efficient multi-agent systems, rather than only the selection of group size under a fixed message-sharing protocol.

Q5. Interpreting the shared-window constraint in practical inference systems

The paper's formulation assumes that reading and generation draw from a common window, so increasing the amount of retained context directly reduces the generation room available to an agent.

This assumption is central to the theoretical results. However, many inference systems expose separate input-context and output-generation limits, and their monetary costs depend on actual input and output tokens rather than solely on a combined maximum length.

The authors do investigate different input/output token price ratios, which is a useful practical extension. Still, it would help to clarify the relationship between the theoretical window constraint and the resource constraints encountered in real deployments.

Under what deployment conditions should the shared-window model be considered an accurate approximation? If the context limit is not binding, but monetary cost or latency is constrained, does the same group-size prediction remain applicable?

I would also appreciate a clearer separation between the maximum generation allowance imposed by the experiment and the number of tokens actually generated. An agent may stop well before its cap, so the opportunity cost of allocating additional context need not always translate into an equivalent reduction in realized reasoning computation.

Clarifying this distinction would make the scope of the theoretical and practical conclusions more precise.

Q6. Identifying when the closed-form predictor is preferable to empirical alternatives

The results suggest that TB-MAD is particularly attractive when calibration resources are scarce. At larger calibration budgets, however, the paper reports that Successive Halving can overtake TB-MAD, and the heterogeneous experiments show a small advantage for Learned beta.

Could the authors characterize the regime in which the closed-form predictor provides the strongest practical advantage?

For example, is there a consistent relationship between the available calibration budget, the complexity of the single-agent ability curve, and the relative performance of TB-MAD versus empirical or more flexible learned predictors?

It would also be useful to understand whether the advantage comes primarily from requiring fewer calibration questions, avoiding debate runs, or reducing the variance of the operating-point estimate.

A clearer account of these trade-offs would help practitioners decide when to use TB-MAD and when a more expensive but potentially more accurate calibration procedure is justified.

6 comments

We thank the reviewer for their careful reading.

Q1. Value of a message.

We agree that the theory should account for message content. Reading a peer can raise the accuracy ceiling (Δpmax⁡\Delta p_{\max}) but also consumes thinking room and debate is beneficial only when the former outweighs the latter. We measured Δpmax⁡\Delta p_{\max} per agent and found small, cell-dependent gains that change sign (Table 15). We also tested three message protocols: full responses (capped at mm=320 tokens), self-generated summaries, and conflict-only messages containing peers' final answers that differ from the reader's (Table V1). Summaries are generated by the same Qwen2.5-7B agent in at most 128 tokens using a fixed prompt and their tokens are excluded from the agents' context windows. Conflict messages require no additional LLM calls.

Table V1. Cost and effect of peer-message content (Qwen2.5-7B, three pooled seeds). Gain per agent is the accuracy change after reading a peer; bold marks the largest mean gain.

What a peer showsTokens per peerGain per agent, mean (%)GSM8K (%)MATH500 (%)Best nn, measuredBest nn, predictedWithin one agent (%)
Full289-0.5-2.1+1.04.95.193
Summary93+0.4-1.7+2.46.37.079
Conflict3-3.5-6.5-0.47.48.079

Insight. Shorter messages allow larger groups, and the prediction captures this trend. However, they do not consistently improve accuracy: gains remain small, vary across tasks, and are negative for conflict-only messages. Prediction accuracy also declines for summaries and conflict-only messages relative to full responses.

Actions. We will make the accuracy ceiling protocol-dependent, writing it as β(g;π)\beta(g;\pi), in Section 3.2. A new appendix will detail the three protocols and the summary-generation procedure.

Q2. Predicted vs. observed curves.

We agree. We will add appendix plots overlaying predicted and measured accuracy against group size nn for every cell. Since our deployed predictor (Eq. 9) outputs a margin rather than accuracy, we use exact majority voting from each question's measured answer distribution to construct accuracy curves as a diagnostic. Across 140 points (20 cells × 7 windows):

  • Peak location error: Distance between predicted n∗n^\ast and the measured optimal nn, averaging 1.3 agents; within one agent in 72% of points.
  • Peak height error: Exact-vote peak accuracy minus measured peak accuracy, averaging +0.5% (+2.2% absolute).
  • Post-peak decay: Exact voting increasingly underestimates accuracy for larger groups, with signed errors of -1.1% for n<6n<6 and -7.2% for n≥6n\geq6. We use 6 to illustrate where the discrepancy grows.

Where it disagrees, and why. The largest errors occur on Gemma3-12B/CSQA and Llama3.2-3B/MMLU-Pro. Replacing round-0 measurements with post-debate values closes only 2% of the peak-location error when updating correlation, but 20% when updating per-agent accuracy, also closing 55% of the regret gap. Thus, prediction errors arise primarily from changes in agent accuracy after peer interaction, rather than correlation shifts or poor ability estimates.

Actions. We will add per-cell overlay plots and formally define all three error measures in the appendix.

Q3. Correlation under debate.

We agree that correlation increases, but this does not explain the prediction error. After one debate round, error correlation rises from 0.56 to 0.68, while the fraction of wrong-agent pairs producing the same wrong answer increases from 54% to 65% (Table V2). However, prediction error is essentially unrelated to the shift in answer distributions (Spearman ρ=−0.07\rho=-0.07). As shown in Q2, changes in per-agent accuracy explain the prediction error better.

Table V2. Error correlation under debate, averaged over 20 cells (5 models × 4 benchmarks). Bold indicates post-debate values.

MetricRound 0After one round
Error correlation0.560.68
Two wrong agents give the same wrong answer (%)5465
Spearman: prediction error vs. answer-distribution shift-0.07

Actions. We will report round-0 and post-debate error correlation, same-wrong-answer rates, and the rationale for this restriction in the appendix.

Q4. Reading cost independent of nn.

We agree that this is the right test. We varied the message cap mm at fixed group size and window across six cells. As mm increases, both measured and predicted optimal group sizes decrease (Table V3). We also tested Qwen3-8B in thinking mode with its native 32k window and no message cap, using MATH500 (levels 4-5) and GPQA-Diamond, where reasoning traces naturally reach 5k-10k tokens. Peers shared either their full traces or final answers only. Both predicted and measured optima respond to this reduction in message length: on MATH500, both increase, while on GPQA both move from n=1n=1 with full traces to n=4n=4 with answers only (Table V4).

Table V3. Measured and predicted optimal group sizes under different message caps. Bold indicates the change from m=100m=100 to m=700m=700.

Message cap mm (tokens)Best nn, measuredBest nn, predictedWithin one agent (%)
1006.16.276
3204.34.586
7003.43.190
Change (700 − 100)-2.7-3.1Same direction in 86%

Table V4. Native-window results (Qwen3-8B, three pooled seeds). Regret measures accuracy lost relative to the measured optimum.

SettingTokens per peerPredicted best nnMeasured best nnRegret (%)Regret using all agents (%)
MATH500 L4–5, full trace7607220.031.8
MATH500 L4–5, answers only795840.30.0
GPQA, full trace9437110.036.2
GPQA, answers only925440.00.4

Actions. Appendix D.5 and Figure 9 will report the comparison with measured optima, and a new appendix will cover the native-window experiments.

Q5. Shared window vs. separate limits.

We partially agree. Under separate input/output cost or latency constraints, a shared window is unnecessary. Reading cost can instead be expressed in output-token-equivalent units by weighting mm by the input-to-output price ratio, with the optimum determined by cost (Appendix D.3). However, shared windows are not a narrow case. They matter when context is the limiting resource, particularly for reasoning traces, multi-round transcripts, and long inputs. Our native-window experiments confirm this: with full traces and n=8n=8, 57% of MATH500 agents and 76% of GPQA agents exhaust the context window.

Actions. Appendix D.3 already covers cost-constrained settings.

Q6. When to use the predictor.

We agree that this needs clarification. At equal calibration-token budgets, TB-MAD achieves lower regret up to 8M tokens, while Successive Halving overtakes it at 16M (1.12% vs. 1.29%). At equal numbers of calibration questions, Successive Halving leads from approximately K=20K=20, but requires 15-19 times more calibration tokens. Thus, TB-MAD's advantage is lower calibration cost, not lower variance.

Actions. We will clarify this practical selection rule in the text accompanying Figure 2.

AuthorsOpenReview ↗

Response to Reviewer vUFG [3/3]

We uploaded a revised paper (changes in blue). The promised actions now appear in the revision:

  • Q1: protocol-dependent ceiling β(g;π)\beta(g;\pi) in Sec. 3.2. The three protocols and the summary prompt in App. D.12 (Table 34).
  • Q2: overlay plots in Fig. 11. The three error measures are defined in App. D.10 and reported in Table 30.
  • Q3: round-0 vs. post-debate correlation, same-wrong-answer rates and the restriction to W≥3.2W\ge3.2k, n≤5n\le5 in App. D.10.
  • Q4: mm sweep against measured optima in App. D.6 (Fig. 9); native-window experiments in App. D.11.
  • Q5: unchanged, App. D.3.
  • Q6: selection rule in Sec. 4.2, next to Fig. 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We sincerely thank all reviewers for their thoughtful and constructive feedback.

We appreciate the reviewers' recognition of several strengths of our work:

  • A practical and important design problem: reading peers and reasoning compete for one token window (vUFG, 3Amf, Wj7B).
  • A theory that predicts the best group size from single-agent measurements, without running debate for calibration (vUFG, 3Amf, Wj7B).
  • Broad evaluation across five models, four tasks and several calibration strategies (vUFG, 3Amf).
  • A useful comparison with self-consistency under a limited budget (vUFG).

Below, we address the main concerns raised by the reviewers, with details provided in the individual responses.

  • Realism: evaluated Qwen3-8B at its native 32k context window, with no artificial cap. Full reasoning traces fill the window naturally and reproduce the collapse (Wj7B, vUFG).
  • When to use the predictor: clarified Table 20 and added a matched-budget comparison (TB-MAD performs best up to 8M tokens) (Wj7B, vUFG).
  • Theory: renamed "closed form", proved uniqueness for the deployed Hill form, made Algorithm 1 an exact integer search, and compared the surrogate with exact voting (3Amf).
  • Behaviour under debate: added predicted versus observed curves and a swap test identifying errors in per-agent accuracy (3Amf, vUFG).
  • New experiments: added message-protocol comparisons, a message-cap sweep, results on configurations that fit the window, and three-round debate (vUFG, 3Amf, Wj7B).
  • Limitations: acknowledged the failure of three-round debate on MATH500 and clarified that our contribution is trade-off analysis, not gains from communication (Wj7B, vUFG).
  • Follow-up: analysed regret versus per-agent accuracy, protocol gains versus message length, and near-optimal selection at the native window (vUFG).

We welcome any further questions before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.