acceptodds
Under review as a conference paper at ICLR 2027

How Communication Topology Density Shapes Performance in LLM-Based Multi-Agent Systems

Abstract

Communication topology is a key design variable in LLM-based multi-agent reasoning. Prior work has shown that sparse communication can outperform full connectivity, but the mechanisms linking communication density and degree allocation to performance remain incompletely understood. We propose a method for analyzing how communication topology affects multi-agent reasoning performance. It first exactly decomposes per-round changes in agent-level answer information into corrective gain and misleading loss, then introduces a graph-based propagation model of evidence coverage, error dilution, and same-direction error reinforcement. Under homogeneous fixed-state assumptions, we derive an explicit optimality condition for communication density and show when it has a unique interior solution: corrective gain increases with diminishing returns, whereas misleading loss reflects a competition between error dilution and same-direction error reinforcement. Under a fixed edge budget and the stated homogeneity assumptions, regular or near-regular degree allocations maximize predicted performance among feasible graphs without isolated nodes. Experiments with two open-source models on six reasoning datasets provide qualitative support: the best accuracy among tested conditions occurs at an intermediate density in all 12 model–dataset combinations, and a fixed-budget comparison favors balanced degrees. Together, these results provide a mechanistic basis for communication-topology design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-based multi-agent systems, multi-agent debate, communication topology, graph density, information propagation, error propagation, collective reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_S9DvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why more communication does not always improve multi-agent reasoning. It models useful information sharing and the spread of errors, and derives conditions favoring an intermediate communication density and a balanced distribution of connections. Experiments with two models on six datasets provide qualitative support for these predictions.

Critical strengths
  • The paper asks a useful question rather than simply proposing another graph-learning method. Explaining when communication helps or hurts could give clearer guidance on how to organize agents and avoid unnecessary exchanges.

  • The controlled experiments are a meaningful strength. The authors separately test the effects of adding correct neighbors and repeating the same wrong answer. This gives more support for the proposed explanation than accuracy comparisons alone.

Critical weaknesses
  • The additional novelty needs a sharper explanation. The paper acknowledges that sparse communication can outperform full connectivity and that communication can spread both correct and incorrect information. The main addition appears to be a particular mathematical model. It is not yet clear which important insight this model provides beyond those existing explanations.

  • The theory and experiments measure different things. The analysis concerns a simplified information model, while the experiments measure self-reported confidence and voting accuracy. The paper acknowledges this distinction, but matching broad trends does not show that the proposed model accurately explains the observed performance. This limits the strength of the theoretical validation.

Questions for authors
  • Which prediction most clearly distinguishes your model from the earlier information-propagation explanations discussed in the introduction?

  • Can model parameters estimated from some experimental runs predict which densities perform better on separate runs, without fitting the model separately to each density?

3 comments
AuthorsOpenReview ↗

Response to Reviewer S9Dv — Part 1/3

Q1. Distinguishing predictions and novelty

Thank you for asking which prediction most clearly distinguishes our model from the earlier information-propagation explanations. The works discussed in our introduction examine beneficial and harmful information propagation, correction and corruption, conformity, and uncertainty in debate. Our contribution connects these mechanisms explicitly to communication density and degree allocation. The clearest additional prediction is that, under homogeneous fixed-state propagation and a fixed edge budget, balancing node degrees maximizes the modeled net-information objective among feasible graphs without isolated nodes; regular or near-regular degree allocations are therefore optimal under these assumptions. The model also yields a parameter-dependent optimality condition for communication density, characterizing how evidence strength, dilution, and error-direction concentration affect the optimum. These results provide explicit, conditional communication-design principles beyond the shared observation that sparse communication can outperform full connectivity, and can guide the selection of topological density, with practical application value.

Q2. Parameter estimation and held-out density prediction

Thank you for asking whether propagation parameters can be estimated or calibrated from observed agent behavior to predict the optimal communication density on held-out data. Under the homogeneous fixed-state model, the model-based optimal communication density is governed by three effective parameters, namely the evidence-revelation probability θ\theta, relative coverage strength κ\kappa, and relative dilution strength β\beta.

θ=Pr⁡(a neighbor reveals a given evidence item),0≤θ≤1,\theta = \Pr(\text{a neighbor reveals a given evidence item}),\quad\quad 0 \leq \theta \leq 1,

κ=ACω3η2Sπ,β=τζ=ω2ω3(1ηSπ−1).\kappa = \frac{\mathcal{A}}{\mathcal{C}\omega_{3}\eta^{2}S_{\pi}},\quad\quad\beta = \tau\zeta = \frac{\omega_{2}}{\omega_{3}}\left( \frac{1}{\eta S_{\pi}} - 1 \right).

Here A=λ∑tAt\mathcal{A} = \lambda\sum_{t}^{}A_{t} and C=μ∑tCt\mathcal{C} = \mu\sum_{t}^{}C_{t} are the aggregate scales in Eq. (A.30), and Sπ=∑sπs2S_{\pi} = \sum_{s}^{}\pi_{s}^{2} is error-direction concentration. The ratios above apply when the reinforcement scale Cω3η2Sπ\mathcal{C}\omega_{3}\eta^{2}S_{\pi} is positive.

We estimate effective counterparts of these parameters from first-update answer-level gains and losses on the calibration questions. We estimate θ\theta by grid search, choosing the value that minimizes the joint least-squares fitting error. For each candidate θ\theta, nonnegative least squares estimates the coverage, dilution, and pair-reinforcement coefficients, together with density-independent offsets. We obtain κ^\widehat{\kappa} by dividing the fitted coverage coefficient by the reinforcement coefficient, and β^\widehat{\beta} by dividing the fitted dilution coefficient by the same coefficient.

Substituting the fitted parameters into the graph-level coverage and disturbance expressions in Eqs. (7)–(8) and (A.19), evaluated at the observed node degrees, gives the predicted net-information score. Its highest-ranked density is

ρ^model∈arg,max⁡ρ∈RB^(Gρ)−M^(Gρ),R=0.2,0.4,0.6,0.8,1.0.{\widehat{\rho}}_{model} \in \operatorname*{arg\\,max}_{\rho \in \mathcal{R}}\\{\widehat{B}(G_{\rho}) - \widehat{M}(G_{\rho})\\},\quad\quad\mathcal{R} = \\{ 0.2,0.4,0.6,0.8,1.0\\}.

AuthorsOpenReview ↗

Response to Reviewer S9Dv — Part 2/3

Ranking uses the unnormalised fitted objective, which also admits boundary solutions. We retain its top two densities and select between them using round-three calibration accuracy. The selected density is then evaluated on held-out test questions. With Qwen2.5-7B-Instruct, we use five fixed splits of 250 questions per task into 100 calibration and 150 disjoint test questions on SciQ, Belebele (English), and ReClor.

Calibration and test questions are sampled from the same task distribution, assuming comparable difficulty and domain composition. Five splits assess sensitivity to the calibration sample. Substantial distribution shifts call for task-specific recalibration.

Table R1. Density choices for five calibration/test splits on three selected tasks. Candidate densities are ordered by predicted net information.

DatasetSplitModel-shortlisted densitiesSelected ρ\rhoTest-best ρ\rhoHit
SciQ11.0, 0.80.80.8Yes
SciQ21.0, 0.80.80.8Yes
SciQ31.0, 0.80.80.8Yes
SciQ40.6, 0.80.80.8Yes
SciQ50.4, 0.20.20.8No
Belebele (English)11.0, 0.80.80.8Yes
Belebele (English)21.0, 0.80.80.8Yes
Belebele (English)31.0, 0.80.80.8Yes
Belebele (English)40.8, 1.01.00.8No
Belebele (English)51.0, 0.81.00.8No
ReClor10.4, 0.20.20.4No
ReClor20.8, 0.60.80.4No
ReClor30.6, 0.40.40.4Yes
ReClor40.4, 0.60.40.4Yes
ReClor50.6, 0.40.40.4Yes

A hit means that the selected density attains the highest test accuracy among the five evaluated densities. Different splits reuse questions from the same task panel.

The selected density attained a test-best accuracy in 4/5 SciQ splits and 3/5 splits on each of Belebele (English) and ReClor, giving 10/15 hits (66.7%) in this selected three-task illustration.

Table R2. Token cost on 100 calibration questions, averaged over five splits. Total tokens include input and output usage, with shared round-0 responses counted once. Values are in millions of tokens.

DatasetModel-guidedFull scanToken saving
SciQ13.80521.28135.1%
Belebele (English)16.18123.40830.9%
ReClor16.51926.83238.4%

Model-guided selection uses one update at all five densities and two more at the shortlisted pair; full scanning uses three at every density. Savings compare both methods on the same calibration questions.

Linking theoretical and measured information quantities

We acknowledge that the original confidence-based proxies provided a loose connection to the theoretical BB and MM. We replace them with Monte Carlo frequency estimates of answer-label conditional mutual information for the first communication round, covering both models, all six datasets, and five communication densities. The estimates use sampled answer transitions and require neither an ϵ\epsilon floor nor a token-probability scoring pass. Let YY denote the correct answer label, and let A0A^{0} and A1A^{1} denote the selected answer labels before and after the first communication update.

B_{ans} & = I(Y;A^{1} \mid A^{0}) \\\\ & = \sum_{y,a,b}^{}p(y,a,b)\log\frac{p(y,a,b)p(a)}{p(y,a)p(a,b)}, \end{aligned}$$ $$\begin{aligned} M_{ans} & = I(Y;A^{0} \mid A^{1}) \\\\ & = \sum_{y,a,b}^{}p(y,a,b)\log\frac{p(y,a,b)p(b)}{p(y,b)p(a,b)}. \end{aligned}$$
AuthorsOpenReview ↗

Response to Reviewer S9Dv — Part 3/3

For each evaluated run, we collect triples (Yq,Ai,q0,Ai,q1)(Y_{q},A_{i,q}^{0},A_{i,q}^{1}) from the sampled task questions and the agents’ generated answers. We approximate the joint probability mass function by their empirical frequencies, giving each question equal weight and valid agents within each question equal weight,

p^r(y,a,b)=1∣Qr∣∑q∈Qr1∣Vq,r∣∑i∈Vq,r1Yq=y,Ai,q0=a,Ai,q1=b.{\widehat{p}}_{r}(y,a,b) = \frac{1}{|\mathcal{Q}_{r}|}\sum_{q \in \mathcal{Q}_{r}}^{}\frac{1}{|V_{q,r}|}\sum_{i \in V_{q,r}}^{}\mathbf{1}\\{ Y_{q} = y,A_{i,q}^{0} = a,A_{i,q}^{1} = b\\}.

Here Qr\mathcal{Q}_{r} contains questions with valid paired answers and Vq,rV_{q,r} their valid agents. Both estimates substitute the same p^r{\widehat{p}}_{r} and its marginals into the expressions above, with natural logarithms and sums over positive-probability cells. We compute them within each run and then average runs equally. This instantiates the theoretical chain-rule decomposition at the observable answer-label level and preserves

B^ans−M^ans=I^(Y;A1)−I^(Y;A0).{\widehat{B}}_{ans} - {\widehat{M}}_{ans} = \widehat{I}(Y;A^{1}) - \widehat{I}(Y;A^{0}).

The following tables report the resulting answer-level corrective gain and misleading loss, respectively.

Corrective gain BansB_{ans} (nats). Columns give communication density

DatasetModel0.20.40.60.81.0
AQuA-RATLlama0.10250.11730.12100.13220.1373
AQuA-RATQwen0.07430.08900.09590.10340.1163
Balanced MMLULlama0.04930.05010.06410.06940.0709
Balanced MMLUQwen0.05550.05850.06790.07230.0770
ARC-ChallengeLlama0.08210.08580.08620.08900.0894
ARC-ChallengeQwen0.02090.02230.02500.02700.0289
HellaSwagLlama0.08220.08090.08510.08950.1186
HellaSwagQwen0.03150.03240.04120.04190.0448
OpenBookQALlama0.06110.07540.07950.08210.0926
OpenBookQAQwen0.03740.03880.03900.04000.0406
Professional MedicineLlama0.07390.07810.08020.08650.0907
Professional MedicineQwen0.04580.04910.05190.05450.0548

Misleading loss MansM_{ans} (nats). Columns give communication density

DatasetModel0.20.40.60.81.0
AQuA-RATLlama0.13800.13620.11900.10580.1061
AQuA-RATQwen0.03660.03460.03440.03540.0373
Balanced MMLULlama0.04780.03410.03850.04580.0544
Balanced MMLUQwen0.01490.01450.01340.01650.0159
ARC-ChallengeLlama0.08170.06910.05480.05410.0552
ARC-ChallengeQwen0.01960.01390.01220.00660.0095
HellaSwagLlama0.05690.05360.05240.05500.0707
HellaSwagQwen0.03900.02740.02530.02770.0278
OpenBookQALlama0.08420.06990.05800.05110.0500
OpenBookQAQwen0.03640.02580.02340.02600.0262
Professional MedicineLlama0.06870.05240.04740.04910.0510
Professional MedicineQwen0.02570.01370.01080.00870.0105

The new estimates range from 0.0209 to 0.1373 nats for corrective gain and 0.0066 to 0.1380 nats for misleading loss. Their numerical magnitudes differ from the original confidence-based scores, while the evidence for density-dependent gain–loss competition remains. Across the 12 model–dataset combinations, gain increases monotonically over the tested grid in 11 combinations, and loss reaches its sampled minimum at an intermediate density in 11 combinations. The main accuracy result is also preserved: the highest observed accuracy occurs at an intermediate tested density in all 12 combinations.

Reviewer_52N6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how communication topology density influences reasoning performance in LLM-based multi-agent systems. The authors introduce an information-theoretic framework that decomposes changes in agent-level answer information into corrective gain and misleading loss. Building on this decomposition, they develop a graph-based propagation model incorporating useful evidence coverage, error dilution, and same-direction error reinforcement.

Under homogeneous and fixed-state assumptions, the model establishes conditions for an interior optimal communication density and predicts that regular or near-regular degree allocations maximize a modeled performance objective under a fixed edge budget.

The empirical evaluation uses Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on six reasoning benchmarks, with 11 agents and three communication rounds. Among the tested configurations, intermediate densities achieve the highest final accuracy in all 12 model–dataset combinations. Additional experiments investigate degree allocation, controlled error propagation, and validation-based density selection.

Critical strengths
  1. An interpretable theoretical perspective on communication topology.

The paper addresses an important question in multi-agent reasoning: why increasing communication connectivity does not necessarily improve performance. Rather than treating communication topology solely as an architecture-search problem, the authors attempt to explain its effects through useful information acquisition and misleading influence.

The separation between corrective gain and misleading loss provides a useful conceptual perspective, and the subsequent coverage and disturbance models offer an interpretable explanation for diminishing communication benefits and potential performance degradation under dense connectivity.

The analytical results are presented with explicit assumptions, and the appendix provides detailed mathematical derivations. The fixed-edge-budget analysis also offers a potentially useful perspective on how communication links should be distributed across agents.

  1. Empirical results that broadly support the qualitative motivation.

The experiments cover two open-source LLM backbones and six reasoning datasets. Intermediate communication densities consistently achieve the highest observed accuracy among the tested configurations, supporting the argument that fully connected communication is not always preferable.

The authors also investigate round-wise changes in correct-answer confidence and conduct controlled interventions involving misleading neighbors. These experiments provide additional evidence that both useful information sharing and harmful social influence can affect multi-agent reasoning.

I appreciate the inclusion of multiple graph realizations, repeated generations, and paired confidence intervals. The validation-based density selection experiments also provide an initial demonstration of practical applicability.

Critical weaknesses
  1. The theoretical framework is not sufficiently connected to the quantities measured in the experiments.

The central information-theoretic decomposition in Eq. (4) follows directly from the mutual-information chain rule. While mathematically valid, this identity does not independently establish the proposed communication mechanisms or provide a predictive relationship between graph topology and downstream accuracy.

The subsequent propagation model introduces specific assumptions about independent evidence revelation, additive evidence coverage, error dilution, and same-direction reinforcement. These assumptions lead to the derived concavity and optimality results, but their correspondence to actual LLM information propagation is not quantitatively established.

In particular, the empirical positive and negative transfer metrics in Eq. (16) are based on changes in self-reported correct-answer probabilities. As the authors acknowledge, these quantities are not numerical estimators of the conditional mutual information appearing in the theoretical decomposition.

Consequently, the experiments primarily demonstrate qualitative patterns consistent with the proposed model rather than directly validating its central theoretical quantities.

Furthermore, the analytical density model is derived under an Erdős–Rényi graph assumption, whereas the main experimental comparisons use regular or near-regular graphs. The authors do not estimate the propagation parameters or demonstrate that the resulting theoretical optimum accurately predicts the empirically optimal density.

These limitations weaken the claimed mechanistic connection between the theoretical framework and the observed multi-agent performance.

  1. The experimental design does not fully isolate the effect of communication topology, and the practical conclusions require stronger validation.

Increasing communication density simultaneously increases the number of neighboring messages received by each agent. Under the evaluated protocol, this also changes input-context length, information redundancy, and potentially the computational burden of reasoning.

The observed degradation under dense connectivity therefore cannot be attributed exclusively to same-direction error reinforcement or other proposed propagation mechanisms without additional controls for communication volume and context length.

Similarly, the controlled local experiments do not fully isolate error reinforcement from neighborhood composition. The dilution condition contains one incorrect and progressively more correct neighbors, while the reinforcement condition contains progressively more neighbors supporting the same incorrect answer. These interventions vary several factors simultaneously. A stronger test would hold the total number of neighbors and incorrect messages fixed while varying only whether the incorrect neighbors agree with one another.

The degree-allocation experiment is also limited to one backbone and one dataset. Because the theoretical model depends primarily on node degrees, it cannot distinguish graphs with identical degree sequences but different clustering, path lengths, or higher-order connectivity. Such structural differences may nevertheless affect multi-round information propagation.

Finally, although Appendix E reports paired confidence intervals, the compared optimal densities are selected based on observed performance within the same experimental comparisons. The statistical analysis should account for this selection and for variation across graph realizations.

The application-oriented experiments use independent validation and test splits, which is appropriate, but involve only two tasks and 150 test questions per task. The accuracy improvements, particularly the 0.67-percentage-point gain on MathQA, are relatively small.

Overall, the empirical evidence supports the practical value of intermediate connectivity, but does not yet establish the proposed mechanisms or optimal-density predictions with sufficient precision.

Questions for authors

Q1. Quantitative connection between theory and empirical measurements.

Can the authors clarify how the mutual-information-based corrective gain and misleading loss in Eqs. (4)–(5) are related quantitatively to the self-reported log-score transfer metrics in Eq. (16)?

Is it possible to estimate or calibrate the propagation parameters from observed agent behavior and predict the optimal communication density on held-out datasets?

This would substantially strengthen the claim that the framework provides a predictive mechanism rather than a qualitative explanation.

Q2. Controlling communication volume and self-revision effects.

Can the authors compare communication densities under a controlled input-token or message-length budget?

In addition, the no-communication baseline performs no subsequent self-update. How would the conclusions change if agents performed the same number of independent self-revision rounds without receiving neighbor messages?

These controls would help separate topology-induced information propagation from effects associated with longer contexts and additional reasoning.

Q3. Isolating error dilution and same-direction reinforcement.

In the controlled intervention experiments, can the authors hold the neighborhood size and the number of incorrect neighbors constant while varying the agreement among incorrect neighbors?

For example, comparing several incorrect neighbors supporting the same wrong answer with an equal number supporting different wrong answers would more directly test same-direction reinforcement.

The current experimental conditions do not fully separate this mechanism from the changing proportion of correct and incorrect messages.

Q4. Graph structure beyond degree distribution.

The fixed-edge-budget result favors balanced degree allocations under homogeneous assumptions. However, graphs with identical degree sequences can have substantially different clustering coefficients, diameters, and information-propagation paths.

Can the authors compare multiple graphs with the same number of nodes, edges, and degree sequence but different higher-order structures?

Such an experiment would help determine whether degree balance alone explains the observed performance or whether additional graph properties are important in multi-round reasoning.

Q5. Statistical reliability and practical density selection.

Can the authors provide selection-adjusted uncertainty estimates for the observed best-performing densities and account for variation across graph realizations?

For the validation-based density selection experiments, how stable are the selected densities across different train–validation splits or random seeds?

It would also be useful to report the inference-token and wall-clock costs of the selected graphs, since reducing the number of communication edges does not necessarily imply a proportional reduction in total inference cost.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 52N6 — Part 1/4

Q1. Theory, measurement, and density prediction

We acknowledge that the original confidence-based proxies provided a loose connection to the theoretical BB and MM. We replace them with Monte Carlo frequency estimates of answer-label conditional mutual information for the first communication round, covering both models, all six datasets, and five communication densities. The estimates use sampled answer transitions and require neither an ϵ\epsilon floor nor a token-probability scoring pass. Let YY denote the correct answer label, and let A0A^{0} and A1A^{1} denote the selected answer labels before and after the first communication update.

B_{ans} & = I(Y;A^{1} \mid A^{0}) \\\\ & = \sum_{y,a,b}^{}p(y,a,b)\log\frac{p(y,a,b)p(a)}{p(y,a)p(a,b)}, \end{aligned}$$ $$\begin{aligned} M_{ans} & = I(Y;A^{0} \mid A^{1}) \\\\ & = \sum_{y,a,b}^{}p(y,a,b)\log\frac{p(y,a,b)p(b)}{p(y,b)p(a,b)}. \end{aligned}$$ For each evaluated run, we collect triples $(Y_{q},A_{i,q}^{0},A_{i,q}^{1})$ from the sampled task questions and the agents’ generated answers. We approximate the joint probability mass function by their empirical frequencies, giving each question equal weight and valid agents within each question equal weight, $${\widehat{p}}_{r}(y,a,b) = \frac{1}{|\mathcal{Q}_{r}|}\sum_{q \in \mathcal{Q}_{r}}^{}\frac{1}{|V_{q,r}|}\sum_{i \in V_{q,r}}^{}\mathbf{1}\\{ Y_{q} = y,A_{i,q}^{0} = a,A_{i,q}^{1} = b\\}.$$ Here $\mathcal{Q}_{r}$ contains questions with valid paired answers and $V_{q,r}$ their valid agents. Both estimates substitute the same ${\widehat{p}}_{r}$ and its marginals into the expressions above, with natural logarithms and sums over positive-probability cells. We compute them within each run and then average runs equally. This instantiates the theoretical chain-rule decomposition at the observable answer-label level and preserves $${\widehat{B}}_{ans} - {\widehat{M}}_{ans} = \widehat{I}(Y;A^{1}) - \widehat{I}(Y;A^{0}).$$ The following tables report the resulting answer-level corrective gain and misleading loss, respectively. **Corrective gain** $B_{ans}$ **(nats). Columns give communication density** |Dataset|Model|0.2|0.4|0.6|0.8|1.0| |---|---|---|---|---|---|---| |AQuA-RAT|Llama|0.1025|0.1173|0.1210|0.1322|0.1373| |AQuA-RAT|Qwen|0.0743|0.0890|0.0959|0.1034|0.1163| |Balanced MMLU|Llama|0.0493|0.0501|0.0641|0.0694|0.0709| |Balanced MMLU|Qwen|0.0555|0.0585|0.0679|0.0723|0.0770| |ARC-Challenge|Llama|0.0821|0.0858|0.0862|0.0890|0.0894| |ARC-Challenge|Qwen|0.0209|0.0223|0.0250|0.0270|0.0289| |HellaSwag|Llama|0.0822|0.0809|0.0851|0.0895|0.1186| |HellaSwag|Qwen|0.0315|0.0324|0.0412|0.0419|0.0448| |OpenBookQA|Llama|0.0611|0.0754|0.0795|0.0821|0.0926| |OpenBookQA|Qwen|0.0374|0.0388|0.0390|0.0400|0.0406| |Professional Medicine|Llama|0.0739|0.0781|0.0802|0.0865|0.0907| |Professional Medicine|Qwen|0.0458|0.0491|0.0519|0.0545|0.0548| **Misleading loss** $M_{ans}$ **(nats). Columns give communication density** |Dataset|Model|0.2|0.4|0.6|0.8|1.0| |---|---|---|---|---|---|---| |AQuA-RAT|Llama|0.1380|0.1362|0.1190|0.1058|0.1061| |AQuA-RAT|Qwen|0.0366|0.0346|0.0344|0.0354|0.0373| |Balanced MMLU|Llama|0.0478|0.0341|0.0385|0.0458|0.0544| |Balanced MMLU|Qwen|0.0149|0.0145|0.0134|0.0165|0.0159| |ARC-Challenge|Llama|0.0817|0.0691|0.0548|0.0541|0.0552| |ARC-Challenge|Qwen|0.0196|0.0139|0.0122|0.0066|0.0095| |HellaSwag|Llama|0.0569|0.0536|0.0524|0.0550|0.0707| |HellaSwag|Qwen|0.0390|0.0274|0.0253|0.0277|0.0278| |OpenBookQA|Llama|0.0842|0.0699|0.0580|0.0511|0.0500| |OpenBookQA|Qwen|0.0364|0.0258|0.0234|0.0260|0.0262| |Professional Medicine|Llama|0.0687|0.0524|0.0474|0.0491|0.0510| |Professional Medicine|Qwen|0.0257|0.0137|0.0108|0.0087|0.0105|
AuthorsOpenReview ↗

Response to Reviewer 52N6 — Part 2/4

The new estimates range from 0.0209 to 0.1373 nats for corrective gain and 0.0066 to 0.1380 nats for misleading loss. Their numerical magnitudes differ from the original confidence-based scores, while the evidence for density-dependent gain–loss competition remains. Across the 12 model–dataset combinations, gain increases monotonically over the tested grid in 11 combinations, and loss reaches its sampled minimum at an intermediate density in 11 combinations. The main accuracy result is also preserved: the highest observed accuracy occurs at an intermediate tested density in all 12 combinations.

Q1 (continued). Parameter estimation and density prediction

Thank you for asking whether propagation parameters can be estimated or calibrated from observed agent behavior to predict the optimal communication density on held-out data. Under the homogeneous fixed-state model, the model-based optimal communication density is governed by three effective parameters, namely the evidence-revelation probability θ\theta, relative coverage strength κ\kappa, and relative dilution strength β\beta.

θ=Pr⁡(a neighbor reveals a given evidence item),0≤θ≤1,\theta = \Pr(\text{a neighbor reveals a given evidence item}),\quad\quad 0 \leq \theta \leq 1,

κ=ACω3η2Sπ,β=τζ=ω2ω3(1ηSπ−1).\kappa = \frac{\mathcal{A}}{\mathcal{C}\omega_{3}\eta^{2}S_{\pi}},\quad\quad\beta = \tau\zeta = \frac{\omega_{2}}{\omega_{3}}\left( \frac{1}{\eta S_{\pi}} - 1 \right).

Here A=λ∑tAt\mathcal{A} = \lambda\sum_{t}^{}A_{t} and C=μ∑tCt\mathcal{C} = \mu\sum_{t}^{}C_{t} are the aggregate scales in Eq. (A.30), and Sπ=∑sπs2S_{\pi} = \sum_{s}^{}\pi_{s}^{2} is error-direction concentration. The ratios above apply when the reinforcement scale Cω3η2Sπ\mathcal{C}\omega_{3}\eta^{2}S_{\pi} is positive.

We estimate effective counterparts of these parameters from first-update answer-level gains and losses on the calibration questions. We estimate θ\theta by grid search, choosing the value that minimizes the joint least-squares fitting error. For each candidate θ\theta, nonnegative least squares estimates the coverage, dilution, and pair-reinforcement coefficients, together with density-independent offsets. We obtain κ^\widehat{\kappa} by dividing the fitted coverage coefficient by the reinforcement coefficient, and β^\widehat{\beta} by dividing the fitted dilution coefficient by the same coefficient.

Substituting the fitted parameters into the graph-level coverage and disturbance expressions in Eqs. (7)–(8) and (A.19), evaluated at the observed node degrees, gives the predicted net-information score. Its highest-ranked density is

ρ^model∈arg,max⁡ρ∈RB^(Gρ)−M^(Gρ),R=0.2,0.4,0.6,0.8,1.0.{\widehat{\rho}}_{model} \in \operatorname*{arg\\,max}_{\rho \in \mathcal{R}}\\{\widehat{B}(G_{\rho}) - \widehat{M}(G_{\rho})\\},\quad\quad\mathcal{R} = \\{ 0.2,0.4,0.6,0.8,1.0\\}.

Ranking uses the unnormalised fitted objective, which also admits boundary solutions. We retain its top two densities and select between them using round-three calibration accuracy. The selected density is then evaluated on held-out test questions. With Qwen2.5-7B-Instruct, we use five fixed splits of 250 questions per task into 100 calibration and 150 disjoint test questions on SciQ, Belebele (English), and ReClor.

Calibration and test questions are sampled from the same task distribution, assuming comparable difficulty and domain composition. Five splits assess sensitivity to the calibration sample. Substantial distribution shifts call for task-specific recalibration.

AuthorsOpenReview ↗

Response to Reviewer 52N6 — Part 3/4

Table R1. Density choices for five calibration/test splits on three selected tasks. Candidate densities are ordered by predicted net information.

DatasetSplitModel-shortlisted densitiesSelected ρ\rhoTest-best ρ\rhoHit
SciQ11.0, 0.80.80.8Yes
SciQ21.0, 0.80.80.8Yes
SciQ31.0, 0.80.80.8Yes
SciQ40.6, 0.80.80.8Yes
SciQ50.4, 0.20.20.8No
Belebele (English)11.0, 0.80.80.8Yes
Belebele (English)21.0, 0.80.80.8Yes
Belebele (English)31.0, 0.80.80.8Yes
Belebele (English)40.8, 1.01.00.8No
Belebele (English)51.0, 0.81.00.8No
ReClor10.4, 0.20.20.4No
ReClor20.8, 0.60.80.4No
ReClor30.6, 0.40.40.4Yes
ReClor40.4, 0.60.40.4Yes
ReClor50.6, 0.40.40.4Yes

A hit means that the selected density attains the highest test accuracy among the five evaluated densities. Different splits reuse questions from the same task panel.

The selected density attained a test-best accuracy in 4/5 SciQ splits and 3/5 splits on each of Belebele (English) and ReClor, giving 10/15 hits (66.7%) in this selected three-task illustration.

Calibration token costs are reported in Table R2 in our response to Q5.

Q2. Communication budget and self-revision

We are sorry for the unclear presentation. We clarify that the experiments in this work use controlled input-token/message-length budgets to examine corrective-gain and misleading-loss propagation under a common communication-budget constraint. Moreover, we add a no-communication baseline with subsequent self-update, in which agents perform three independent self-revision rounds. The table reports changes in final team accuracy relative to the no-communication baseline without subsequent self-update. These changes show that subsequent self-update has a limited effect on accuracy in the evaluated settings.

DatasetLlama change (pp)Qwen change (pp)
MMLU+0.6+0.4
Professional Medicine−0.3−0.2
AQuA-RAT−0.3+0.3
ARC-Challenge0.00.0
HellaSwag−0.2+0.2
OpenBookQA−0.50.0

The largest absolute accuracy difference between the two no-communication baselines is 0.6 percentage points. This is small compared with the several-percentage-point advantages of intermediate-density communication over full connectivity seen in the main HellaSwag results. Moreover, changing the no-communication reference leaves the measured accuracies and rankings of the communicating topologies unchanged. The conclusion that an intermediate tested density outperforms full connectivity therefore remains intact.

Q3. Error dilution and aligned-error reinforcement

Thank you for recommending this control. On Balanced MMLU, we fixed four neighbors, two correct and two incorrect. For two example wrong options B and C, the concentrated group used BB and CC, and the dispersed group used BC and CB. Each incorrect neighbor had a distinct message for each option. This balanced each message's use across groups, with matched group-mean input lengths. Initially correct focal agents switched to an incorrect answer in 12.50% versus 5.83% of updates, a paired difference of 6.67 percentage points. This provides direct behavioral evidence that agreement among incorrect neighbors increases misleading influence when neighbor count and error count are fixed. The primary outcome uses answer correctness and is independent of self-reported confidence and the probability floor.

Error groupWrong-option assignmentsCorrect-to-incorrect rate
ConcentratedBB / CC12.50%
DispersedBC / CB5.83%
AuthorsOpenReview ↗

Response to Reviewer 52N6 — Part 4/4

Q4. Graph structure beyond degree distribution

We added a three-round fixed-degree structural control using Qwen2.5-7B-Instruct on Balanced MMLU. All six connected graphs have 11 nodes, 22 edges, and degree 4 at every node. Three graphs have low clustering and three have high clustering. All graphs have the same mean shortest-path length (18/11) and diameter (3).

Structure groupClustering coefficientFinal team accuracy
Low clustering0.09172.67%
High clustering0.36473.67%

The clustering coefficient increases fourfold, while mean final team accuracy changes by 1.00 percentage point, from 72.67% to 73.67%. These results suggest that the tested graph structures with different clustering coefficients have a limited effect on accuracy under the matched-degree conditions.

Q5. Selection stability and calibration cost

Thank you for your question. Our calibration experiment evaluates density selection on disjoint held-out test questions, with fitting and selection confined to the calibration questions. Across the five fixed splits in Table R1, the most frequently selected density is 0.8 for SciQ (4/5 splits), 0.8 for Belebele (English; 3/5), and 0.4 for ReClor (3/5). The modal density recurs in 60–80% of the splits, indicating a recurring density preference across different calibration/test partitions.

Table R2 reports input-plus-output token costs on 100 calibration questions, averaged over five splits, with shared round-0 responses counted once. Model-guided shortlisting reduces calibration token usage by 35.1% on SciQ, 30.9% on Belebele (English), and 38.4% on ReClor relative to full five-density calibration. These measurements demonstrate substantial token savings during calibration search, reducing the cost of applying the theory to task-specific density selection.

Table R2. Token cost on 100 calibration questions, averaged over five splits. Total tokens include input and output usage, with shared round-0 responses counted once. Values are in millions of tokens.

DatasetModel-guidedFull scanToken saving
SciQ13.80521.28135.1%
Belebele (English)16.18123.40830.9%
ReClor16.51926.83238.4%

Model-guided selection uses one update at all five densities and two further updates at the shortlisted pair. Full scanning completes three updates at all five densities. Savings compare model-guided selection with full scanning on the same calibration questions.

Regular graphs and the ER analysis

Thank you for highlighting the distinction between graph families. The ER assumption gives a tractable density-averaged expression. Before averaging over ER graphs, our model defines coverage and disturbance conditional on node degree, and these expressions can be evaluated on a given graph under the stated assumptions. The fixed-edge-budget degree-allocation result likewise holds without requiring ER sampling.

Regular or near-regular graphs provide controlled density comparisons by fixing the edge count within each density condition and limiting degree heterogeneity, which would otherwise vary across ER realizations. This design focuses the available inference budget on comparisons across densities. Controlled comparisons using regular topologies also have precedent in Li et al. (2024, Sec. 3.1).

Accordingly, our experiments test qualitative topology-dependent predictions under controlled degree allocations. The ER ensemble average and the controlled regular-graph experiments are distinguished in scope.

Reviewer_K6tZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the density of the communication graph affects LLM multi-agent debate. It makes three contributions: It writes the per-round change in each agent’s mutual information with the correct answer as a “corrective gain” minus a “misleading loss” (Eqs. 4–5). It adds a stylised propagation model with three pieces. Corrective gain is concave in density because each neighbour reveals an evidence item independently. Misleading loss is convex because error dilution (∝ 1/d) competes with same-direction reinforcement (∝ d(d−1)). The sum then has a unique interior optimum under stated conditions, and balanced degrees are optimal under a fixed edge budget (via Jensen). It tests these predictions with Llama-3.1-8B and Qwen2.5-7B (11 agents, 3 rounds, six multiple-choice datasets). The tests are density sweeps, a degree-heterogeneity experiment, controlled neighbour-composition interventions and a validation-based density selection study.

Critical strengths
  • The question is useful and the paper is unusually clear about its assumptions and limits. The derivations are careful. I re-derived Eqs. (A.15), (A.32)–(A.39) and (A.43) and they hold.
  • The empirical pattern is consistent. Across 12 model–dataset combinations, the best density is never 0.2 or 1.0 (0.4 in 7 cases, 0.6 in 4, 0.8 in 1). Under an exchangeable-argmax null, avoiding both endpoints in all 12 would have probability about 0.6¹² ≈ 0.2%.
  • The reporting is reasonable: 5 sampled graphs × 3 runs per density, with paired CIs in Appendix E. All the CIs are above zero.
  • The intervention experiment (Fig. 3c–d) is clean. It separates the effect of neighbour composition from neighbour count: one wrong neighbour among correct ones is diluted, while repeated support for the same wrong answer is not.
Critical weaknesses
  • The theory is largely by construction and detached from what is measured.
  • Eq. (4) is the chain rule for mutual information, which holds for any pair of variables. Eq. (5) is its telescoping sum, so it adds nothing topology-specific.
  • The paper notes that the empirical B̃ and M̃ are not estimators of these information terms, and the modelled B and M are a third quantity. The headline “exact decomposition” is never used empirically.
  • Concavity of gain and convexity of loss follow from the chosen functional forms (1−(1−θ)^d, a/d, d(d−1)). The unique interior optimum then depends only on the sign conditions F′(ρ_min) > 0 > F′(1).
  • The model is hard to falsify, and its mechanism is not what explains the data.
  • No parameter (θ, η, λ, μ, ω, S_π) is estimated, and no quantitative prediction is compared with data.
  • The text says that monotone loss, U-shapes and non-concave curves are all “permitted”, and Appendix A.7 allows effectively arbitrary shapes. An inverted U in accuracy is also predicted by other accounts, such as herding or loss of diversity.
  • The paper itself concedes that first-round transfers do not determine the final optimum. For Qwen on AQuA-RAT, positive transfer rises and negative transfer falls through ρ = 1.0, yet accuracy declines beyond 0.8. The accuracy drop at high density is attributed to state dependence and vote correlation, which sit outside the tested model. The extension in A.7 is too general to constrain anything.
  • The theory does have testable implications that go unexamined. By my reading of Eq. (A.32), the optimal mean degree is nearly independent of n. I checked this numerically: for several parameter settings the optimum moves by under 0.2 as n goes from 10 to 200. The paper only uses n = 11.
  • Also untested are the θ sign-flip and the other comparative statics (for example, shifting S_π by seeding aligned errors).
  • The transfer metrics may be dominated by artifacts.
  • B̃ and M̃ are the positive and negative parts of log-score changes using self-reported (verbalised) confidences with a floor of ε = 10⁻⁶. A reported 0% option then costs about 13.8 nats.
  • Llama’s transfers (about 1.2–1.9 nats on AQuA-RAT and Balanced MMLU) are roughly 4–10× Qwen’s. That suggests the metric reflects how often a model reports 0% or non-zero confidence on the correct option.
  • There is no sensitivity analysis over ε or the elicitation method, even though token log-probabilities are available for open-weight models. There is also no noise-floor control, such as self-resampling at ρ = 0 or shuffled neighbour messages, so I cannot tell content-driven transfer from prompt-driven variation.
  • The experimental scope is narrow for a general claim about density.
  • There are two 7–8B models, homogeneous agents, one team size, three rounds and multiple-choice tasks only.
  • The fixed-budget “balanced degrees are best” claim rests on one model and one dataset (Qwen, Balanced MMLU). The accuracy range is 1.9 points. The number of graphs per heterogeneity level and the number of questions are not stated in the main text.
  • The theory is for ER graphs, but the experiments use regular or near-regular graphs, which the authors name as a possible source of mismatch.
  • Section 4.1 says mean individual accuracy is reported. I could not find it. It is needed for the A.5 bridge from individual to team accuracy, and a diversity or agreement measure is needed to support the homogenisation story.
Questions for authors

Can you fit or estimate the model parameters and show they predict ρ* across datasets and models? Does the optimal mean degree stay roughly constant as n varies (for example, n = 5, 11, 21)? Can you report mean individual accuracy and agreement or diversity per density and round, and estimate Δ(ρ) in A.5? How do B̃ and M̃ change with ε, with a token-probability scoring pass, and with a resampling or shuffled-neighbour control?

4 comments
AuthorsOpenReview ↗

Response to Reviewer K6tZ — Part 1/4

1. Predicting communication density through effective model parameters

Thank you for asking how estimated model parameters can guide density prediction. Under the homogeneous fixed-state model, the model-based optimal communication density is governed by three effective parameters, namely the evidence-revelation probability θ\theta, relative coverage strength κ\kappa, and relative dilution strength β\beta.

θ=Pr⁡(a neighbor reveals a given evidence item),0≤θ≤1,\theta = \Pr(\text{a neighbor reveals a given evidence item}),\quad\quad 0 \leq \theta \leq 1,

κ=ACω3η2Sπ,β=τζ=ω2ω3(1ηSπ−1).\kappa = \frac{\mathcal{A}}{\mathcal{C}\omega_{3}\eta^{2}S_{\pi}},\quad\quad\beta = \tau\zeta = \frac{\omega_{2}}{\omega_{3}}\left( \frac{1}{\eta S_{\pi}} - 1 \right).

Here A=λ∑tAt\mathcal{A} = \lambda\sum_{t}^{}A_{t} and C=μ∑tCt\mathcal{C} = \mu\sum_{t}^{}C_{t} are the aggregate scales in Eq. (A.30), and Sπ=∑sπs2S_{\pi} = \sum_{s}^{}\pi_{s}^{2} is error-direction concentration. The ratios above apply when the reinforcement scale Cω3η2Sπ\mathcal{C}\omega_{3}\eta^{2}S_{\pi} is positive.

We estimate effective counterparts of these parameters from first-update answer-level gains and losses on the calibration questions. We estimate θ\theta by grid search, choosing the value that minimizes the joint least-squares fitting error. For each candidate θ\theta, nonnegative least squares estimates the coverage, dilution, and pair-reinforcement coefficients, together with density-independent offsets. We obtain κ^\widehat{\kappa} by dividing the fitted coverage coefficient by the reinforcement coefficient, and β^\widehat{\beta} by dividing the fitted dilution coefficient by the same coefficient.

Substituting the fitted parameters into the graph-level coverage and disturbance expressions in Eqs. (7)–(8) and (A.19), evaluated at the observed node degrees, gives the predicted net-information score. Its highest-ranked density is

ρ^model∈arg,max⁡ρ∈RB^(Gρ)−M^(Gρ),R=0.2,0.4,0.6,0.8,1.0.{\widehat{\rho}}_{model} \in \operatorname*{arg\\,max}_{\rho \in \mathcal{R}}\\{\widehat{B}(G_{\rho}) - \widehat{M}(G_{\rho})\\},\quad\quad\mathcal{R} = \\{ 0.2,0.4,0.6,0.8,1.0\\}.

Ranking uses the unnormalised fitted objective, which also admits boundary solutions. We retain its top two densities and select between them using round-three calibration accuracy. The selected density is then evaluated on held-out test questions. With Qwen2.5-7B-Instruct, we use five fixed splits of 250 questions per task into 100 calibration and 150 disjoint test questions on SciQ, Belebele (English), and ReClor.

Table R1. Density choices for five calibration/test splits on three selected tasks. Candidate densities are ordered by predicted net information.

DatasetSplitModel-shortlisted densitiesSelected ρ\rhoTest-best ρ\rhoHit
SciQ11.0, 0.80.80.8Yes
SciQ21.0, 0.80.80.8Yes
SciQ31.0, 0.80.80.8Yes
SciQ40.6, 0.80.80.8Yes
SciQ50.4, 0.20.20.8No
Belebele (English)11.0, 0.80.80.8Yes
Belebele (English)21.0, 0.80.80.8Yes
Belebele (English)31.0, 0.80.80.8Yes
Belebele (English)40.8, 1.01.00.8No
Belebele (English)51.0, 0.81.00.8No
ReClor10.4, 0.20.20.4No
ReClor20.8, 0.60.80.4No
ReClor30.6, 0.40.40.4Yes
ReClor40.4, 0.60.40.4Yes
ReClor50.6, 0.40.40.4Yes

A hit means that the selected density attains the highest test accuracy among the five evaluated densities. The selected density attains test-best accuracy in 4/5 SciQ splits and 3/5 splits on each of Belebele (English) and ReClor, giving 10/15 hits. Calibration and test questions are sampled from the same task distribution with comparable difficulty and domain composition. The five splits reuse questions from the same task panel and assess sensitivity to the calibration sample.

AuthorsOpenReview ↗

Response to Reviewer K6tZ — Part 2/4

Table R2. Token cost on 100 calibration questions, averaged over five splits. Total tokens include input and output usage, with shared round-0 responses counted once. Values are in millions of tokens.

DatasetModel-guidedFull scanToken saving
SciQ13.80521.28135.1%
Belebele (English)16.18123.40830.9%
ReClor16.51926.83238.4%

Model-guided selection uses one update at all five densities and two further updates at the shortlisted pair. Full scanning completes three updates at all five densities. Savings compare model-guided selection with full scanning on the same calibration questions.

2. Team size and the optimal mean degree

Thank you for this valuable comment. Eq. (A.32) depends on team size through N=n−1N = n - 1 and the mean degree d=Nρd = N\rho. Expressing its interior optimality condition in terms of dd gives

κθ(1−θdn−1)n−2+βd2−(d−12)=0.\kappa\theta\left( 1 - \frac{\theta d}{n - 1} \right)^{n - 2} + \frac{\beta}{d^{2}} - \left( d - \frac{1}{2} \right) = 0.

For fixed effective parameters and bounded dd, the coverage factor approaches e−θde^{- \theta d} as nn increases. The limiting condition consequently depends primarily on neighbor count, explaining the weak dependence on team size identified by the reviewer. The finite-size equation retains its nn dependence and the feasibility constraint d≤n−1d \leq n - 1.

We evaluate Qwen2.5-7B-Instruct on the same Balanced MMLU questions with 5, 11, and 21 agents, using three communication rounds and matched prompts and decoding settings. Table R3 reports the best tested mean degree selected by final team accuracy. It changes from 4 to 4 to 6 as team size increases, while the corresponding density decreases from 1.0 to 0.4 to 0.3. This weak size dependence is consistent with the degree-based interpretation of Eq. (A.32).

Table R3. Best tested mean degree and its corresponding communication density on Balanced MMLU with Qwen.

Team size nBest tested mean degreeCorresponding density ρ\rho
541.0
1140.4
2160.3

3. Mean individual accuracy and the aggregation residual

We are sorry for omitting the individual-level results. Table R4 now reports the final-round mean individual accuracy at each density for both models and all six datasets. The density dependence broadly follows that of final team accuracy, with their changes agreeing in direction in 47 of 48 adjacent-density comparisons.

Table R4. Final-round mean individual accuracy (%). Density 0 denotes the fixed round-0 no-communication baseline.

DatasetModel0.00.20.40.60.81.0
AQuA-RATLlama58.4257.9757.8157.3456.6755.85
AQuA-RATQwen60.0360.9561.9362.9663.1762.15
Balanced MMLULlama54.0454.6855.1353.6254.1853.57
Balanced MMLUQwen56.6258.5158.7358.0957.2157.11
ARC-ChallengeLlama67.1968.7070.7568.8967.7268.50
ARC-ChallengeQwen72.0871.6973.5570.8370.1171.26
HellaSwagLlama57.5758.3460.5561.1360.9058.92
HellaSwagQwen61.7763.5164.4564.8963.9662.22
OpenBookQALlama66.3167.4768.1667.1266.8665.49
OpenBookQAQwen65.5365.7066.5666.8366.3065.01
Professional MedicineLlama62.2063.0463.3763.5862.1960.17
Professional MedicineQwen60.2861.8862.8162.4661.9162.08

Table R5 summarises agreement in the directions of individual- and team-accuracy changes across the four adjacent-density steps from 0.2 to 1.0. The same-direction proportion is 97.92%.

Table R5. Agreement in the directions of adjacent-density changes in final-round individual and team accuracy.

Direction of changeCountPercentage
Same direction4797.92%
Opposite directions12.08%

We estimate Δ(ρ)\Delta(\rho) using the separately measured individual and team outcomes on the evaluation questions. We first average agent correctness to obtain p^(ρ)\widehat{p}(\rho) and evaluate the team’s majority-vote answers to obtain A^team(ρ){\widehat{A}}_{team}(\rho). We then evaluate the independent homogeneous strict-majority baseline in Appendix A.5 and compute

B11(p)=∑k=611(11k)pk(1−p)11−k,Δ^(ρ)=A^team(ρ)−B11(p^(ρ)).B_{11}(p) = \sum_{k = 6}^{11}\binom{11}{k}p^{k}(1 - p)^{11 - k},\quad\quad\widehat{\Delta}(\rho) = {\widehat{A}}_{team}(\rho) - B_{11}(\widehat{p}(\rho)).

AuthorsOpenReview ↗

Response to Reviewer K6tZ — Part 3/4

Table R6 reports the resulting empirical aggregation residual. Its range is -1.28 to +2.22 percentage points, with a median absolute change of 0.06 percentage points between adjacent densities and a maximum of 0.78. These results quantify the aggregation term alongside the measured individual-accuracy trends.

Table R6. Final-round aggregation residual Δ(ρ)\Delta(\rho) in percentage points.

DatasetModel0.00.20.40.60.81.0
AQuA-RATLlama0.260.901.681.822.072.11
AQuA-RATQwen1.781.791.761.751.741.73
Balanced MMLULlama-0.01-0.080.360.09-0.58-1.28
Balanced MMLUQwen1.782.032.132.132.132.09
ARC-ChallengeLlama0.830.790.900.940.891.04
ARC-ChallengeQwen0.500.460.460.520.500.48
HellaSwagLlama1.941.781.792.221.872.08
HellaSwagQwen1.721.671.641.721.691.65
OpenBookQALlama1.141.040.891.131.131.26
OpenBookQAQwen1.191.311.171.171.171.25
Professional MedicineLlama1.641.481.591.571.671.85
Professional MedicineQwen1.991.931.931.861.871.88

4. Answer-level Monte Carlo estimates and message controls

We agree that the original confidence-based scores provided a loose connection to the theoretical BB and MM. We address this with Monte Carlo frequency estimates of answer-label conditional mutual information. The estimates use sampled answer transitions and require neither an ϵ\epsilon floor nor a token-probability scoring pass. Let YY denote the correct answer label, and let A0A^{0} and A1A^{1} denote the selected answer labels before and after the first communication update.

B_{ans} & = I(Y;A^{1} \mid A^{0}) \\\\ & = \sum_{y,a,b}^{}p(y,a,b)\log\frac{p(y,a,b)p(a)}{p(y,a)p(a,b)}, \end{aligned}$$ $$\begin{aligned} M_{ans} & = I(Y;A^{0} \mid A^{1}) \\\\ & = \sum_{y,a,b}^{}p(y,a,b)\log\frac{p(y,a,b)p(b)}{p(y,b)p(a,b)}. \end{aligned}$$ For each evaluated run, we collect triples $(Y_{q},A_{i,q}^{0},A_{i,q}^{1})$ from the sampled task questions and the agents’ generated answers. We approximate the joint probability mass function by their empirical frequencies, giving each question equal weight and valid agents within each question equal weight, $${\widehat{p}}_{r}(y,a,b) = \frac{1}{|\mathcal{Q}_{r}|}\sum_{q \in \mathcal{Q}_{r}}^{}\frac{1}{|V_{q,r}|}\sum_{i \in V_{q,r}}^{}\mathbf{1}\\{ Y_{q} = y,A_{i,q}^{0} = a,A_{i,q}^{1} = b\\}.$$ Here $\mathcal{Q}_{r}$ contains questions with valid paired answers and $V_{q,r}$ their valid agents. Both estimates substitute the same ${\widehat{p}}_{r}$ and its marginals into the expressions above, with natural logarithms and sums over positive-probability cells. We compute them within each run and then average runs equally. This instantiates the theoretical chain-rule decomposition at the observable answer-label level and preserves $${\widehat{B}}_{ans} - {\widehat{M}}_{ans} = \widehat{I}(Y;A^{1}) - \widehat{I}(Y;A^{0}).$$ Table R7. Answer-level corrective gain $B_{ans}$ (nats). Columns give communication density. |Dataset|Model|0.2|0.4|0.6|0.8|1.0| |---|---|---|---|---|---|---| |AQuA-RAT|Llama|0.1025|0.1173|0.1210|0.1322|0.1373| |AQuA-RAT|Qwen|0.0743|0.0890|0.0959|0.1034|0.1163| |Balanced MMLU|Llama|0.0493|0.0501|0.0641|0.0694|0.0709| |Balanced MMLU|Qwen|0.0555|0.0585|0.0679|0.0723|0.0770| |ARC-Challenge|Llama|0.0821|0.0858|0.0862|0.0890|0.0894| |ARC-Challenge|Qwen|0.0209|0.0223|0.0250|0.0270|0.0289| |HellaSwag|Llama|0.0822|0.0809|0.0851|0.0895|0.1186| |HellaSwag|Qwen|0.0315|0.0324|0.0412|0.0419|0.0448| |OpenBookQA|Llama|0.0611|0.0754|0.0795|0.0821|0.0926| |OpenBookQA|Qwen|0.0374|0.0388|0.0390|0.0400|0.0406| |Professional Medicine|Llama|0.0739|0.0781|0.0802|0.0865|0.0907| |Professional Medicine|Qwen|0.0458|0.0491|0.0519|0.0545|0.0548|
AuthorsOpenReview ↗

Response to Reviewer K6tZ — Part 4/4

Table R8. Answer-level misleading loss MansM_{ans} (nats). Columns give communication density.

DatasetModel0.20.40.60.81.0
AQuA-RATLlama0.13800.13620.11900.10580.1061
AQuA-RATQwen0.03660.03460.03440.03540.0373
Balanced MMLULlama0.04780.03410.03850.04580.0544
Balanced MMLUQwen0.01490.01450.01340.01650.0159
ARC-ChallengeLlama0.08170.06910.05480.05410.0552
ARC-ChallengeQwen0.01960.01390.01220.00660.0095
HellaSwagLlama0.05690.05360.05240.05500.0707
HellaSwagQwen0.03900.02740.02530.02770.0278
OpenBookQALlama0.08420.06990.05800.05110.0500
OpenBookQAQwen0.03640.02580.02340.02600.0262
Professional MedicineLlama0.06870.05240.04740.04910.0510
Professional MedicineQwen0.02570.01370.01080.00870.0105

The new estimates range from 0.0209 to 0.1373 nats for gain and 0.0066 to 0.1380 nats for loss. Their numerical magnitudes change with the estimand, while the evidence for density-dependent gain–loss competition remains. Across the 12 model–dataset combinations, gain increases monotonically over the tested grid in 11 combinations, and loss reaches its sampled minimum at an intermediate density in 11 combinations.

Message content and independent self-revision

We compare normal neighbor messages, cross-question shuffled messages, and independent self-revision for the first update of 11 Qwen agents on 100 Professional Medicine questions at densities 0.2, 0.6, and 1.0. Conditions share initial answers. Normal and shuffled groups preserve neighbor counts and have matching mean input lengths within each density. Using the new answer-level estimates, we define net information as J=Bans−MansJ = B_{ans} - M_{ans}. Normal communication exceeds shuffled communication by 0.1407, 0.0603, and 0.0861 nats at the three densities, respectively. All three paired 95% question-bootstrap intervals lie above zero.

Table R9. Paired differences in first-update net answer information. Positive differences favor normal communication.

DensityComparisonDifference (nats)95% interval
0.2Normal minus self-revision0.0258[-0.0022, 0.0517]
0.6Normal minus self-revision0.0431[0.0204, 0.0733]
1.0Normal minus self-revision0.0644[0.0340, 0.1010]
0.2Normal minus shuffled0.1407[0.0852, 0.2082]
0.6Normal minus shuffled0.0603[0.0321, 0.0989]
1.0Normal minus shuffled0.0861[0.0482, 0.1340]

Intervals are pointwise 95% question-bootstrap intervals from 10,000 resamples, conditional on the evaluated graph and message permutation.

Regular graphs and the ER analysis

Thank you for highlighting the distinction between graph families. The ER assumption gives a tractable density-averaged expression. Before averaging over ER graphs, our model defines coverage and disturbance conditional on node degree, and these expressions can be evaluated on a given graph under the stated assumptions. The fixed-edge-budget degree-allocation result likewise holds without requiring ER sampling.

Regular or near-regular graphs provide controlled density comparisons by fixing the edge count within each density condition and limiting degree heterogeneity, which would otherwise vary across ER realizations. This design focuses the available inference budget on comparisons across densities. Controlled comparisons using regular topologies also have precedent in Li et al. (2024, Sec. 3.1).

Accordingly, our experiments test qualitative topology-dependent predictions under controlled degree allocations. The ER ensemble average and the controlled regular-graph experiments are distinguished in scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.