- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why more communication does not always improve multi-agent reasoning. It models useful information sharing and the spread of errors, and derives conditions favoring an intermediate communication density and a balanced distribution of connections. Experiments with two models on six datasets provide qualitative support for these predictions.
Critical strengths
-
The paper asks a useful question rather than simply proposing another graph-learning method. Explaining when communication helps or hurts could give clearer guidance on how to organize agents and avoid unnecessary exchanges.
-
The controlled experiments are a meaningful strength. The authors separately test the effects of adding correct neighbors and repeating the same wrong answer. This gives more support for the proposed explanation than accuracy comparisons alone.
Critical weaknesses
-
The additional novelty needs a sharper explanation. The paper acknowledges that sparse communication can outperform full connectivity and that communication can spread both correct and incorrect information. The main addition appears to be a particular mathematical model. It is not yet clear which important insight this model provides beyond those existing explanations.
-
The theory and experiments measure different things. The analysis concerns a simplified information model, while the experiments measure self-reported confidence and voting accuracy. The paper acknowledges this distinction, but matching broad trends does not show that the proposed model accurately explains the observed performance. This limits the strength of the theoretical validation.
Questions for authors
-
Which prediction most clearly distinguishes your model from the earlier information-propagation explanations discussed in the introduction?
-
Can model parameters estimated from some experimental runs predict which densities perform better on separate runs, without fitting the model separately to each density?
3 comments
Response to Reviewer S9Dv — Part 2/3
Ranking uses the unnormalised fitted objective, which also admits boundary solutions. We retain its top two densities and select between them using round-three calibration accuracy. The selected density is then evaluated on held-out test questions. With Qwen2.5-7B-Instruct, we use five fixed splits of 250 questions per task into 100 calibration and 150 disjoint test questions on SciQ, Belebele (English), and ReClor.
Calibration and test questions are sampled from the same task distribution, assuming comparable difficulty and domain composition. Five splits assess sensitivity to the calibration sample. Substantial distribution shifts call for task-specific recalibration.
Table R1. Density choices for five calibration/test splits on three selected tasks. Candidate densities are ordered by predicted net information.
| Dataset | Split | Model-shortlisted densities | Selected | Test-best | Hit |
|---|---|---|---|---|---|
| SciQ | 1 | 1.0, 0.8 | 0.8 | 0.8 | Yes |
| SciQ | 2 | 1.0, 0.8 | 0.8 | 0.8 | Yes |
| SciQ | 3 | 1.0, 0.8 | 0.8 | 0.8 | Yes |
| SciQ | 4 | 0.6, 0.8 | 0.8 | 0.8 | Yes |
| SciQ | 5 | 0.4, 0.2 | 0.2 | 0.8 | No |
| Belebele (English) | 1 | 1.0, 0.8 | 0.8 | 0.8 | Yes |
| Belebele (English) | 2 | 1.0, 0.8 | 0.8 | 0.8 | Yes |
| Belebele (English) | 3 | 1.0, 0.8 | 0.8 | 0.8 | Yes |
| Belebele (English) | 4 | 0.8, 1.0 | 1.0 | 0.8 | No |
| Belebele (English) | 5 | 1.0, 0.8 | 1.0 | 0.8 | No |
| ReClor | 1 | 0.4, 0.2 | 0.2 | 0.4 | No |
| ReClor | 2 | 0.8, 0.6 | 0.8 | 0.4 | No |
| ReClor | 3 | 0.6, 0.4 | 0.4 | 0.4 | Yes |
| ReClor | 4 | 0.4, 0.6 | 0.4 | 0.4 | Yes |
| ReClor | 5 | 0.6, 0.4 | 0.4 | 0.4 | Yes |
A hit means that the selected density attains the highest test accuracy among the five evaluated densities. Different splits reuse questions from the same task panel.
The selected density attained a test-best accuracy in 4/5 SciQ splits and 3/5 splits on each of Belebele (English) and ReClor, giving 10/15 hits (66.7%) in this selected three-task illustration.
Table R2. Token cost on 100 calibration questions, averaged over five splits. Total tokens include input and output usage, with shared round-0 responses counted once. Values are in millions of tokens.
| Dataset | Model-guided | Full scan | Token saving |
|---|---|---|---|
| SciQ | 13.805 | 21.281 | 35.1% |
| Belebele (English) | 16.181 | 23.408 | 30.9% |
| ReClor | 16.519 | 26.832 | 38.4% |
Model-guided selection uses one update at all five densities and two more at the shortlisted pair; full scanning uses three at every density. Savings compare both methods on the same calibration questions.
Linking theoretical and measured information quantities
We acknowledge that the original confidence-based proxies provided a loose connection to the theoretical and . We replace them with Monte Carlo frequency estimates of answer-label conditional mutual information for the first communication round, covering both models, all six datasets, and five communication densities. The estimates use sampled answer transitions and require neither an floor nor a token-probability scoring pass. Let denote the correct answer label, and let and denote the selected answer labels before and after the first communication update.
B_{ans} & = I(Y;A^{1} \mid A^{0}) \\\\ & = \sum_{y,a,b}^{}p(y,a,b)\log\frac{p(y,a,b)p(a)}{p(y,a)p(a,b)}, \end{aligned}$$ $$\begin{aligned} M_{ans} & = I(Y;A^{0} \mid A^{1}) \\\\ & = \sum_{y,a,b}^{}p(y,a,b)\log\frac{p(y,a,b)p(b)}{p(y,b)p(a,b)}. \end{aligned}$$Response to Reviewer S9Dv — Part 3/3
For each evaluated run, we collect triples from the sampled task questions and the agents’ generated answers. We approximate the joint probability mass function by their empirical frequencies, giving each question equal weight and valid agents within each question equal weight,
Here contains questions with valid paired answers and their valid agents. Both estimates substitute the same and its marginals into the expressions above, with natural logarithms and sums over positive-probability cells. We compute them within each run and then average runs equally. This instantiates the theoretical chain-rule decomposition at the observable answer-label level and preserves
The following tables report the resulting answer-level corrective gain and misleading loss, respectively.
Corrective gain (nats). Columns give communication density
| Dataset | Model | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 |
|---|---|---|---|---|---|---|
| AQuA-RAT | Llama | 0.1025 | 0.1173 | 0.1210 | 0.1322 | 0.1373 |
| AQuA-RAT | Qwen | 0.0743 | 0.0890 | 0.0959 | 0.1034 | 0.1163 |
| Balanced MMLU | Llama | 0.0493 | 0.0501 | 0.0641 | 0.0694 | 0.0709 |
| Balanced MMLU | Qwen | 0.0555 | 0.0585 | 0.0679 | 0.0723 | 0.0770 |
| ARC-Challenge | Llama | 0.0821 | 0.0858 | 0.0862 | 0.0890 | 0.0894 |
| ARC-Challenge | Qwen | 0.0209 | 0.0223 | 0.0250 | 0.0270 | 0.0289 |
| HellaSwag | Llama | 0.0822 | 0.0809 | 0.0851 | 0.0895 | 0.1186 |
| HellaSwag | Qwen | 0.0315 | 0.0324 | 0.0412 | 0.0419 | 0.0448 |
| OpenBookQA | Llama | 0.0611 | 0.0754 | 0.0795 | 0.0821 | 0.0926 |
| OpenBookQA | Qwen | 0.0374 | 0.0388 | 0.0390 | 0.0400 | 0.0406 |
| Professional Medicine | Llama | 0.0739 | 0.0781 | 0.0802 | 0.0865 | 0.0907 |
| Professional Medicine | Qwen | 0.0458 | 0.0491 | 0.0519 | 0.0545 | 0.0548 |
Misleading loss (nats). Columns give communication density
| Dataset | Model | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 |
|---|---|---|---|---|---|---|
| AQuA-RAT | Llama | 0.1380 | 0.1362 | 0.1190 | 0.1058 | 0.1061 |
| AQuA-RAT | Qwen | 0.0366 | 0.0346 | 0.0344 | 0.0354 | 0.0373 |
| Balanced MMLU | Llama | 0.0478 | 0.0341 | 0.0385 | 0.0458 | 0.0544 |
| Balanced MMLU | Qwen | 0.0149 | 0.0145 | 0.0134 | 0.0165 | 0.0159 |
| ARC-Challenge | Llama | 0.0817 | 0.0691 | 0.0548 | 0.0541 | 0.0552 |
| ARC-Challenge | Qwen | 0.0196 | 0.0139 | 0.0122 | 0.0066 | 0.0095 |
| HellaSwag | Llama | 0.0569 | 0.0536 | 0.0524 | 0.0550 | 0.0707 |
| HellaSwag | Qwen | 0.0390 | 0.0274 | 0.0253 | 0.0277 | 0.0278 |
| OpenBookQA | Llama | 0.0842 | 0.0699 | 0.0580 | 0.0511 | 0.0500 |
| OpenBookQA | Qwen | 0.0364 | 0.0258 | 0.0234 | 0.0260 | 0.0262 |
| Professional Medicine | Llama | 0.0687 | 0.0524 | 0.0474 | 0.0491 | 0.0510 |
| Professional Medicine | Qwen | 0.0257 | 0.0137 | 0.0108 | 0.0087 | 0.0105 |
The new estimates range from 0.0209 to 0.1373 nats for corrective gain and 0.0066 to 0.1380 nats for misleading loss. Their numerical magnitudes differ from the original confidence-based scores, while the evidence for density-dependent gain–loss competition remains. Across the 12 model–dataset combinations, gain increases monotonically over the tested grid in 11 combinations, and loss reaches its sampled minimum at an intermediate density in 11 combinations. The main accuracy result is also preserved: the highest observed accuracy occurs at an intermediate tested density in all 12 combinations.
Response to Reviewer S9Dv — Part 1/3
Q1. Distinguishing predictions and novelty
Thank you for asking which prediction most clearly distinguishes our model from the earlier information-propagation explanations. The works discussed in our introduction examine beneficial and harmful information propagation, correction and corruption, conformity, and uncertainty in debate. Our contribution connects these mechanisms explicitly to communication density and degree allocation. The clearest additional prediction is that, under homogeneous fixed-state propagation and a fixed edge budget, balancing node degrees maximizes the modeled net-information objective among feasible graphs without isolated nodes; regular or near-regular degree allocations are therefore optimal under these assumptions. The model also yields a parameter-dependent optimality condition for communication density, characterizing how evidence strength, dilution, and error-direction concentration affect the optimum. These results provide explicit, conditional communication-design principles beyond the shared observation that sparse communication can outperform full connectivity, and can guide the selection of topological density, with practical application value.
Q2. Parameter estimation and held-out density prediction
Thank you for asking whether propagation parameters can be estimated or calibrated from observed agent behavior to predict the optimal communication density on held-out data. Under the homogeneous fixed-state model, the model-based optimal communication density is governed by three effective parameters, namely the evidence-revelation probability θ, relative coverage strength κ, and relative dilution strength β.
θ=Pr(a neighbor reveals a given evidence item),0≤θ≤1,
κ=Cω3η2SπA,β=τζ=ω3ω2(ηSπ1−1).
Here A=λ∑tAt and C=μ∑tCt are the aggregate scales in Eq. (A.30), and Sπ=∑sπs2 is error-direction concentration. The ratios above apply when the reinforcement scale Cω3η2Sπ is positive.
We estimate effective counterparts of these parameters from first-update answer-level gains and losses on the calibration questions. We estimate θ by grid search, choosing the value that minimizes the joint least-squares fitting error. For each candidate θ, nonnegative least squares estimates the coverage, dilution, and pair-reinforcement coefficients, together with density-independent offsets. We obtain κ by dividing the fitted coverage coefficient by the reinforcement coefficient, and β by dividing the fitted dilution coefficient by the same coefficient.
Substituting the fitted parameters into the graph-level coverage and disturbance expressions in Eqs. (7)–(8) and (A.19), evaluated at the observed node degrees, gives the predicted net-information score. Its highest-ranked density is
ρmodel∈arg,maxρ∈RB(Gρ)−M(Gρ),R=0.2,0.4,0.6,0.8,1.0.