- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper tests whether multi agent llms actually help each other or mislead each other with a proposed framework ICR. The authors have each model answer on its own, then see another model’s message and revise. They track two things: (1) how often wrong answers get fixed; (2) how often correct answers get lost in the way. Through evaluations, they find that systems with similar aggregated accuracy can behave very differently: sharing more reasoning fixes more mistakes but also breaks more correct answers. To what extent communication between agents help depends on two things: (a) what is sent; (2) how the receiving model is told to use it.
Critical strengths
- I like the problem the authors are targeting at, along with the clean way to measure it. The authors does this by tracking separately how often messages fix wrong answers and how often they break correct ones, against a no-message baseline. This helps the field to move evaluation from coarsed-grained aggregated final accuracy eval to a more fine-grained eval.
- Related to the first point of the clean eval, the paper manages to show that a channel's quality is not fixed. This helps move the conversation forward because it challenges the common practice of ranking communication methods as simply better or worse.
Critical weaknesses
- I'm quite iffy about the novelty of the paper. I quite a lot of self-correction, self-play, multi-agent debate research, tracking how often answers flip from wrong to right and right to wrong is already common practice, some of these work the authors have already cited. Adn the finding that the richer the messages, the more the models defer in both directions is also fairly expected. So I feel the conceptual advance is quite limited.
- The evidence is quite limited. Almost all experiments are based on one "outdated" small model Qwen3-4B, and the "different" agents are just the same model taking to each other. This makes is quite hard to support more general claims about multi-agent systems. And relatedly, for the latent reasoning part, if I'm understanding correctly, the authors are using an adaptation of the original methos instead of the original one. This is fine, but if doing this, should at least check if the method itself reproduce the original method's results.
- Related to the second point, many claims in the paper have somewhat weak support. Many sample sizes are quite small, and I suggest the authors check out the confidence interval, many includes 0.
Questions for authors
- Can you at least try a few newer and larger models to see if the results hold? I'm in general fine with limited models tested, but in this case, the results mostly rely on one small model, which makes it hard for me to believe the generality of the findings. And further, ideally with mixed sender and receiver models, and show whether the Full Text vs. Answer Only trade-off still appears.
- For the latent channels, can you run a condition that sends a payload from a different question, or random noise? That would show whether their correction and preservation profiles come from the message itself or just from disturbing the receiver.
- Can you apply the random-delivery comparison to every channel, not just Answer Only condition? That would show whether any channel corrects more than a content-blind mix of full text and no messaeg at the same preservation rate.
6 comments
Response to Reviewer ceSK (Part 2/3)
3. Wrong-question payload controls
Following the reviewer’s suggestion, we added a wrong-question payload condition for StateBridge and LatentMAS using Qwen3-4B across three runs.
Each receiver retained its original question and prior response. We replaced the normal sender payload with one constructed from a different question in the same dataset and run, using the same sender-agent identity. Donor selection was independent of correctness, and both channels used the same donor mapping. The normal, wrong-question, and NM conditions were evaluated on the same receiver events.
Tables 5–6 report mean ± SD (%). “Same question” denotes the normal communication payload; “Different question” denotes the replacement payload.
Table 5. MedQA: wrong-question payload control.
| Channel | Payload source | CR ↑ | PR ↑ | SI ↑ |
|---|---|---|---|---|
| No Message | None | 6.51 ± 1.35 | 96.92 ± 1.30 | 51.72 ± 1.13 |
| StateBridge | Same question | 55.49 ± 1.22 | 66.30 ± 4.89 | 60.89 ± 2.72 |
| StateBridge | Different question | 2.60 ± 1.80 | 99.07 ± 0.16 | 50.83 ± 0.83 |
| LatentMAS | Same question | 57.26 ± 3.76 | 53.09 ± 2.97 | 55.18 ± 1.74 |
| LatentMAS | Different question | 8.29 ± 4.03 | 94.96 ± 2.28 | 51.62 ± 2.45 |
Table 6. GPQA-Diamond: wrong-question payload control.
| Channel | Payload source | CR ↑ | PR ↑ | SI ↑ |
|---|---|---|---|---|
| No Message | None | 14.75 ± 1.97 | 93.78 ± 4.24 | 54.26 ± 1.37 |
| StateBridge | Same question | 65.87 ± 1.16 | 59.37 ± 5.34 | 62.62 ± 2.70 |
| StateBridge | Different question | 10.86 ± 4.40 | 90.79 ± 2.51 | 50.82 ± 2.56 |
| LatentMAS | Same question | 50.04 ± 0.85 | 75.56 ± 1.84 | 62.80 ± 1.17 |
| LatentMAS | Different question | 8.67 ± 3.35 | 94.42 ± 1.68 | 51.55 ± 1.77 |
Replacing the payload with one from another question sharply reduces correction in both channels on both datasets. Preservation increases at the same time.
These results support the role of question-related message content in shaping the observed profiles. The normal payload contributes both to correcting initially wrong receivers and to changing initially correct receivers into incorrect ones. The control therefore helps distinguish task-related communication effects from simply introducing an additional latent payload.
4. Matched-PR random-delivery comparisons across channels
We extended the random-delivery comparison to Answer Only, StateBridge, and LatentMAS on both datasets across three runs. NM and FT form the two endpoints of the reference mixture.
For each target channel and each run, we choose a probability of delivering FT, with NM otherwise, so that the mixture has the same PR as the target channel. Delivery is independent of the individual question, message content, and correctness. We then compare CR at this matched preservation level. The mixture’s expected metrics are calculated from the existing paired FT and NM outputs.
Tables 7–8 report mean ± SD (%). Each reference row is separately calibrated to the channel immediately above it.
Table 7. MedQA: matched-preservation comparison.
| Condition | CR ↑ | PR ↑ | SI ↑ |
|---|---|---|---|
| Answer Only | 22.07 ± 1.81 | 86.60 ± 0.73 | 54.33 ± 1.23 |
| FT/NM mixture matched to AO | 19.75 ± 3.38 | 86.60 ± 0.73 | 53.17 ± 1.36 |
| StateBridge | 55.49 ± 1.22 | 66.30 ± 4.89 | 60.89 ± 2.72 |
| FT/NM mixture matched to StateBridge | 45.95 ± 7.62 | 66.30 ± 4.89 | 56.12 ± 1.74 |
| LatentMAS | 57.26 ± 3.76 | 53.09 ± 2.97 | 55.18 ± 1.74 |
| FT/NM mixture matched to LatentMAS | 62.78 ± 5.92 | 53.09 ± 2.97 | 57.93 ± 1.70 |
Table 8. GPQA-Diamond: matched-preservation comparison.
| Condition | CR ↑ | PR ↑ | SI ↑ |
|---|---|---|---|
| Answer Only | 45.71 ± 2.01 | 78.17 ± 8.77 | 61.94 ± 4.40 |
| FT/NM mixture matched to AO | 37.31 ± 6.45 | 78.17 ± 8.77 | 57.74 ± 1.23 |
| StateBridge | 65.87 ± 1.16 | 59.37 ± 5.34 | 62.62 ± 2.70 |
| FT/NM mixture matched to StateBridge | 65.24 ± 5.05 | 59.37 ± 5.34 | 62.30 ± 3.22 |
| LatentMAS | 50.04 ± 0.85 | 75.56 ± 1.84 | 62.80 ± 1.17 |
| FT/NM mixture matched to LatentMAS | 41.63 ± 4.05 | 75.56 ± 1.84 | 58.59 ± 2.77 |
On MedQA, the mean CR differences relative to the matched mixture are +2.32 points for AO, +9.54 for StateBridge, and −5.51 for LatentMAS. On GPQA-Diamond, they are +8.40, +0.63, and +8.42 points, respectively.
We report these as descriptive matched-preservation comparisons. They show where each channel lies relative to a reference that changes communication frequency without inspecting message content, extending the original AO analysis to both latent channels.
Response to Reviewer ceSK (Part 3/3)
5. Independent repetitions and empirical support
We additionally repeated the five-channel Qwen3-4B evaluation with seeds 101, 102, and 103, regenerating initial answers and revisions and reconstructing the correctness-conditioned strata for each run. The full CR/PR/SI tables are provided in our response to Reviewer hG1S.
Across the three runs, the evaluation contains 656 directed mixed-correctness events on MedQA and 620 on GPQA-Diamond, drawn from 94 and 87 unique questions, respectively. The analyses retain the grouping of directions from the same question.
FT increased CR and decreased PR relative to AO in every run on both datasets. The paired differences, reported as mean ± between-run SD in percentage points, were:
- MedQA: ΔCR = +62.20 ± 1.36; ΔPR = −50.21 ± 4.36; ΔSI = +6.00 ± 1.54.
- GPQA-Diamond: ΔCR = +33.03 ± 3.69; ΔPR = −28.07 ± 3.91; ΔSI = +2.48 ± 1.48.
The CR increase and PR decrease also persisted after jointly excluding questions with truncated initial or revision outputs across all conditions and runs. These repetitions provide direct evidence about generation variability, alongside the paired question-level analyses.
6. Contribution of the audit protocol
ICR provides a controlled protocol for examining how communication changes a receiver’s answer. It combines frozen initial trajectories, paired communication interventions, conditioning on both agents’ initial correctness, and a no-message revision control. Together, these components enable a common behavioral comparison of textual and latent communication.
The added experiments demonstrate the diagnostic value of this protocol. Heterogeneous-agent experiments reveal different communication profiles across sender–receiver directions. Wrong-question controls show that question-related payload content contributes to both correction and corruption. Matched-PR comparisons evaluate correction at a common preservation level, while independent runs and the larger-model evaluation show the recurrence of the FT–AO correction–preservation trade-off.
These findings help identify which communication choices promote correction, which preserve existing correct answers, and how that balance varies across settings. We will sharpen this contribution in the revised manuscript, expand the discussion of related work, and incorporate the additional experimental results and methodological details.
We hope the above explanations address your questions. Please let us know if any concerns remain; if our responses have resolved them, we would be grateful if you would consider revisiting your assessment. Thank you again for your thoughtful feedback, which will help us improve both the presentation and analysis of the paper.
Response to Reviewer ceSK (Part 1/3)
We thank the reviewer for recognizing the value of separating correction from preservation and evaluating communication channels through their behavioral profiles. Following the suggestions, we added a larger-model evaluation, heterogeneous-agent experiments, wrong-question payload controls, and matched-PR comparisons for all communication channels.
1. Evaluation with a larger model
We evaluated Qwen3-32B-AWQ (4-bit) on 300 MedQA and 198 GPQA-Diamond questions, using three agents, one run, and all five communication conditions under the Critical Evaluation policy. Each CR/PR stratum contains 76 events on MedQA and 102 on GPQA-Diamond.
Table 1. MedQA: Qwen3-32B-AWQ, one run (%).
Table 2. GPQA-Diamond: Qwen3-32B-AWQ, one run (%).
The FT–AO correction–preservation trade-off persists on both datasets. FT increases CR by 59.21 points on MedQA and 43.14 points on GPQA-Diamond, while decreasing PR by 39.47 and 18.63 points, respectively.
The latent channels also exhibit distinct profiles. StateBridge preserves more initially correct answers than FT while correcting fewer errors on both datasets. LatentMAS has lower CR and PR than FT on MedQA, but lower CR and slightly higher PR on GPQA-Diamond. These results extend the audit to a larger model and demonstrate the value of inspecting correction and preservation separately.
2. Mixed sender and receiver models
We conducted three independent runs with Qwen3-4B and Mistral-7B-Instruct-v0.3, evaluating both sender–receiver directions on MedQA and GPQA-Diamond. Tables 3–4 report mean ± sample SD across runs (%). Qwen and Mistral denote these two models.
Table 3. MedQA: heterogeneous communication.
Table 4. GPQA-Diamond: heterogeneous communication.
For Qwen→Mistral, FT increases CR and decreases PR relative to AO on both datasets, with the same direction in every run. Mistral→Qwen shows a different profile: on GPQA-Diamond, FT has lower mean CR and higher mean PR than AO.
These results show that communication behavior depends on the sender–receiver pairing. The audit reveals both the recurring trade-off and the directional differences that matter when selecting communication formats for heterogeneous systems.