acceptodds
Under review as a conference paper at ICLR 2027

Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems

Abstract

Multi-agent communication aims to help agents benefit from one another's information. Yet improvements in system performance leave a fundamental ambiguity: do they reflect effective communication, a favorable agent architecture, or simply additional reasoning? Because communication methods are commonly evaluated within the systems they were designed for, these factors are difficult to disentangle. Final accuracy further merges corrected errors and corrupted answers into a single outcome, obscuring how communication changes decisions. We introduce Independent–Communicate–Revise (ICR), a controlled framework that evaluates communication as answer revision following independent reasoning. ICR fixes initial reasoning trajectories, measures correction and preservation conditional on both agents' initial correctness, and uses a no-message revision control to quantify gains beyond additional reasoning. Across four reasoning benchmarks, our audit of textual and latent communication reveals that similar aggregate accuracy can conceal substantially different revision behaviors. Compared with transmitting answers alone, full reasoning increases correction while reducing preservation on all four benchmarks, so richer messages amplify beneficial and harmful influence alike. Receiver-policy comparisons on MedQA and GPQA-D further show that a structured verification policy shifts every channel toward greater preservation and lower correction, while its effect on selectivity varies across channels and tasks. These findings challenge treating communication quality as an intrinsic property of a channel. ICR therefore recenters evaluation on selective revision, providing a unified framework for examining how message content and receiver policies jointly produce benefits and harms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Communication, Large Language Model Agents, Communication Evaluation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ceSKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether multi agent llms actually help each other or mislead each other with a proposed framework ICR. The authors have each model answer on its own, then see another model’s message and revise. They track two things: (1) how often wrong answers get fixed; (2) how often correct answers get lost in the way. Through evaluations, they find that systems with similar aggregated accuracy can behave very differently: sharing more reasoning fixes more mistakes but also breaks more correct answers. To what extent communication between agents help depends on two things: (a) what is sent; (2) how the receiving model is told to use it.

Critical strengths
  • I like the problem the authors are targeting at, along with the clean way to measure it. The authors does this by tracking separately how often messages fix wrong answers and how often they break correct ones, against a no-message baseline. This helps the field to move evaluation from coarsed-grained aggregated final accuracy eval to a more fine-grained eval.
  • Related to the first point of the clean eval, the paper manages to show that a channel's quality is not fixed. This helps move the conversation forward because it challenges the common practice of ranking communication methods as simply better or worse.
Critical weaknesses
  • I'm quite iffy about the novelty of the paper. I quite a lot of self-correction, self-play, multi-agent debate research, tracking how often answers flip from wrong to right and right to wrong is already common practice, some of these work the authors have already cited. Adn the finding that the richer the messages, the more the models defer in both directions is also fairly expected. So I feel the conceptual advance is quite limited.
  • The evidence is quite limited. Almost all experiments are based on one "outdated" small model Qwen3-4B, and the "different" agents are just the same model taking to each other. This makes is quite hard to support more general claims about multi-agent systems. And relatedly, for the latent reasoning part, if I'm understanding correctly, the authors are using an adaptation of the original methos instead of the original one. This is fine, but if doing this, should at least check if the method itself reproduce the original method's results.
  • Related to the second point, many claims in the paper have somewhat weak support. Many sample sizes are quite small, and I suggest the authors check out the confidence interval, many includes 0.
Questions for authors
  • Can you at least try a few newer and larger models to see if the results hold? I'm in general fine with limited models tested, but in this case, the results mostly rely on one small model, which makes it hard for me to believe the generality of the findings. And further, ideally with mixed sender and receiver models, and show whether the Full Text vs. Answer Only trade-off still appears.
  • For the latent channels, can you run a condition that sends a payload from a different question, or random noise? That would show whether their correction and preservation profiles come from the message itself or just from disturbing the receiver.
  • Can you apply the random-delivery comparison to every channel, not just Answer Only condition? That would show whether any channel corrects more than a content-blind mix of full text and no messaeg at the same preservation rate.
6 comments
AuthorsOpenReview ↗

Response to Reviewer ceSK (Part 1/3)

We thank the reviewer for recognizing the value of separating correction from preservation and evaluating communication channels through their behavioral profiles. Following the suggestions, we added a larger-model evaluation, heterogeneous-agent experiments, wrong-question payload controls, and matched-PR comparisons for all communication channels.


1. Evaluation with a larger model

We evaluated Qwen3-32B-AWQ (4-bit) on 300 MedQA and 198 GPQA-Diamond questions, using three agents, one run, and all five communication conditions under the Critical Evaluation policy. Each CR/PR stratum contains 76 events on MedQA and 102 on GPQA-Diamond.

Table 1. MedQA: Qwen3-32B-AWQ, one run (%).

ConditionCR ↑PR ↑SI ↑
No Message3.9597.3750.66
Answer Only26.3293.4259.87
Full Text85.5353.9569.74
StateBridge51.3281.5866.45
LatentMAS77.6343.4260.53

Table 2. GPQA-Diamond: Qwen3-32B-AWQ, one run (%).

ConditionCR ↑PR ↑SI ↑
No Message12.7596.0854.41
Answer Only38.2476.4757.35
Full Text81.3757.8469.61
StateBridge56.8664.7160.78
LatentMAS67.6558.8263.24

The FT–AO correction–preservation trade-off persists on both datasets. FT increases CR by 59.21 points on MedQA and 43.14 points on GPQA-Diamond, while decreasing PR by 39.47 and 18.63 points, respectively.

The latent channels also exhibit distinct profiles. StateBridge preserves more initially correct answers than FT while correcting fewer errors on both datasets. LatentMAS has lower CR and PR than FT on MedQA, but lower CR and slightly higher PR on GPQA-Diamond. These results extend the audit to a larger model and demonstrate the value of inspecting correction and preservation separately.


2. Mixed sender and receiver models

We conducted three independent runs with Qwen3-4B and Mistral-7B-Instruct-v0.3, evaluating both sender–receiver directions on MedQA and GPQA-Diamond. Tables 3–4 report mean ± sample SD across runs (%). Qwen and Mistral denote these two models.

Table 3. MedQA: heterogeneous communication.

Sender → ReceiverConditionCR ↑PR ↑SI ↑
Qwen → MistralNo Message3.91 ± 2.4693.79 ± 3.8148.85 ± 2.88
Qwen → MistralAnswer Only34.55 ± 4.5471.03 ± 20.6952.79 ± 12.08
Qwen → MistralFull Text87.00 ± 4.5816.53 ± 6.9351.77 ± 5.70
Mistral → QwenNo Message3.98 ± 3.5199.57 ± 0.7451.77 ± 2.12
Mistral → QwenAnswer Only30.86 ± 9.8494.78 ± 0.7762.82 ± 4.60
Mistral → QwenFull Text38.80 ± 10.8691.90 ± 2.3465.35 ± 4.28

Table 4. GPQA-Diamond: heterogeneous communication.

Sender → ReceiverConditionCR ↑PR ↑SI ↑
Qwen → MistralNo Message4.59 ± 1.3788.85 ± 3.9846.72 ± 1.88
Qwen → MistralAnswer Only32.97 ± 6.9753.85 ± 8.0243.41 ± 6.61
Qwen → MistralFull Text80.25 ± 7.6530.42 ± 3.4155.33 ± 4.20
Mistral → QwenNo Message5.40 ± 4.8998.34 ± 1.4351.87 ± 3.00
Mistral → QwenAnswer Only54.62 ± 6.3590.85 ± 4.6572.74 ± 0.94
Mistral → QwenFull Text39.62 ± 8.1994.18 ± 2.8266.90 ± 5.33

For Qwen→Mistral, FT increases CR and decreases PR relative to AO on both datasets, with the same direction in every run. Mistral→Qwen shows a different profile: on GPQA-Diamond, FT has lower mean CR and higher mean PR than AO.

These results show that communication behavior depends on the sender–receiver pairing. The audit reveals both the recurring trade-off and the directional differences that matter when selecting communication formats for heterogeneous systems.

AuthorsOpenReview ↗

Response to Reviewer ceSK (Part 2/3)

3. Wrong-question payload controls

Following the reviewer’s suggestion, we added a wrong-question payload condition for StateBridge and LatentMAS using Qwen3-4B across three runs.

Each receiver retained its original question and prior response. We replaced the normal sender payload with one constructed from a different question in the same dataset and run, using the same sender-agent identity. Donor selection was independent of correctness, and both channels used the same donor mapping. The normal, wrong-question, and NM conditions were evaluated on the same receiver events.

Tables 5–6 report mean ± SD (%). “Same question” denotes the normal communication payload; “Different question” denotes the replacement payload.

Table 5. MedQA: wrong-question payload control.

ChannelPayload sourceCR ↑PR ↑SI ↑
No MessageNone6.51 ± 1.3596.92 ± 1.3051.72 ± 1.13
StateBridgeSame question55.49 ± 1.2266.30 ± 4.8960.89 ± 2.72
StateBridgeDifferent question2.60 ± 1.8099.07 ± 0.1650.83 ± 0.83
LatentMASSame question57.26 ± 3.7653.09 ± 2.9755.18 ± 1.74
LatentMASDifferent question8.29 ± 4.0394.96 ± 2.2851.62 ± 2.45

Table 6. GPQA-Diamond: wrong-question payload control.

ChannelPayload sourceCR ↑PR ↑SI ↑
No MessageNone14.75 ± 1.9793.78 ± 4.2454.26 ± 1.37
StateBridgeSame question65.87 ± 1.1659.37 ± 5.3462.62 ± 2.70
StateBridgeDifferent question10.86 ± 4.4090.79 ± 2.5150.82 ± 2.56
LatentMASSame question50.04 ± 0.8575.56 ± 1.8462.80 ± 1.17
LatentMASDifferent question8.67 ± 3.3594.42 ± 1.6851.55 ± 1.77

Replacing the payload with one from another question sharply reduces correction in both channels on both datasets. Preservation increases at the same time.

These results support the role of question-related message content in shaping the observed profiles. The normal payload contributes both to correcting initially wrong receivers and to changing initially correct receivers into incorrect ones. The control therefore helps distinguish task-related communication effects from simply introducing an additional latent payload.


4. Matched-PR random-delivery comparisons across channels

We extended the random-delivery comparison to Answer Only, StateBridge, and LatentMAS on both datasets across three runs. NM and FT form the two endpoints of the reference mixture.

For each target channel and each run, we choose a probability of delivering FT, with NM otherwise, so that the mixture has the same PR as the target channel. Delivery is independent of the individual question, message content, and correctness. We then compare CR at this matched preservation level. The mixture’s expected metrics are calculated from the existing paired FT and NM outputs.

Tables 7–8 report mean ± SD (%). Each reference row is separately calibrated to the channel immediately above it.

Table 7. MedQA: matched-preservation comparison.

ConditionCR ↑PR ↑SI ↑
Answer Only22.07 ± 1.8186.60 ± 0.7354.33 ± 1.23
FT/NM mixture matched to AO19.75 ± 3.3886.60 ± 0.7353.17 ± 1.36
StateBridge55.49 ± 1.2266.30 ± 4.8960.89 ± 2.72
FT/NM mixture matched to StateBridge45.95 ± 7.6266.30 ± 4.8956.12 ± 1.74
LatentMAS57.26 ± 3.7653.09 ± 2.9755.18 ± 1.74
FT/NM mixture matched to LatentMAS62.78 ± 5.9253.09 ± 2.9757.93 ± 1.70

Table 8. GPQA-Diamond: matched-preservation comparison.

ConditionCR ↑PR ↑SI ↑
Answer Only45.71 ± 2.0178.17 ± 8.7761.94 ± 4.40
FT/NM mixture matched to AO37.31 ± 6.4578.17 ± 8.7757.74 ± 1.23
StateBridge65.87 ± 1.1659.37 ± 5.3462.62 ± 2.70
FT/NM mixture matched to StateBridge65.24 ± 5.0559.37 ± 5.3462.30 ± 3.22
LatentMAS50.04 ± 0.8575.56 ± 1.8462.80 ± 1.17
FT/NM mixture matched to LatentMAS41.63 ± 4.0575.56 ± 1.8458.59 ± 2.77

On MedQA, the mean CR differences relative to the matched mixture are +2.32 points for AO, +9.54 for StateBridge, and −5.51 for LatentMAS. On GPQA-Diamond, they are +8.40, +0.63, and +8.42 points, respectively.

We report these as descriptive matched-preservation comparisons. They show where each channel lies relative to a reference that changes communication frequency without inspecting message content, extending the original AO analysis to both latent channels.

AuthorsOpenReview ↗

Response to Reviewer ceSK (Part 3/3)

5. Independent repetitions and empirical support

We additionally repeated the five-channel Qwen3-4B evaluation with seeds 101, 102, and 103, regenerating initial answers and revisions and reconstructing the correctness-conditioned strata for each run. The full CR/PR/SI tables are provided in our response to Reviewer hG1S.

Across the three runs, the evaluation contains 656 directed mixed-correctness events on MedQA and 620 on GPQA-Diamond, drawn from 94 and 87 unique questions, respectively. The analyses retain the grouping of directions from the same question.

FT increased CR and decreased PR relative to AO in every run on both datasets. The paired differences, reported as mean ± between-run SD in percentage points, were:

  • MedQA: ΔCR = +62.20 ± 1.36; ΔPR = −50.21 ± 4.36; ΔSI = +6.00 ± 1.54.
  • GPQA-Diamond: ΔCR = +33.03 ± 3.69; ΔPR = −28.07 ± 3.91; ΔSI = +2.48 ± 1.48.

The CR increase and PR decrease also persisted after jointly excluding questions with truncated initial or revision outputs across all conditions and runs. These repetitions provide direct evidence about generation variability, alongside the paired question-level analyses.


6. Contribution of the audit protocol

ICR provides a controlled protocol for examining how communication changes a receiver’s answer. It combines frozen initial trajectories, paired communication interventions, conditioning on both agents’ initial correctness, and a no-message revision control. Together, these components enable a common behavioral comparison of textual and latent communication.

The added experiments demonstrate the diagnostic value of this protocol. Heterogeneous-agent experiments reveal different communication profiles across sender–receiver directions. Wrong-question controls show that question-related payload content contributes to both correction and corruption. Matched-PR comparisons evaluate correction at a common preservation level, while independent runs and the larger-model evaluation show the recurrence of the FT–AO correction–preservation trade-off.

These findings help identify which communication choices promote correction, which preserve existing correct answers, and how that balance varies across settings. We will sharpen this contribution in the revised manuscript, expand the discussion of related work, and incorporate the additional experimental results and methodological details.


We hope the above explanations address your questions. Please let us know if any concerns remain; if our responses have resolved them, we would be grateful if you would consider revisiting your assessment. Thank you again for your thoughtful feedback, which will help us improve both the presentation and analysis of the paper.

Reviewer_jrUkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Independent-Communicate-Revise (ICR), a controlled evaluation framework designed to disentangle the impact of inter-agent communication from confounding factors such as agent topology and additional inference in LLM multi-agent systems. The authors observe that existing communication protocols have primarily been assessed through final task accuracy within their respective system designs. Under such evaluation practices, determining whether performance improvements stem from actual information exchange is challenging, while aggregate accuracy merges the correction of wrong answers and the contamination of correct answers into a single scalar value. To address this limitation, the authors formulate four diagnostic metrics (CR, PR, SR, and SI) and conduct empirical analyses using Qwen3-4B and Qwen3-8B across the MedQA, ARC-C, GSM8K, and GPQA-Diamond benchmarks.

Critical strengths
  • The problem addressed in this work is an important topic within the current research field. The authors make a valid point that evaluating multi-agent systems solely on final accuracy fails to reveal whether inter-agent communication is genuinely beneficial. By decomposing the process of reaching a correct answer into four distinct metrics, the authors propose an effective method to resolve this evaluation gap.
  • Overall, the proposed metrics, methodological design, and experimental setup are well-established. Through strict control of confounding variables, the authors empirically demonstrate their intended claims in a clear and convincing manner.
  • The authors present interesting empirical findings that will be potentially valuable to the research community. In particular, the analysis clarifies that the benefits of communication among LLMs do not stem from incorrect agents jointly deriving a new correct answer, but rather from selectively adopting information from an agent that already holds the correct answer. Furthermore, the finding that longer messages containing reasoning traces amplify not only the persuasiveness of correct information but also the harmful influence of incorrect information provides meaningful insight for utilizing agent systems.
  • The authors appear to have achieved the best possible results within their defined scope.
Critical weaknesses
  • All experiments are restricted to a single model family (Qwen3-4B and Qwen3-8B), which limits the generalizability of the empirical findings. Since the authors sample three responses from the exact same model, further discussion is required regarding how well this experimental setup aligns with realistic multi-agent scenarios and whether the conclusions generalize to broader settings.
    • When utilizing three sampled outputs from an identical model as done in the experiments, the sender and receiver share the exact same parameters and knowledge, differing only through sampling noise. Consequently, this configuration resembles self-consistency rather than genuine multi-agent collaboration.
    • ICR evaluates each directional pair (such as A→B) in isolation and does not propagate revised answers to other agent pairs. Furthermore, an agent receives a message from only a single peer at a time, which fails to capture the diverse interaction dynamics that occur during practical multi-agent deployment.
    • The primary benchmarks consist of multiple-choice and numerical tasks, where adopting a peer response reduces to copying a single letter or number. Whether these findings hold for tasks requiring free-form text generation remains questionable under the current experimental scope.
    • Given that the paper draws broad conclusions about how message content and receiver policies shape answer revision across multi-agent systems, a wider experimental setup is necessary to substantiate these claims.
  • The findings presented in the paper largely synthesize observations that have already been explored in numerous prior studies, making it difficult to identify substantial novelty.
    • https://aclanthology.org/2025.naacl-long.412.pdf
    • https://aclanthology.org/2023.findings-emnlp.795.pdf
    • In particular, the main empirical findings appear fairly intuitive. The observation that providing richer information induces more frequent answer revisions (higher CR and lower PR), whereas applying a conservative prompt suppresses revisions, could be reasonably anticipated prior to experimentation.
    • While characterizing these phenomena has some empirical value, it is regrettable that the paper stops short of analyzing their root causes. Examining which specific message properties trigger correction versus corruption (such as sender confidence, reasoning length, or error types) and discussing actionable mitigation strategies would have made the paper much stronger. Merely documenting surface-level phenomena offers limited novelty given the volume of similar studies in the current literature.
  • The main text seems to be not sufficiently self-contained, which makes it difficult to interpret the cross-channel comparisons.
    • Nearly all implementation details regarding the latent channels are deferred to Appendix B. The main text does not explain what each latent channel actually transmits or where the representations are injected, and including these explanations in the main body is strongly recommended.
    • The receiver policies employed in the main audit are also discussed only superficially in the main text.
    • The manuscript omits the rationale for evaluating Answer Only exclusively on mixed-correctness directions in Qwen3-4B, as well as the reason for evaluating all conditions solely on mixed-correctness directions in Qwen3-8B.
    • Information that is essential for understanding the core methodology and results must reside in the main text rather than the appendix. Relegating such details to the supplementary material appears to be merely a workaround to bypass the page limit.
Questions for authors

please see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer jrUk (Part 1/3)

We thank the reviewer for recognizing the value of separating helpful correction from harmful corruption and for the constructive suggestions on experimental coverage, practical implications, and presentation. We have added independent generation runs, heterogeneous-agent experiments, two-round interaction, a larger-model evaluation, and message-content controls.


1. Independent repetitions and heterogeneous agents

ICR evaluates how an individual receiver revises its answer after receiving a peer’s message, with No Message providing a paired self-revision control. Self-consistency instead aggregates independently sampled answers. Using a shared model in the original experiments holds model parameters fixed while comparing communication interventions.

We first examined sampling stability through three independent runs with Qwen3-4B on 300 MedQA and 198 GPQA-Diamond questions. Each run generated fresh initial answers for agents A/B/C and fresh revisions, then reconstructed the correctness-conditioned evaluation pairs. Within each run, all conditions shared the frozen initial answers, evaluation pairs, and event-specific revision seeds.

Tables 1–2 report mean ± sample SD across the three runs, in percent.

Table 1. MedQA: three independent runs.

ConditionCR ↑PR ↑SI ↑
No Message6.51 ± 1.3596.92 ± 1.3051.72 ± 1.13
Answer Only22.07 ± 1.8186.60 ± 0.7354.33 ± 1.23
Full Text84.28 ± 3.0136.38 ± 3.6960.33 ± 1.21
StateBridge55.49 ± 1.2266.30 ± 4.8960.89 ± 2.72
LatentMAS57.26 ± 3.7653.09 ± 2.9755.18 ± 1.74

Table 2. GPQA-Diamond: three independent runs.

ConditionCR ↑PR ↑SI ↑
No Message14.75 ± 1.9793.78 ± 4.2454.26 ± 1.37
Answer Only45.71 ± 2.0178.17 ± 8.7761.94 ± 4.40
Full Text78.73 ± 2.2650.10 ± 4.9064.42 ± 3.49
StateBridge65.87 ± 1.1659.37 ± 5.3462.62 ± 2.70
LatentMAS50.04 ± 0.8575.56 ± 1.8462.80 ± 1.17

FT increased CR and decreased PR relative to AO in every run on both datasets. The correction–preservation trade-off therefore recurred across independently sampled initial answers and revisions.

To directly examine communication between agents with different model parameters and training histories, we additionally evaluated Qwen3-4B and Mistral-7B-Instruct-v0.3 in both sender–receiver directions, also over three independent runs. Tables 3–4 report mean ± SD (%); Qwen and Mistral refer to these two models.

Table 3. MedQA: heterogeneous communication.

Sender → ReceiverConditionCR ↑PR ↑SI ↑
Qwen → MistralNo Message3.91 ± 2.4693.79 ± 3.8148.85 ± 2.88
Qwen → MistralAnswer Only34.55 ± 4.5471.03 ± 20.6952.79 ± 12.08
Qwen → MistralFull Text87.00 ± 4.5816.53 ± 6.9351.77 ± 5.70
Mistral → QwenNo Message3.98 ± 3.5199.57 ± 0.7451.77 ± 2.12
Mistral → QwenAnswer Only30.86 ± 9.8494.78 ± 0.7762.82 ± 4.60
Mistral → QwenFull Text38.80 ± 10.8691.90 ± 2.3465.35 ± 4.28

Table 4. GPQA-Diamond: heterogeneous communication.

Sender → ReceiverConditionCR ↑PR ↑SI ↑
Qwen → MistralNo Message4.59 ± 1.3788.85 ± 3.9846.72 ± 1.88
Qwen → MistralAnswer Only32.97 ± 6.9753.85 ± 8.0243.41 ± 6.61
Qwen → MistralFull Text80.25 ± 7.6530.42 ± 3.4155.33 ± 4.20
Mistral → QwenNo Message5.40 ± 4.8998.34 ± 1.4351.87 ± 3.00
Mistral → QwenAnswer Only54.62 ± 6.3590.85 ± 4.6572.74 ± 0.94
Mistral → QwenFull Text39.62 ± 8.1994.18 ± 2.8266.90 ± 5.33

For Qwen→Mistral, FT increases correction and reduces preservation relative to AO on both datasets. The reverse direction shows a different profile: on GPQA-Diamond, FT produces lower CR and higher PR than AO.

These results highlight the importance of evaluating the intended sender–receiver pairing. Reporting both directions reveals communication differences that would be hidden by pooling the model pair into a single score.

AuthorsOpenReview ↗

Response to Reviewer jrUk (Part 2/3)

2. Tracking revised answers through two rounds within each agent pair

We extended the Qwen3-4B experiments to two synchronous communication rounds over the same three runs. In round two, each agent uses its first-round answer as its new prior and receives its peer’s first-round answer. Revised answers therefore directly influence subsequent revisions.

The original pairs remain fixed across both rounds, including pairs that become both correct or both incorrect after round one. No Message performs two self-revisions. CR and PR remain defined over the original correctness strata, allowing the same trajectories to be tracked over time.

Tables 5–6 report second-round mean ± SD (%). The corresponding first-round results appear in Tables 1–2.

Table 5. MedQA: after two communication rounds.

ConditionCR ↑PR ↑SI ↑
No Message8.93 ± 1.2096.83 ± 1.4052.88 ± 0.82
Answer Only25.25 ± 3.2384.45 ± 0.2954.85 ± 1.76
Full Text42.63 ± 2.8680.64 ± 1.5261.63 ± 0.90
StateBridge53.90 ± 5.3574.13 ± 2.6964.01 ± 3.47
LatentMAS55.87 ± 2.2359.52 ± 3.0457.69 ± 2.44

Table 6. GPQA-Diamond: after two communication rounds.

ConditionCR ↑PR ↑SI ↑
No Message19.39 ± 3.4590.80 ± 6.4755.10 ± 3.06
Answer Only48.41 ± 0.2277.71 ± 9.5763.06 ± 4.69
Full Text55.62 ± 7.5476.78 ± 1.2566.20 ± 4.13
StateBridge52.08 ± 3.6170.95 ± 5.5561.52 ± 4.44
LatentMAS65.29 ± 1.2759.22 ± 3.5962.26 ± 2.42

For FT, SI changes only from 60.33 to 61.63 on MedQA and from 64.42 to 66.20 on GPQA-Diamond, while correction and preservation change substantially in opposite directions.

Tracking individual trajectories explains this behavior. FT retains 49.60% of its first-round corrections on MedQA and 65.81% on GPQA-Diamond. At the same time, it recovers 69.93% and 59.49% of the initially correct answers that became incorrect in round one.

This distinguishes obtaining a correction from sustaining it through further interaction. Evaluating additional communication rounds should therefore track both correction retention and recovery from earlier corruption.


3. Evaluation with a larger model

We also evaluated Qwen3-32B-AWQ (4-bit) on 300 MedQA and 198 GPQA-Diamond questions, using three agents, one run, and all five communication conditions. The resulting mixed-correctness groups contain 76 events per CR/PR stratum on MedQA and 102 on GPQA-Diamond. Tables 7–8 report percentages.

Table 7. MedQA: Qwen3-32B-AWQ.

ConditionCR ↑PR ↑SI ↑
No Message3.9597.3750.66
Answer Only26.3293.4259.87
Full Text85.5353.9569.74
StateBridge51.3281.5866.45
LatentMAS77.6343.4260.53

Table 8. GPQA-Diamond: Qwen3-32B-AWQ.

ConditionCR ↑PR ↑SI ↑
No Message12.7596.0854.41
Answer Only38.2476.4757.35
Full Text81.3757.8469.61
StateBridge56.8664.7160.78
LatentMAS67.6558.8263.24

FT again increases CR and decreases PR relative to AO on both datasets. StateBridge shows higher preservation and lower correction than FT, while LatentMAS exhibits different preservation profiles across the two tasks. These results extend the audit to a larger model and illustrate the distinct correction–preservation profiles of the communication channels.


4. Code-generation results in the existing supplement

The original submission already includes a code-generation evaluation on HumanEval+ problems in Appendix F, Table 14. Receivers generate Python implementations, and correctness is evaluated by executing the extracted programs against the loader-provided tests.

All conditions share the same 168 retained directions, including 28 correction opportunities and 28 preservation cases. Table 9 reproduces the corresponding correction and preservation results.

Table 9. Existing code-generation supplement.

ConditionCorrected / 28Preserved / 28SI (%)
No Message162675.00
Full Text212683.93
StateBridge192782.14
LatentMAS241773.21

FT corrects more programs than NM while preserving the same number of initially correct programs. LatentMAS achieves the most corrections but preserves fewer correct programs, resulting in a lower SI than NM.

Although generation truncation and lengthy code verification limited the scale of our current code-generation evaluation, we will expand the evaluation to additional code-generation and open-ended tasks beyond single-answer formats, with improved generation and verification procedures.

AuthorsOpenReview ↗

Response to Reviewer jrUk (Part 3/3)

5. Relationship to prior work and analysis of message effects

Thank you for pointing us to Wang et al. (2023), “Can ChatGPT Defend its Belief in Truth?”, and Stengel-Eskin et al. (2025), “Teaching Models to Balance Resisting and Accepting Persuasion.” The former examines maintaining correct beliefs under opposing arguments; the latter develops training to balance beneficial and harmful persuasion. We will discuss both explicitly.

ICR contributes a common audit protocol that compares textual and latent communication using frozen sender–receiver trajectories, correctness-conditioned outcomes, and a paired no-message revision control. This enables controlled comparisons of message delivery while keeping the initial trajectories fixed.

To investigate the role of message content, we added a wrong-question payload control for both latent channels. Each receiver retained its original question and prior answer, while the payload came from a different question in the same dataset and run. Donor selection did not use correctness.

Across three runs, replacing the normal payload with a wrong-question payload reduced CR:

  • MedQA: StateBridge, 55.49 → 2.60%; LatentMAS, 57.26 → 8.29%.
  • GPQA-Diamond: StateBridge, 65.87 → 10.86%; LatentMAS, 50.04 → 8.67%.

PR increased in all four comparisons. The normal, question-related payload therefore contributes to both helpful correction and harmful influence on initially correct answers.

Together, the added experiments identify three practical evaluation priorities: checking incoming message relevance and factual support, evaluating the intended sender–receiver pairing, and tracking whether corrections survive subsequent communication. We will organize the discussion around these findings and their implications for communication design.


6. Making the main text self-contained

We will move the essential channel definitions, message sources, injection locations, receiver-policy descriptions, and evaluation-population rationale into the main text.

Table 10 summarizes the communication mechanisms used in the added content-matched experiments. FT and both latent channels use the sender’s frozen post-thinking response as their source. Round one uses the initial response; round two uses the first-round response from the same pair and condition.

Table 10. Message sources and receiver interfaces.

ConditionSourceConstruction and receiver interface
No MessageNo sender messageReceiver revises using the question and its own prior response
Answer OnlySender’s parsed final answerAnswer label supplied in the external-message slot
Full TextSender’s frozen post-thinking response, including its explanation and final answerResponse supplied as text in the external-message slot
StateBridgeThe same source response as Full TextFreshly encode the response; align up to 64 final-layer hidden states to the embedding space and insert them at the external-message slot
LatentMASThe same source response as Full Text, together with its clean source-question contextConstruct a fresh KV cache, perform 10 latent steps, and attach the cache as a causal prefix before the receiver’s revision prompt

We will also explain the receiver policies directly:

  • Critical Evaluation: assess the receiver’s previous reasoning and the peer message against the original problem, revising when a concrete error or better-supported evidence is identified.
  • Structured Verification: first check the peer message against the problem, then independently check the receiver’s previous reasoning, and select the answer supported by the surviving claims.

The original Qwen3-4B AO comparison and Qwen3-8B evaluation focus on mixed-correctness pairs because these directly define CR and PR: one direction measures correction and the reverse measures preservation. We will state this rationale alongside the results.

We have additionally completed a MedQA full-pair experiment covering all six directions per question for every channel (see our response to Reviewer iWuS, Section 1). It provides directly measured accuracy from 1,800 revisions per condition, including both-correct and both-wrong pairs.


We will incorporate the additional results, expand the related-work discussion, and revise the main-text presentation accordingly. We thank the reviewer again for these suggestions, which will help us improve both the presentation and analysis of the paper.

Reviewer_iWuSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ICR, a protocol for evaluating communication between LLM agents as answer revision. Three agents first answer each question independently, these trajectories are frozen, and each ordered sender–receiver pair is then given one revision step under a fixed communication condition. The audit reports the correction rate (CR) when a wrong receiver meets a correct sender and the preservation rate (PR) when a correct receiver meets a wrong sender, with a no-message revision as the control. With Qwen3-4B on MedQA, ARC-C, GSM8K and GPQA-D, sending full reasoning instead of the answer alone raises CR and lowers PR on all four tasks, and a structured verification prompt shifts every channel toward higher PR and lower CR. Two latent channels, adapted from StateBridge and LatentMAS, are audited under the same protocol.

Critical strengths
  1. The protocol is clean. Freezing the initial trajectories, conditioning on both agents' initial correctness and comparing against a no-message revision separates the effect of the message from the effect of an extra reasoning pass.

  2. The statistics are careful and the scope is stated clearly. Intervals use paired question-cluster bootstrap, the paper says what they do not cover (line 1023), and the appendix reports sensitivity checks such as the GPQA-D truncation analysis.

  3. The audit gives textual and latent channels a common behavioral target without decoding the latent messages. And the code is provided.

Critical weaknesses
  1. My main concern is that the headline finding, that aggregate accuracy hides revision behavior, is largely arithmetic. Most questions have three correct agents (Table 5), so full-set accuracy barely moves, and most of it is imputed: on GSM8K only 10% of the directions are observed (Table 15), so the 0.11-point spread across conditions (line 363) mainly reflects the imputation assumption. On the mixed pairs, SI equals accuracy by the paper's own identity (line 238), so the new information is CR and PR, which are correctness-conditioned flip rates.

  2. Most of the selectivity results are unresolved, and generation variance is not measured. All four ΔSI intervals for Full Text against Answer Only include zero (Table 3), all four random-delivery intervals include zero (line 431), and all GPQA-D policy intervals include zero (Table 4). Each stratum has 92 to 116 events from 46 to 58 questions, the intervals are conditional on one sampled run (line 1023), and dropping nine truncated GPQA-D questions moves SI by about 3 to 4.5 points and puts LatentMAS below No Message (Table 7).

  3. The findings that do hold are expected, and some related work is missing. Full reasoning makes the receiver change its answer more often in both directions (valid change rate 29% to 69% on MedQA, Table 12). A more conservative prompt makes it change less. Studies of sycophancy and belief maintenance already measure answer flips conditional on correctness (Sharma et al., 2024; Wang et al., 2023). The paper should say what ICR reveals beyond this trade-off.

  4. The channel comparisons confound information content with representation. Full Text sends the post-thinking response (line 666), LatentMAS rebuilds a KV cache from the full sequence including the thinking segment (line 910), and StateBridge sends 64 vectors from the final portion of the trajectory (line 905). Both latent methods are adaptations rather than their native systems (line 254), so differences between textual and latent channels cannot be attributed to the channel itself.

  5. The setting is far from the multi-agent systems that motivate the paper. The agents are three samples of the same Qwen3-4B on multiple-choice and numeric questions, with a single revision step, no roles, no tools and no multi-round interaction, while the motivation cites planner–critic–judge pipelines (lines 54–58). The Qwen3-8B check uses 46 and 54 questions (line 311), and on HumanEval+ the PR pattern of the main text does not hold (lines 1239–1241).

Questions for authors
  1. How do CR, PR and SI change when the initial and revision generations are repeated with different seeds?
  2. Can the authors compare channels with matched content, for example Full Text that includes the thinking segment, or a LatentMAS cache built from the post-thinking response only?
  3. Which ICR finding is not already visible in sycophancy studies that condition answer flips on correctness? how would it change the design of a multi-agent system?
  4. Do the conclusions hold for heterogeneous agents (different models or roles) or for more than one revision round?
  5. Can the main table report measured retained-subset accuracy next to Acc.†, and how sensitive is Acc.† to the imputation assumption?
5 comments
AuthorsOpenReview ↗

Response to Reviewer iWuS (Part 1/3)

We thank the reviewer for the detailed feedback and for recognizing the controlled protocol and no-message baseline. We have added independent generation runs, content-matched comparisons, fully measured accuracy, heterogeneous agents, two-round interaction, and a larger-model evaluation. Here are the details.


1. Measured accuracy and the role of the decomposition

We agree that SI equals accuracy on balanced mixed-correctness pairs. Its role is to summarize CR and PR, while the diagnostic information comes from separating helpful correction from harmful corruption.

To address the imputation concern, we evaluated all six directed pairs on each of 300 MedQA questions, using Qwen3-4B, three agents, and one run. Each condition has 1,800 actual revision outputs, including both-correct and both-wrong pairs. No outcomes are imputed.

Table 1. MedQA: fully measured accuracy and conditional revision metrics (%).

ConditionMeasured accuracyCR ↑PR ↑SI ↑
No Message70.175.0895.7650.42
Answer Only70.9422.8887.2955.08
Full Text71.2886.4432.2059.32
StateBridge71.5054.2466.9560.59
LatentMAS71.1757.6355.9356.78

Accuracy uses all 1,800 directions; CR and PR each use 118 mixed-correctness directions. All columns come from the same run.

FT exceeds AO by only 0.33 percentage points in measured accuracy, while increasing CR by 63.56 points and decreasing PR by 55.08 points. Thus, similar aggregate accuracy can accompany substantially different revision behavior even when every outcome is directly measured.

As a sensitivity calculation on this new run, carrying initial correctness forward for all non-mixed directions changes each channel’s accuracy by at most 0.17 points relative to direct measurement. This is a new validation experiment, not a reconstruction of the original Acc.† estimates. We will distinguish measured and imputed results explicitly and avoid using the original imputed accuracy spread as primary evidence. We will also report directly measured retained-subset accuracy alongside Acc.† in the revised tables, clearly identifying the evaluation population for each.


2. Independent generation runs and robustness

We repeated the Qwen3-4B experiments with three independent generation seeds, regenerating both initial answers and revisions under the Critical Evaluation policy. Correctness-conditioned strata were reconstructed separately for each run. Within each run, conditions shared the frozen initial answers, evaluation pairs, and event-specific revision seeds.

The numbers of mixed-correctness questions were 59/45/60 on MedQA and 64/55/36 on GPQA-Diamond. Tables 2–3 report mean ± sample SD across runs (%).

Table 2. MedQA: three independent runs.

ConditionCR ↑PR ↑SI ↑
No Message6.51 ± 1.3596.92 ± 1.3051.72 ± 1.13
Answer Only22.07 ± 1.8186.60 ± 0.7354.33 ± 1.23
Full Text84.28 ± 3.0136.38 ± 3.6960.33 ± 1.21
StateBridge (adapted)55.49 ± 1.2266.30 ± 4.8960.89 ± 2.72
LatentMAS (adapted)57.26 ± 3.7653.09 ± 2.9755.18 ± 1.74

Table 3. GPQA-Diamond: three independent runs.

ConditionCR ↑PR ↑SI ↑
No Message14.75 ± 1.9793.78 ± 4.2454.26 ± 1.37
Answer Only45.71 ± 2.0178.17 ± 8.7761.94 ± 4.40
Full Text78.73 ± 2.2650.10 ± 4.9064.42 ± 3.49
StateBridge (adapted)65.87 ± 1.1659.37 ± 5.3462.62 ± 2.70
LatentMAS (adapted)50.04 ± 0.8575.56 ± 1.8462.80 ± 1.17

“Adapted” means using each method’s communication mechanism within ICR’s paired revision protocol, with representations reconstructed from the same post-thinking response used by Full Text.

Tables 2–3 report both the average CR, PR, and SI and their between-run variation for all five channels. Each run independently regenerated the initial answers and revisions and reconstructed the correctness-conditioned strata, so these results capture generation variability beyond resampling a single set of outputs.

The main FT–AO trade-off recurred in every run on both datasets: Full Text corrected more initially incorrect answers, while preserving fewer initially correct answers. Its SI was also higher in each observed run, although this does not establish a general advantage across settings.

The CR increase and PR decrease persisted after jointly excluding questions with any truncated initial or revision output across all conditions and runs. These additional experiments evaluate the Critical Evaluation policy. We also extended the matched-PR analysis to AO, StateBridge, and LatentMAS across three runs. These descriptive comparisons examine correction at the same preservation level; the complete results are provided in our response to Reviewer ceSK, Section 4.

AuthorsOpenReview ↗

Response to Reviewer iWuS (Part 2/3)

3. Contribution beyond answer-flip observations

We agree that sycophancy and belief-maintenance studies, including Sharma et al. (2024) and Wang et al. (2023), already examine correctness-conditioned answer changes. We will expand the related-work discussion and clarify that our contribution is the controlled communication audit, rather than the discovery of answer flips or the SI identity.

ICR combines frozen, independently generated sender–receiver trajectories, conditioning on both agents’ initial correctness, paired communication interventions, and an actual no-message revision baseline. This provides a common behavioral evaluation for textual and latent communication without decoding latent messages.

The additional experiments show how this audit can inform system design:

  • Channel selection depends on the model pairing. (Table 5) On heterogeneous GPQA-Diamond, FT has higher CR than AO for Qwen→Mistral, but lower CR for Mistral→Qwen.
  • Track whether corrections survive further communication. Our trajectory analysis in Section 5 shows that FT retains 49.60% of its first-round corrections on MedQA and 65.81% on GPQA-Diamond, despite small changes in aggregate SI between rounds.
  • Preservation alone does not establish better communication. A conservative channel or policy may preserve more correct answers while foregoing useful corrections; the NM and matched-PR controls make this distinction explicit.

Our objective is therefore not to identify a universally superior channel. It is to reveal when communication helps, when it harms, and whether aggregate scores conceal those differences. These observations guide evaluation choices.


4. Content-matched channel comparisons

Following the reviewer’s suggestion, we used the same frozen sender’s post-thinking response, including its explanation and final answer but excluding the thinking segment, as the source content for FT, StateBridge, and LatentMAS.

FT transmits this response as text. StateBridge constructs hidden-state payloads from a fresh encoding of the same response, while LatentMAS builds a fresh KV cache from it. Neither latent channel reuses the sender’s original thinking trajectory. Tables 2–3 report these content-matched comparisons across three independent runs.


5. Heterogeneous agents, multiple rounds, and a larger model

Heterogeneous agents. We evaluated Qwen3-4B and Mistral-7B-Instruct-v0.3 in both sender–receiver directions over three runs. Results below are mean ± SD (%), using the audited, frozen answer-extraction rules.

Table 4. MedQA: heterogeneous communication.

Sender → ReceiverConditionCR ↑PR ↑SI ↑
Qwen → MistralNo Message3.91 ± 2.4693.79 ± 3.8148.85 ± 2.88
Qwen → MistralAnswer Only34.55 ± 4.5471.03 ± 20.6952.79 ± 12.08
Qwen → MistralFull Text87.00 ± 4.5816.53 ± 6.9351.77 ± 5.70
Mistral → QwenNo Message3.98 ± 3.5199.57 ± 0.7451.77 ± 2.12
Mistral → QwenAnswer Only30.86 ± 9.8494.78 ± 0.7762.82 ± 4.60
Mistral → QwenFull Text38.80 ± 10.8691.90 ± 2.3465.35 ± 4.28

Table 5. GPQA-Diamond: heterogeneous communication.

Sender → ReceiverConditionCR ↑PR ↑SI ↑
Qwen → MistralNo Message4.59 ± 1.3788.85 ± 3.9846.72 ± 1.88
Qwen → MistralAnswer Only32.97 ± 6.9753.85 ± 8.0243.41 ± 6.61
Qwen → MistralFull Text80.25 ± 7.6530.42 ± 3.4155.33 ± 4.20
Mistral → QwenNo Message5.40 ± 4.8998.34 ± 1.4351.87 ± 3.00
Mistral → QwenAnswer Only54.62 ± 6.3590.85 ± 4.6572.74 ± 0.94
Mistral → QwenFull Text39.62 ± 8.1994.18 ± 2.8266.90 ± 5.33

The FT−AO correction–preservation pattern recurs for Qwen→Mistral on both datasets, but is not universal across directions. On GPQA-Diamond, Mistral→Qwen shows the opposite mean direction. Restricting evaluation to pairs with uniquely parseable initial answers preserves these directional observations. This supports auditing the actual sender–receiver configuration rather than assuming a fixed channel ranking.

AuthorsOpenReview ↗

Response to Reviewer iWuS (Part 3/3)

5. Heterogeneous agents, multiple rounds, and a larger model (continued)

Two-round interaction. We continued the same pairs for a second synchronous round. Each agent used its first-round answer as its new prior and its peer’s first-round answer as the message source. All original pairs were retained, regardless of first-round outcomes; NM received a second self-revision.

Tables 6–7 show second-round results; first-round results appear in Tables 2–3. CR and PR remain defined over the original correctness strata, so these are cumulative outcomes.

Table 6. MedQA: after two rounds, mean ± SD (%).

ConditionCR ↑PR ↑SI ↑
No Message8.93 ± 1.2096.83 ± 1.4052.88 ± 0.82
Answer Only25.25 ± 3.2384.45 ± 0.2954.85 ± 1.76
Full Text42.63 ± 2.8680.64 ± 1.5261.63 ± 0.90
StateBridge (adapted)53.90 ± 5.3574.13 ± 2.6964.01 ± 3.47
LatentMAS (adapted)55.87 ± 2.2359.52 ± 3.0457.69 ± 2.44

Table 7. GPQA-Diamond: after two rounds, mean ± SD (%).

ConditionCR ↑PR ↑SI ↑
No Message19.39 ± 3.4590.80 ± 6.4755.10 ± 3.06
Answer Only48.41 ± 0.2277.71 ± 9.5763.06 ± 4.69
Full Text55.62 ± 7.5476.78 ± 1.2566.20 ± 4.13
StateBridge (adapted)52.08 ± 3.6170.95 ± 5.5561.52 ± 4.44
LatentMAS (adapted)65.29 ± 1.2759.22 ± 3.5962.26 ± 2.42

For FT, SI changes by only +1.30 points on MedQA and +1.78 on GPQA-Diamond between rounds, while CR decreases by 41.65/23.12 points and PR increases by 44.25/26.69 points. The trajectory audit identifies both lost first-round corrections and recovery from first-round corruption. Thus, small net changes can conceal substantial changes in revision behavior across rounds. Specifically, FT retains 49.60% of its first-round corrections on MedQA and 65.81% on GPQA-Diamond. It also recovers 69.93% and 59.49%, respectively, of the initially correct answers that became incorrect in round one.

Larger model. We additionally evaluated Qwen3-32B-AWQ (4-bit) with three agents and one run on the same 300 MedQA and 198 GPQA-Diamond questions. Each CR/PR stratum contains 76 events on MedQA and 102 on GPQA-Diamond.

Table 8. MedQA: Qwen3-32B-AWQ, one run (%).

ConditionCR ↑PR ↑SI ↑
No Message3.9597.3750.66
Answer Only26.3293.4259.87
Full Text85.5353.9569.74
StateBridge (adapted)51.3281.5866.45
LatentMAS (adapted)77.6343.4260.53

Table 9. GPQA-Diamond: Qwen3-32B-AWQ, one run (%).

ConditionCR ↑PR ↑SI ↑
No Message12.7596.0854.41
Answer Only38.2476.4757.35
Full Text81.3757.8469.61
StateBridge (adapted)56.8664.7160.78
LatentMAS (adapted)67.6558.8263.24

The correction–preservation trade-off persists: compared with AO, FT increases CR by 59.21/43.14 points and decreases PR by 39.47/18.63 points on MedQA/GPQA-Diamond.

The latent-channel results reveal distinct correction–preservation profiles. StateBridge achieves higher PR and lower CR than FT on both datasets. LatentMAS achieves lower CR and PR than FT on MedQA, but lower CR and slightly higher PR on GPQA-Diamond. These differences highlight the value of auditing correction and preservation separately when comparing communication methods.

Code generation. The existing HumanEval+ supplement provides another informative profile: FT corrects 21/28 programs versus 16/28 for NM, while both preserve 26/28 initially correct programs (Appendix F, Table 14). Here, additional correction occurs without a reduction in preservation, illustrating how the audit identifies task-dependent communication behavior.

Together, the heterogeneous-agent, two-round, and larger-model experiments extend the audit across model pairings, interaction rounds, and model scale. They show how communication behavior varies across these settings and provide additional evidence for evaluating both helpful correction and harmful corruption.

We will incorporate these results and the corresponding methodological details into the revised manuscript, expand the related-work discussion, and clarify the interpretation of the findings.


We hope the above explanations and additional experiments address your concerns. Please let us know if any questions remain; if these additions resolve your main concerns, we would be grateful if you would consider reassessing the paper. We thank you again for your thoughtful suggestions, which will help us improve both the presentation and analysis of the paper.

Reviewer_hG1SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Independent-Communicate-Revise (ICR), a controlled protocol for auditing how one LLM revises a frozen independently generated answer after receiving another agent’s message. It measures correction and preservation conditional on the sender and receiver’s initial correctness, with a no-message revision control. Experiments compare textual and latent communication conditions across four reasoning benchmarks and receiver policies.

Critical strengths

The paper gives a clean, useful decomposition of communication-mediated revision. Section 3 formalizes CR and PR in Equation 5, separately reports both-incorrect recovery and both-correct preservation in Equation 6, and correctly shows in Equation 8 why aggregate accuracy depends on the prevalence of each initial-correctness stratum. This is consequential because it turns a vague concern about peer influence into measurable outcomes that distinguish helpful correction from harmful corruption.

Critical weaknesses

The central empirical patterns are based on a single stochastic realization of independent reasoning and revision. Appendix A.1 uses stochastic decoding with temperature 0.6, and Appendix C.1 explicitly states that bootstrap intervals “do not capture variation across unobserved generation seeds.” Question-cluster bootstrapping is appropriate for dependence among the six directions from a question, but it cannot establish that the headline Full Text versus Answer Only pattern, or the channel-policy profiles in Tables 1 and 4, is stable under a new draw of agent trajectories and sampled revisions. This matters because the strata themselves are selected by the sampled initial answers, and the conclusions concern behavior conditional on these potentially seed-sensitive strata. At minimum, rerunning a representative subset of datasets and conditions over several independent generation seeds, then reporting between-seed variation of CR, PR, and the paired contrasts, would substantially increase confidence in the general empirical conclusions.

Questions for authors

Appendix C.1 notes that the bootstrap intervals condition on the evaluated generation runs and omit seed variation. Have you run any independent initial-generation and revision seeds for a representative subset of the Full Text, Answer Only, and No Message comparisons? Evidence that the CR increase and PR decrease persist across seeds would materially increase my confidence in the headline cross-benchmark conclusion.

1 comment
AuthorsOpenReview ↗

Response to Reviewer hG1S

We sincerely thank the reviewer for the careful and constructive feedback. We agree that bootstrap uncertainty alone does not capture generation variability, and have added independent generation runs to address this concern.


1. Variability across independent generation runs

We repeated the experiments with Qwen3-4B on 300 MedQA and 198 GPQA-Diamond questions using three seeds (101, 102, and 103), under the Critical Evaluation policy. Each run independently generated initial answers for three agents and subsequent revisions, retaining stochastic decoding at temperature 0.6. Within each run, all conditions shared the frozen initial answers, evaluation pairs, and event-specific revision seeds. The following tables report mean ± sample SD across the three runs (%).

Table 1. MedQA.

ConditionCR ↑PR ↑SI ↑
No Message6.51 ± 1.3596.92 ± 1.3051.72 ± 1.13
Answer Only22.07 ± 1.8186.60 ± 0.7354.33 ± 1.23
Full Text84.28 ± 3.0136.38 ± 3.6960.33 ± 1.21
StateBridge (adapted)55.49 ± 1.2266.30 ± 4.8960.89 ± 2.72
LatentMAS (adapted)57.26 ± 3.7653.09 ± 2.9755.18 ± 1.74

Table 2. GPQA-Diamond.

ConditionCR ↑PR ↑SI ↑
No Message14.75 ± 1.9793.78 ± 4.2454.26 ± 1.37
Answer Only45.71 ± 2.0178.17 ± 8.7761.94 ± 4.40
Full Text78.73 ± 2.2650.10 ± 4.9064.42 ± 3.49
StateBridge (adapted)65.87 ± 1.1659.37 ± 5.3462.62 ± 2.70
LatentMAS (adapted)50.04 ± 0.8575.56 ± 1.8462.80 ± 1.17

“Adapted” means using each method’s communication mechanism within ICR’s paired revision protocol, with representations reconstructed from the same post-thinking response used by Full Text.


2. Sensitivity of correctness-conditioned strata to initial sampling

We reconstructed the correctness-conditioned strata separately from each run’s newly generated initial answers. Thus, the repetitions capture changes in which questions enter the evaluation, rather than holding the original selected pairs fixed.

The numbers of mixed-correctness questions were 59/45/60 on MedQA and 64/55/36 on GPQA-Diamond, yielding 656 and 620 directed events across runs, respectively. Within each run, every condition was evaluated on the same events.


3. Stability of the main channel comparison

Full Text increased CR and decreased PR relative to Answer Only in all three runs on both datasets. The paired differences, reported as mean ± between-run SD in percentage points, were:

  • MedQA: ΔCR = +62.20 ± 1.36; ΔPR = −50.21 ± 4.36; ΔSI = +6.00 ± 1.54.
  • GPQA-Diamond: ΔCR = +33.03 ± 3.69; ΔPR = −28.07 ± 3.91; ΔSI = +2.48 ± 1.48.

These results support the recurrence of the correction–preservation trade-off across independently sampled initial answers and revisions.


We thank the reviewer again for these suggestions, which will help us improve both the presentation and analysis of the paper. We will incorporate these results into the revision and clarify their scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.