- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Social Circuits, a framework using message interventions and receiver activation replacement to trace communication effects in LLM-based multi-agent systems. It further introduces Circuit-Guided Deliberation(CGD), a learned message selector based on activation and message features. The paper provides theoretical analysis and evaluates the method on a synthetic task and four benchmarks, demonstrating strong answer-change recovery on 2Wiki and competitive performance with fewer generated tokens.
Critical strengths
S1. The paper investigates an important problem in multi-agent communication. Its intervention-based framework provides a useful approach to analyzing message effects, achieving approximately 97% answer-change recovery on 2Wiki, substantially exceeding other message controls.
S2. CGD demonstrates promising message-selection performance with limited training data. Controlled candidate-message comparisons and cross-benchmark evaluations provide evidence of its effectiveness.
Critical weaknesses
W1. Insufficiently controlled comparisons and efficiency evaluation. In the 2Wiki two-answer evaluation, CGD receives sender passages and responses, while collaboration baselines exchange only generated responses. Single-agent baselines also receive less evidence, making performance gains difficult to attribute to message selection. Moreover, generated-token comparisons exclude answer scoring and activation recording, while CGD requires receiver evaluations over exponentially many message subsets. Its actual efficiency advantage remains unclear.
W2. Limited validation of the proposed mechanism and feature construction. Activation replacement reproduces message effects but does not clearly isolate mechanisms specific to echo-chamber reinforcement. Answer recovery on Correlated-Trap is only 1.33%-16.39%, substantially below 2Wiki, although the protocols differ. Furthermore, CGD's activation-change features are algebraically related to those available to the Activation Products baseline, making the contribution of its specific parameterization unclear. Activation Products also slightly outperforms CGD in the Gemma free-generation setting.
Questions for authors
Q1. Can the authors provide evidence-matched comparisons in which all collaboration baselines receive the same sender passages and responses, including a single-agent baseline with access to all available evidence?
Q2. What are CGD's end-to-end latency and computational costs, including receiver processing, answer scoring, activation extraction, and selection? Does CGD retain its efficiency advantage under matched total compute, accounting for cache reuse and computations performed only during offline evaluation?
Q3. Does CGD consistently outperform Activation Products under matched input features, preprocessing, regularization, and multiple evaluation or training seeds?
Q4. What explains the substantial answer-recovery gap between 2Wiki and Correlated-Trap? Does it persist under matched replacement protocols, and are the terms in Theorem 2's bound empirically small enough to make it informative?
6 comments
Response to Reviewer p3yh Part 2
Weakness 1 (continued)
Table 1 (continued)
TruthfulQA (A/B answers)
| Model | Metric | Single | SC6 | Majority vote | MAD | MAD-M2 | MADC | MOC | SafeSieve | FGD |
|---|---|---|---|---|---|---|---|---|---|---|
| Gemma-3-4B | Accuracy ↑ | 62.67±1.76 | 63.83±1.26 | 68.00±1.32 | 66.83±2.75 | 70.17±0.29 | 66.83±2.75 | 68.67±1.04 | 67.17±1.26 | 70.67±1.04 |
| Tokens ↓ | 67 | 405 | 236 | 583 | 418 | 583 | 449 | 617 | 191 | |
| Qwen3-4B | Accuracy ↑ | 76.17±1.53 | 75.83±0.76 | 79.00±1.80 | 79.83±1.44 | 79.00±1.00 | 79.83±1.44 | 82.00±1.32 | 80.00±1.50 | 83.33±1.76 |
| Tokens ↓ | 105 | 627 | 353 | 828 | 691 | 828 | 677 | 716 | 208 | |
| Qwen3-8B | Accuracy ↑ | 82.67±0.58 | 82.00±0.50 | 83.67±1.04 | 85.00±2.29 | 82.50±1.50 | 85.00±2.29 | 86.33±2.08 | 84.83±0.58 | 87.00±1.50 |
| Tokens ↓ | 96 | 576 | 305 | 746 | 590 | 746 | 608 | 698 | 236 |
Table 2. Free-generation accuracy (%) on 240 2Wiki tasks under matched total generation budgets.
Single-agent: Single and SC6. Multi-agent: Majority vote, MAD, MAD-M2, MADC, MOC, SafeSieve, and FGD.
| Model | Single | SC6 | Majority vote | MAD | MAD-M2 | MADC | MOC | SafeSieve | FGD |
|---|---|---|---|---|---|---|---|---|---|
| Gemma-3-4B | 24.72±1.05 | 25.42±1.67 | 26.39±2.77 | 7.22±1.88 | 20.97±2.10 | 7.22±1.88 | 20.28±0.48 | 25.42±4.23 | 42.92±0.83 |
| Qwen3-4B | 21.94±0.24 | 22.22±0.48 | 25.97±1.97 | 23.33±2.53 | 25.14±3.13 | 23.33±2.53 | 29.31±2.44 | 40.56±2.37 | 51.81±0.64 |
| Qwen3-8B | 23.61±2.10 | 23.89±1.05 | 30.00±1.67 | 14.17±0.83 | 30.83±0.83 | 14.17±0.83 | 28.75±5.42 | 29.86±5.50 | 61.53±0.48 |
Table 10. A/B accuracy (%) on 240 new 2Wiki tasks for the 4B models and the original 240 tasks for Qwen3-8B, reported as mean ± sample standard deviation. FGD achieves the highest mean accuracy across all three models.
| Strategy | Gemma-3-4B | Qwen3-4B | Qwen3-8B |
|---|---|---|---|
| No message | 84.17±0.00 | 78.75±0.00 | 81.67±0.00 |
| Sender 1 only | 86.11±1.27 | 81.67±1.10 | 84.58±2.17 |
| Sender 2 only | 82.50±1.91 | 79.58±1.44 | 77.50±0.83 |
| Both messages | 85.28±1.34 | 83.89±1.34 | 82.78±1.97 |
| FGD | 89.72±1.20 | 91.11±0.64 | 87.36±0.64 |
Table 16 reports time per question, including prompt processing, activation extraction, scoring, selection, and decoding. FGD takes less time than MAD and SafeSieve on both models under their original protocols. FGD generates sender responses and the final answer, while those baselines add one communication round after their initial responses. Answers for unselected combinations are generated after timing.
Table 16. Time per question under the original protocols on A100 40GB. We show the timing columns from the manuscript. Entries show the mean and sample standard deviation across three seeds on 2Wiki.
| Approach | Gemma-3-4B: Time (s) | Qwen3-4B: Time (s) |
|---|---|---|
| FGD | 10.65±2.35 | 19.46±11.40 |
| MAD | 39.25±9.07 | 30.37±7.08 |
| SafeSieve | 27.21±6.23 | 24.69±0.14 |
Weakness 2: W2. Limited validation of the proposed mechanism and feature construction. Activation replacement reproduces message effects but does not clearly isolate mechanisms specific to echo-chamber reinforcement. Answer recovery on Correlated-Trap is only 1.33%-16.39%, substantially below 2Wiki, although the protocols differ. Furthermore, CGD's activation-change features are algebraically related to those available to the Activation Products baseline, making the contribution of its specific parameterization unclear. Activation Products also slightly outperforms CGD in the Gemma free-generation setting.
Author's Response: Thank you for pointing out this connection. FGD uses the same weight, with opposite signs, for the sender–receiver products before and after a message. Activation products learns a separate weight for each. Section 4.1 now explains this difference.
Table 11 shows higher mean A/B accuracy for FGD across all three models after matching candidate messages, training data, input scaling, and weight penalty. The advantage is modest, and the ordering varies across seeds. In free generation, Table 13 shows higher accuracy for FGD on Qwen3-8B and for Activation products on Gemma-3-4B. The Qwen3-8B FGD entry uses a separate run with the same tasks, seeds, and scoring.
Response to Reviewer p3yh Part 3
Weakness 2 (continued)
Table 11. A/B accuracy (%) on the original 240 2Wiki tasks for each model, before and after matching the scaling and penalty to FGD. Entries show the mean and sample standard deviation across three seeds.
| Approach | Gemma-3-4B | Qwen3-4B | Qwen3-8B |
|---|---|---|---|
| Activation products | 88.06±2.29 | 87.92±0.83 | 85.00±1.50 |
| Activation products (adjusted) | 89.31±1.05 | 88.75±1.50 | 86.94±1.27 |
| FGD | 89.72±1.46 | 89.86±1.68 | 87.36±0.64 |
Table 13. Free-generation accuracy (%) on 2Wiki, reported as mean ± sample standard deviation. The Qwen3-8B FGD entry uses a separate run with the same evaluation tasks, seeds, and scoring. Bold marks the highest mean in each column.
| Approach | Gemma-3-4B | Qwen3-8B |
|---|---|---|
| Text-only | 38.33±1.67 | 56.67±1.10 |
| Receiver-only | 40.69±1.58 | 60.42±0.42 |
| Text + receiver | 38.47±1.05 | 59.17±0.72 |
| Additive | 42.78±1.27 | 53.61±1.05 |
| Activation products | 43.33±1.91 | 57.36±0.24 |
| FGD | 42.92±0.83 | 61.53±0.48 |
Table 3 in Section 5.2 shows high answer recovery on both tasks under the common answer prefix and A/B scoring, with higher recovery for the original message than for controls matched in norm. The recorded activations precede either answer token. Both replacement position and scoring change from the original Correlated-Trap protocol.
Table 3. Answer recovery (%) after replacement at the common answer prefix on Gemma-3-4B. Answer recovery uses records where the original and reference messages produce different answers.
| Task | Reference | Original message | Control |
|---|---|---|---|
| 2Wiki | Message removal | 95.92±2.35 | 44.13±17.80 |
| Shuffled message | 97.81±2.13 | 33.53±8.90 | |
| Correlated-Trap | Message removal | 99.85±0.26 | 0.00±0.00 |
| Shuffled message | 99.84±0.28 | 0.00±0.00 |
Table 4 reports a further round of 2Wiki debate after the agents have given the same wrong answer. Among answers changed by shuffling the sender message, restoring the original activations recovers the original answer more often than both controls matched in norm in every seed. Recovered answers include the previously shared error. Section 5.2 presents the results, and Appendix E.3 describes the settings.
Table 4. Answer recovery (%) of further MAD debate on 2Wiki with Gemma-3-4B. We replace activations from the sender message through the final prompt token, using a shuffled message as the reference. Entries show the mean and sample standard deviation across three seeds.
| Activation changes | Answer recovery (%) |
|---|---|
| Original message | 49.70±6.70 |
| Different message | 37.60±5.46 |
| Shuffled activation changes | 15.77±14.52 |
Table 8(b) reports the low recovery under the original Correlated-Trap protocol, which replaces tokens before the answer prefix under a shuffled reference. The original 2Wiki protocol replaces the last common answer-prefix token after message removal. Fig. 13 reports message-effect recovery, including negative recovery for correction on Qwen3-4B.
Table 8. Receiver answers after activation replacement, reported as mean ± sample standard deviation. Agreement uses the same matched message records for both replacements: 1,268 for Gemma-3-4B and 1,334 for Qwen3-4B on 2Wiki, and 1,440 per model on Correlated-Trap. Answer recovery uses only records where the message changes the answer, including both corrections and changes to incorrect answers. Bold marks the higher mean recovery within each model and dataset.
(a) 2Wiki
| Model | Activation changes | Answer agreement (%) | Answer recovery (%) |
|---|---|---|---|
| Gemma-3-4B | Original message | 99.61±0.27 | 97.81±0.23 |
| Different message (same task) | 88.89±1.88 | 41.07±13.20 | |
| Qwen3-4B | Original message | 99.78±0.00 | 96.73±0.22 |
| Different message (same task) | 94.23±1.48 | 55.06±11.67 |
(b) Correlated-Trap
| Model | Activation changes | Answer agreement (%) | Answer recovery (%) |
|---|---|---|---|
| Gemma-3-4B | Original message | 23.68±2.68 | 16.39±1.67 |
| Swapped-role changes | 6.74±1.15 | 0.23±0.00 | |
| Qwen3-4B | Original message | 18.47±1.07 | 1.33±0.92 |
| Swapped-role changes | 14.51±1.05 | 0.27±0.27 | |
| Qwen3-8B | Original message | 40.90±1.89 | 4.88±1.54 |
| Swapped-role changes | 30.49±1.73 | 0.00±0.00 |
Response to Reviewer p3yh Part 1
We thank the reviewer for the careful comments on the comparison conditions, efficiency, and feature construction. We describe the clarifications and additional comparisons in the revised manuscript below.
Weakness 1: W1. Insufficiently controlled comparisons and efficiency evaluation. In the 2Wiki two-answer evaluation, CGD receives sender passages and responses, while collaboration baselines exchange only generated responses. Single-agent baselines also receive less evidence, making performance gains difficult to attribute to message selection. Moreover, generated-token comparisons exclude answer scoring and activation recording, while CGD requires receiver evaluations over exponentially many message subsets. Its actual efficiency advantage remains unclear.
Author's Response: Thank you for highlighting the comparison conditions. With three senders using the same passages, FGD is more accurate than Single with all passages. With two senders using passages, FGD is more accurate than self-consistency under the time budget set by FGD. We have added these comparisons in Appendix E.5 and complete time measurements in Appendix E.6.
Table 1 compares the complete approaches, with FGD using passages and responses and the collaboration baselines using responses. Appendix D.1 specifies shared passage access in their initial free-generation messages. Table 2 compares matched generation budgets, and Table 10 fixes candidate messages and receiver settings.
Table 1. Accuracy (%, mean ± standard deviation over three seeds) and mean tokens per task for generated messages and final answers under the original generation settings. On 2Wiki A/B, FGD sends passages and responses, and collaboration baselines send responses alone.
Single-agent: Single and SC6. Multi-agent: Majority vote, MAD, MAD-M2, MADC, MOC, SafeSieve, and FGD.
2Wiki (A/B answers)
GSM8K (free generation)
MATH-500 (A/B answers)