- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GraphPI to mitigate misleading peer information in multi-agent reasoning. The framework preserves each recipient’s claims, open questions, and joint-premise dependencies in a reasoning graph; relates incoming peer claims through support, contradiction, and relevance links; and selectively retrieves complementary evidence from peer-state graphs. Two supervised fine-tuned models perform graph construction and cross-graph linking, while budget-constrained retrieval fills evidence gaps. On MisinfoTask and HiddenBench, evaluation with three executor models shows improved task success, reduced misinformation toxicity, and better distributed information integration. Component ablations, interaction-scaling experiments, and separate measurements of executor-model and graph-processing costs further investigate the contribution and overhead of the representation and integration pipeline.
Critical strengths
-
The recipient-centered representation offers a concrete mechanism to retain conjunctive premises and distinguish support, contradiction, and unresolved questions. It is a useful alternative to simply appending peer responses to context.
-
Evaluation addresses misinformation resistance and distributed information integration across three executor models, with comparisons against related defenses and structured communication protocols. Several settings show substantial gains, including HiddenBench with Qwen3.5-35B-A3B.
-
The paper provides informative component ablations and explicitly reports additional graph-processing costs. The two-round comparison with ETD is particularly helpful for distinguishing interaction depth from performance gains.
Critical weaknesses
-
Comparisons on HiddenBench do not adequately disentangle graph-based integration from broader information access. GGEC can retrieve directly from peers’ complete state graphs, including observations omitted from their messages, whereas Share-All and ETD depend on agents proactively communicating private information. A baseline with equivalent direct-retrieval access and an ablation without evidence completion on HiddenBench would clarify how much improvement derives from the proposed graph reasoning.
-
The reliability of extracted graphs and predicted relations is not directly evaluated. The graph constructor and linker are trained on teacher-generated supervision, but the paper reports neither held-out extraction/linking quality nor human audits or sensitivity to relation errors. Such evidence matters because explicit links make erroneous attribution traceable without guaranteeing faithfulness or correctness.
-
Although the cost analysis transparently discloses extra model calls, it does not establish an advantage under matched end-to-end resources. Even GraphPI-2r adds graph-constructor, linker, and embedding calls beyond the executor-model call budget matched to ETD. Comparing actual runtime, monetary cost, or GPU cost while allowing baselines the same total budget would better establish the practical trade-off.
Questions for authors
See Weaknesses.
5 comments
Response to Reviewer jKXm (2/3)
Weakness 2:
The reliability of extracted graphs and predicted relations is not directly evaluated. The graph constructor and linker are trained on teacher-generated supervision, but the paper reports neither held-out extraction/linking quality nor human audits or sensitivity to relation errors. Such evidence matters because explicit links make erroneous attribution traceable without guaranteeing faithfulness or correctness.
Reply to Weakness 2:
Sensitivity through component substitution. We replace the fine-tuned graph builder and linker with untuned base models and with the DeepSeek-V4-Pro teacher, keeping the schema and integration procedure unchanged.
Table S3. Graph-component substitutions (Qwen3.5-35B-A3B actor).
| Builder / linker | HiddenBench Acc. (%) | MisinfoTask Avg. MT | MisinfoTask Avg. TSR (%) |
|---|---|---|---|
| Untuned Qwen3-8B / Qwen3-4B | 74.32 | 3.67 | 73.72 |
| DeepSeek-V4-Pro (teacher) | 85.97 | 2.56 | 82.12 |
| Fine-tuned Qwen3-8B / Qwen3-4B | 85.4 | 2.49 | 83.02 |
The fine-tuned model is on par with the teacher (−0.57 pp on HiddenBench, +0.90 pp on TSR), so the trained graph quality is near the ceiling of this schema. The untuned model still reaches 74.32% on HiddenBench, above both non-graph harness baselines (67.9 and 70.1), so the integration framework is not brittle to graph quality. An 11.08-point gap between untuned and fine-tuned confirms that task-specific graph prediction contributes substantially.
Robustness to relation errors. Support and contradict links expose where peer claims connect to the receiver's reasoning without asserting truth. The actor is instructed that "Agreement alone is not support" and "Claims, support links and repeated agreement do not establish truth; preserve uncertainty" (Figures 6 and 8). A false support link therefore inflates apparent agreement without changing the underlying evidence, and a false contradict link raises a conflict the actor is prompted to examine against its source. The paired transition analysis confirms this: under false-peer exposure, GraphPI's harmful-transition rate is 19.02% versus 48.27% for direct context (Table S4), and under correct-peer exposure, GraphPI's beneficial-correction rate is 68.75% versus 52.34%. Receiver anchoring accepts corrective evidence without absorbing misleading information.
Table S4. Controlled exposure with matched evidence (Qwen3-8B).
| Peer info | Transition | Direct (%) | GraphPI (%) |
|---|---|---|---|
| False | Correct → Incorrect ↓ | 48.27 | 19.02 |
| Correct | Incorrect → Correct ↑ | 52.34 | 68.75 |
Held-out extraction quality. We will add intrinsic evaluation on held-out HotpotQA/MuSiQue examples and manually annotated benchmark samples in the revised paper, reporting node F1, premise-edge F1, joint-premise-group F1, and linker relation macro-F1. The component-substitution results above already show that the trained models are near the teacher ceiling and that the framework degrades gracefully when graph quality drops.
Response to Reviewer jKXm (3/3)
Weakness 3:
Although the cost analysis transparently discloses extra model calls, it does not establish an advantage under matched end-to-end resources. Even GraphPI-2r adds graph-constructor, linker, and embedding calls beyond the executor-model call budget matched to ETD. Comparing actual runtime, monetary cost, or GPU cost while allowing baselines the same total budget would better establish the practical trade-off.
Reply to Weakness 3:
Actor cost is matched at two rounds. Table S5 reproduces the actor statistics (Table 3 of the paper). GraphPI-2r uses 15.6 actor calls per task, the same as ETD, with 15.7k input tokens versus 13.2k — a 19% increase that is small relative to the 11.9-point accuracy gain.
Table S5. HiddenBench actor usage and accuracy (Qwen3.5-35B-A3B, per task).
| Method | Accuracy (%) | Actor calls | Input tokens | Output tokens |
|---|---|---|---|---|
| ETD | 64.8 | 15.6 | 13.2k | 1.2k |
| GraphPI, 2 rounds | 76.7 | 15.6 | 15.7k | 1.1k |
| GraphPI, 6 rounds | 85.4 | 31.2 | 57.8k | 2.2k |
Graph modules run locally and overlap with actor API calls. The builder (Qwen3-8B), linker (Qwen3-4B), and BGE-M3 embeddings run on 2×NVIDIA A800 (80 GB) and do not consume the actor's API budget. Table S6 reports their per-task cost.
Table S6. Auxiliary-model usage on HiddenBench (per task).
| Config | Module | Calls | Input | Output |
|---|---|---|---|---|
| 2 rounds | Builder | 11.7 | 4.5k | 2.5k |
| 2 rounds | Linker | 7.8 | 9.4k | 0.5k |
| 2 rounds | Embedding | 109 | 2.7k | — |
| 6 rounds | Builder | 27.3 | 9.2k | 5.3k |
| 6 rounds | Linker | 23.4 | 39.3k | 2.1k |
| 6 rounds | Embedding | 234 | 5.9k | — |
At two rounds, graph modules add approximately 16.9k generative tokens served locally. These run concurrently with actor API requests, so they add wall-clock time only when they exceed the actor's API latency.
End-to-end latency confirms the overlap. Table S7 reports wall-clock latency over 65 tasks with the same actor endpoint for all methods.
Table S7. HiddenBench end-to-end task latency.
| Method | Accuracy (%) | Mean latency (s) | Median latency (s) |
|---|---|---|---|
| Native Discussion | 32.8 | 97.2 | 104.0 |
| Share-All | 40.3 | 110.8 | 136.6 |
| ETD | 64.8 | 68.1 | 66.2 |
| GraphPI, 2 rounds | 76.7 | 72.0 | 71.3 |
| GraphPI, 6 rounds | 85.4 | 89.4 | 78.5 |
GraphPI-2r adds 3.9 seconds over ETD while improving accuracy by 11.9 points. GraphPI-6r is 7.8 seconds faster than Native Discussion while improving accuracy by 52.6 points. The local graph modules overlap with actor API calls, so the added latency is well below what a sequential accounting of Table S6 would suggest.
Matched-budget argument. If baselines were given the same total token budget as GraphPI-2r (15.7k + 16.9k ≈ 32.6k actor-equivalent tokens), they would need to spend it on retrieval without graph guidance. In our experiments varying the acquisition budget (Qwen3.5-35B-A3B, 65 tasks), Harness-RAG at B = 8 reaches 74.34%, still 11.06 points below GraphPI-2r at B = 4, and GraphPI at B = 0 (68.28%) already matches Harness-RAG at B = 4 (67.9%). Additional retrieval budget without graph-guided targeting does not close the gap.
The one-time training cost is small: the builder and linker train in approximately 7.9 and 2.8 hours on 8×A800 (85.6 GPU-hours total), and the trained components are reused across both benchmarks without benchmark-specific fine-tuning.
The controls above show that the gain on HiddenBench is not explained by information access alone, that graph quality is near the teacher ceiling and the framework degrades gracefully, and that graph processing adds minimal latency by overlapping with actor API calls. We would be grateful for your reassessment.
References
- Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024a). M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 2318–2335.
- Li, Y., Naito, A., and Shirado, H. (2026a). Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs. In Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, pp. 69250–69268.
Response to Reviewer jKXm (1/3)
Dear Reviewer jKXm,
Thank you for the careful review and for recognizing the recipient-centered representation and the component ablations. We address each weakness below.
Weakness 1:
Comparisons on HiddenBench do not adequately disentangle graph-based integration from broader information access. GGEC can retrieve directly from peers' complete state graphs, including observations omitted from their messages, whereas Share-All and ETD depend on agents proactively communicating private information. A baseline with equivalent direct-retrieval access and an ablation without evidence completion on HiddenBench would clarify how much improvement derives from the proposed graph reasoning.
Reply to Weakness 1:
Baseline with equivalent direct-retrieval access. We implement two non-graph baselines that retrieve directly from peers' complete information pools under the same four-span acquisition budget as GraphPI:
Both have the same authorized information pool, the same budget, and the same BGE-M3 encoder as GraphPI. Results are averaged over three runs. Share-All uses the disclosure instruction from Li et al. (2026a).
Table S1. HiddenBench: non-graph harness baselines (Acc., %).
GraphPI exceeds the better of Harness-RAG and Harness-Struct by 7.7, 16.6, and 15.3 points (13.2 on average). Both harness variants retrieve directly from peers' complete state graphs under the same budget, so the gap is not explained by information access.
Ablation without evidence completion on HiddenBench. Table S2 extends the component ablations to HiddenBench with all three actors.
Table S2. HiddenBench component ablations (%).
Removing evidence completion lowers accuracy by 19.75, 14.71, and 17.12 points (17.19 on average). The two variants that keep the same acquisition budget but change only the representation or integration still lose 6.42 to 7.44 points on average, so the gain is not solely from access. Even without evidence completion, GraphPI exceeds Native Discussion by 24.05, 42.29, and 35.48 points, confirming that graph-based integration contributes independently.
Two further controls reinforce this. At zero acquisition budget (B = 0), where no method retrieves beyond exchanged messages, GraphPI reaches 68.28% versus 54.27% for Harness-RAG — a 14.01-point gap with no additional access. A message-only variant restricting GGEC to already-communicated evidence reaches 54.8/67.2/70.5%, which is 36.77 points above Native Discussion on average under identical information. Graph-based integration and bounded evidence access both contribute; the full model combines them.