acceptodds
Under review as a conference paper at ICLR 2027

GraphPI: Graph-based Peer Integration for Multi-Agent Collaboration

Abstract

Multi-agent collaboration has emerged as a promising approach to improving LLM reasoning by enabling agents to exchange information and iteratively refine their solutions. However, peer information is often incorporated directly into a receiver's context, implicitly shaping its subsequent judgments. Such influence is difficult to attribute and control, potentially allowing misleading peer information to shift individual judgments and reinforce erroneous consensus through repeated interaction. To address this issue, we propose Graph-based Peer Integration (GraphPI), which preserves each receiver's reasoning state as a graph, making its dependencies and information gaps (e.g., missing supporting evidence for an agent's reasoning) explicit. It then integrates peer messages by establishing explicit cross-graph relations with the receiver's preserved state, grounding peer influence in the receiver's existing reasoning. The resulting graph then guides selective retrieval of complementary peer information to address remaining gaps before subsequent reasoning. Experiments on multi-agent collaboration benchmarks show that GraphPI consistently improves collaborative reasoning while reducing susceptibility to misleading peer information. Our code is available at https://anonymous.4open.science/r/GrapPI_review/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Systems, Multi-Agent Collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jKXmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GraphPI to mitigate misleading peer information in multi-agent reasoning. The framework preserves each recipient’s claims, open questions, and joint-premise dependencies in a reasoning graph; relates incoming peer claims through support, contradiction, and relevance links; and selectively retrieves complementary evidence from peer-state graphs. Two supervised fine-tuned models perform graph construction and cross-graph linking, while budget-constrained retrieval fills evidence gaps. On MisinfoTask and HiddenBench, evaluation with three executor models shows improved task success, reduced misinformation toxicity, and better distributed information integration. Component ablations, interaction-scaling experiments, and separate measurements of executor-model and graph-processing costs further investigate the contribution and overhead of the representation and integration pipeline.

Critical strengths
  1. The recipient-centered representation offers a concrete mechanism to retain conjunctive premises and distinguish support, contradiction, and unresolved questions. It is a useful alternative to simply appending peer responses to context.

  2. Evaluation addresses misinformation resistance and distributed information integration across three executor models, with comparisons against related defenses and structured communication protocols. Several settings show substantial gains, including HiddenBench with Qwen3.5-35B-A3B.

  3. The paper provides informative component ablations and explicitly reports additional graph-processing costs. The two-round comparison with ETD is particularly helpful for distinguishing interaction depth from performance gains.

Critical weaknesses
  1. Comparisons on HiddenBench do not adequately disentangle graph-based integration from broader information access. GGEC can retrieve directly from peers’ complete state graphs, including observations omitted from their messages, whereas Share-All and ETD depend on agents proactively communicating private information. A baseline with equivalent direct-retrieval access and an ablation without evidence completion on HiddenBench would clarify how much improvement derives from the proposed graph reasoning.

  2. The reliability of extracted graphs and predicted relations is not directly evaluated. The graph constructor and linker are trained on teacher-generated supervision, but the paper reports neither held-out extraction/linking quality nor human audits or sensitivity to relation errors. Such evidence matters because explicit links make erroneous attribution traceable without guaranteeing faithfulness or correctness.

  3. Although the cost analysis transparently discloses extra model calls, it does not establish an advantage under matched end-to-end resources. Even GraphPI-2r adds graph-constructor, linker, and embedding calls beyond the executor-model call budget matched to ETD. Comparing actual runtime, monetary cost, or GPU cost while allowing baselines the same total budget would better establish the practical trade-off.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer jKXm (1/3)

Dear Reviewer jKXm,

Thank you for the careful review and for recognizing the recipient-centered representation and the component ablations. We address each weakness below.

Weakness 1:

Comparisons on HiddenBench do not adequately disentangle graph-based integration from broader information access. GGEC can retrieve directly from peers' complete state graphs, including observations omitted from their messages, whereas Share-All and ETD depend on agents proactively communicating private information. A baseline with equivalent direct-retrieval access and an ablation without evidence completion on HiddenBench would clarify how much improvement derives from the proposed graph reasoning.

Reply to Weakness 1:

Baseline with equivalent direct-retrieval access. We implement two non-graph baselines that retrieve directly from peers' complete information pools under the same four-span acquisition budget as GraphPI:

  • Harness-RAG: each agent maintains a persistent compressed summary and generates unresolved questions; BGE-M3 (Chen et al., 2024a) retrieves from authorized peer pools with the same unique-source-span accounting as GraphPI; retrieved spans are appended as plain text.
  • Harness-Struct: same as Harness-RAG plus a structured message schema (claim / evidence / uncertainty / open question).

Both have the same authorized information pool, the same budget, and the same BGE-M3 encoder as GraphPI. Results are averaged over three runs. Share-All uses the disclosure instruction from Li et al. (2026a).

Table S1. HiddenBench: non-graph harness baselines (Acc., %).

ActorETDHarness-RAGHarness-StructShare-AllGraphPI
GPT-4o-mini67.662.464.247.471.9
DeepSeek-V369.261.759.258.578.3
Qwen3.5-35B-A3B64.867.970.140.385.4

GraphPI exceeds the better of Harness-RAG and Harness-Struct by 7.7, 16.6, and 15.3 points (13.2 on average). Both harness variants retrieve directly from peers' complete state graphs under the same budget, so the gap is not explained by information access.

Ablation without evidence completion on HiddenBench. Table S2 extends the component ablations to HiddenBench with all three actors.

Table S2. HiddenBench component ablations (%).

Actorw/o Graph Inputw/o Receiver Anchoringw/o Evidence CompletionGraphPI
GPT-4o-mini67.4365.2852.1571.9
DeepSeek-V369.6170.2463.5978.3
Qwen3.5-35B-A3B76.2480.8368.2885.4

Removing evidence completion lowers accuracy by 19.75, 14.71, and 17.12 points (17.19 on average). The two variants that keep the same acquisition budget but change only the representation or integration still lose 6.42 to 7.44 points on average, so the gain is not solely from access. Even without evidence completion, GraphPI exceeds Native Discussion by 24.05, 42.29, and 35.48 points, confirming that graph-based integration contributes independently.

Two further controls reinforce this. At zero acquisition budget (B = 0), where no method retrieves beyond exchanged messages, GraphPI reaches 68.28% versus 54.27% for Harness-RAG — a 14.01-point gap with no additional access. A message-only variant restricting GGEC to already-communicated evidence reaches 54.8/67.2/70.5%, which is 36.77 points above Native Discussion on average under identical information. Graph-based integration and bounded evidence access both contribute; the full model combines them.

AuthorsOpenReview ↗

Response to Reviewer jKXm (2/3)

Weakness 2:

The reliability of extracted graphs and predicted relations is not directly evaluated. The graph constructor and linker are trained on teacher-generated supervision, but the paper reports neither held-out extraction/linking quality nor human audits or sensitivity to relation errors. Such evidence matters because explicit links make erroneous attribution traceable without guaranteeing faithfulness or correctness.

Reply to Weakness 2:

Sensitivity through component substitution. We replace the fine-tuned graph builder and linker with untuned base models and with the DeepSeek-V4-Pro teacher, keeping the schema and integration procedure unchanged.

Table S3. Graph-component substitutions (Qwen3.5-35B-A3B actor).

Builder / linkerHiddenBench Acc. (%)MisinfoTask Avg. MTMisinfoTask Avg. TSR (%)
Untuned Qwen3-8B / Qwen3-4B74.323.6773.72
DeepSeek-V4-Pro (teacher)85.972.5682.12
Fine-tuned Qwen3-8B / Qwen3-4B85.42.4983.02

The fine-tuned model is on par with the teacher (−0.57 pp on HiddenBench, +0.90 pp on TSR), so the trained graph quality is near the ceiling of this schema. The untuned model still reaches 74.32% on HiddenBench, above both non-graph harness baselines (67.9 and 70.1), so the integration framework is not brittle to graph quality. An 11.08-point gap between untuned and fine-tuned confirms that task-specific graph prediction contributes substantially.

Robustness to relation errors. Support and contradict links expose where peer claims connect to the receiver's reasoning without asserting truth. The actor is instructed that "Agreement alone is not support" and "Claims, support links and repeated agreement do not establish truth; preserve uncertainty" (Figures 6 and 8). A false support link therefore inflates apparent agreement without changing the underlying evidence, and a false contradict link raises a conflict the actor is prompted to examine against its source. The paired transition analysis confirms this: under false-peer exposure, GraphPI's harmful-transition rate is 19.02% versus 48.27% for direct context (Table S4), and under correct-peer exposure, GraphPI's beneficial-correction rate is 68.75% versus 52.34%. Receiver anchoring accepts corrective evidence without absorbing misleading information.

Table S4. Controlled exposure with matched evidence (Qwen3-8B).

Peer infoTransitionDirect (%)GraphPI (%)
FalseCorrect → Incorrect ↓48.2719.02
CorrectIncorrect → Correct ↑52.3468.75

Held-out extraction quality. We will add intrinsic evaluation on held-out HotpotQA/MuSiQue examples and manually annotated benchmark samples in the revised paper, reporting node F1, premise-edge F1, joint-premise-group F1, and linker relation macro-F1. The component-substitution results above already show that the trained models are near the teacher ceiling and that the framework degrades gracefully when graph quality drops.

AuthorsOpenReview ↗

Response to Reviewer jKXm (3/3)

Weakness 3:

Although the cost analysis transparently discloses extra model calls, it does not establish an advantage under matched end-to-end resources. Even GraphPI-2r adds graph-constructor, linker, and embedding calls beyond the executor-model call budget matched to ETD. Comparing actual runtime, monetary cost, or GPU cost while allowing baselines the same total budget would better establish the practical trade-off.

Reply to Weakness 3:

Actor cost is matched at two rounds. Table S5 reproduces the actor statistics (Table 3 of the paper). GraphPI-2r uses 15.6 actor calls per task, the same as ETD, with 15.7k input tokens versus 13.2k — a 19% increase that is small relative to the 11.9-point accuracy gain.

Table S5. HiddenBench actor usage and accuracy (Qwen3.5-35B-A3B, per task).

MethodAccuracy (%)Actor callsInput tokensOutput tokens
ETD64.815.613.2k1.2k
GraphPI, 2 rounds76.715.615.7k1.1k
GraphPI, 6 rounds85.431.257.8k2.2k

Graph modules run locally and overlap with actor API calls. The builder (Qwen3-8B), linker (Qwen3-4B), and BGE-M3 embeddings run on 2×NVIDIA A800 (80 GB) and do not consume the actor's API budget. Table S6 reports their per-task cost.

Table S6. Auxiliary-model usage on HiddenBench (per task).

ConfigModuleCallsInputOutput
2 roundsBuilder11.74.5k2.5k
2 roundsLinker7.89.4k0.5k
2 roundsEmbedding1092.7k—
6 roundsBuilder27.39.2k5.3k
6 roundsLinker23.439.3k2.1k
6 roundsEmbedding2345.9k—

At two rounds, graph modules add approximately 16.9k generative tokens served locally. These run concurrently with actor API requests, so they add wall-clock time only when they exceed the actor's API latency.

End-to-end latency confirms the overlap. Table S7 reports wall-clock latency over 65 tasks with the same actor endpoint for all methods.

Table S7. HiddenBench end-to-end task latency.

MethodAccuracy (%)Mean latency (s)Median latency (s)
Native Discussion32.897.2104.0
Share-All40.3110.8136.6
ETD64.868.166.2
GraphPI, 2 rounds76.772.071.3
GraphPI, 6 rounds85.489.478.5

GraphPI-2r adds 3.9 seconds over ETD while improving accuracy by 11.9 points. GraphPI-6r is 7.8 seconds faster than Native Discussion while improving accuracy by 52.6 points. The local graph modules overlap with actor API calls, so the added latency is well below what a sequential accounting of Table S6 would suggest.

Matched-budget argument. If baselines were given the same total token budget as GraphPI-2r (15.7k + 16.9k ≈ 32.6k actor-equivalent tokens), they would need to spend it on retrieval without graph guidance. In our experiments varying the acquisition budget (Qwen3.5-35B-A3B, 65 tasks), Harness-RAG at B = 8 reaches 74.34%, still 11.06 points below GraphPI-2r at B = 4, and GraphPI at B = 0 (68.28%) already matches Harness-RAG at B = 4 (67.9%). Additional retrieval budget without graph-guided targeting does not close the gap.

The one-time training cost is small: the builder and linker train in approximately 7.9 and 2.8 hours on 8×A800 (85.6 GPU-hours total), and the trained components are reused across both benchmarks without benchmark-specific fine-tuning.

The controls above show that the gain on HiddenBench is not explained by information access alone, that graph quality is near the teacher ceiling and the framework degrades gracefully, and that graph processing adds minimal latency by overlapping with actor API calls. We would be grateful for your reassessment.

References

  • Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024a). M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 2318–2335.
  • Li, Y., Naito, A., and Shirado, H. (2026a). Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs. In Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, pp. 69250–69268.
Reviewer_2T4aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the problem of implicit peer influence in LLM-based multi-agent collaboration, where misleading information from other agents may distort individual judgments and lead to erroneous consensus. Motivated by empirical observations of this phenomenon, the authors propose GraphPI, a graph-based peer integration framework consisting of three components: Receiver State Graph Construction (RSGC), Receiver-Anchored Peer Integration (RAPI), and Graph-Guided Evidence Completion (GGEC). The framework uses fine-tuned LLMs to represent agents' reasoning states as graphs, explicitly model relationships between peer information and the receiver's existing reasoning, and selectively retrieve complementary evidence from other agents. Experiments on MisinfoTask and HiddenBench demonstrate improvements in task performance and robustness to misinformation across different LLM backbones.

Critical strengths
  1. The paper focus on an important problem in multi-agent collaboration. The empirical analysis of judgment shifts and erroneous consensus provides useful motivation for investigating how peer information should be integrated.

  2. The main argument of the article was verified in terms of experimental design and execution.

Critical weaknesses
  1. The necessity and novelty of the proposed graph-based framework are not sufficiently established. Many of the functionalities provided by GraphPI, such as preserving agents' reasoning states, identifying information gaps, sharing relevant evidence, and integrating peer information, could potentially be achieved through existing multi-agent harness mechanisms, including context compression, structured information sharing, persistent memory, and retrieval-based communication. It is therefore unclear what benefits are specifically attributable to the graph representation and the fine-tuned graph construction/linking models. Although the paper includes several ablation studies, a comparison against a strong, non-graph-based multi-agent framework with similar information management capabilities is missing. Such a comparison would help clarify the essential methodological contribution of GraphPI.

  2. The proposed graph representation introduces a relatively strong inductive bias, which raises concerns about generalizability. GraphPI represents agent states primarily through claims, open questions, and premise dependencies, while cross-agent relationships are modeled as support, contradiction, or relevance. This representation appears particularly suitable for evidence-based reasoning and decision-making tasks. However, multi-agent collaboration also includes task-oriented settings where agents primarily execute actions, invoke tools, manipulate artifacts, or coordinate workflows rather than exchange judgments. It is unclear how well the proposed representation captures these types of agent states and interactions. Since both evaluation benchmarks primarily involve reasoning and decision-making, the current experiments provide limited evidence for the broader applicability of GraphPI to general multi-agent collaboration.

  3. The source of the performance improvements is not fully disentangled. In addition to introducing graph-based representations, GraphPI allows agents to retrieve complementary information from other agents' full state graphs, including evidence that was not explicitly communicated in their messages. This additional information access could itself substantially improve performance, especially on HiddenBench, where successful decisions depend on integrating distributed private information. Consequently, the observed gains may partly result from more effective information retrieval rather than the proposed graph-based integration mechanism. A comparison with non-graph baselines that have access to the same information and use comparable retrieval budgets would help establish the specific contribution of graph-based peer integration.

  4. The connection between explicit graph relations and reduced misinformation influence could be better justified. Although GraphPI makes relationships between claims explicit, identifying a support or contradiction relation does not necessarily establish the reliability of the underlying information. Misleading claims can also be incorporated into the receiver's graph and potentially influence subsequent reasoning. Therefore, it remains unclear to what extent GraphPI actually controls peer influence rather than simply making information integration more structured. Additional analysis distinguishing beneficial corrections from harmful judgment shifts would strengthen the claimed mechanism.

A few minor comments:

  • A detailed case study illustrating the complete process of graph construction, cross-graph relation prediction, evidence retrieval, and final judgment revision would help readers better understand the practical advantages of the proposed framework.
  • The computational cost analysis could be more comprehensive. Although the paper reports additional calls and token usage for graph-related modules, end-to-end latency and total computational overhead would provide a clearer picture of the practical efficiency of GraphPI.
Questions for authors
  1. Could the authors provide comparisons with stronger non-graph-based multi-agent frameworks, such as approaches using structured context compression, persistent agent memory, and retrieval-based information sharing? In particular, what advantages does GraphPI offer beyond functionalities that can be implemented through a conventional multi-agent harness?

  2. How does the proposed framework generalize to task-oriented multi-agent systems in which agents primarily perform actions, use tools, or coordinate workflows rather than exchange claims and judgments? Could the authors discuss the limitations introduced by the predefined graph ontology or provide additional evaluations on such tasks?

  3. Could the authors better isolate the contribution of graph-based integration from that of additional information access? For example, how would a non-graph baseline perform if it could retrieve the same complementary evidence from other agents?

  4. How does GraphPI distinguish between beneficial and harmful peer influence? In particular, when an agent's initial judgment is incorrect, can the proposed receiver-anchored integration mechanism effectively incorporate corrective evidence from other agents?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 2T4a (1/10)

Dear Reviewer 2T4a,

Thank you for the careful review and for recognizing the problem's importance. Your questions give us a useful chance to clarify the role of the graph representation and the evidence in GraphPI. Since Questions 1–4 correspond closely to Weaknesses 1–4, we address each pair together below.

Weakness 1:

Contribution beyond conventional multi-agent harness mechanisms. The paper combines several techniques, including persistent memory, context compression, and retrieval-augmented generation, into a framework called GraphPI. The core contribution over existing multi-agent harness approaches is not entirely clear.

Reply to Weakness 1 & Question 1:

GraphPI's contribution is a specific principle for peer integration: incoming information is interpreted relative to the receiver's preserved reasoning state. Persistent memory, context compression, and retrieval are useful infrastructure for retaining and accessing information. GraphPI specifies how the retained reasoning, incoming claims, and newly acquired evidence are connected and presented for subsequent judgment.

The distinction is therefore operational. RAPI links incoming claims to the receiver's existing reasoning through explicit support, contradiction, and relevance relations. GGEC traces premise and support dependencies backward from the receiver's latest message to identify open questions and conflicting claims; when these are absent, it uses the fallback evidence targets described in Appendix B.5. Retrieved claims are expanded with their source context and premise dependencies, including complete joint premise groups, before budgeted selection and integration. These operations connect information acquisition to the receiver's current reasoning needs.

GraphPI is compatible with a harness that provides persistent storage, compression, and communication. The methodological question is whether its explicit relational organization improves collaboration beyond strong implementations of those capabilities. Our existing ablations provide evidence for the contribution of the individual design choices:

Table R1. Ablations on MisinfoTask with Qwen3.5-35B-A3B (from Table 2).

MethodAvg. MT ↓Avg. TSR ↑TSR decrease relative to full GraphPI (pp)
w/o Receiver Anchoring3.4171.6011.42
w/o Subgraph Expansion2.8675.317.71
w/o Evidence Completion2.7074.388.64
w/o Joint Premise3.5169.7513.27
w/o Graph Input2.5373.159.87
GraphPI2.4983.02—

The reductions associated with removing receiver anchoring, joint premises, and graph input show that the reported performance depends on how evidence is represented and integrated, in addition to whether evidence completion is available. Figure 3 also compares GraphPI with structured communication protocols: its HiddenBench accuracy is 71.9/78.3/85.4 across the three actors, compared with 67.6/69.2/64.8 for ETD and 47.4/58.5/40.3 for Share-All.

AuthorsOpenReview ↗

Response to Reviewer 2T4a (2/10)

Non-graph baselines with matched information access and acquisition budgets. To directly examine the reviewer's proposed comparison, we use two non-graph harness configurations:

  • Harness-RAG: each agent maintains its own observations and a persistent compressed reasoning summary. After each round, the actor LLM is called once to compress the agent's observation list and current reasoning into a summary of at most 150 tokens (context compression). Before evidence acquisition, the actor is prompted with its current context and asked to produce up to three unresolved questions in JSON format (gap identification). Each question is encoded with BGE-M3 (Chen et al., 2024a), and the top-24 source spans from authorized peer information pools are retrieved by cosine similarity and filtered to the acquisition budget using the same unique-source-span accounting as GraphPI. Retrieved spans are prepended with their peer identifier and appended to the context as plain text, with no structural linking to the receiver's existing reasoning.
  • Harness-Struct: the same memory compression, question-generation, and BGE-M3 retrieval components as Harness-RAG. All exchanged messages additionally follow a structured schema with four fields: claim (the agent's main assertion), evidence (supporting observations), uncertainty (unresolved concerns), and open question (a question addressed to peers). This provides explicit structured information sharing, in the spirit of the structured exchange protocol of Li et al. (2026a), without a maintained graph of cross-source relations and premise dependencies.
  • Share-All-RAG (HiddenBench): combines the Share-All disclosure instruction—each agent is explicitly told to include all of its held observations in every message (Li et al., 2026a)—with Harness-RAG's question-driven BGE-M3 retrieval and the same acquisition budget. Since Share-All is a HiddenBench protocol, this baseline is evaluated on HiddenBench only. It tests whether adding retrieval on top of maximal voluntary disclosure can close the gap to GraphPI.

The comparison uses the same actor, initial task information, communication schedule, authorized peer sources, and source-span accounting within each benchmark. Each agent has an acquisition budget of four new source spans, shared across the episode on HiddenBench (Appendix B.5) and renewed at each completion step on MisinfoTask. The same BGE-M3 embedding model (Chen et al., 2024a) used in GraphPI's evidence retrieval is used in all harness baselines for retrieval encoding, so embedding quality does not vary across methods. Matching the available information and budget leaves each method free to select evidence according to its own integration procedure.

ETD and ARGUS are benchmark-specific baselines, so each is reported only on its own benchmark. We present the comparison as one table per benchmark. Harness-RAG, Harness-Struct, and Share-All-RAG results are averaged over three runs.

Table R2-HB. HiddenBench: non-graph harness baselines (Acc., %).

ActorMethodAcc. ↑
GPT-4o-miniETD*67.6
GPT-4o-miniHarness-RAG62.4
GPT-4o-miniHarness-Struct64.2
GPT-4o-miniShare-All-RAG†72.3
GPT-4o-miniGraphPI*71.9
DeepSeek-V3ETD*69.2
DeepSeek-V3Harness-RAG61.7
DeepSeek-V3Harness-Struct59.2
DeepSeek-V3Share-All-RAG†68.4
DeepSeek-V3GraphPI*78.3
Qwen3.5-35B-A3BETD*64.8
Qwen3.5-35B-A3BHarness-RAG67.9
Qwen3.5-35B-A3BHarness-Struct70.1
Qwen3.5-35B-A3BShare-All-RAG†71.6
Qwen3.5-35B-A3BGraphPI*85.4

Table R2-MT. MisinfoTask: non-graph harness baselines (Avg. over three attack settings).

ActorMethodAvg. MT ↓Avg. TSR ↑
GPT-4o-miniARGUS*4.7270.68
GPT-4o-miniHarness-RAG5.6458.96
GPT-4o-miniHarness-Struct5.8765.14
GPT-4o-miniGraphPI*3.9177.47
DeepSeek-V3ARGUS*3.4170.06
DeepSeek-V3Harness-RAG3.8969.15
DeepSeek-V3Harness-Struct4.1268.45
DeepSeek-V3GraphPI*2.5482.72
Qwen3.5-35B-A3BARGUS*2.6078.70
Qwen3.5-35B-A3BHarness-RAG3.4368.25
Qwen3.5-35B-A3BHarness-Struct3.1273.29
Qwen3.5-35B-A3BGraphPI*2.4983.02

*ETD, ARGUS, and GraphPI results are from the submitted paper. Matched experimental comparisons use the same evaluated task set and configuration. †Share-All-RAG combines full voluntary disclosure (Li et al., 2026a) with Harness-RAG retrieval and therefore has broader peer-information access than the other harness rows.

AuthorsOpenReview ↗

Response to Reviewer 2T4a (3/10)

GraphPI outperforms both non-graph harness variants on every actor and on both benchmarks under matched information access and acquisition budgets. On HiddenBench (Table R2-HB), GraphPI exceeds the better of Harness-RAG and Harness-Struct by 7.7, 16.6, and 15.3 points on GPT-4o-mini, DeepSeek-V3, and Qwen3.5-35B-A3B, respectively, an average of 13.2 points. On MisinfoTask (Table R2-MT), relative to the best harness variant on each metric, GraphPI lowers Avg. MT by 1.73, 1.35, and 0.63 (1.24 on average) and raises Avg. TSR by 12.33, 13.57, and 9.73 points (11.88 on average). Share-All-RAG, in which every agent additionally discloses all of its held observations, reaches 72.3/68.4/71.6 on HiddenBench; GraphPI is ahead of it by 9.9 points on DeepSeek-V3 and 13.8 points on Qwen3.5-35B-A3B, is comparable on GPT-4o-mini (71.9 vs. 72.3), and leads by 7.8 points on average. These baselines already provide persistent memory, context compression, structured messaging, and question-driven retrieval over the same peer information and the same four-span budget. The remaining gap comes from what GraphPI adds: relating incoming information to the receiver's preserved reasoning state and acquiring dependency-preserving evidence for that state's needs.

Role of the fine-tuned graph components. We separately examine how the graph builder and linker affect the framework. The component comparison replaces both modules with the corresponding untuned base models or the teacher model. It retains the graph schema, relation definitions, and downstream integration procedure. Qwen3.5-35B-A3B serves as the actor.

Table R3. Graph-component substitutions.

Graph builder / linkerHiddenBench Acc. ↑MisinfoTask Avg. MT ↓MisinfoTask Avg. TSR ↑
Untuned Qwen3-8B / Qwen3-4B74.323.6773.72
DeepSeek-V4-Pro for both modules85.972.5682.12
Fine-tuned Qwen3-8B / Qwen3-4B85.42.4983.02

The untuned and DeepSeek-V4-Pro rows are means over three runs on completed instances; the fine-tuned row is the result reported in the submitted paper.

Replacing the fine-tuned builder and linker with their untuned base models lowers HiddenBench accuracy by 11.08 percentage points (85.4 to 74.32), raises Avg. MT by 1.18 (2.49 to 3.67), and lowers Avg. TSR by 9.30 percentage points (83.02 to 73.72), so task-specific graph prediction contributes substantially to the final performance. The fine-tuned 8B/4B configuration is on par with the DeepSeek-V4-Pro teacher: it differs by -0.57 percentage points on HiddenBench (85.4 vs. 85.97), by +0.90 percentage points on Avg. TSR (83.02 vs. 82.12), and by 0.07 on Avg. MT (2.49 vs. 2.56), and it has the lowest Avg. MT and the highest Avg. TSR of the three configurations. This level of performance comes at a much lower serving cost. Graph construction and linking for one six-round HiddenBench task involve approximately 55.9k input and output tokens (Table R12: 9.2k + 5.3k for the builder and 39.3k + 2.1k for the linker). The teacher configuration would route these tokens through a hosted API, whereas the fine-tuned configuration serves them from local Qwen3-8B and Qwen3-4B models. The untuned configuration also remains ahead of the non-graph harnesses on HiddenBench, with 74.32 compared with 67.9 for Harness-RAG and 70.1 for Harness-Struct under the same Qwen3.5-35B-A3B actor (Table R2-HB), and its Avg. TSR of 73.72 is above both harness variants (68.25 and 73.29; Table R2-MT). The advantage of graph-based integration over the non-graph harnesses on these two measures is therefore already present with untuned components, while fine-tuning adds a further gain. These measurements distinguish the contribution of task-specific graph prediction from that of the downstream integration procedure.

Reviewer_Z4tJOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GraphPI, a graph-based framework for integrating peer information in multi-agent collaboration. A trained graph builder represents each agent’s reasoning state using claims, open questions, and premise dependencies. A trained cross-graph linker connects peer messages to the receiver’s existing graph, after which graph-guided retrieval selects complementary evidence from peer state graphs under a budget. The updated graph is then provided to the actor for subsequent reasoning.

Critical strengths
  1. Explicit representation of peer information. Graph nodes retain source spans, claims, and open questions, while premise and cross-graph relations make the intended information dependencies explicit.
  2. Component and interaction analyses. The paper includes ablations of receiver anchoring, evidence completion, joint premises, and graph input, as well as tests that vary interaction rounds and agent counts.
Critical weaknesses
  1. Limited modeling of relation strength. Although the graph includes premise dependencies and support, contradict, and relate links, the predicted semantic relations remain discrete and do not explicitly express their strength or confidence. It would be useful to investigate weighted relations and their role in graph-based evidence integration.
  2. Limited actor-model coverage. The evaluation uses only three actor models, with the GPT-family comparison relying solely on GPT-4o-mini. Experiments with additional model families and a broader range of GPT models would strengthen the evidence for the method’s generality.
  3. Insufficient graph-level analysis. The evaluation mainly reports task-level outcomes. More graph-specific analysis is needed, such as round-by-round examples showing how peer claims are linked, which evidence subgraphs are retrieved, and how these changes affect each agent’s judgment.
Questions for authors

Just see weeknesses

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Reviewers, AC, SAC, and PC,

We thank the reviewers for their time and constructive feedback. Below we summarize the key strengths recognized and the new experiments and revisions conducted in response to the comments.

(* We refer to Reviewer Z4tJ as R1, Reviewer 2T4a as R2, and Reviewer jKXm as R3.)


Key strengths noted by the reviewers:

  • S1 — Important and well-motivated problem: the empirical analysis of judgment shifts and erroneous consensus provides a clear basis for investigating explicit peer integration (R2).
  • S2 — Recipient-centered representation that retains conjunctive premises and distinguishes support, contradiction, and unresolved questions, as a concrete alternative to appending peer responses to context (R3).
  • S3 — Informative component ablations and separate cost reporting; the two-round ETD comparison is noted as particularly helpful (R3).
  • S4 — Consistent improvements across three actor models on both MisinfoTask and HiddenBench (R2, R3).

Key revisions and new experiments:

[R2 W1/W3, R3 W1] Non-graph harness baselines under matched information access (Appendix C, E)

We constructed three non-graph baselines — Harness-RAG, Harness-Struct, and Share-All-RAG — sharing GraphPI's authorized peer-information pool, four-span acquisition budget, and BGE-M3 encoder.

  • GraphPI exceeds the better of Harness-RAG/Struct by 13.2 pp on average on HiddenBench and lowers Avg. MT by 1.24 on average on MisinfoTask.
  • Share-All-RAG (full disclosure + retrieval) still trails GraphPI by 7.8 pp on average on HiddenBench.
  • Budget sweep B ∈ {0,1,2,4,8}: at B=0 (no additional access), GraphPI already leads Harness-RAG by 14.01 pp; at B=1 it already surpasses Harness-RAG at B=8.

[R2 W4] Judgment transition analysis (Appendix F)

Paired transition analysis decomposes accuracy changes into beneficial corrections (C) and harmful shifts (H):

  • GraphPI: C = 69.6 pp, H = 2.4 pp (ETD: C = 54.1, H = 5.8).
  • Controlled exposure experiment: under false-peer exposure, harmful-transition rate drops from 48.27% to 19.02% (−29.25 pp); under correct-peer exposure, beneficial-correction rate rises from 52.34% to 68.75% (+16.41 pp).

[R3 W2] Held-out graph quality evaluation (Appendix G)

  • Graph builder: Node F1 76.28%, Premise-edge F1 70.32%, Joint-premise-group F1 64.09%
  • Cross-graph linker: Relation macro-F1 72.53%
  • Fine-tuned model trails DeepSeek-V4-Pro teacher by only 0.57 pp on HiddenBench.

[R3 W3, R2 Minor 2] End-to-end latency (Section 5, Computational Cost Analysis)

  • GraphPI-2r adds only 3.9 s over ETD (+11.9 pp accuracy).
  • GraphPI-6r is 7.8 s faster than Native Discussion (+52.6 pp accuracy).
  • Graph modules run concurrently with actor API calls.

[R1 W2] Extended actor-model coverage (Appendix H)

Seven additional actor models evaluated on MisinfoTask: GPT-4o, GPT-4.1-mini, GPT-4.1, DeepSeek-V4-Flash, DeepSeek-V4.1-Flash, Gemini 2.5 Flash, GLM-4.7-Flash.

  • GraphPI achieves the lowest Avg. MT and highest Avg. TSR across all seven, with gains over ARGUS of 0.19–0.40 MT and 2.77–5.86 pp TSR.

[R1 W1] Confidence-weighted relation ablation (Appendix I)

Evaluated LLM-estimated and label-probability confidence variants. Discrete relations achieve the highest Avg. TSR across all three actors, confirming the design choice.

[R1 W3, R2 Minor 1] Case Study (Section 5; Appendix J)

A step-by-step trace of one HiddenBench task through RSGC, RAPI, and GGEC, connecting graph-level operations to individual judgment revisions.


Thank you again for your time, careful evaluation, and for managing the review process. We appreciate your effort, and we hope this summary is helpful.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.