- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes privacy-aware collaboration among LLM agents with heterogeneous private memories. RAD selectively abstracts sensitive query spans using memory-relevance losses and a query-local Gaussian mixture; CEI ranks peer evidence by geometric complementarity before generating reasoning summaries. Experiments on constructed MedQA and LegalQA tasks with three LLM backbones evaluate accuracy, fact-reconstruction protection, and their trade-off, alongside component and sensitivity analyses.
Critical strengths
- The two communication directions receive distinct, useful treatment. Separating outgoing disclosure from incoming evidence selection addresses both unnecessary exposure and redundant collaboration. The modular comparisons help identify their different effects on utility and privacy.
- There is concrete evidence of useful collaboration. With Qwen3-30B, the full method improves accuracy over a single agent from 81.88% to 84.79% on MedQA and from 42.86% to 45.36% on LegalQA (Table 1). These gains support the practical value of exploiting heterogeneous memories.
Critical weaknesses
- The privacy boundary is incomplete at the central CEI step. Equations (10)–(12) and Algorithms 1/3 require joint access to target- and peer-memory representations, yet the paper does not specify where this computation occurs or what representations cross institutions. Appendix C evaluates attacks on the protected query and returned summaries only. Without accounting for the information needed to compute CEI, neither distributed implementability nor the reported protection is established for the complete protocol.
- The aggregate metric does not establish a convincing privacy–utility advantage. OP pools query and retrieved-memory facts, allowing 94.86% protection despite QP = 0 for the Qwen single-agent baseline (Table 1). This weighting can obscure query exposure. Moreover, the full method's DeepSeek average TO is below RAD alone (79.19 versus 79.98), and uncertainty is unreported. Thus, the benefit of adding collaboration to disclosure protection remains insufficiently resolved.
Questions for authors
- Protocol clarification: Where is CEI computed, what exact messages or embeddings are exchanged, and what protection results hold when the attacker sees that complete transcript?
- Evaluation clarification: Which models and prompts perform reconstruction and entailment judging? Does the conclusion hold when query and memory protection are assessed separately using a common source-fact universe across methods?
- Evidence needed for assessment: What are the evaluation sizes and paired uncertainty estimates for the full method versus RAD alone and RAD + PrivCollab-Agent across memory partitions and inference runs?
5 comments
Response to Reviewer keYf (2/3)
W2.2: Effectiveness of Collaboration Mechanisms
We thank you for raising this important point regarding the benefits of collaboration.
First, collaboration improves task utility. As shown in Table E, introducing peer collaboration consistently improves TU across different query protection strategies.
Table E: Effect of collaboration on TU using DeepSeek-V4-Flash.
| Privacy Mechanism | Collaboration Strategy | MedQA TU (↑) |
|---|---|---|
| MNI-Gate | Without Collaboration | 89.97 |
| MNI-Gate | + Reasoning Share | 92.23 (+2.26) |
| PromptObfus | Without Collaboration | 92.56 |
| PromptObfus | + Reasoning Share | 94.17 (+1.61) |
| RAD | Without Collaboration | 94.50 |
| RAD | + CEI (Ours) | 96.44 (+1.94) |
Second, effective collaboration requires complementary expertise utilization. Unlike direct Reasoning Share, CEI explicitly prioritizes complementary peer evidence beyond the target's existing knowledge. For example, RAD + CEI achieves 96.44 TU on MedQA, outperforming RAD + PrivCollab-Agent (94.82), demonstrating the benefit of complementarity-aware reasoning.
Q2: Clarification on Reconstruction Evaluation
We sincerely thank you for highlighting this point. We clarify our reconstruction protocol from two aspects.
-
(1) Reconstruction Model and Prompts.
Both Query and Memory Reconstruction use DeepSeek-V4-Flash as the attack model, with prompts tailored to their respective communication boundaries:
- Query Reconstruction: The attacker reconstructs private query facts from the peer-visible protected query. The key prompt instructions are:
You are a protocol-aware privacy reconstruction attacker auditing a protected medical QA query. ... provide exactly one best-guess replacement sentence or clause using its surrounding context ... Do not enumerate alternatives. Do not select or reveal the correct answer option...
- Memory Reconstruction: The attacker reconstructs private-memory facts from the target-visible original query and peer reasoning summaries. The key prompt instructions are:
You are a privacy reconstruction attacker. The user message contains exactly the task context known to a target client and the messages it received from two peers. ... Reconstruct every atomic, case-specific fact about those hidden peer memories that the target can state or reasonably infer.
-
(2) Entailment Judging.
We use a fixed microsoft/deberta-base-mnli classifier to evaluate reconstruction success, with reconstructed statements as premises and source facts as hypotheses. A fact is recovered only when the argmax prediction is
ENTAILMENT. This fixed criterion reduces evaluation variability and ensures consistent assessment across methods.
Q3: Evaluation Robustness Across Memory Partitions and Inference Runs
We thank you for suggesting additional uncertainty estimates. To assess robustness, we conduct additional experiments across random seeds and memory heterogeneity levels using DeepSeek-V4-Flash
- (1) Robustness Across Random Seeds.
MedQA and LegalQA contain 103 and 133 test questions, respectively. With three agents each serving as the target, we obtain 309 and 399 target–question evaluations per run. We report results with 3 and 5 additional seeds (4 and 6 runs in total).
Table F: Performance across additional random seeds at (mean ± SD).
| Dataset | Added Seeds | Method | TU (↑) | OP (↑) | TO (↑) |
|---|---|---|---|---|---|
| MedQA | 3 | RAD | 95.31 ± 0.81 | 95.04 ± 0.03 | 95.17 ± 0.39 |
| RAD + PrivCollab-Agent | 96.52 ± 1.44 | 94.30 ± 0.17 | 95.40 ± 0.79 | ||
| Ours | 97.17 ± 0.67 | 93.98 ± 0.09 | 95.55 ± 0.33 | ||
| 5 | RAD | 95.42 ± 0.66 | 95.04 ± 0.03 | 95.23 ± 0.32 | |
| RAD + PrivCollab-Agent | 97.30 ± 1.27 | 94.33 ± 0.14 | 95.79 ± 0.69 | ||
| Ours | 97.36 ± 0.63 | 93.92 ± 0.12 | 95.61 ± 0.28 | ||
| LegalQA | 3 | RAD | 48.37 ± 2.44 | 95.22 ± 0.20 | 64.15 ± 2.15 |
| RAD + PrivCollab-Agent | 46.49 ± 4.67 | 93.54 ± 0.35 | 62.11 ± 4.17 | ||
| Ours | 49.94 ± 0.87 | 91.04 ± 0.06 | 64.50 ± 0.73 | ||
| 5 | RAD | 48.12 ± 2.45 | 95.25 ± 0.16 | 63.94 ± 2.16 | |
| RAD + PrivCollab-Agent | 48.08 ± 3.77 | 93.56 ± 0.28 | 63.52 ± 3.29 | ||
| Ours | 49.87 ± 1.38 | 90.93 ± 0.18 | 64.42 ± 1.17 |
Our framework achieves the batter mean TU across all settings and the completive mean TO in three of four cases, with consistently lower reported variability in TU and TO. Although RAD + PrivCollab-Agent achieves slightly higher TO on MedQA with five additional seeds, its substantially larger standard deviation indicates less stable performance. These results highlight our framework's stronger task utility and greater robustness across random runs.
Response to Reviewer keYf (3/3)
Q3: Evaluation Robustness Across Memory Partitions and Inference Runs
- (2) Robustness Across Memory Heterogeneity.
We further evaluate different Dirichlet parameters (), using the original experimental results at .
Table G: TU under different private-memory heterogeneity levels on MedQA.
| Method | |||
|---|---|---|---|
| RAD | 93.53 | 94.50 | 92.56 |
| RAD + PrivCollab-Agent | 93.85 | 94.82 | 93.53 |
| Ours | 96.12 | 96.44 | 96.44 |
As decreases from 0.5 to 0.1, stronger data heterogeneity limits each agent's knowledge coverage, causing larger TU drops for RAD and RAD + PrivCollab-Agent than for Ours. CEI mitigates this degradation by identifying complementary peer expertise beyond the target's knowledge. At and , broader knowledge coverage provides more stable expertise, while CEI suppresses redundant evidence, maintaining consistently high TU.
[1] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS, 2020.
[2] MASS-RAG: Multi-Agent Synthesis Retrieval-Augmented Generation. Findings of ACL, 2026.
Response to Reviewer keYf (1/3)
W1&Q1: Clarification on privacy boundary at CEI and RAD
We sincerely appreciate your insightful observation and apologize for the insufficient clarification of CEI's communication protocol. To clarify the communication boundaries of our framework, we summarize the complete workflow below.
Table A. Communication protocol of our framework.
Importantly, CEI operates on the peer side without requiring centralized access to raw private memories. As formulated in Sec. 4.2 (Eqs. 9–12), each agent locally computes query-relative geometric descriptors (θ,u) for its retrieved memories. The target shares its descriptors with peers, which locally compute pairwise complementarity scores and prioritize complementary evidence for reasoning. To further examine the privacy–utility trade-offs of RAD and CEI, we conduct additional comparisons on MedQA using DeepSeek-V4-Flash.
Table B: Comparison of query abstraction strategies for RAD.
Table C: Comparison of CEI deployment strategies.
Task Sharing (RAD). RAD protects the original query, with QP measuring sensitive information recoverable from the protected query. As shown in Table B, All Abstraction maximizes QP but reduces TU, while No Abstraction preserves TU without query protection. Our RAD adaptively preserves expertise-relevant information while protecting sensitive details, achieving a better privacy–utility balance.
Output Sharing (CEI). CEI enables complementary reasoning sharing without exchanging raw private memories, with MP measuring information leakage from peer summaries. As shown in Table C, Target-side CEI reweights only local memories after receiving unguided peer reasoning. In contrast, Peer-side CEI directly guides complementary peer reasoning, improving TU, MP, and TO while keeping raw memories local.
We have added these experiments in Paper’s 30th page.
W2.1: Clarification on OP Aggregation
We sincerely thank you for highlighting this important point. To further examine the effect of privacy aggregation, we consider an alternative Mean-based OP alongside our Fact-weighted OP:
OPmean=2QP+MP
OPours=1−∣FQ∣+∣FM∣∣FQ∣+∣FM∣
The key distinction lies in the unit of privacy evaluation. Mean-based OP assigns equal weight to QP and MP, regardless of the number of private facts in each. In contrast, our OP assigns equal weight to each evaluated private fact rather than each privacy dimension.
This distinction is particularly relevant to multi-agent collaboration. A single query may involve multiple peer agents, each retrieving several private memories containing distinct atomic facts [1, 2]. Consequently, query and memory facts may differ substantially in quantity. While Mean-based OP always assigns 50% weight to each privacy dimension, our Fact-weighted OP accounts for the actual number of facts involved. We further compare the two aggregation strategies using Qwen3-30B and DeepSeek-V4-Flash on MedQA:
Table D: Performance under different OP aggregation strategies.
50.0094.8673.2993.0350.0093.0772.3891.96Under Mean-based OP, our framework surpasses Single Agent by 16.53 and 17.07 TO points on Qwen3-30B and DeepSeek-V4-Flash, respectively. These larger gains arise because Mean-based OP equally weights QP and MP, eliminating the influence of unequal fact counts. Nevertheless, we retain Fact-weighted OP to reflect actual fact-level privacy exposure, under which our framework still outperforms Single Agent on Qwen3-30B and achieves comparable TO on DeepSeek-V4-Flash, with higher TU on both.