- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes PPM-RAG, a model-frozen multi-agent framework for privacy-aware multimodal retrieval-augmented generation over distributed local data sources. The main idea is to separate the discovery of relevant information sources from the disclosure of query-specific evidence.
The proposed framework introduces a central–local Federated Schema Directory (FSD). Each local specialist extracts document-level schemas consisting of summaries, keywords, and synthetic document-to-query representations, and organizes them into hierarchical retrieval trees. Only the summaries and metadata of local root nodes are shared with the central agent. During inference, the central agent decomposes the input query into sub-queries, routes them to relevant specialists using hybrid dense–sparse retrieval and best-first traversal, and integrates query-specific evidence generated by the selected local agents.
The paper evaluates PPM-RAG on WebQA and MMDocRAG, comparing it with six baselines in terms of answer quality, retrieval effectiveness, retrieval efficiency, and privacy-related relevance metrics. The reported results indicate improvements in retrieval recall, answer quality, and retrieval latency. The paper also includes component ablations, scalability experiments, and an evaluation of different foundation model backbones.
Critical strengths
S1. Well-motivated problem setting. The paper addresses an important practical challenge: performing multimodal retrieval over distributed personal data without requiring a central server to access all original documents. Separating source discovery from evidence disclosure is a useful architectural principle for privacy-aware RAG systems.
S2. A coherent hierarchical retrieval framework. The central–local FSD provides a structured mechanism for coordinating distributed specialists. The combination of document summaries, keywords, and document-to-query representations is intuitive and potentially useful for handling heterogeneous modalities and reducing unnecessary retrieval operations.
S3. Promising empirical performance. The reported results on WebQA and MMDocRAG show competitive answer quality and retrieval recall, together with substantial improvements in the reported retrieval latency. For example, Table 1 reports a WebQA recall of 0.932 and a QA-All score of 0.539 for the proposed method. These results suggest that the framework can achieve effective retrieval without task-specific backbone fine-tuning.
Critical weaknesses
W1. The privacy-preserving claim is not sufficiently substantiated.
The central claim of the paper is privacy-preserving multimodal RAG, but the privacy evaluation primarily measures retrieval relevance rather than actual sensitive-information disclosure. In Appendix A.2, the Irrelevant Information Ratio (IIR) is defined as 1 − BLEU. However, lexical similarity to a reference does not directly quantify privacy leakage. Query-irrelevant information is not necessarily sensitive, and highly query-relevant information may still contain private attributes that should not be disclosed.
Similarly, UMBRELA evaluates relevance rather than confidentiality. Neither metric establishes that sensitive information cannot be reconstructed from the uploaded FSD summaries or the evidence generated by local agents. The paper also lacks a clearly specified threat model, privacy budget, formal disclosure constraints, or systematic privacy-attack evaluation. Therefore, the reported improvements in IIR and UMBRELA support better relevance filtering, but do not conclusively establish privacy preservation.
W2. Potential information leakage through the shared FSD is not addressed.
The proposed architecture uploads root-level summaries and metadata to the central agent. However, these representations are generated from private local documents and may themselves contain personally identifiable or otherwise sensitive information. Moreover, Section 4.1 specifies that parent-node keywords and document-to-query metadata are constructed by merging and deduplicating information from child nodes. This aggregation may preserve sensitive lexical details even when the corresponding raw documents remain local.
The paper does not quantify the amount of information exposed through the root-level summaries and metadata, evaluate reconstruction or attribute-inference attacks, or establish a principled bound on the disclosed information. Consequently, keeping original documents local is not sufficient to guarantee the claimed privacy properties.
Questions for authors
Privacy threat model and leakage measurement. What adversarial capabilities are assumed for the central agent? Is the central server considered honest-but-curious, or can it actively issue adaptive queries to extract sensitive information? How does the proposed framework prevent private attributes from being inferred through repeated queries or root-level FSD metadata? Could the authors provide direct privacy-leakage measurements rather than relying exclusively on retrieval-relevance metrics?
Interpretation of IIR. Appendix A.2 defines IIR as 1 − BLEU. What text is used as the BLEU reference, and how is the metric computed for multimodal evidence? Why should this metric be interpreted as a measure of irrelevant information exposure rather than lexical mismatch? Could the authors validate the claimed privacy improvements using annotated sensitive attributes, exposure rates, or privacy-oriented attack evaluations?
2 comments
Dear Reviewer Wkmb,
Regarding the discussion of privacy preservation in our manuscript, we would like to clarify that our work is motivated by a practical central–local setting in which users submit queries involving their private data and leverage the reasoning capabilities of a central agent powered by a cloud-based LLM to fulfill their requests. In this setting, we consider user-authorized queries that involve accessing personal data distributed across local agents, such as smartphones and personal computers. Meanwhile, our method seeks to minimize the exposure of query-irrelevant information that may contain sensitive personal data and pose privacy risks to the central agent. Specifically, we distinguish between authorizing access to personal data for a particular user query and disclosing all information contained in the accessed documents to the central agent. Accordingly, our privacy-preserving contribution focuses on minimizing the exposure of query-irrelevant sensitive information to the central agent during authorized query processing, rather than filtering all private information or enforcing authorization for potentially malicious queries.
To address this issue, PPM-RAG incorporates query-conditioned evidence generation at local specialists, which serves as an information-minimization gatekeeper before evidence is transmitted to the central agent. Specifically, the central agent decomposes the query into sub-queries and uses the central FSD to identify relevant local specialists. Each selected specialist retrieves documents locally and generates only the essential evidence for its assigned sub-query, rather than sharing the original documents or their full contents. The central agent then aggregates this task-specific evidence for final reasoning, while the original multimodal documents remain on the local. We would like to clarify that our method aims to minimize the unnecessary disclosure of query-irrelevant sensitive information while preserving information necessary to answer authorized queries. This contribution is supported by the additional privacy evaluation presented in our previous response, using two complementary metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). Our method achieves a SAER of 0.007 and an IPLR of 0.148, representing average reductions of 91.5% and 42.1%, respectively, compared with the evaluated baselines. These results empirically demonstrate the effectiveness of our method in minimizing the exposure of query-irrelevant sensitive information under the considered setting. All additional experimental results and clarifications regarding our privacy-preserving claims have been incorporated into the revised manuscript.
Finally, we welcome further discussion and would be happy to address any additional questions or concerns. We sincerely appreciate your time and thoughtful consideration.
We sincerely appreciate your insightful and valuable reviews.
Clarification of IIR Computation
We clarify that IIR is computed over the concatenated retrieved context, with non-textual modalities converted into text using a multimodal LLM.
Additional Evaluation Highlighting Privacy-Preserving Contribution
We apologize for the confusion arised in the description of FSD construction and the central agent's assumptions. During FSD construction, bottom-up merging operations extract high-level, generalizable information rather than preserving sensitive details from child nodes. We also clarify that the central agent is assumed to be honest-but-curious. Specifically, we assume that all queries are benign and do not explicitly request the disclosure of query-irrelevant sensitive information. We conducted additional experiments to evaluate the privacy-preserving performance of our method. We annotated query-irrelevant sensitive attributes in raw documents following established guidelines from prior studies [1, 2] and introduce additional metrics to directly quantify their disclosure in retrieved contexts.
Concretely, we annotated sensitive attributes in the WebQA dataset using a combination of regular expression matching and GPT-6-based annotation, covering four categories: (i) identity and account identifiers (e.g., account IDs and email addresses), (ii) private locations (e.g., residential addresses), (iii) financial information (e.g., payment and banking details), and (iv) personal and biometric information (e.g., medical records and identifiable facial features). We employ two complementary privacy metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). SAER measures the proportion of queries in which at least one annotated sensitive attribute is disclosed to the central agent, defined as:
SAER=∣Q+∣∑q∈Q+I[∃a∈Aq:Expose(a,Zq)=1].
Here, Q+ denotes queries whose retrieved documents contain sensitive attributes, Aq is the corresponding annotated attribute set, and Zq represents the evidence returned to the central agent. Expose(a,Zq)=1 indicates that attribute a is disclosed in Zq.
IPLR employs an independent LLM-as-a-Judge to assess the disclosure of query-irrelevant sensitive attributes before final answer generation. Both metrics range from 0 to 1, with lower values indicating better privacy preservation.
The results show that our method achieves average reductions of 91.5% in SAER and 42.1% in IPLR compared with the baselines, primarily attributable to the local generation of essential query-relevant evidence by each specialist, which effectively minimizes the disclosure of sensitive information. The additional experimental results and analyses have been updated into the revison. All annotated documents and corresponding evaluation prompts will be made publicly available in repository to facilitate future research.
Similarly, we further employ FSD Sensitive Attribute Exposure Rate (FSD-SAER) to quantify sensitive information leakage through summary and metadata (D2Q, Keywords) in the FSD, defined as:
\frac{\sum_{v \in \mathcal{V}} \mathbb{I}[\exists a \in \mathcal{A}_v : \mathrm{Expose}(a,M_v^{(t)})=1]}{|\mathcal{V}|}.$$ Here, $\mathcal{V}$ denotes the set of all central FSD nodes, $t \in [\mathrm{Summary}, \mathrm{D2Q}, \mathrm{Keywords}]$ specifies the text type, $M_v^{(t)}$ represents the corresponding text of node $v$, and $\mathcal{A}_v$ denotes the set of annotated sensitive attributes in its underlying documents. The metric measures the proportion of FSD nodes whose metadata of type $t$ exposes at least one sensitive attribute, with lower values indicating better privacy preservation. ||Summary|D2Q|Keywords -|-|-|- FSD-SAER|0.001|0.008|0.005 The results indicate that both FSD summaries and metadata (D2Q and keywords) primarily capture high-level, abstract information rather than sensitive details. Nevertheless, we acknowledge that they still pose potential privacy leakage risks. **Discussion of Limitations** We acknowledge that the honest-but-curious assumption regarding the central agent limits the applicability of our privacy-preserving approach under malicious attacks and that the central FSD poses potential privacy leakage risks. We have stated the limitation in the revision. [1] Pilán, I., et al. "The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization." Comp. Ling. 2022. [2] Orekondy, T., et al. "Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images." ICCV2017.