acceptodds
Under review as a conference paper at ICLR 2027

Where to Look, What to Reveal: Privacy Preserving Multimodal RAG with Multiple Agents

Abstract

Modern large language model (LLM) systems increasingly rely on centrally incorporating task-relevant external context at inference time, in addition to scaling model capacity. Retrieval-augmented generation (RAG) provides a general mechanism for such context augmentation, yet existing cloud-centric RAG systems often require fine-grained user data to be centrally accessible, creating unnecessary privacy exposure when personalized information is distributed across heterogeneous local sources. We propose PPM-RAG, a model-frozen multi-agent multimodal RAG framework that enables collaborative retrieval over distributed private data while keeping raw documents local. At its core, PPM-RAG introduces a central–local Federated Schema Directory (FSD), where local agent specialists organize modality-specific documents into hierarchical semantic directories and expose only compact root-level summaries and metadata to the central agent. During inference, the central agent decomposes a user query into sub-queries and traverses the central FSD to identify relevant specialists. Selected specialists then perform hierarchical retrieval within their local FSDs and return only query-relevant evidence for final evidence-grounded reasoning. This design separates the discovery of where relevant knowledge resides from the disclosure of what information is required, enabling efficient and privacy-aware collaboration across distributed multimodal sources. Extensive experiments on WebQA and MMDocRAG against six baselines show that PPM-RAG improves retrieval Recall by 12.4% on average and answer quality by up to 32.9%, while reducing retrieval time by 90.6% compared with OpenRAG. PPM-RAG further reduces IIR by 10.8% and improves UMBRELA by 28.1%, demonstrating effective, efficient, and privacy-aware multimodal retrieval.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLMs, Multimodal QA, Data Privacy, Distributed System, Edge Computing

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_WkmbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PPM-RAG, a model-frozen multi-agent framework for privacy-aware multimodal retrieval-augmented generation over distributed local data sources. The main idea is to separate the discovery of relevant information sources from the disclosure of query-specific evidence.

The proposed framework introduces a central–local Federated Schema Directory (FSD). Each local specialist extracts document-level schemas consisting of summaries, keywords, and synthetic document-to-query representations, and organizes them into hierarchical retrieval trees. Only the summaries and metadata of local root nodes are shared with the central agent. During inference, the central agent decomposes the input query into sub-queries, routes them to relevant specialists using hybrid dense–sparse retrieval and best-first traversal, and integrates query-specific evidence generated by the selected local agents.

The paper evaluates PPM-RAG on WebQA and MMDocRAG, comparing it with six baselines in terms of answer quality, retrieval effectiveness, retrieval efficiency, and privacy-related relevance metrics. The reported results indicate improvements in retrieval recall, answer quality, and retrieval latency. The paper also includes component ablations, scalability experiments, and an evaluation of different foundation model backbones.

Critical strengths

S1. Well-motivated problem setting. The paper addresses an important practical challenge: performing multimodal retrieval over distributed personal data without requiring a central server to access all original documents. Separating source discovery from evidence disclosure is a useful architectural principle for privacy-aware RAG systems.

S2. A coherent hierarchical retrieval framework. The central–local FSD provides a structured mechanism for coordinating distributed specialists. The combination of document summaries, keywords, and document-to-query representations is intuitive and potentially useful for handling heterogeneous modalities and reducing unnecessary retrieval operations.

S3. Promising empirical performance. The reported results on WebQA and MMDocRAG show competitive answer quality and retrieval recall, together with substantial improvements in the reported retrieval latency. For example, Table 1 reports a WebQA recall of 0.932 and a QA-All score of 0.539 for the proposed method. These results suggest that the framework can achieve effective retrieval without task-specific backbone fine-tuning.

Critical weaknesses

W1. The privacy-preserving claim is not sufficiently substantiated.

The central claim of the paper is privacy-preserving multimodal RAG, but the privacy evaluation primarily measures retrieval relevance rather than actual sensitive-information disclosure. In Appendix A.2, the Irrelevant Information Ratio (IIR) is defined as 1 − BLEU. However, lexical similarity to a reference does not directly quantify privacy leakage. Query-irrelevant information is not necessarily sensitive, and highly query-relevant information may still contain private attributes that should not be disclosed.

Similarly, UMBRELA evaluates relevance rather than confidentiality. Neither metric establishes that sensitive information cannot be reconstructed from the uploaded FSD summaries or the evidence generated by local agents. The paper also lacks a clearly specified threat model, privacy budget, formal disclosure constraints, or systematic privacy-attack evaluation. Therefore, the reported improvements in IIR and UMBRELA support better relevance filtering, but do not conclusively establish privacy preservation.

W2. Potential information leakage through the shared FSD is not addressed.

The proposed architecture uploads root-level summaries and metadata to the central agent. However, these representations are generated from private local documents and may themselves contain personally identifiable or otherwise sensitive information. Moreover, Section 4.1 specifies that parent-node keywords and document-to-query metadata are constructed by merging and deduplicating information from child nodes. This aggregation may preserve sensitive lexical details even when the corresponding raw documents remain local.

The paper does not quantify the amount of information exposed through the root-level summaries and metadata, evaluate reconstruction or attribute-inference attacks, or establish a principled bound on the disclosed information. Consequently, keeping original documents local is not sufficient to guarantee the claimed privacy properties.

Questions for authors

Privacy threat model and leakage measurement. What adversarial capabilities are assumed for the central agent? Is the central server considered honest-but-curious, or can it actively issue adaptive queries to extract sensitive information? How does the proposed framework prevent private attributes from being inferred through repeated queries or root-level FSD metadata? Could the authors provide direct privacy-leakage measurements rather than relying exclusively on retrieval-relevance metrics?

Interpretation of IIR. Appendix A.2 defines IIR as 1 − BLEU. What text is used as the BLEU reference, and how is the metric computed for multimodal evidence? Why should this metric be interpreted as a measure of irrelevant information exposure rather than lexical mismatch? Could the authors validate the claimed privacy improvements using annotated sensitive attributes, exposure rates, or privacy-oriented attack evaluations?

2 comments

We sincerely appreciate your insightful and valuable reviews.

Clarification of IIR Computation

We clarify that IIR is computed over the concatenated retrieved context, with non-textual modalities converted into text using a multimodal LLM.

Additional Evaluation Highlighting Privacy-Preserving Contribution

We apologize for the confusion arised in the description of FSD construction and the central agent's assumptions. During FSD construction, bottom-up merging operations extract high-level, generalizable information rather than preserving sensitive details from child nodes. We also clarify that the central agent is assumed to be honest-but-curious. Specifically, we assume that all queries are benign and do not explicitly request the disclosure of query-irrelevant sensitive information. We conducted additional experiments to evaluate the privacy-preserving performance of our method. We annotated query-irrelevant sensitive attributes in raw documents following established guidelines from prior studies [1, 2] and introduce additional metrics to directly quantify their disclosure in retrieved contexts.

Concretely, we annotated sensitive attributes in the WebQA dataset using a combination of regular expression matching and GPT-6-based annotation, covering four categories: (i) identity and account identifiers (e.g., account IDs and email addresses), (ii) private locations (e.g., residential addresses), (iii) financial information (e.g., payment and banking details), and (iv) personal and biometric information (e.g., medical records and identifiable facial features). We employ two complementary privacy metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). SAER measures the proportion of queries in which at least one annotated sensitive attribute is disclosed to the central agent, defined as:

SAER=∑q∈Q+I[∃a∈Aq:Expose(a,Zq)=1]∣Q+∣.\mathrm{SAER} = \frac{\sum_{q \in \mathcal{Q}^{+}} \mathbb{I}[\exists a \in \mathcal{A}_q : \mathrm{Expose}(a,Z_q)=1]}{|\mathcal{Q}^{+}|}.

Here, Q+\mathcal{Q}^{+} denotes queries whose retrieved documents contain sensitive attributes, Aq\mathcal{A}_q is the corresponding annotated attribute set, and ZqZ_q represents the evidence returned to the central agent. Expose(a,Zq)=1\mathrm{Expose}(a,Z_q)=1 indicates that attribute aa is disclosed in ZqZ_q.

IPLR employs an independent LLM-as-a-Judge to assess the disclosure of query-irrelevant sensitive attributes before final answer generation. Both metrics range from 0 to 1, with lower values indicating better privacy preservation.

MethodSAER$\downarrow$IPLR$\downarrow$
OpenRAG0.1770.392
LightRAG0.1830.389
OFA0.0330.181
PromptCap0.1750.314
MMAPG0.1310.288
QDMG0.0480.190
Ours0.0070.148

The results show that our method achieves average reductions of 91.5% in SAER and 42.1% in IPLR compared with the baselines, primarily attributable to the local generation of essential query-relevant evidence by each specialist, which effectively minimizes the disclosure of sensitive information. The additional experimental results and analyses have been updated into the revison. All annotated documents and corresponding evaluation prompts will be made publicly available in repository to facilitate future research.

Similarly, we further employ FSD Sensitive Attribute Exposure Rate (FSD-SAER) to quantify sensitive information leakage through summary and metadata (D2Q, Keywords) in the FSD, defined as:

\frac{\sum_{v \in \mathcal{V}} \mathbb{I}[\exists a \in \mathcal{A}_v : \mathrm{Expose}(a,M_v^{(t)})=1]}{|\mathcal{V}|}.$$ Here, $\mathcal{V}$ denotes the set of all central FSD nodes, $t \in [\mathrm{Summary}, \mathrm{D2Q}, \mathrm{Keywords}]$ specifies the text type, $M_v^{(t)}$ represents the corresponding text of node $v$, and $\mathcal{A}_v$ denotes the set of annotated sensitive attributes in its underlying documents. The metric measures the proportion of FSD nodes whose metadata of type $t$ exposes at least one sensitive attribute, with lower values indicating better privacy preservation. ||Summary|D2Q|Keywords -|-|-|- FSD-SAER|0.001|0.008|0.005 The results indicate that both FSD summaries and metadata (D2Q and keywords) primarily capture high-level, abstract information rather than sensitive details. Nevertheless, we acknowledge that they still pose potential privacy leakage risks. **Discussion of Limitations** We acknowledge that the honest-but-curious assumption regarding the central agent limits the applicability of our privacy-preserving approach under malicious attacks and that the central FSD poses potential privacy leakage risks. We have stated the limitation in the revision. [1] Pilán, I., et al. "The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization." Comp. Ling. 2022. [2] Orekondy, T., et al. "Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images." ICCV2017.

Dear Reviewer Wkmb,

Regarding the discussion of privacy preservation in our manuscript, we would like to clarify that our work is motivated by a practical central–local setting in which users submit queries involving their private data and leverage the reasoning capabilities of a central agent powered by a cloud-based LLM to fulfill their requests. In this setting, we consider user-authorized queries that involve accessing personal data distributed across local agents, such as smartphones and personal computers. Meanwhile, our method seeks to minimize the exposure of query-irrelevant information that may contain sensitive personal data and pose privacy risks to the central agent. Specifically, we distinguish between authorizing access to personal data for a particular user query and disclosing all information contained in the accessed documents to the central agent. Accordingly, our privacy-preserving contribution focuses on minimizing the exposure of query-irrelevant sensitive information to the central agent during authorized query processing, rather than filtering all private information or enforcing authorization for potentially malicious queries.

To address this issue, PPM-RAG incorporates query-conditioned evidence generation at local specialists, which serves as an information-minimization gatekeeper before evidence is transmitted to the central agent. Specifically, the central agent decomposes the query into sub-queries and uses the central FSD to identify relevant local specialists. Each selected specialist retrieves documents locally and generates only the essential evidence for its assigned sub-query, rather than sharing the original documents or their full contents. The central agent then aggregates this task-specific evidence for final reasoning, while the original multimodal documents remain on the local. We would like to clarify that our method aims to minimize the unnecessary disclosure of query-irrelevant sensitive information while preserving information necessary to answer authorized queries. This contribution is supported by the additional privacy evaluation presented in our previous response, using two complementary metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). Our method achieves a SAER of 0.007 and an IPLR of 0.148, representing average reductions of 91.5% and 42.1%, respectively, compared with the evaluated baselines. These results empirically demonstrate the effectiveness of our method in minimizing the exposure of query-irrelevant sensitive information under the considered setting. All additional experimental results and clarifications regarding our privacy-preserving claims have been incorporated into the revised manuscript.

Finally, we welcome further discussion and would be happy to address any additional questions or concerns. We sincerely appreciate your time and thoughtful consideration.

Reviewer_1UY7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multimodal retrieval-augmented generation over documents held by separate private data sources. PPM-RAG operates without model fine-tuning and uses a central agent to locate relevant local sources while keeping the original documents local. Each local specialist builds a hierarchy from document summaries, keywords, and synthetic questions that its documents can answer, and uploads only root summaries and metadata. At query time, the central agent decomposes the question and searches its directory to select specialists. Those specialists retrieve locally and return evidence generated for their assigned subqueries. Experiments on sampled WebQA and MMDocRAG collections examine answer quality, retrieval, scalability, and disclosure of unrelated information. The results support the retrieval design, but the privacy evaluation does not directly establish whether sensitive information is protected.

Critical strengths

1.The architecture separates locating a data source from sharing its contents. Sections 4.1-4.2 give a concrete workflow: the central agent searches a directory, while local specialists keep the original documents and return selected evidence. This can reduce the need to collect all raw documents centrally. That architectural benefit is useful for distributed private-data applications, even though keeping documents local does not by itself guarantee privacy.

2.The experiments show useful gains and identify an important retrieval component. Table 1 reports a WebQA QA-All score of 0.539, compared with 0.512 for the best listed baseline on that metric. In Table 2, removing document-to-query (D2Q) metadata lowers this score to 0.328. D2Q means synthetic questions generated from a document to help match it to a user's question. Figure 4 also shows relatively stable recall when the collection grows from 4,000 to 32,000 documents. These results support the usefulness of the document representations and directory search.

Critical weaknesses

1.The privacy evaluation measures relevance, which is insufficient to establish protection of sensitive information. Section 5.3 uses IIR and UMBRELA to assess privacy. Appendix A.2 defines IIR as 1 minus BLEU and UMBRELA as a query-evidence relevance score. These metrics can indicate more focused disclosure, but cannot establish whether disclosed facts are sensitive or authorized: information can be both relevant and private. Section 4.1 also merges child keywords and generated questions into root metadata that is uploaded centrally, without directly evaluating what private details it reveals. The paper needs to define what information should remain protected and assess disclosure against that definition. This gap matters because privacy protection is a central contribution, whereas the current evaluation primarily supports reduced disclosure of unrelated information.

2.The added value of the directory and the scope of the timing results are not sufficiently clear. Table 2 tests individual retrieval signals, question decomposition, and traversal strategies. However, it does not compare the proposed directory with a simpler distributed retriever using the same D2Q metadata, local evidence generation, and retrieval budget. The results therefore do not show how much additional benefit comes from the directory itself. Some margins are small, such as MMDocRAG recall of 0.612 versus 0.609 in Table 1. The reported retrieval times of 0.006 and 0.010 seconds also need a clear statement of whether query encoding, decomposition, local evidence generation, and communication are included. These uncertainties limit the evidence for the distinctive system-level benefit and contribute to my weak-reject recommendation.

Questions for authors

1.What information should remain protected, which disclosures are authorized, and what can the central agent learn from uploaded root metadata and repeated queries? How do IIR and UMBRELA establish protection under that definition, or what direct disclosure measurements are available?

2.With D2Q metadata, local evidence generation, and retrieval budgets held fixed, how does the directory compare with a simpler distributed hybrid retriever? This would clarify the directory's additional contribution.

3.Which stages are included in Table 1's retrieval time? Please specify the hardware, query encoding, directory search, model calls, and communication assumptions, and report full query-processing time if available.

5 comments

We sincerely appreciate your valuable feedback.

Additional Evaluation Highlighting Privacy-Preserving Contribution

We apologize for the confusion arised in the description of Federated Schema Directory (FSD) construction. The merging operations during FSD construction are bottom-up to extract high-level, more genralizable information rather than keeping sensitive details in the children nodes. Specifically, we assume that all queries are benign and do not explicitly request the disclosure of query-irrelevant sensitive information. We conducted additional experiments to evaluate the privacy-preserving performance of our method. We annotated sensitive attributes in the WebQA dataset using a combination of regular expression matching and GPT-6-based annotation, covering four categories: (i) identity and account identifiers (e.g., account IDs and email addresses), (ii) private locations (e.g., residential addresses), (iii) financial information (e.g., payment and banking details), and (iv) personal and biometric information (e.g., medical records and identifiable facial features). We employ two complementary privacy metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). SAER measures the proportion of queries in which at least one annotated sensitive attribute is disclosed to the central agent, defined as:

SAER=∑q∈Q+I[∃a∈Aq:Expose(a,Zq)=1]∣Q+∣.\mathrm{SAER} = \frac{\sum_{q \in \mathcal{Q}^{+}} \mathbb{I}[\exists a \in \mathcal{A}_q : \mathrm{Expose}(a,Z_q)=1]}{|\mathcal{Q}^{+}|}.

Here, Q+\mathcal{Q}^{+} denotes queries whose retrieved documents contain sensitive attributes, Aq\mathcal{A}_q is the corresponding annotated attribute set, and ZqZ_q represents the evidence returned to the central agent. Expose(a,Zq)=1\mathrm{Expose}(a,Z_q)=1 indicates that attribute aa is disclosed in ZqZ_q.

IPLR employs an independent LLM-as-a-Judge to assess the disclosure of query-irrelevant sensitive attributes before final answer generation. Both metrics range from 0 to 1, with lower values indicating better privacy preservation.

MethodSAER$\downarrow$IPLR$\downarrow$
OpenRAG0.1770.392
LightRAG0.1830.389
OFA0.0330.181
PromptCap0.1750.314
MMAPG0.1310.288
QDMG0.0480.190
Ours0.0070.148

The results show that our method achieves average reductions of 91.5% in SAER and 42.1% in IPLR compared with the baselines, primarily attributable to the local generation of essential query-relevant evidence by each specialist, which effectively minimizes the disclosure of sensitive information. The additional experimental results and analyses have been incorporated into the revision. All annotated documents and corresponding evaluation prompts will be made publicly available in repository to facilitate future research.

Ablation Study of the FSD

To evaluate the effectiveness of the central–local FSD, we conducted an ablation study using PPM-RAG w/o FSD on WebQA, a variant that removes both FSDs and performs retrieval directly at each specialist.

MethodQA-AccQA-FluQA-AllRecallRet.Time↓TTFT↓
PPM-RAG0.6630.6770.5390.9320.0060.287
-w/o FSD0.5890.6460.4940.8610.0210.462

The results demonstrate that our method consistently outperforms PPM-RAG w/o FSD, as FSD-guided retrieval enables the central agent to accurately identify relevant specialists, allowing each specialist to efficiently retrieve the most relevant documents and generate query-relevant local evidence.

Clarification of Retrieval Time and Additional Reporting of TTFT

To clarify, Average Retrieval Time (Avg. Ret. Time) measures the time required to traverse the central and local FSDs, from the completion of sub-query decomposition to the start of local evidence generation. We additionally report Time-to-First-Token (TTFT), which encompasses query decomposition, FSD-guided specialist routing, local evidence generation, and final answer reasoning up to the first output token, and compare it with relevant baselines across both datasets.

MethodWebQAMMDocRAG
Avg. Ret. Time (s)TTFT (s)Avg. Ret. Time (s)TTFT (s)
OpenRAG0.1070.7050.1084.700
LightRAG0.1862.3190.1932.768
OFA0.7560.8940.7810.931
PromptCap0.9341.2570.9261.394
MMAPG0.6010.7910.7281.083
QDMG0.2320.3130.2460.359
Ours0.0060.2870.0100.316

The results show that, our method achieves an average reduction of 63.6% in TTFT. The reduction in TTFT achieved by our method can be partially attributed to its efficient retrieval process. More importantly, each specialist generates only essential query-relevant information, reducing the amount of information the central agent needs to process during final answer generation and thereby further decreasing TTFT. The additional results and analysis have been updated into the revision.

Discussion of Limitations

We acknowledge that the central FSD poses potential privacy leakage risks. We have stated the limitation in the revison.

Thank you for the additional privacy leakage evaluation. However, I am still concerned that the paper does not clearly define what privacy protection means. Even if certain information is relevant to a user's query, it does not necessarily mean that it should be disclosed. In addition, repeated queries may gradually reveal sensitive information. I hope the authors can further clarify these issues. I wish you do not need to conduct any additional experiments to support your claims.

Thank you for your thoughtful comments regarding the privacy-preserving capabilities of our method. We appreciate the opportunity to further clarify the scope of our contributions and the practical scenario addressed by our method. Our work is motivated by a practical central–local setting in which users may submit queries involving their private data and leverage the reasoning capabilities of a central agent powered by a cloud-based LLM to fulfill their requests. In this setting, we consider user-authorized queries that involve accessing personal data distributed across local agents, such as smartphones and personal computers. Meanwhile, our method seeks to minimize the exposure of query-irrelevant information that may contain sensitive personal data and pose privacy risks to the central agent. Specifically, we distinguish between authorizing access to personal data for a particular user query and disclosing all information contained in the accessed documents to the central agent. Accordingly, our privacy-preserving contribution focuses on minimizing the exposure of query-irrelevant sensitive information to the central agent during authorized query processing, rather than filtering all private information or enforcing authorization for potentially malicious queries.

To address this issue, PPM-RAG incorporates query-conditioned evidence generation at local specialists, which serves as an information-minimization gatekeeper before evidence is transmitted to the central agent. Specifically, the central agent decomposes the query into sub-queries and uses the central FSD to identify relevant local specialists. Each selected specialist retrieves documents locally and generates only the essential evidence for its assigned sub-query, rather than sharing the original documents or their full contents. The central agent then aggregates this task-specific evidence for final reasoning, while the original multimodal documents remain on the local. We would like to clarify that our method aims to minimize the unnecessary disclosure of query-irrelevant sensitive information while preserving information necessary to answer authorized queries. This contribution is supported by the additional privacy evaluation presented in our previous response, using two complementary metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). Our method achieves a SAER of 0.007 and an IPLR of 0.148, representing average reductions of 91.5% and 42.1%, respectively, compared with the evaluated baselines. These results empirically demonstrate the effectiveness of our method in minimizing the exposure of query-irrelevant sensitive information under the considered setting. All additional experimental results and clarifications regarding our privacy-preserving claims have been incorporated into the revised manuscript.

Reviewer_FVHWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PPM-RAG, a model-frozen multi-agent framework for privacy-aware multimodal retrieval-augmented generation over distributed data sources. The key idea of this work is a central-local Federated Schema Directory (FSD), in which local agents construct hierarchical semantic indexes and share only root-level summaries and metadata with a central agent. During inference, the central agent decomposes queries and routes them to relevant specialists, which perform local retrieval and return query-specific evidence for answer generation. The authors evaluate on WebQA and MMDocRAG, reporting improvements in retrieval effectiveness, answer quality, and retrieval efficiency compared with the baselines.

Critical strengths

1. Relevant and well-motivated problem. The paper addresses an important practical challenge in multimodal RAG, which is retrieving information distributed across heterogeneous local sources while limiting unnecessary central access to user data.

2. Coherent system architecture. The proposed central-local FSD provides an intuitive framework for coordinating multiple local specialists, without fine-tuning the underlying models.

3. Good empirical performance. PPM-RAG achieves competitive or improved answer quality and retrieval Recall on both WebQA and MMDocRAG. The reported retrieval-time reductions are particularly encouraging.

Critical weaknesses

1. The central 'privacy-preservation' claim is insufficiently validated. The proposed IIR metric is defined as 1 − BLEU, which measures lexical overlap rather than disclosure of sensitive information. UMBRELA similarly evaluates relevance rather than privacy leakage. Moreover, root-level summaries and metadata are shared with the central agent, creating a potential privacy exposure that is not evaluated. Without a clear threat model or direct leakage measurements, the experiments primarily demonstrate query-focused retrieval rather than privacy preservation.

2. Limited technical novelty and missing architectural comparisons. Most individual components (hierarchical summarization, hybrid retrieval, document-to-query generation, query decomposition, and best-first search) are established techniques. While their integration into a distributed FSD is potentially useful, the paper lacks comparisons and ablations that isolate the contribution of the central-local hierarchy from conventional hierarchical or federated retrieval alternatives.

3. Insufficiently explained retrieval-efficiency results. The reported retrieval times of 6–10ms are impressive, but it is unclear whether they include query decomposition, communication between agents, local model inference, and evidence generation. The omission of offline index construction costs and an end-to-end latency breakdown makes it difficult to assess the practical efficiency advantages of the proposed architecture.

4. Insufficient discussion of limitations. The paper does not meaningfully address potential limitations, which could include: information leakage through shared metadata, dependence on generated summaries, potential routing errors, or the costs of updating distributed indexes.

5. Minor formatting issue. A.1 Dataset Statistics in the appendix is only showing a table without any text, where the table is even mingled with the reference. It is better to write explanation text of the table, and start appendix in a new page.

Questions for authors

No questions

5 comments

We thank for your insightful reviews.

Additional Evaluation Highlighting Privacy-Preserving Contribution

We apologize for the confusion about the scope of our privacy-preserving contributions. Specifically, we assume that all queries are benign and do not explicitly request the disclosure of query-irrelevant sensitive information. We conducted additional experiments to evaluate the privacy-preserving performance. We annotated sensitive attributes in the WebQA dataset using a combination of regular expression matching and GPT-6-based annotation, covering four categories: (i) identity and account identifiers (e.g., account IDs and email addresses), (ii) private locations (e.g., residential addresses), (iii) financial information (e.g., payment and banking details), and (iv) personal and biometric information (e.g., medical records and identifiable facial features). We employ two complementary privacy metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). SAER measures the proportion of queries in which at least one annotated sensitive attribute is disclosed to the central agent, defined as:

SAER=∑q∈Q+I[∃a∈Aq:Expose(a,Zq)=1]∣Q+∣.\mathrm{SAER} = \frac{\sum_{q \in \mathcal{Q}^{+}} \mathbb{I}[\exists a \in \mathcal{A}_q : \mathrm{Expose}(a,Z_q)=1]}{|\mathcal{Q}^{+}|}.

Here, Q+\mathcal{Q}^{+} denotes queries whose retrieved documents contain sensitive attributes, Aq\mathcal{A}_q is the corresponding annotated attribute set, and ZqZ_q represents the evidence returned to the central agent. Expose(a,Zq)=1\mathrm{Expose}(a,Z_q)=1 indicates that attribute aa is disclosed in ZqZ_q.

IPLR employs an independent LLM-as-a-Judge to assess the disclosure of query-irrelevant sensitive attributes before final answer generation. Both metrics range from 0 to 1, with lower values indicating better privacy preservation.

MethodSAER$\downarrow$IPLR$\downarrow$
OpenRAG0.1770.392
LightRAG0.1830.389
OFA0.0330.181
PromptCap0.1750.314
MMAPG0.1310.288
QDMG0.0480.190
Ours0.0070.148

The results show that our method achieves average reductions of 91.5% in SAER and 42.1% in IPLR compared with the baselines, primarily attributable to the local generation of essential query-relevant evidence by each specialist, which effectively minimizes the disclosure of sensitive information. The additional experimental results and analyses have been incorporated into the revised manuscript. All annotated documents and corresponding evaluation prompts will be made publicly available in repository to facilitate future research.

Ablation Study of the Federated Schema Directory (FSD)

To evaluate the effectiveness of the central–local FSD, we conducted an ablation study using PPM-RAG w/o FSD on WebQA, a variant that removes both FSDs and performs retrieval directly at each specialist.

MethodQA-AccQA-FluQA-AllRecallRet.Time↓TTFT↓
PPM-RAG0.6630.6770.5390.9320.0060.287
-w/o FSD0.5890.6460.4940.8610.0210.462

The results demonstrate that our method consistently outperforms PPM-RAG w/o FSD, as FSD-guided retrieval enables the central agent to accurately identify relevant specialists, allowing each specialist to efficiently retrieve the most relevant documents and generate query-relevant local evidence.

Clarification of Retrieval Time and Additional Reporting of TTFT

To clarify, Average Retrieval Time (Avg. Ret. Time) measures the time required to traverse the central and local FSDs, from the completion of sub-query decomposition to the start of local evidence generation. We additionally report Time-to-First-Token (TTFT), which encompasses query decomposition, FSD-guided specialist routing, local evidence generation, and final answer reasoning up to the first output token, and compare it with relevant baselines across both datasets.

MethodWebQAMMDocRAG
Avg. Ret. Time (s)TTFT (s)Avg. Ret. Time (s)TTFT (s)
OpenRAG0.1070.7050.1084.700
LightRAG0.1862.3190.1932.768
OFA0.7560.8940.7810.931
PromptCap0.9341.2570.9261.394
MMAPG0.6010.7910.7281.083
QDMG0.2320.3130.2460.359
Ours0.0060.2870.0100.316

The results show that, our method achieves an average reduction of 63.6% in TTFT. The reduction in TTFT achieved by our method can be partially attributed to its efficient retrieval process. More importantly, each specialist generates only essential query-relevant information, reducing the amount of information the central agent needs to process during final answer generation and thereby further decreasing TTFT. The additional results and analysis have been incorporated into the revision.

Discussion of Limitations and Formatting Corrections

We acknowledge the non-negligible computational cost of offline preprocessing and the potential risks of privacy leakage. We have stated these limitations in the revised manuscript, along with the formatting corrections.

Dear Reviewer FVHW,

Regarding the discussion of privacy preservation in our manuscript, we would like to clarify that our work is motivated by a practical central–local setting in which users submit queries involving their private data and leverage the reasoning capabilities of a central agent powered by a cloud-based LLM to fulfill their requests. In this setting, we consider user-authorized queries that involve accessing personal data distributed across local agents, such as smartphones and personal computers. Meanwhile, our method seeks to minimize the exposure of query-irrelevant information that may contain sensitive personal data and pose privacy risks to the central agent. Specifically, we distinguish between authorizing access to personal data for a particular user query and disclosing all information contained in the accessed documents to the central agent. Accordingly, our privacy-preserving contribution focuses on minimizing the exposure of query-irrelevant sensitive information to the central agent during authorized query processing, rather than filtering all private information or enforcing authorization for potentially malicious queries.

To address this issue, PPM-RAG incorporates query-conditioned evidence generation at local specialists, which serves as an information-minimization gatekeeper before evidence is transmitted to the central agent. Specifically, the central agent decomposes the query into sub-queries and uses the central FSD to identify relevant local specialists. Each selected specialist retrieves documents locally and generates only the essential evidence for its assigned sub-query, rather than sharing the original documents or their full contents. The central agent then aggregates this task-specific evidence for final reasoning, while the original multimodal documents remain on the local. We would like to clarify that our method aims to minimize the unnecessary disclosure of query-irrelevant sensitive information while preserving information necessary to answer authorized queries. This contribution is supported by the additional privacy evaluation presented in our previous response, using two complementary metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). Our method achieves a SAER of 0.007 and an IPLR of 0.148, representing average reductions of 91.5% and 42.1%, respectively, compared with the evaluated baselines. These results empirically demonstrate the effectiveness of our method in minimizing the exposure of query-irrelevant sensitive information under the considered setting. All additional experimental results and clarifications regarding our privacy-preserving claims have been incorporated into the revised manuscript.

Finally, we welcome further discussion and would be happy to address any additional questions or concerns. We sincerely appreciate your time and thoughtful consideration.

Thank you for the detailed rebuttal from the authors. My concern regarding the 'privacy-preservation' claim is somewhat addressed. However, another concern regarding limited technical novelty still remains as a crucial holdback. Therefore, I'll maintain my score of 2.

Reviewer_7RR3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a scenario in which privacy preservation is crucial and multimodal external information is distributed across different local sources. They propose PPM-RAG, a model-frozen multi-agent multimodal RAG framework that employs a central–local FSD mechanism design. During online inference, the framework retrieves only query-relevant evidence for final evidence-grounded reasoning. Experimental results demonstrate that PPM-RAG outperforms baseline methods in both accuracy, efficiency, and privacy preservation.

Critical strengths
  1. The problem setting and the proposed method are described clearly and easy to understand. The core challenge stated as “where relevant knowledge resides” and “what information should be revealed” is intuitive.

  2. The proposed PPM-RAG framework is carefully designed to address the targeted scenario. And the methodology is presented with sufficient details.

Critical weaknesses
  1. The methodological novelty is limited, particularly regarding the design of the FSD. Constructing the FSD seems need substantial preprocessing and computation overhead, which may undermine the claimed efficiency advantages.
  2. The experimental evaluation is not sufficient, only use two datasets. In particular, WebQA is commonly used for VQA evaluation, and may not have enough difficulty and scale for retrieval over heterogeneous documents. Also, the selected baselines does not include more recent federal learning related agentic methods.
  3. The privacy-aware evaluation is very limited. Since privacy preservation is presented as one of the paper’s core contributions, more extensive analyses and comparisons with relevant privacy-preserving baselines are needed to substantiate the claims.
Questions for authors
  1. What is the offline cost of constructing the local and central FSDs, including the time and token consumption.
  2. Why only consider the retrieval time during online inference, how about the LLM generation time cost?
4 comments

We sincerely appreciate your valuable feedback.

Clarification of Retrieval Time and Additional Reporting of TTFT

To further evaluate the inference efficiency of PPM-RAG, we additionally report its Time-to-First-Token (TTFT), including query decomposition, FSD-guided specialist routing, local evidence generation and final answer reasoning up to the generation of the first output token, and compare it with relevant baselines across both datasets.

MethodWebQAMMDocRAG
Avg. Ret. Time (s) ↓\downarrowTTFT (s) ↓\downarrowAvg. Ret. Time (s) ↓\downarrowTTFT (s) ↓\downarrow
OpenRAG0.1070.7050.1084.700
LightRAG0.1862.3190.1932.768
OFA0.7560.8940.7810.931
PromptCap0.9341.2570.9261.394
MMAPG0.6010.7910.7281.083
QDMG0.2320.3130.2460.359
Ours0.0060.2870.0100.316

The results demonstrate that, beyond significantly reducing average retrieval time (Avg. Ret. Time), our method achieves an average reduction of 63.6% in TTFT compared with all baselines. The reduction in TTFT achieved by our method can be partially attributed to the efficient FSD-guided retrieval process, which traverses only O(log⁡N)O(\log N) layers with at most KK nodes per layer, requiring fewer comparisons than exhaustive retrieval across all documents. More importantly, each specialist generates only essential query-relevant information, reducing the amount of information the central agent needs to process during final answer generation and thereby further decreasing TTFT. The additional results and analysis have been incorporated into the revision to further demonstrate the inference efficiency of our method.

Clarification of the Offline FSD Construction Process

We acknowledge that the computational cost of offline preprocessing is non-negligible. We have stated the cost of offline FSD construction in the revised manuscript to clarify the efficiency claims. Meanwhile, it is worth noting that offline preprocessing to facilitate online retrieval is common in existing RAG studies and applications. For instance, LightRAG constructs a document-based knowledge graph while MMAPG preprocesses documents to build modality-specific knowledge bases. Similarly, our FSD is constructed only once before online retrieval to improve inference efficiency.

Extensive Comparisons with Federated RAG Baselines

We conducted additional experiments to further evaluate the effectiveness of our proposed PPM-RAG against two federated RAG baselines, FedRRAG [1] and FedMos [2].

MethodWebQAMMDocRAG
QA-AccQA-FluQA-AllRecallAvg. Ret. Time$\downarrow$TTFT$\downarrow$BLEURougeLLLM-ScoreRecallAvg. Ret. Time$\downarrow$TTFT$\downarrow$
FedRRAG0.5010.5440.3690.7390.1170.3160.0860.1670.4650.4210.2010.364
FedMos0.4860.5310.3730.7210.2080.3650.1040.2130.4980.4090.2330.386
Ours0.6630.6770.5390.9320.0060.2870.1500.2720.6560.6120.0100.316

The results show that our method consistently outperforms the two federated RAG baselines in terms of both performance and online efficiency across both datasets.

Additional Evaluation of Privacy-Preserving Performance

We conducted additional experiments to evaluate privacy-preserving performance of our method. We annotated sensitive attributes on WebQA dataset, considering four categories: (i) identity and account identifiers (e.g., account IDs and email addresses), (ii) private locations (e.g., residential addresses), (iii) financial information (e.g., payment and banking details), and (iv) personal and biometric information (e.g., medical records and identifiable facial features). We employ two metrics: Sensitive Attribute Exposure Rate (SAER), which measures the proportion of annotated sensitive attributes explicitly disclosed to the central agent, and Irrelevant Privacy Leakage Rate (IPLR), which uses an independent LLM-as-a-Judge to assess the disclosure of query-irrelevant sensitive attributes before final answer generation. Both metrics range from 0 to 1, with lower values indicating better privacy preservation.

MethodSAER$\downarrow$IPLR$\downarrow$
OpenRAG0.1770.392
LightRAG0.1830.389
OFA0.0330.181
PromptCap0.1750.314
MMAPG0.1310.288
QDMG0.0480.190
FedRRAG0.1470.265
FedMos0.0700.203
Ours0.0070.148

The results show that our method achieves average reductions of 91.5% in SAER and 42.1% in IPLR compared with the baselines, primarily attributable to the local generation of essential query-relevant evidence by each specialist, which effectively minimizes the disclosure of sensitive information.

[1] Peng, C., et al. "Beyond Raw Context Transfer: Representation-based Federated Retrieval-Augmented Generation." NIPS, 2026.

[2] Liang, Z., et al. "FedMosaic: Federated Retrieval-Augmented Generation via Parametric Adapters." ACM SIGIR, 2026.

Dear Reviewer 7RR3,

Regarding the discussion of privacy preservation in our manuscript, we would like to clarify that our work is motivated by a practical central–local setting in which users submit queries involving their private data and leverage the reasoning capabilities of a central agent powered by a cloud-based LLM to fulfill their requests. In this setting, we consider user-authorized queries that involve accessing personal data distributed across local agents, such as smartphones and personal computers. Meanwhile, our method seeks to minimize the exposure of query-irrelevant information that may contain sensitive personal data and pose privacy risks to the central agent. Specifically, we distinguish between authorizing access to personal data for a particular user query and disclosing all information contained in the accessed documents to the central agent. Accordingly, our privacy-preserving contribution focuses on minimizing the exposure of query-irrelevant sensitive information to the central agent during authorized query processing, rather than filtering all private information or enforcing authorization for potentially malicious queries.

To address this issue, PPM-RAG incorporates query-conditioned evidence generation at local specialists, which serves as an information-minimization gatekeeper before evidence is transmitted to the central agent. Specifically, the central agent decomposes the query into sub-queries and uses the central FSD to identify relevant local specialists. Each selected specialist retrieves documents locally and generates only the essential evidence for its assigned sub-query, rather than sharing the original documents or their full contents. The central agent then aggregates this task-specific evidence for final reasoning, while the original multimodal documents remain on the local. We would like to clarify that our method aims to minimize the unnecessary disclosure of query-irrelevant sensitive information while preserving information necessary to answer authorized queries. This contribution is supported by the additional privacy evaluation presented in our previous response, using two complementary metrics: Sensitive Attribute Exposure Rate (SAER) and Irrelevant Privacy Leakage Rate (IPLR). Our method achieves a SAER of 0.007 and an IPLR of 0.148, representing average reductions of 91.5% and 42.1%, respectively, compared with the evaluated baselines. These results empirically demonstrate the effectiveness of our method in minimizing the exposure of query-irrelevant sensitive information under the considered setting. All additional experimental results and clarifications regarding our privacy-preserving claims have been incorporated into the revised manuscript.

Finally, we welcome further discussion and would be happy to address any additional questions or concerns. We sincerely appreciate your time and thoughtful consideration.

Thanks for the detailed response from authors. The experiment comparison with federated RAG baselines are helpful. Have you not yet uploaded the revised manuscript since I could not find the cost of offline FSD construction? My concern regarding evaluation on additional benchmarks remains unaddressed. I will maintain my score.

We sincerely thank the reviewer for acknowledging the improvements made in our revised manuscript.

Additional Experimental Evaluation on OpenRAGBench

We apologize for reporting these results at the final stage of the rebuttal period, as the additional experiments required substantial computational effort.

To address this concern, we conducted additional experiments on the OpenRAGBench dataset [1], which contains multimodal documents comprising text, images, and tables. Compared with the WebQA and MMDocRAG datasets used in our original experiments, OpenRAGBench additionally incorporates the table modality, providing a more comprehensive evaluation of our method's effectiveness and generalizability across heterogeneous multimodal documents.

MethodROUGE-L ↑Recall ↑Avg. Ret. Time (s) ↓TTFT (s) ↓
OpenRAG0.2990.8720.1060.965
LightRAG0.2870.8690.2012.683
OFA0.3260.9200.7410.996
PromptCap0.2630.8830.7191.172
MMAPG0.2950.8960.8781.034
QDMG0.3140.9180.3740.867
FedRRAG0.3080.9010.3400.793
FedMos0.3010.8970.3120.876
Ours0.3420.9320.0120.643

Compared with all eight baselines on Qwen3.5-9B, our method achieves average relative improvements of 14.75% in ROUGE-L and 4.23% in Recall, while reducing Time-to-First-Token (TTFT) by 37.39% on average. These results demonstrate that our method improves both answer quality and retrieval effectiveness while substantially reducing inference latency.

Offline Construction Cost

Given a collection of N documents, our FSD construction consists of two stages. First, each document requires only one LLM call to jointly extract its summary and associated metadata, including keywords and document-to-query (D2Q) information, resulting in N calls. Second, during bottom-up FSD construction, parent-node summaries and metadata are generated hierarchically. During bottom-up construction of the balanced binary FSD, each parent node requires one LLM call to aggregate the summaries and metadata of its child nodes, resulting in an additional (N-1) LLM calls. Therefore, the total number of LLM calls is N + N -1 = O(N). The number of LLM calls is comparable to that required by existing LLM-assisted offline preprocessing approaches, such as LightRAG, one of our baselines, which employs LLMs to extract entities and relationships for graph-based index construction. The discussion of offline construction costs has been included in the Limitations section of the appendix in the revision.

[1] Vectara. "OpenRAGBench: A Multimodal Benchmark for Retrieval-Augmented Generation." Hugging Face, 2025. https://huggingface.co/datasets/vectara/open_ragbench.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.