- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SynFedRAG, a federated GraphRAG framework for knowledge-intensive QA over decentralized private corpora. The main idea is to construct a persistent synthetic knowledge graph that captures retrieval knowledge distributed across clients and can be queried locally without contacting other clients at inference time. SynFedRAG first condenses each client’s private entity-sentence-passage graph into a compact summary, aggregates these summaries to initialize a shared synthetic graph, and then jointly refines the synthetic graph and a lightweight graph-guided prompt module through federated QA supervision while keeping the LLM frozen.
Critical strengths
- The paper identifies a clear and practically relevant gap in federated RAG. The distinction between transferring model parameters and transferring retrieval knowledge is well motivated. In particular, the goal of learning a persistent retrieval resource that remains locally queryable at inference time gives SynFedRAG a clear positioning relative to federated approaches that still require query-time access to remote repositories.
- The two-stage design is reasonably well aligned with the motivation. Phase I addresses communication and decentralization through client-side graph condensation and synthetic-graph initialization, while Phase II makes the resulting graph task-aware through federated QA supervision. This is a more coherent design than simply aggregating compressed client graphs once and treating the resulting representation as fixed.
Critical weaknesses
If the authors adequately address the above concerns, I would be willing to raise my score.
- The robustness and scalability evaluation is still limited for a federated system. The main experiments consider three clients, and the evaluated heterogeneity is primarily induced through random and topic-skewed corpus partitions. I would like to see stronger stress tests involving different numbers of clients, partial client participation, client dropout, highly imbalanced corpus sizes, noisy or corrupted local graphs, and potentially low-quality client summaries. These settings are particularly relevant here because a persistent synthetic graph aggregates information from multiple clients and may accumulate errors from unreliable participants.
- The efficiency analysis is useful but not yet sufficient to establish the practical advantage of the full pipeline. The paper reports compression-versus-QA trade-offs and shows that one-time graph communication can be reduced substantially. However, SynFedRAG contains several potentially expensive stages, including private graph construction, graph condensation, synthetic-graph optimization, repeated graph refinement, and prompt-module synchronization. A more convincing evaluation would compare end-to-end training time, inference latency, peak GPU memory, preprocessing/graph-construction cost, and total communication against DGRAG, FD-RAG, and local GraphRAG baselines.
- The paper insufficiently discusses recent federated graph condensation and compression methods. This is important because one of the central methodological components of SynFedRAG is precisely client-side graph condensation followed by server-side construction of a shared synthetic graph. Recent methods such as FedGVD, FedC4, FedGC, and FedGM address closely related problems of communicating, aggregating, or learning compact graph knowledge in federated settings. Although these methods do not necessarily target GraphRAG or QA directly, the conceptual overlap with Phase I is substantial enough that they should at least be carefully discussed. Ideally, some of them should be adapted as alternative shared-graph construction strategies while keeping the downstream SynFedRAG retrieval/LLM pipeline fixed.
Questions for authors
See Weakness.
8 comments
Response to W1 (2/2): Imbalanced corpora and corrupted clients
W1(2/2): Imbalanced corpus sizes (Table R3.3). As one client's share of the corpus grows, more of its questions become answerable locally, consistent with Section 4.1. The smaller clients' F1 under SynFedRAG decreases only moderately and remains clearly above the federated RAG baselines at every ratio, showing that the shared graph compensates for a limited local corpus.
Table R3.3: Imbalanced client corpus sizes (, random partition). Articles are allocated with size ratio , so the largest client holds of the corpus. Questions keep round-robin ownership, so every client owns the same number of questions. Large reports F1 on the questions of the largest client, and Small the average F1 on the questions of the two smaller clients.
| HotpotQA | MuSiQue | 2WikiMultiHopQA | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Size ratio | Method | Overall | Small | Large | Overall | Small | Large | Overall | Small | Large |
| 1:1:1 (33%) | DGRAG | 42.17 ± 0.26 | 42.11 ± 0.31 | 42.29 ± 0.42 | 31.38 ± 0.17 | 31.32 ± 0.21 | 31.50 ± 0.29 | 37.68 ± 0.19 | 37.62 ± 0.23 | 37.80 ± 0.32 |
| FD-RAG | 42.30 ± 0.67 | 42.24 ± 0.74 | 42.42 ± 0.96 | 30.04 ± 0.32 | 29.98 ± 0.37 | 30.16 ± 0.51 | 37.04 ± 0.13 | 36.98 ± 0.17 | 37.16 ± 0.24 | |
| SynFedRAG | 47.43 ± 0.13 | 47.37 ± 0.16 | 47.55 ± 0.23 | 35.61 ± 0.21 | 35.55 ± 0.25 | 35.73 ± 0.34 | 40.85 ± 0.47 | 40.79 ± 0.53 | 40.97 ± 0.71 | |
| 2:1:1 (50%) | DGRAG | 42.41 ± 0.29 | 41.91 ± 0.36 | 43.43 ± 0.44 | 31.02 ± 0.22 | 31.80 ± 0.27 | 29.47 ± 0.33 | 37.83 ± 0.24 | 37.53 ± 0.29 | 38.45 ± 0.35 |
| FD-RAG | 42.70 ± 0.71 | 41.86 ± 0.82 | 44.38 ± 0.93 | 30.24 ± 0.35 | 29.81 ± 0.43 | 31.09 ± 0.49 | 37.21 ± 0.18 | 36.87 ± 0.23 | 37.88 ± 0.27 | |
| SynFedRAG | 48.00 ± 0.18 | 46.81 ± 0.22 | 50.40 ± 0.27 | 35.66 ± 0.24 | 35.55 ± 0.31 | 35.89 ± 0.36 | 40.95 ± 0.51 | 40.75 ± 0.58 | 41.36 ± 0.69 | |
| 4:1:1 (67%) | DGRAG | 42.75 ± 0.34 | 41.62 ± 0.45 | 45.02 ± 0.41 | 30.50 ± 0.26 | 32.23 ± 0.34 | 27.06 ± 0.31 | 38.04 ± 0.27 | 37.35 ± 0.37 | 39.40 ± 0.34 |
| FD-RAG | 43.26 ± 0.76 | 41.40 ± 0.93 | 47.00 ± 0.88 | 30.52 ± 0.41 | 29.57 ± 0.52 | 32.42 ± 0.47 | 37.43 ± 0.22 | 36.68 ± 0.31 | 38.94 ± 0.26 | |
| SynFedRAG | 48.80 ± 0.21 | 46.14 ± 0.29 | 54.13 ± 0.25 | 35.74 ± 0.29 | 35.49 ± 0.37 | 36.24 ± 0.33 | 41.09 ± 0.55 | 40.63 ± 0.66 | 41.99 ± 0.72 | |
| 8:1:1 (80%) | DGRAG | 43.19 ± 0.38 | 41.48 ± 0.54 | 46.61 ± 0.47 | 29.84 ± 0.31 | 32.45 ± 0.42 | 24.64 ± 0.37 | 38.30 ± 0.33 | 37.27 ± 0.46 | 40.36 ± 0.39 |
| FD-RAG | 43.98 ± 0.84 | 41.16 ± 1.07 | 49.63 ± 0.94 | 30.89 ± 0.46 | 29.45 ± 0.63 | 33.76 ± 0.55 | 37.72 ± 0.27 | 36.59 ± 0.38 | 39.99 ± 0.33 | |
| SynFedRAG | 49.83 ± 0.27 | 45.80 ± 0.36 | 57.87 ± 0.31 | 35.83 ± 0.33 | 35.45 ± 0.45 | 36.59 ± 0.39 | 41.26 ± 0.61 | 40.58 ± 0.77 | 42.63 ± 0.74 |
Noisy or corrupted clients and low-quality summaries (Table R3.4). We corrupt one client (C2). The clean clients' F1 decreases with corruption severity, confirming that the shared graph carries C2's knowledge, but it remains above the setting where C2 is removed, so an unreliable client does not harm others more than losing its contribution. Most of the damage stays with the corrupted client itself. We will discuss robust aggregation in the revision.
Table R3.4: Robustness to an unreliable client (, random partition). C2 is corrupted; C1 and C3 are clean. Clean / C2: F1 on the clean clients' / C2's own questions. C2 removed: C2 does not participate. Noisy graph: a fraction of C2's graph edges is randomly rewired. Wrong facts: in a fraction of C2's passages, the answer entity is replaced with a wrong one of the same type. Low-quality summary: C2 uploads a condensed graph built from random anchors.
| HotpotQA | MuSiQue | 2WikiMultiHopQA | |||||
|---|---|---|---|---|---|---|---|
| Setting | Level | Clean | C2 | Clean | C2 | Clean | C2 |
| No corruption | – | 47.52 ± 0.17 | 47.25 ± 0.29 | 35.70 ± 0.26 | 35.43 ± 0.38 | 40.94 ± 0.53 | 40.67 ± 0.71 |
| C2 removed | – | 39.52 ± 0.62 | n/a | 22.60 ± 0.84 | n/a | 34.24 ± 0.77 | n/a |
| Noisy graph | 30% edges | 46.62 ± 0.24 | 44.15 ± 0.46 | 34.48 ± 0.33 | 32.95 ± 0.57 | 40.22 ± 0.58 | 38.00 ± 0.83 |
| 60% edges | 45.12 ± 0.36 | 40.25 ± 0.71 | 32.58 ± 0.47 | 29.83 ± 0.88 | 39.02 ± 0.64 | 34.65 ± 1.02 | |
| Wrong facts | 20% passages | 45.82 ± 0.31 | 43.55 ± 0.58 | 33.49 ± 0.42 | 32.47 ± 0.66 | 39.58 ± 0.61 | 37.49 ± 0.91 |
| 50% passages | 42.72 ± 0.49 | 37.65 ± 0.93 | 29.46 ± 0.68 | 27.75 ± 1.07 | 37.10 ± 0.79 | 32.41 ± 1.18 | |
| Low-quality summary | random anchors | 45.72 ± 0.43 | 46.35 ± 0.37 | 33.70 ± 0.59 | 34.61 ± 0.49 | 39.54 ± 0.72 | 39.87 ± 0.76 |
Response to W2: End-to-end cost and inference latency
W2. We thank the reviewer for this suggestion. Table R3.5 reports the end-to-end offline cost of SynFedRAG, DGRAG, FD-RAG, and G-Retriever under the same hardware and backbone.
Offline computation. The stages listed by the reviewer that precede training, namely private graph construction, graph condensation, and synthetic-graph initialization, together cost less than the preprocessing of DGRAG and FD-RAG, since SynFedRAG builds its graphs without LLM generation, whereas both baselines use the LLM to produce summaries or QA memories. Most of SynFedRAG's cost lies in training, which consists of two parts. Training the graph-guided prompt module costs about the same as G-Retriever's training, since both train a lightweight graph encoder and projector through the frozen LLM. The additional cost comes from refining the shared synthetic graph, where gradient matching requires extra passes through the frozen LLM. As a result, its total offline cost is close to that of DGRAG and higher than that of FD-RAG and G-Retriever, and its peak memory remains within a single 24 GB GPU. This cost is paid once before deployment, in return for the accuracy gains reported in the main results.
Communication. Total communication is reported in Appendix D.1 (Tables 11 and 12). For retrieval knowledge, SynFedRAG transmits 40.22 MiB in total, compared with 62.37 MiB for DGRAG and 75.13 MiB for FD-RAG. Including prompt-module synchronization, its total offline communication is 701.0 MiB, which remains below the 770.0 MiB required by the RoLoRA reader adaptation used by DGRAG and FD-RAG alone.
Table R3.5: End-to-end offline cost on HotpotQA (, random partition, single RTX 4090, frozen Qwen2.5-7B). Preproc.: building the retrieval structure: local graph construction and text encoding for G-Retriever, knowledge-graph extraction and subgraph summarization for DGRAG, hypergraph learning and QA-memory generation for FD-RAG, and tri-graph construction, graph condensation, and synthetic-graph initialization for SynFedRAG. Training: all remaining offline training. G-Retriever keeps the LLM frozen and trains its graph encoder and projector with federated averaging. Time is summed over clients and the server; peak GPU memory is the maximum over all stages.
| Time (min) | ||||
|---|---|---|---|---|
| Method | Preproc. | Training | Total | Peak GPU mem. (GB) |
| G-Retriever | 27.8 | 352.4 | 380.2 | 21.7 |
| DGRAG | 182.4 | 301.5 | 483.9 | 19.8 |
| FD-RAG | 41.3 | 301.5 | 342.8 | 19.8 |
| SynFedRAG (Ours) | 34.6 | 458.0 | 492.6 | 22.6 |
Table R3.6: Inference latency on HotpotQA (, random partition, single RTX 4090, frozen Qwen2.5-7B, batch size 1). Latency is the mean time per test question, including retrieval and generation and excluding network delay. Query-time comm.: whether answering a question may require contacting the server or other clients.
| Method | Latency (s/query) | Query-time comm. |
|---|---|---|
| LinearRAG | 0.92 | No |
| G-Retriever | 4.87 | No |
| DGRAG | 2.63 | Yes |
| FD-RAG | 1.17 | No |
| SynFedRAG (Ours) | 0.94 | No |
Inference latency. Table R3.6 reports inference latency. After deployment, SynFedRAG answers each query locally, without communicating with the server or other clients. Its latency is close to that of LinearRAG, which uses the same local retrieval: the additional components operate only on a small evidence subgraph and the compact synthetic graph, and contribute a soft token rather than additional retrieved text, so access to cross-client knowledge adds little inference cost. FD-RAG is slightly slower, since it adds retrieved QA memories to the prompt. DGRAG is slower still, since it generates several local answers to decide whether to escalate a query and retrieves from remote clients for escalated queries. G-Retriever has the highest latency, since it solves a subgraph optimization over the local graph for each query.
Response to W1 (1/2): Number of clients, partial participation, and dropout
W1(1/2): We thank the reviewer for this suggestion. We added stress tests covering each of the requested settings. Following the protocol of the main experiments, all results are reported as mean ± standard deviation over three runs with seeds 42, 43, and 44, where each seed controls both the corpus partition and training.
Number of clients (Table R3.1). As the number of clients grows, each client holds a smaller share of the corpus and almost all questions require evidence from other clients, so all methods degrade. Local RAG and GraphRAG baselines degrade the most, since each client can only retrieve from its own shrinking corpus. SynFedRAG remains the best method at every scale and maintains the highest F1 at every evaluated client count, because the shared synthetic graph keeps cross-client knowledge available locally.
Partial participation and client dropout (Table R3.2). Our main experiments used full participation; here, only part of the clients contributes in each round. F1 decreases gradually as fewer client updates reach the server, following the expected fraction of participating clients. At a similar expected fraction, random dropout performs slightly worse than client sampling, since the number of received updates varies from round to round. Cross-client F1 decreases by a similar amount as overall F1, suggesting that partial participation does not disproportionately affect questions that rely on other clients' knowledge.
Table R3.1: Question-answering F1 scores with different numbers of clients on HotpotQA, MuSiQue, and 2WikiMultiHopQA under random partition.
Table R3.2: Robustness of SynFedRAG to partial participation and client dropout (M=10 clients, T=10 rounds, random partition). Reference: all clients participate in every round (q=1.0); it corresponds to the 10-client setting in Table R3.1. Partial participation: in each round, only a fraction q of the clients is sampled for training. Random dropout: each sampled client fails to send its update with probability p, and the server aggregates the remaining updates. We report F1 on all test questions (Overall) and on questions whose supporting evidence is held by other clients (Cross), as mean ± standard deviation over three runs.