- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes Byzantine-robust conformal prediction for federated RAG with frozen scoring pipelines and a fixed, unknown set of malicious nodes. All nodes score shared calibration questions. The hub considers every possible honest group consistent with a declared corruption budget and retains a candidate if the same group supports it during both calibration and prediction. Validation includes synthetic experiments, four multiple-choice benchmarks using GPT-2, and an OpenBookQA experiment using prompt-injected Llama-3.2-3B-Instruct nodes.
Critical strengths
- This paper explores a realistic hospital scenario involving federated RAG and sensitive patient information.
- The experiments vary attacks, corruption budgets, dependence, and quantization, and include deliberate assumption violations.
Critical weaknesses
- The four benchmark experiments use GPT-2 (Tab. 16), whose accuracy is approximately 24–28% against 25% chance, with prediction sets retaining roughly 3.6 of four options. The stronger-model evaluation is limited to Llama-3.2-3B-Instruct on OpenBookQA. Without stronger LLM/RAG baselines under comparable evidence access and coverage targets, it remains unclear whether the method produces useful predictions beyond reducing overhead in weak scoring systems.
- Worst case inflation can produce full answer sets, while exact enumeration scales combinatorially. The paper establishes a focused theoretical improvement, but does not sufficiently demonstrate how consequential that improvement is in realistic federated deployments.
- The experimental presentation is too fragmented, key results are relegated to extensive appendices, requiring repeated navigation between sections. A consolidated main-text table would make the practical significance of the improvements easier to assess.
Questions for authors
- Do authors have results with stronger models on the medical benchmarks, or comparisons against stronger LLM/RAG systems at matched coverage and comparable evidence access?
- Why must the method keep every answer that a possible honest group would accept, instead of simply meeting the target coverage?
- In practice, how would an operator estimate the maximum number of malicious nodes and ensure that the honest nodes remain the same during calibration and prediction?
8 comments
Response to Reviewer tHFz (2/7): W1
W1. Weak scorers, and stronger baselines with comparable evidence and coverage
Thank you for pressing on this. With a strong scorer, the method gives useful predictions, not just lower overhead. Its sets are small where the scorer is accurate, often a single answer, and they stay valid under attack, where averaging and the median do not. We re-ran the four benchmarks with three stronger scorers (the 7B on three) and compared fixed-set, at the same 90% target, with pipelines built on the same corpus (no retrieval, a single node, the unguarded 16-node mean, and retrieval from the whole pooled corpus) and with seven aggregation rules at the hub. Single-pipeline methods such as TRAQ, C-RAG and RobustRAG give no coverage guarantee when some of the hub's sources may send arbitrary reports; each can run inside a node, so they complement this layer.
- Useful predictions. With Qwen2.5-14B at every node and two of 16 nodes corrupt, fixed-set returns a single answer for 83% of OpenBookQA and 63% of MMLU-Med questions (1.18 and 1.49 answers on average; GPT-2 kept 3.60–3.66), at coverage 0.91–0.92 (Table 1).
- Matched baselines at the same target, with nobody attacking (Table 24). Mean answers kept at 90%:
| Benchmark, scorer | No retrieval | Whole corpus | 16-node mean | Fixed-set () |
|---|---|---|---|---|
| OpenBookQA, Llama-3.2-3B | 1.752 | 1.803 | 1.719 | 1.849 |
| OpenBookQA, Qwen2.5-7B | 1.282 | 1.576 | 1.412 | 1.530 |
| OpenBookQA, Qwen2.5-14B | 1.101 | 1.159 | 1.113 | 1.175 |
| MMLU-Med, Llama-3.2-3B | 1.862 | 1.959 | 1.944 | 2.098 |
| MMLU-Med, Qwen2.5-7B | 1.536 | 1.788 | 1.738 | 1.854 |
| MMLU-Med, Qwen2.5-14B | 1.392 | 1.414 | 1.390 | 1.476 |
| MedQA-US, Llama-3.2-3B | 2.602 | 2.669 | 2.633 | 2.786 |
| MedQA-US, Qwen2.5-7B | 2.457 | 2.566 | 2.550 | 2.662 |
| MedQA-US, Qwen2.5-14B | 2.198 | 2.249 | 2.176 | 2.291 |
| MedMCQA, Llama-3.2-3B | 2.648 | 2.789 | 2.733 | 2.882 |
| MedMCQA, Qwen2.5-14B | 2.588 | 2.612 | 2.662 | 2.766 |
On these pairs, the Byzantine guarantee costs fixed-set only 0.06–0.15 answers per question over the unguarded 16-node mean (at most 0.23 on any row of Table 24, the literature tasks included), and 0.07–0.32 over the model without retrieval. A single node, which sees only its own shard, is 0.9–4.6 points less accurate than the 16-node mean on every exam pair, and with the Qwen scorers it keeps more answers than fixed-set on all seven pairs.
- Under attack, the averaging rules and the median fall below the target; the proposed rules stay at 0.90–0.91 (Table 25). Coverage at 90% under low–high reporting, over the eleven benchmark–scorer pairs:
| Aggregation rule at the hub | Coverage |
|---|---|
| Plain mean | 0.65–0.86 |
| Winsorized mean | 0.82–0.86 |
| Trimmed mean | 0.82–0.87 |
| Median of means | 0.82–0.86 |
| Coordinate-wise median | 0.86–0.88 |
| Krum | 0.89–0.91 |
| Local-marginal rule (no alignment across nodes) | 0.91–0.96 |
| Fixed-set, joint-threshold, deletion () | 0.90–0.91 |
Krum, which follows one selected node, stays near the target here but has no guarantee: when the two nodes send the maximum at calibration only, it falls to 0.879 (7B, MedQA-US). The local-marginal rule, which ignores which calibration questions the nodes share, has no guarantee either, and its sets are larger than fixed-set's on ten of the eleven pairs. The two other rules that are valid by construction keep larger sets: under this attack, the -value merger keeps 2.051–4.000 answers with the Qwen scorers, and the guarded symmetric rule (a trimmed mean plus a worst-case guard) all or nearly all four on six of the seven Qwen pairs.
In the revision: Table 1 (p. 10) and Section 5 (lines 451–458 and 512); Tables 24–25 (Appendix T.2, pp. 62–63); Section 1 and Appendix A for TRAQ, C-RAG and RobustRAG.
Response to Reviewer tHFz (3/7): W1 (cont.) and Q1
W1 (continued). Two tasks where retrieval matters, and what the exam benchmarks show
On two literature tasks, the federation adds accuracy as well as validity under attack. We added the two tasks of the same medical suite that need the literature: BioASQ yes/no (518 held-out questions, 2 options) and PubMedQA* (400 questions, 3 options). Each of 16 nodes retrieves four passages from its own shard of PubMed (Table 41):
| Task, scorer | Accuracy: no retrieval / 16-node mean / whole corpus | Answers kept at 90%: no retrieval / fixed-set (2 corrupt) | Low–high coverage: fixed-set / plain mean |
|---|---|---|---|
| BioASQ yes/no, Llama-3.2-3B | 53.3 / 71.0 / 83.8% | 1.750 / 1.512 | 0.901 / 0.825 |
| BioASQ yes/no, Qwen2.5-14B | 51.4 / 73.9 / 87.3% | 1.735 / 1.457 | 0.901 / 0.823 |
| PubMedQA*, Llama-3.2-3B | 36.0 / 52.5 / 76.0% | 2.236 / 2.303 | 0.906 / 0.796 |
| PubMedQA*, Qwen2.5-14B | 9.5 / 39.2 / 78.5% | 2.837 / 2.635 | 0.911 / 0.633 |
- Smaller sets than the model alone, at the guarantee. With two corrupt nodes, fixed-set keeps fewer answers than the model without retrieval on three of the four pairs, at coverage 0.93–0.95. Under low–high it covers 0.90–0.91, where the plain mean falls to 0.63–0.83.
- Retrieval helps when it finds the evidence. A node whose four passages include an article of the question's gold evidence answers 75.7–91.1% of the questions correctly; the other nodes answer 36.7–68.4% correctly.
- Replicating articles across nodes helps further. The one exception above, PubMedQA* with Llama-3.2-3B (2.303 answers against 2.236 without retrieval), arises because each article sits with only one node of 16, and averaging dilutes it. When each article is held by 2, 4 or 8 nodes instead, the mean is right on 58.8, 63.2 and 71.0% of the questions (52.5% with one), and fixed-set keeps 2.135–2.192 answers, fewer than without retrieval. On BioASQ, accuracy rises from 71.0% to 81.1%, and fixed-set's sets shrink from 1.510 to 1.278 answers.
- Pooling the whole corpus is more accurate still (76.0–87.3%), but that is exactly what a federation cannot do; replication closes part of the gap.
- What the four exam benchmarks test. On OpenBookQA and the three medical exam benchmarks, the 16-node mean is at most 0.7 points more accurate than the model without retrieval, and its sets are at most 0.034 answers smaller (Table 24). These benchmarks therefore test the aggregation rule, the paper's contribution; the two literature tasks test what federated retrieval adds.
In the revision: Table 1 (p. 10) and Section 5 (lines 457–458 and 512); Table 41 (Appendix T.10, p. 81).
Q1. Stronger models on the medical benchmarks; stronger systems at matched coverage
Yes to both. With Qwen2.5-14B, fixed-set keeps 1.49 answers on MMLU-Med at 90%, a single answer for 63% of the questions, and 2.31 and 2.77 on MedQA-US and MedMCQA, where that scorer is right on 64% and 54% of the questions (Table 1, which also has Llama-3.2-3B and Qwen2.5-7B). Our response to W1 gives the comparison at matched coverage and evidence: with nobody corrupt, the guarantee costs fixed-set 0.06–0.23 answers per question over the unguarded 16-node mean, and when the corrupt nodes report low scores at calibration and high ones at the query, the averaging rules and the median fall below the target on the exam benchmarks while the proposed rules stay at 0.90–0.91 (Tables 24–25).
In the revision: Table 1 (p. 10); Tables 24–25 (Appendix T.2, pp. 62–63).
Response to Reviewer tHFz (1/7)
Dear Reviewer tHFz,
Thank you for the careful and specific review, and for the comparisons you proposed: stronger models on the medical benchmarks, baselines at matched coverage and evidence, including retrieval from the whole pooled corpus, and one consolidated table. We ran or added each of them, and took the presentation score to heart.
In brief.
Terms.
New text is in blue in the revised PDF.