- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work investigates backdoor attacks against retrievers in agentic Retrieval-Augmented Generation (RAG). The proposed attack assumes that an attacker can provide a malicious retriever checkpoint without modifying the search agent or knowledge database.
The authors propose three attacks: an untargeted attack that suppresses relevant evidence, a targeted attack that repeatedly retrieves a selected document, and a round-control attack that prolongs the agent's search process. The paper further proposes Decoy Overwrite–Unlearn (DOU), which injects and removes an auxiliary backdoor to reduce the detectability of the original backdoor while preserving its malicious behavior.
The authors evaluate the attacks on HotpotQA and TriviaQA using E5 and Contriever. The attacks substantially reduce answer accuracy, achieve nearly 100% persistent target retrieval, and increase search costs. DOU also reduces detection signals across several evaluated backdoor detectors.
Critical strengths
-
This work investigates retriever backdoors in agentic RAG, considering both answer manipulation and search efficiency. The three attack objectives demonstrate different security risks introduced by compromised retrievers.
-
The proposed DOU method is interesting because it exploits backdoor purification to conceal malicious behavior. The paper also evaluates multiple detection methods and reports the trade-off between concealment and attack effectiveness.
Critical weaknesses
-
The attack effectiveness evaluation is limited to HotpotQA and TriviaQA using a single agentic search framework. Furthermore, the attack relies on Agentic-R's design, which includes the original question in every retrieval round, allowing the trigger to persist. Its effectiveness in other agentic search frameworks remains unclear.
-
The backdoor detection evaluation is limited. The TriviaQA and MuSiQue detection sets contain only 8 and 30 naturally triggered questions, respectively. Although the experiments are repeated with different seeds, they reuse the same question sets. Furthermore, Neural Cleanse and PICCOLO share the same trigger-inversion procedure, limiting the diversity of evaluated detectors.
Questions for authors
-
Can the authors evaluate whether the attacks remain effective when the retriever receives only agent-generated queries? Since Agentic-R includes the original question in every retrieval round, the backdoor trigger remains present throughout the search process.
-
Can the authors provide additional end-to-end evaluations on benchmarks such as NQ, MuSiQue, or 2WikiMultiHopQA to demonstrate generalizability?
-
How reliable are the detection results given the limited number of naturally triggered questions? Can the authors provide evidence using larger evaluation sets and more diverse detection methods?
2 comments
Response to Reviewer g181 — Weaknesses
W1. Limited benchmarks and dependence on Agentic-R’s inclusion of the original question.
The attack transfers beyond Agentic-R to DeFT-RAG and remains effective on the additional 2WikiMultiHopQA benchmark. DeFT-RAG reformulates subsequent queries after receiving the original question in the first round. Using the same poisoned E5 on 1,000 paired HotpotQA questions, triggering increases target retrieval from 4.80% to 100% and reduces F1 from 50.00 to 33.22. Target hits occur in 64.44% of retrieval rounds.
We added these results to Section 4.2.2, “Transfer to DeFT-RAG,” and Table 5.
| Setting | EM | F1 | Target Hit | Hit/Round | Top-1 Hit |
|---|---|---|---|---|---|
| Backdoor-Clean | 37.70 | 50.00 | 4.80 | 5.65 | 4.30 |
| Backdoor-Trigger | 23.70 | 33.22 | 100.00 | 64.44 | 100.00 |
Target Hit and Top-1 Hit are sample-level metrics; Hit/Round uses executed retrieval rounds.
We also evaluated 200 paired 2WikiMultiHopQA questions using K2-Horizon-7B and Qwen3.5-27B, without retraining the poisoned E5. Triggering reduces F1 by 31.58 and 34.68 points, respectively, relative to Backdoor-Clean. The target ranks first in every executed triggered retrieval round.
We added these results to Section 4.2.2, “Transfer to additional search agents and a multi-hop benchmark,” and Table 4.
| Agent | Setting | EM | F1 | Target Hit | Target@1/Round |
|---|---|---|---|---|---|
| K2-Horizon-7B | Original Retriever | 46.50 | 57.31 | 0.00 | 0.00 |
| K2-Horizon-7B | Backdoor-Clean | 44.00 | 55.27 | 0.50 | 0.24 |
| K2-Horizon-7B | Backdoor-Trigger | 17.50 | 23.69 | 99.50 | 100.00 |
| Qwen3.5-27B | Original Retriever | 57.50 | 66.49 | 0.00 | 0.00 |
| Qwen3.5-27B | Backdoor-Clean | 55.00 | 63.57 | 0.00 | 0.00 |
| Qwen3.5-27B | Backdoor-Trigger | 23.50 | 28.89 | 97.50 | 100.00 |
W2. Small detection sets and shared trigger inversion in NC/PICCOLO.
The concealment effect persists with more naturally triggered questions and across different detection mechanisms. We expanded detection to the full MuSiQue development set: 2,411 unique questions, including 73 naturally triggered positives. DOU reduces separation for six of seven evaluated detectors.
We added these results to Appendix D.3, “Concealment persists on the full MuSiQue development set,” and Table 14, complementing the main-text results in Table 9.
| Detector | Original backdoor | DOU80 |
|---|---|---|
| Activation Clustering | 50.00 | 10.89 |
| Black-box TopK Instability | 44.00 | 9.86 |
| ONION | 6.67 | 6.67 |
| PICCOLO | 39.68 | 20.72 |
| RAP | 11.69 | 5.16 |
| STRIP | 20.14 | 15.56 |
| Spectral Signatures | 50.00 | 22.53 |
Values are detector separation ; lower values indicate weaker separation. DOU80 uses 80 overwrite and 80 unlearn steps.
NC and PICCOLO originate from distinct detection methods, although both use trigger inversion and our retrieval adaptations share this component. Our evaluation also includes Activation Clustering, Spectral Signatures, Black-box TopK Instability, RAP, STRIP, and ONION. Thus, the evidence for concealment extends across multiple detection mechanisms rather than relying solely on NC and PICCOLO.
Response to Reviewer g181 — Questions
Thank you for your careful review and constructive suggestions. We address your questions below.
Q1. Can the attacks remain effective when the retriever receives agent-generated queries?
Yes. The attack remains effective under DeFT-RAG, which receives the original question in the first round and reformulates subsequent queries. Using the same poisoned E5 on 1,000 paired HotpotQA questions, triggering increases target retrieval from 4.80% to 100% and reduces F1 from 50.00 to 33.22. Target hits occur in 64.44% of retrieval rounds, with an average of 4.586 rounds per triggered run.
We added these results to Section 4.2.2, “Transfer to DeFT-RAG,” and Table 5.
Target Hit and Top-1 Hit are sample-level metrics; Hit/Round uses executed retrieval rounds.
Q2. Can you provide additional end-to-end evaluations on benchmarks such as NQ, MuSiQue, or 2WikiMultiHopQA?
Yes. We added end-to-end evaluations on 2WikiMultiHopQA with two additional search agents, K2-Horizon-7B and Qwen3.5-27B. On 200 paired questions, without retraining the poisoned E5, triggering reduces F1 by 31.58 and 34.68 points, respectively, relative to Backdoor-Clean. The target ranks first in every executed triggered retrieval round.
We added these results to Section 4.2.2, “Transfer to additional search agents and a multi-hop benchmark,” and Table 4.
Q3. How reliable are the detection results with few naturally triggered questions? Can you evaluate larger sets and more diverse detectors?
The reduction in detector separation persists on a larger evaluation set and across different detection mechanisms. We expanded detection to the full MuSiQue development set. DOU reduces separation for six of seven detectors, including representation- and perturbation-based methods; the expanded evaluation also includes ONION.
We added these results to Appendix D.3, “Concealment persists on the full MuSiQue development set,” and Table 14, complementing the main-text results in Table 9.
Values are detector separation Δ=meani∣AUROCi−50∣; lower values indicate weaker separation. DOU80 uses 80 overwrite and 80 unlearn steps.