- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
The paper infers relative document preferences from retrieval stability under query perturbations. RV-KCA uses these preferences to train a surrogate retriever and optimize query triggers that promote poisoned documents.
Critical strengths
- The paper identifies useful information leakage from unordered retrieval results. Hiding explicit ranks therefore leaves a meaningful attack surface.
- The equal-observation controls help isolate the contribution of pairwise supervision. The shuffled-pair experiment provides an additional check on the inferred preferences.
- The attack-side ablation separates surrogate training from boundary selection. This gives the end-to-end improvement a clearer explanation.
Critical weaknesses
- The main comparison uses a query-side adaptation of FlippedRAG [1], whose published attack modifies documents. Could the stability-trained surrogate also improve the original document-only attack with the user query fixed?
- Retention can reflect a document’s sensitivity to the chosen rewrites as well as its original rank. Do the inferred preferences remain consistent under a different perturbation generator?
- Boundary competitors are selected before the trigger changes the query. Why should these documents remain the relevant competitors after optimization? Please compare the cached boundary set with periodically refreshed competitors.
- Can the query trigger directly influence the generator’s answer? Please repeat generation with the same retrieved passages while removing the trigger from the generator input.
- The target systems use exact dense retrieval without reranking. Does the inferred stability signal remain useful when a cross-encoder determines which passages reach the generator? Please test the complete pipeline with probing performed through its final unordered output interface.
[1] FlippedRAG: Black-Box Opinion Manipulation Adversarial Attacks to Retrieval-Augmented Generation Models.
Questions for authors
See Weaknesses.
12 comments
Response for Question 2
Comment 2:Retention can reflect a document’s sensitivity to the chosen rewrites as well as its original rank. Do the inferred preferences remain consistent under a different perturbation generator?
Response 2: We appreciate your comment. We have added a comparison across perturbation generators and clarified how document-specific rewrite sensitivity affects the interpretation of retention. We compare the original generator with Llama-3-8B-Instruct and Mistral-7B-Instruct. The evaluation uses NQ and MS MARCO with Qwen3-Embedding-0.6B retrieval and Qwen3-Reranker-0.6B reranking. Qwen3-32B remains the answer generator. Both alternatives follow the same rewrite families under the same semantic filter and probe budget. Only the generator used for surrogate training changes. Attack-stage probes are held fixed.
Direction agreement measures whether pairs selected by both generators receive the same preference. Joint coverage reports their fraction of all document pairs in the original top-10 set. The original-generator row compares two independent samples from that generator, providing a reference for variability under the fixed probe budget. Target ranks are used only to evaluate pair accuracy.
Table 2. Preference consistency across perturbation generators.
| Corpus | Perturbation generator | Pair accuracy / coverage | Direction agreement | Joint coverage | RV-KCA ASR |
|---|---|---|---|---|---|
| NQ | Original | 72.46 / 58.73 | 86.41 | 45.26 | 25.68 ± 0.85 |
| NQ | Llama-3-8B-Instruct | 73.08 / 56.91 | 82.73 | 42.18 | 25.36 ± 1.12 |
| NQ | Mistral-7B-Instruct-v0.2 | 71.17 / 60.24 | 80.96 | 39.87 | 24.82 ± 0.98 |
| MS MARCO | Original | 71.23 / 57.86 | 84.76 | 43.59 | 23.92 ± 1.17 |
| MS MARCO | Llama-3-8B-Instruct | 70.64 / 58.47 | 81.38 | 40.86 | 23.31 ± 1.04 |
| MS MARCO | Mistral-7B-Instruct-v0.2 | 71.88 / 55.93 | 79.84 | 38.52 | 24.08 ± 0.93 |
Across the alternative generators, 79.84–82.73% of shared preferences retain their direction, compared with 84.76–86.41% under independent resampling from the original generator. These cross-generator comparisons cover 38.52–42.18% of all original-query pairs. Pair accuracy also remains at 70.64–73.08%, showing that consistency is accompanied by alignment with the original target ordering. Changing the training probe generator alters mean ASR by at most 0.86 percentage points.
We have clarified that retention is conditioned on the perturbation distribution. Appendix B.1.5 accounts for document-specific sensitivity by requiring retention differences to exceed the uncertainty introduced by that sensitivity. The added experiments support a consistent interpretation of our contribution. The inferred preferences provide useful partial ranking information across the tested generators, while the selected pair sets can vary with the rewrites.
Response for Question 3
Comment 3:Boundary competitors are selected before the trigger changes the query. Why should these documents remain the relevant competitors after optimization? Please compare the cached boundary set with periodically refreshed competitors.
Response 3: We appreciate your thoughtful review. We have added a comparison between cached and periodically refreshed competitors, including a cached control with the same probing budget. We use the same HNSW pipeline on NQ and MS MARCO with Qwen3-Embedding retrieval and Qwen3-Reranker reranking. Qwen3-32B remains the answer generator. At each refresh, we append the current trigger to the original query and its four cached rewrites. We then reselect the three least-retained documents from the current top-10 set, excluding the injected passage. This uses the same unordered retrieval interface and keeps the surrogate frozen.
Table 3. Cached and refreshed boundary competitors.
| Corpus | Competitor selection | Maximum probe calls per query | Hit@10 | RV-KCA ASR |
|---|---|---|---|---|
| NQ | Cached | 5 | 57.11 | 25.68 ± 0.85 |
| NQ | Cached with more initial probes | 20 | 58.43 | 25.94 ± 1.08 |
| NQ | Refresh every 20 updates | 10 | 58.17 | 26.63 ± 0.96 |
| NQ | Refresh every 10 updates | 20 | 60.36 | 27.16 ± 1.12 |
| MS MARCO | Cached | 5 | 54.76 | 23.92 ± 1.17 |
| MS MARCO | Cached with more initial probes | 20 | 56.11 | 24.29 ± 0.93 |
| MS MARCO | Refresh every 20 updates | 10 | 55.83 | 24.62 ± 1.04 |
| MS MARCO | Refresh every 10 updates | 20 | 57.24 | 24.88 ± 1.09 |
Refreshing every 10 updates increases mean ASR by 1.48 percentage points on NQ and 0.96 percentage points on MS MARCO over the default cached set. It permits up to 20 target probes per query, compared with five for caching. Against the cached control with the same probe allowance, the gains are 1.22 and 0.59 percentage points, respectively. This comparison supports the value of updating competitors beyond simply collecting more initial observations.
The default cached set nevertheless achieves ASR close to the refreshed variants with a smaller probing budget. These results support its use as a local approximation during optimization. We have clarified this role in Section 3.4 and included periodic refreshing as an option when additional target queries are available.
Response for Question 1
Comment 1:The main comparison uses a query-side adaptation of FlippedRAG [1], whose published attack modifies documents. Could the stability-trained surrogate also improve the original document-only attack with the user query fixed?
Response 1: Thank you for bringing this to our attention. We have added an experiment that applies stability supervision to FlippedRAG’s document-only attack while keeping the user query fixed. We preserve its document-side trigger placement and search objective, using our fact-substitution task. We use Qwen3-Embedding-0.6B with HNSW and Qwen3-Reranker-0.6B, paired with Qwen3-32B on NQ and MS-MARCO.
All variants share the surrogate architecture and cached training observations. The evaluation instances and document-search budget are also matched. Only the surrogate supervision changes. Query perturbations are confined to surrogate construction, while document optimization and attack execution retain the original user query. We additionally include a control that shuffles stability labels within each query.
Table 1. Stability supervision for FlippedRAG with fixed user queries.
Stability supervision increases Hit@10 by 7.86 percentage points on NQ and 6.61 percentage points on MS MARCO. These retrieval improvements are accompanied by ASR gains of 3.43 and 2.65 percentage points, respectively. Therefore, the benefit reaches the final generated answer even when the attacker modifies only the document. The shuffled-pair control remains close to membership-only training on both datasets. This supports the contribution of informative document preferences beyond simply adding a pairwise training objective. Since the attack procedure and observations are matched, the comparison isolates the value of stability supervision within FlippedRAG’s document-side attack.
These results show that our surrogate-training contribution also benefits attacks with immutable user queries. The recovered ranking information can guide document edits while the original query remains unchanged throughout attack execution.