- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose PoisonVax, a maintainer-side defense against corpus poisoning in retrieval-augmented generation. The key idea is to represent each trusted fact using two text views: an index view optimized to improve retrieval exposure and a generation view containing the trusted fact and an answer directive to improve answer adoption. The two views are linked through a fixed field mapping, while the retriever and generator remain frozen. The paper further argues that retrieval exposure and answer adoption are distinct failure modes and should be handled separately. Experiments compare PoisonVax with several existing RAG defenses and include matched single-text controls, unseen questions, two corpora, different retrievers and generators, adaptive attacks, corpus-scale experiments, and analyses of non-target interference. The results generally show strong protection under the evaluated poisoning settings, although the practical protection–utility trade-off remains an important concern.
Critical strengths
- Clear and useful decomposition of the defense problem. The separation between retrieval exposure and answer adoption is conceptually meaningful. The paper does more than report end-to-end accuracy: the controlled interventions on retrieval scores, exposure, ordering, and attacker removal provide evidence that successfully suppressing one attacker does not necessarily restore the trusted answer.
- The empirical evaluation is unusually extensive. The paper includes matched single-text controls with multiple optimization and generation seeds, experiments on unseen questions and different corpora, retriever and generator transfer, defense-aware attacks, large-corpus evaluation, and explicit analysis of non-target utility. The matched representation experiment is particularly important because it shows that the dual-view advantage is not explained solely by the attention-attractor spans.
Critical weaknesses
- The main comparison against existing defenses does not fully isolate the benefit of the proposed defense mechanism from the additional maintainer capabilities. In Table 1, existing defenses receive ordinary trusted records, whereas PoisonVax receives per-fact optimized retrieval keys, a separate generation view, and a maintainer-controlled field mapping. This makes the comparison practically relevant, but it is not a strictly capability-matched comparison. The single-text experiments control the representation question well, but they do not replace a comparison against simpler maintainer-side alternatives that exploit comparable privileges. For example, it would be useful to compare against simpler schemes such as explicitly boosting trusted records, reserving a trusted retrieval slot, or using the same two-field infrastructure with an unoptimized trusted fact. Without such controls, it remains unclear how much of the gain comes specifically from PoisonVax rather than from giving trusted records stronger infrastructure-level privileges.
- The protection–utility trade-off is not yet resolved by a single deployable configuration. The default non-selective records interfere substantially with unrelated queries, while the unconditional generation directive can introduce unrelated protected answers. The paper proposes selective retrieval keys and conditional generation directives to reduce these effects, but these two mechanisms are evaluated separately rather than as one complete end-to-end system. In addition, the million-passage experiment uses the non-selective version, where protected records are retrieved for a large fraction of unrelated queries. This is important because the practical contribution depends not only on protecting target facts but also on avoiding degradation of normal RAG behavior. The current experiments demonstrate that each source of interference can be reduced independently, but they do not yet establish the overall operating point of the complete mitigated system. A related concern is generator dependence. The complete PoisonVax record improves results for several generators but performs worse than both a plain trusted fact and no defense with Llama-2-13B, indicating that the generation-side mechanism is not universally reliable.
Questions for authors
- Can the authors add capability-matched maintainer baselines? In particular, how does PoisonVax compare with simpler approaches that use the same maintainer authority, such as trusted-record score boosting, reserved trusted slots, or the same index-to-generation field mapping without an optimized retrieval key?
- Can the authors evaluate one complete deployable variant that combines the selective retrieval key and the conditional generation directive? Please report target protection and non-target retrieval/generation utility simultaneously, preferably also at the million-passage scale.
- How should the Llama-2 failure be interpreted? Can the authors provide a principled criterion for selecting or adapting the generation view to a new generator without tuning on the protected evaluation questions?
- All semantic answer labels appear to be machine-generated. Can the authors provide human validation on a representative subset, particularly for ambiguous outputs containing both correct and attacker-related information? The paper explicitly states that the answer labels and rechecks are machine-produced, so agreement with human judgment would strengthen confidence in the headline accuracy comparisons.