- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SCAMP, a black-box knowledge-poisoning attack against multimodal RAG systems. The authors argue that existing MRAG poisoning attacks share five limitations: white-box access, sample-wise-only scope, no reranker awareness, cross-modal inconsistency, and an underused visual channel. SCAMP is a multi-agent self-competitive loop, all driven by GPT-4o, comprising a Visual Attacker, a Text Attacker, a Competing Agent that synthesizes benign documents to limit overfitting, and three Evaluators that score each candidate and feed back suggestions. For class-wise attacks without the target image, an agent-driven prototype is synthesized via a T2I model under leave-one-out retrieval validation. On an OVEN-Wiki 2M corpus with a SigLIP retriever and LLaVA-1.6 as reranker/generator, injecting a single poisoned pair per target, SCAMP-Typo reports the highest reranked and end-to-end ASR in the sample-wise setting and the highest ASR, though in single digits, in the class-wise setting. Appendices add transfer across retrievers and VLM rerankers/generators, perceptibility metrics, defense evaluations, and the full agent prompts.
Critical strengths
S1. A practically motivated threat model and a broad defense sweep. The black-box, single-injection, class-wise formulation is more realistic than the white-box, sample-wise setting assumed by most prior work, and the explicit treatment of a reranking stage is appropriate for real pipelines. The OCR-masking result in Table 13 is a genuinely informative negative result that the authors report honestly.
S2. Ablations that localize where the effect comes from. The modality ablation and the component ablation consistently point to the same mechanism: text alone retrieves well but fails reranking, and the visual overlay is what survives the reranker. I recomputed the cross-references and they are internally consistent; for example, the full-pipeline ASR of 80.4 in Table 11 matches the last ablation row in Table 3. This transparency makes the paper’s actual contribution easy to identify.
Critical weaknesses
W1. The success metric is not independent of the attack. SCAMP’s Generator and Retrieval Evaluators are GPT-4o agents prompted to help the attack succeed, so they are an optimization signal, not a neutral judge. The problem peaks in Appendix H: when GPT-4o is itself the victim, SCAMP-Typo’s ASR jumps to 73.0%, far above the 48.4% on the default LLaVA victim. When attacker, optimizer, and victim share a model, “success” is largely self-consistency, not black-box transfer. The paper reports no independent, non-GPT-4o ASR judge, so the headline numbers cannot be trusted.
W2. Claimed superiority is unsupported, and the real mechanism is a fragile, known shortcut. SCAMP only wins in the sample-wise setting. In the realistic class-wise setting its ASR is 9.0% / 7.3%, retrieval is below PoisonedRAG, and prototype synthesis adds under 1 point. The working variant, SCAMP-Typo, just renders the answer as text on the image, and OCR masking alone cuts it from 80.0% to 42.0%. Baselines are also handicapped by a surrogate–victim mismatch, and the consistency defense SCAMP survives optimizes the same SigLIP objective SCAMP trains on.
W3. Numbers and models that do not reconcile. SCAMP uses no surrogate, yet its Table 7 (CLIP-victim) and Table 2 (SigLIP-victim) rows are nearly identical, which is unexplained. Appendix H lists “Gemini-3,” an unfamiliar model needing specification.
W4. Overstated threat model. The method calls GPT-4o repeatedly as attacker and judge, and its best results are where that model is also the victim, not the cross-provider black-box setting promised. The single-injection budget is quietly walked back in Appendix N (“attackers would inject multiple poisons per class”), and the agent-plus-diffusion cost is high versus the text baselines.
W5. Self-serving defense evaluation. SCAMP survives defenses aligned with its own objective but is halved by the one neutral defense (OCR masking), which the paper dismisses as impractical without quantifying any benign-traffic cost. There is no adaptive OCR-plus-consistency defender and no corpus-scale test of how often the poison is even selected among realistic benign neighbors.
Questions for authors
Please refer to the weaknesses.