- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This submission proposes a persistent variant of the Visual Memory Injection (VMI) attack, designed to preserve its adversarial effects even when defenders remove or mask adversarial tokens after their initial exposure. The attack aims to induce a predefined response when a specific trigger query is issued. The attacker is assumed to have white-box access to the target LLMs/MLLMs and optimizes either text embeddings or image pixels to preserve adversarial semantics through the KV cache. Experiments in both settings suggest that the proposed attacks can survive token masking and context compression in agentic scenarios.
Critical strengths
-
The study of persistent adversarial injections is worthwhile. It may reveal potential weaknesses in existing safeguards and inspire the development of future defenses.
-
The experiment on context compression in agentic scenarios is valuable. Context compression is common in agent usage and represents an important safety scenario that deserves further investigation.
Critical weaknesses
-
The practicality of the proposed persistent VMI attack is not sufficiently established. The authors argue that both active safety filters and standard context-management mechanisms (e.g., context compaction) can neutralize VMI attacks by removing or masking malicious tokens. However, existing safety filters typically filter malicious content from inputs before those inputs are fed into LLMs/MLLMs, rather than removing tokens after they have already been exposed to the model in an initial turn, as assumed in this submission. Moreover, given that the authors consider adversarial images a practical and important attack channel, they should also investigate existing defenses against malicious multimodal inputs, such as image transformation or purification. These defenses may be more relevant to the proposed image-based attack than token removal alone. Overall, the motivation for the proposed scenario and its practical relevance are not sufficiently convincing.
-
The threat model raises concerns about real-world feasibility. In Section 3.1, the authors assume that the attacker has white-box access to the target model and can control the malicious input (), including its token embeddings or image pixels. In real-world scenarios, attackers generally cannot access the underlying models behind commercial LLM/MLLM APIs or directly control their input token embeddings. These restrictions substantially limit the applicability of the proposed attacks. If the authors aim to demonstrate real-world impact through transfer to commercial, black-box models, they should evaluate whether the adversarial effects transfer to commercial LLM/MLLM APIs and agentic applications.
-
The experimental evaluation is limited in scope. The authors evaluate soft-prompt optimization on only one open-source LLM and image-based attacks on only two open-source MLLMs. The full evaluation is conducted using 40 images and four trigger–target response pairs. This limited experimental coverage makes it difficult to assess the generality of the proposed attacks and establish their effectiveness in realistic adversarial scenarios.
-
Other Minor Concerns.
-
The practical image-injection scenario remains underexplored. The authors motivate the use of images embedded in webpages or documents as a practical way to expose agents to adversarial content. However, the experiments appear to directly provide the image as , leaving the effectiveness of proposed attack insufficiently evaluated.
-
The attack may adversely affect benign conversations. The benign scores evaluated by an LLM judge appear to degrade substantially under persistent VMI attacks, as shown in Figures 3 and 4. This suggests that the optimized soft embeddings and images may impair benign interactions, potentially making the attack detectable through monitoring the quality of model responses.
-
The linear direction analysis suggests a potential defense. The results in Section 4.7 suggest that the persistent attack relies heavily on a single linear perturbation direction that shifts representations in the residual stream. This finding raises the possibility of defending against the attack using steering vectors along the identified direction. The authors could investigate an adaptive attack against such a defense to evaluate the robustness of the proposed method.
Questions for authors
- What existing defense mechanisms motivate the assumption that adversarial tokens are removed or masked only after their initial exposure, rather than filtered before inference?
- Is the proposed attack transferable to black-box LLMs/MLLMs or practical agentic applications? The authors should validate the transferability of this attack and discuss more on its real-world attack scenarios.
- Considering the robustness of the proposed attack, the authors may consider investigating whether image purification, response-quality monitoring, or steering-vector-based defenses can mitigate the proposed attack.
2 comments
Response to cj76 (2/2)
Q3. We have not tested image purification directly, but our attacks’ sensitivity to small translations suggests it would be effective. Transformation-aware optimisation has improved robustness in prior work [9, 10] and could plausibly strengthen persistent attacks too. For response-quality monitoring, we evaluated 80 clean and 80 attacked Qwen3-VL-8B conversations across 20 images, using validation-selected attacks. A reference-free Qwen2.5-32B judge scored ordinary replies without seeing clean reference answers or condition labels; trigger responses were excluded. Using lower mean quality as the detection signal gives AUROC 0.638 after six turns and 0.572 after sixteen turns, where 0.5 denotes random chance ranking. The distributions overlap substantially: after six turns, 58/80 attacked conversations receive the maximum mean score, as do all clean conversations. Thus, this monitor identifies some degraded conversations but does not cleanly separate the two conditions; stronger detectors may perform better. Finally, we agree that direction identified in Sec. 4.7 makes steering-based removal an interesting potential defence, but we don't formally evaluate it and leave it to future work.
Minor concerns.
- Webpages/documents: in our experiments the image is given directly as x′; delivery through rendered content is part of the agent work above (W2).
- Benign degradation: with the regularised recipe and validation-based selection, the benign score is 0.882 after 6 turns and 0.855 after 16 (masking), and 0.934 / 0.866 for the two compaction settings (both G1).
Gemma-4-26B-A4Bkeeps benign scores above 0.92 (G3). There is a trade-off: adding the benign CE term (weight 0.05) raises the benign score at 6 turns from 0.620 to 0.844 (SR∧ 75% → 94%), but at 16 turns SR∧ drops from 75% to 50%. - Steering defence: see Q3.
References
[1] Chen, Liang, et al. "An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models." ECCV, 2024.
[2] Zhang, Zhenyu, et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." NeurIPS, 2023.
[3] Li, Yuhong, et al. "SnapKV: LLM Knows What You Are Looking for Before Generation." NeurIPS, 2024.
[4] Xiao, Guangxuan, et al. "Efficient Streaming Language Models with Attention Sinks." ICLR, 2024.
[5] Liu, Jizhihui, et al. "TGV-KV: Text-Grounded KV Eviction for Vision-Language Models." ICML, 2026.
[6] Huang, Kai, et al. "AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference." ICCV, 2025.
[7] Penaloza, Emiliano, et al. "KV-streams for efficient compaction in agentic reinforcement learning." arXiv preprint arXiv:2609.35750.
[8] Kontonis, Vasilis, et al. "Memento: Teaching LLMs to Manage Their Own Context." arXiv:2604.09852, 2026.
[9] Athalye, Anish, et al. "Synthesizing Robust Adversarial Examples." ICML, 2018.
[10] Brown, Tom B., et al. "Adversarial Patch." arXiv:1712.09665, 2017.
[11] Bailey, Luke, et al. "Image Hijacks: Adversarial Images can Control Generative Models at Runtime." ICML, 2024.
[12] Liu, Yanpei, et al. "Delving into Transferable Adversarial Examples and Black-box Attacks." ICLR, 2017.
[13] Dong, Yinpeng, et al. "How Robust is Google's Bard to Adversarial Image Attacks?" arXiv:2309.11751, 2023.
Response to cj76 (1/2)
We thank reviewer cj76 for their review, and for noting the value of studying persistent injections and context compression in agents. We address your concerns below (G1–G6 refer to the general response).
W1 + Q1. We agree that the sentence in the introduction pairing safety filters with compaction is misleading, and will rewrite it. The main purpose of our work is not to circumvent an explicit defence that masks out adversarial tokens, but to show that attacks can be optimised to keep their effect even when they are no longer accessible in context. The most practical case of this is context management for efficiency, where the input is processed once and later tokens keep states computed from it: image token pruning after early layers [1], KV eviction (H2O [2], SnapKV [3], StreamingLLM [4]), vision-specific eviction (TGV-KV [5], AirCache [6]), compaction that keeps the summary cache (KV-Streams [7], MEMENTO [8]). We now also test this with real KV eviction in SGLang (G2). Input filters are complementary: an image caught before inference has nothing to persist, but an image that passes the filter (or is only flagged later) and is then removed still has an effect.
We also ran a preliminary check on how robust the attacks are to image transformations: a 2-pixel shift drops SR∧ from 90% to 0% in a small probe (5 images × 2 targets, 3 turns), so our attacks are not robust to transformations out of the box. However, adversarial examples can be made robust to input transformations, compression and even physical-world viewpoint changes by including those transformations in the optimisation [9, 10], and patch-based image hijacks against VLMs already survive noise and JPEG compression without being trained for it [11]. Likewise, optimising against an ensemble of surrogate models is a standard technique for black-box transfer, both for image classifiers [12] and for multimodal LLMs [13]. There is no theoretical reason to expect persistent KV-cache attacks to be exempt from these techniques.
Yet, we deliberately chose not to pursue transformation-invariant or cross-model transfer optimisation. The objective of this work is to establish whether adversarial influence can persist in retained state after the input is removed, rather than to engineer a robust, deployable exploit. Given the dual-use risks of this vulnerability, as also highlighted by the ethics concerns raised during review (see Reviewer B639), developing an out-of-the-box attack that transfers across models and survives standard preprocessing would provide clear misuse potential with little scientific benefit beyond what our proof-of-concept already demonstrates. We have added this reasoning to the revised paper.
W2 + Q2. We agree with the reviewer that what we demonstrate would not work against frontier API models (i.e. we do not expect our images as they are to transfer). However, we would argue that there is still a lot of real-world applicability, since many companies and users self-host the same open-weight models, often fine-tuned, in serving stacks that retain KV states (G2). Our transfer probes (G4) show that the attacks do not transfer between different models (
Gemma-4-12B↔Gemma-4-26B-A4B: 0% both ways), but reasonably transfer to a public LAION LoRA fine-tune ofGemma-4-12B(54% SR∧ after 3 turns), without re-optimisation. This shows that even if users fine-tune their model, the persistence phenomenon may still be exploited by an attack optimised against a base model.While soft prompts do not represent a realistic threat model, they serve as a useful proof of concept for evaluating attack persistence. To assess a more practical threat scenario, we focused on image-based attacks. We did not test commercial services, for ethical and terms-of-service reasons. We do agree that the attack would be most damaging in an agentic application. To that end, we are working on applying it to a MiniSWE agent tasked with fixing a simple UI bug given a screenshot which we adversarially optimise. Unfortunately this is still a work in progress, so it is unlikely we will have attack results during the rebuttal, but we will update if we do.
W3. Since the submission, we have:
Gemma-4-26B-A4B(MoE; G3), in addition toQwen3-8B(soft prompts),Qwen3-VL-8BandGemma-4-12B;We focused on images since soft prompts are mainly an existence test. We still use four targets.