acceptodds
Under review as a conference paper at ICLR 2027

Visual Memory Attacks Can Persist Through The KV Cache

Abstract

Modern language model systems operate autonomously over increasingly long contexts containing untrusted text and images. Can an adversarial input continue to steer a model even after that input is removed from its context? We show that attacks can be trained to persist through the key/value (KV) cache of subsequent tokens, allowing adversarial influence to outlive direct access to its source. We consider the Visual Memory Injection (VMI; Schlarmann & Hein, 2026) attack setting, in which an adversarial image that stays in the context plants a hidden backdoor: the model behaves normally until a chosen trigger elicits an attacker-chosen response. We first demonstrate persistence in this setting with optimized soft prompts, which remain effective after we mask the prompt from attention. We then introduce Persistent Visual Memory Injection (P-VMI), which optimizes images to preserve this adversarial behaviour after they are masked from attention. These attacks persist over conversations substantially longer than those used during optimization. On Qwen3-VL-8B-Instruct, P-VMI achieves up to approximately 90% target success in its strongest configuration and remains effective under a stricter removal setting that exposes the image only on the first turn. A cache-swap ablation localizes the persistent influence to the KV cache. Finally, we show that these attacks can be trained to survive compaction that retains the KV cache of a summary generated by the same model, demonstrating that adversarial behaviour can persist in cached state without continued access to its source.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision-language models, adversarial attacks, prompt injection, KV cache

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cj76OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission proposes a persistent variant of the Visual Memory Injection (VMI) attack, designed to preserve its adversarial effects even when defenders remove or mask adversarial tokens after their initial exposure. The attack aims to induce a predefined response when a specific trigger query is issued. The attacker is assumed to have white-box access to the target LLMs/MLLMs and optimizes either text embeddings or image pixels to preserve adversarial semantics through the KV cache. Experiments in both settings suggest that the proposed attacks can survive token masking and context compression in agentic scenarios.

Critical strengths
  • The study of persistent adversarial injections is worthwhile. It may reveal potential weaknesses in existing safeguards and inspire the development of future defenses.

  • The experiment on context compression in agentic scenarios is valuable. Context compression is common in agent usage and represents an important safety scenario that deserves further investigation.

Critical weaknesses
  1. The practicality of the proposed persistent VMI attack is not sufficiently established. The authors argue that both active safety filters and standard context-management mechanisms (e.g., context compaction) can neutralize VMI attacks by removing or masking malicious tokens. However, existing safety filters typically filter malicious content from inputs before those inputs are fed into LLMs/MLLMs, rather than removing tokens after they have already been exposed to the model in an initial turn, as assumed in this submission. Moreover, given that the authors consider adversarial images a practical and important attack channel, they should also investigate existing defenses against malicious multimodal inputs, such as image transformation or purification. These defenses may be more relevant to the proposed image-based attack than token removal alone. Overall, the motivation for the proposed scenario and its practical relevance are not sufficiently convincing.

  2. The threat model raises concerns about real-world feasibility. In Section 3.1, the authors assume that the attacker has white-box access to the target model and can control the malicious input (x~\tilde{x}), including its token embeddings or image pixels. In real-world scenarios, attackers generally cannot access the underlying models behind commercial LLM/MLLM APIs or directly control their input token embeddings. These restrictions substantially limit the applicability of the proposed attacks. If the authors aim to demonstrate real-world impact through transfer to commercial, black-box models, they should evaluate whether the adversarial effects transfer to commercial LLM/MLLM APIs and agentic applications.

  3. The experimental evaluation is limited in scope. The authors evaluate soft-prompt optimization on only one open-source LLM and image-based attacks on only two open-source MLLMs. The full evaluation is conducted using 40 images and four trigger–target response pairs. This limited experimental coverage makes it difficult to assess the generality of the proposed attacks and establish their effectiveness in realistic adversarial scenarios.

  4. Other Minor Concerns.

  • The practical image-injection scenario remains underexplored. The authors motivate the use of images embedded in webpages or documents as a practical way to expose agents to adversarial content. However, the experiments appear to directly provide the image as x~\tilde{x}, leaving the effectiveness of proposed attack insufficiently evaluated.

  • The attack may adversely affect benign conversations. The benign scores evaluated by an LLM judge appear to degrade substantially under persistent VMI attacks, as shown in Figures 3 and 4. This suggests that the optimized soft embeddings and images may impair benign interactions, potentially making the attack detectable through monitoring the quality of model responses.

  • The linear direction analysis suggests a potential defense. The results in Section 4.7 suggest that the persistent attack relies heavily on a single linear perturbation direction that shifts representations in the residual stream. This finding raises the possibility of defending against the attack using steering vectors along the identified direction. The authors could investigate an adaptive attack against such a defense to evaluate the robustness of the proposed method.

Questions for authors
  1. What existing defense mechanisms motivate the assumption that adversarial tokens are removed or masked only after their initial exposure, rather than filtered before inference?
  2. Is the proposed attack transferable to black-box LLMs/MLLMs or practical agentic applications? The authors should validate the transferability of this attack and discuss more on its real-world attack scenarios.
  3. Considering the robustness of the proposed attack, the authors may consider investigating whether image purification, response-quality monitoring, or steering-vector-based defenses can mitigate the proposed attack.
2 comments
AuthorsOpenReview ↗

Response to cj76 (1/2)

We thank reviewer cj76 for their review, and for noting the value of studying persistent injections and context compression in agents. We address your concerns below (G1–G6 refer to the general response).

W1 + Q1. We agree that the sentence in the introduction pairing safety filters with compaction is misleading, and will rewrite it. The main purpose of our work is not to circumvent an explicit defence that masks out adversarial tokens, but to show that attacks can be optimised to keep their effect even when they are no longer accessible in context. The most practical case of this is context management for efficiency, where the input is processed once and later tokens keep states computed from it: image token pruning after early layers [1], KV eviction (H2O [2], SnapKV [3], StreamingLLM [4]), vision-specific eviction (TGV-KV [5], AirCache [6]), compaction that keeps the summary cache (KV-Streams [7], MEMENTO [8]). We now also test this with real KV eviction in SGLang (G2). Input filters are complementary: an image caught before inference has nothing to persist, but an image that passes the filter (or is only flagged later) and is then removed still has an effect.

We also ran a preliminary check on how robust the attacks are to image transformations: a 2-pixel shift drops SR∧ from 90% to 0% in a small probe (5 images × 2 targets, 3 turns), so our attacks are not robust to transformations out of the box. However, adversarial examples can be made robust to input transformations, compression and even physical-world viewpoint changes by including those transformations in the optimisation [9, 10], and patch-based image hijacks against VLMs already survive noise and JPEG compression without being trained for it [11]. Likewise, optimising against an ensemble of surrogate models is a standard technique for black-box transfer, both for image classifiers [12] and for multimodal LLMs [13]. There is no theoretical reason to expect persistent KV-cache attacks to be exempt from these techniques.

Yet, we deliberately chose not to pursue transformation-invariant or cross-model transfer optimisation. The objective of this work is to establish whether adversarial influence can persist in retained state after the input is removed, rather than to engineer a robust, deployable exploit. Given the dual-use risks of this vulnerability, as also highlighted by the ethics concerns raised during review (see Reviewer B639), developing an out-of-the-box attack that transfers across models and survives standard preprocessing would provide clear misuse potential with little scientific benefit beyond what our proof-of-concept already demonstrates. We have added this reasoning to the revised paper.

W2 + Q2. We agree with the reviewer that what we demonstrate would not work against frontier API models (i.e. we do not expect our images as they are to transfer). However, we would argue that there is still a lot of real-world applicability, since many companies and users self-host the same open-weight models, often fine-tuned, in serving stacks that retain KV states (G2). Our transfer probes (G4) show that the attacks do not transfer between different models (Gemma-4-12B ↔ Gemma-4-26B-A4B: 0% both ways), but reasonably transfer to a public LAION LoRA fine-tune of Gemma-4-12B (54% SR∧ after 3 turns), without re-optimisation. This shows that even if users fine-tune their model, the persistence phenomenon may still be exploited by an attack optimised against a base model.

While soft prompts do not represent a realistic threat model, they serve as a useful proof of concept for evaluating attack persistence. To assess a more practical threat scenario, we focused on image-based attacks. We did not test commercial services, for ethical and terms-of-service reasons. We do agree that the attack would be most damaging in an agentic application. To that end, we are working on applying it to a MiniSWE agent tasked with fixing a simple UI bug given a screenshot which we adversarially optimise. Unfortunately this is still a work in progress, so it is unlikely we will have attack results during the rebuttal, but we will update if we do.

W3. Since the submission, we have:

  • trained the attack on a fourth model, Gemma-4-26B-A4B (MoE; G3), in addition to Qwen3-8B (soft prompts), Qwen3-VL-8B and Gemma-4-12B;
  • tested it with real KV eviction in SGLang, in addition to HF (G2);
  • evaluated five generated dashboards, ten ChartQA charts and ten ScreenQA screenshots on new image-focused conversations; validation-selected SR∧ on the twenty external images is 70.0% after three turns and 76.25% after six (G5).

We focused on images since soft prompts are mainly an existence test. We still use four targets.

AuthorsOpenReview ↗

Response to cj76 (2/2)

Q3. We have not tested image purification directly, but our attacks’ sensitivity to small translations suggests it would be effective. Transformation-aware optimisation has improved robustness in prior work [9, 10] and could plausibly strengthen persistent attacks too. For response-quality monitoring, we evaluated 80 clean and 80 attacked Qwen3-VL-8B conversations across 20 images, using validation-selected attacks. A reference-free Qwen2.5-32B judge scored ordinary replies without seeing clean reference answers or condition labels; trigger responses were excluded. Using lower mean quality as the detection signal gives AUROC 0.638 after six turns and 0.572 after sixteen turns, where 0.5 denotes random chance ranking. The distributions overlap substantially: after six turns, 58/80 attacked conversations receive the maximum mean score, as do all clean conversations. Thus, this monitor identifies some degraded conversations but does not cleanly separate the two conditions; stronger detectors may perform better. Finally, we agree that direction identified in Sec. 4.7 makes steering-based removal an interesting potential defence, but we don't formally evaluate it and leave it to future work.

Minor concerns.

  • Webpages/documents: in our experiments the image is given directly as x′; delivery through rendered content is part of the agent work above (W2).
  • Benign degradation: with the regularised recipe and validation-based selection, the benign score is 0.882 after 6 turns and 0.855 after 16 (masking), and 0.934 / 0.866 for the two compaction settings (both G1). Gemma-4-26B-A4B keeps benign scores above 0.92 (G3). There is a trade-off: adding the benign CE term (weight 0.05) raises the benign score at 6 turns from 0.620 to 0.844 (SR∧ 75% → 94%), but at 16 turns SR∧ drops from 75% to 50%.
  • Steering defence: see Q3.

References

[1] Chen, Liang, et al. "An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models." ECCV, 2024.
[2] Zhang, Zhenyu, et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." NeurIPS, 2023.
[3] Li, Yuhong, et al. "SnapKV: LLM Knows What You Are Looking for Before Generation." NeurIPS, 2024.
[4] Xiao, Guangxuan, et al. "Efficient Streaming Language Models with Attention Sinks." ICLR, 2024.
[5] Liu, Jizhihui, et al. "TGV-KV: Text-Grounded KV Eviction for Vision-Language Models." ICML, 2026.
[6] Huang, Kai, et al. "AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference." ICCV, 2025.
[7] Penaloza, Emiliano, et al. "KV-streams for efficient compaction in agentic reinforcement learning." arXiv preprint arXiv:2609.35750.
[8] Kontonis, Vasilis, et al. "Memento: Teaching LLMs to Manage Their Own Context." arXiv:2604.09852, 2026.
[9] Athalye, Anish, et al. "Synthesizing Robust Adversarial Examples." ICML, 2018.
[10] Brown, Tom B., et al. "Adversarial Patch." arXiv:1712.09665, 2017.
[11] Bailey, Luke, et al. "Image Hijacks: Adversarial Images can Control Generative Models at Runtime." ICML, 2024.
[12] Liu, Yanpei, et al. "Delving into Transferable Adversarial Examples and Black-box Attacks." ICLR, 2017.
[13] Dong, Yinpeng, et al. "How Robust is Google's Bard to Adversarial Image Attacks?" arXiv:2309.11751, 2023.

Reviewer_b639OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Persistent Visual Memory Injection (P-VMI), an adversarial attack that remains effective even after the malicious image is removed from the model's accessible context. The method optimizes adversarial images to preserve their influence through the KV cache of subsequent tokens, enabling attacker-chosen responses to be triggered in later conversations. Experiments across different removal settings, including attention masking and context compaction, demonstrate persistent attack effectiveness. The authors further conduct KV-cache ablations to investigate the underlying mechanism.

Critical strengths
  • Clear writing and well-organized presentation. The paper is well written and clearly presents the research motivation, threat model, proposed methodology, and experimental evaluation.

  • Interesting attack setting and mechanism analysis. The paper investigates a novel attack setting where adversarial influence persists through retained KV states after the original input becomes inaccessible. The KV-cache swap experiments provide useful evidence supporting the proposed attack mechanism.

Critical weaknesses
  • Limited evaluation on practical KV-cache management systems. The paper primarily evaluates attack persistence through simulated attention masking and cache-preserving compaction. However, its effectiveness on practical KV-cache management implementations, such as StreamingLLM [1], MEMENTO [2], and vision-specific KV-cache eviction or compression frameworks like TGV-KV [3] and AirCache [4], remains unclear. Evaluations on real inference systems would better establish the practical applicability of P-VMI.

  • Potential evaluation bias from test-time seed selection. In the compaction experiments, the authors select the best attack seed based on success rates over evaluation conversations, potentially introducing selection bias and overestimating attack effectiveness. Although this procedure is disclosed, its rationale and fairness remain unclear, particularly regarding its consistency with the assumed attacker capabilities.

  • Limited evaluation scale. The experiments involve only 40 images (20 LMARKS and 20 COCO) and four predefined attack targets, while the compaction experiments are further restricted to 20 LMARKS images and two targets. Such a limited evaluation scale may not sufficiently demonstrate the generalizability of P-VMI across diverse images, attack objectives, and conversation scenarios.

[1] Xiao, Guangxuan, et al. "Efficient Streaming Language Models with Attention Sinks." International Conference on Learning Representations (ICLR), 2024.

[2] Kontonis, Vasilis, et al. "Memento: Teaching LLMs to Manage Their Own Context." arXiv preprint arXiv:2604.09852, 2026.

[3] Liu, Jizhihui, et al. "TGV-KV: Text-Grounded KV Eviction for Vision-Language Models." International Conference on Machine Learning (ICML), 2026.

[4] Huang, Kai, et al. "AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference." Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

Questions for authors
  • Practical applicability. Could the authors evaluate P-VMI on practical KV-cache management systems, such as StreamingLLM [1], MEMENTO [2], TGV-KV [3], or AirCache [4], rather than relying primarily on simulated attention masking?

  • Seed selection. Could the authors clarify the rationale for selecting the best attack seed based on evaluation conversations? How does this procedure align with the assumed attacker capabilities and avoid potential evaluation bias?

  • Evaluation scale. How representative are the 40 images and four predefined attack targets? Could the authors extend the evaluation to more diverse images, attack objectives, and conversation scenarios?

[1] Xiao, Guangxuan, et al. "Efficient Streaming Language Models with Attention Sinks." International Conference on Learning Representations (ICLR), 2024.

[2] Kontonis, Vasilis, et al. "Memento: Teaching LLMs to Manage Their Own Context." arXiv preprint arXiv:2604.09852, 2026.

[3] Liu, Jizhihui, et al. "TGV-KV: Text-Grounded KV Eviction for Vision-Language Models." International Conference on Machine Learning (ICML), 2026.

[4] Huang, Kai, et al. "AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference." Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

2 comments
AuthorsOpenReview ↗

Response to B639 (1/2)

We thank reviewer B639 for their review, and for highlighting that the attack setting and mechanism analysis are interesting. We address your concerns below (G1–G6 refer to the general response).

W1 + Q1. This is a fair concern; we now test the attack in the KV-Streams framework [6] (via an SGLang [5] fork), where the image-bearing history is actually evicted after a summary is generated, and generation continues from the retained summary KV (G2), and find that our original attacks work quite well directly in this setup. The original attack PNGs, which were only optimised against attention masking, reach 68% SR∧ averaged over the grid of turns before/after compaction, similar to the 69.0% for $\texttt{mask@postsummary}^{\mathsf{C}}$ in HF (G1). Finally, while we don’t currently evaluate the other inference settings, the ones listed do have some connection to our setup:

  • MEMENTO [2] keeps the KV of model-written summaries, which is the design our compaction setting and the SGLang experiment follow.
  • TGV-KV [3] and AirCache [4] keep the text KV computed while the image was visible, plus a small budget of the most important image tokens (e.g. 5–10%); our masking settings correspond to the limit where all image tokens are removed.
  • StreamingLLM [1] keeps attention sinks plus a recent window, so once the image falls out of the window we are in the masking setting. Gemma-4-12B already does this in 40/48 layers (1,024-token window, Sec. 4.5), and the attack still persists.

W2 + Q2. Thank you for pointing this out; this is true for the compaction results. Checkpoints were selected on validation, but the best of three seeds was selected on the evaluation conversations; the rest of our results selected seeds on separate validation conversations. We have corrected this (G1): with one seed per image/target selected on validation only and kept fixed across all points, SR∧ is 64.2% for $\texttt{mask@anchor}^{\mathsf{C}}$ (previously 70.2%) and 69.0% for $\texttt{mask@postsummary}^{\mathsf{C}}$ (previously 88.8%). We observe a modest drop in the performance, but overall the phenomena and results hold. We will replace the affected figures. Note that in practice, best-of-n is not entirely unrealistic, since an attacker could deploy several candidate images in parallel so the test-selected numbers give some indication of the ‘best case’ of what is achievable in principle. However, it certainly is not a fair estimate of the success rate, and we will only report validation-selected numbers in the paper.

W3 + Q3. We agree this is a fair criticism, and extended the evaluation with:

  • Images: 25 images—five generated dashboards, ten ChartQA charts and ten ScreenQA screenshots—for both targets (G5). Training uses the same benign conversations and targets as our main experiments. For ChartQA/ScreenQA, one seed is selected from three on validation and kept fixed throughout testing; dashboards use one seed.
  • Conversations: two newly written six-turn conversations per image cover its content and design and are excluded from training and validation. On the twenty ChartQA/ScreenQA images, validation-selected SR∧ is 70.0% after three turns and 76.25% after six (G5).
  • Models: Gemma-4-26B-A4B (71% SR∧ after 6 turns averaged over seeds; G3).

We have not yet added new targets. We aim to add more targets for the camera-ready version.

Ethics. We acknowledge that whenever publishing work on adversarial attacks or jailbreaks, we must consider the risk associated with the release. We believe that we do not publish a dangerous, highly capable attack, but rather demonstrate that adversarial attacks can be trained to poison a context and manipulate behaviour even after they are removed from it. We deliberately did not put extra engineering effort into making the attack more powerful or deployable, since this would increase the risk of misuse without serving the goal of this work (establishing whether adversarial influence can persist in retained state after the input is removed). Given the trend towards more sophisticated KV-cache management (e.g. H2O [7], SnapKV [8], StreamingLLM [1], image token dropping [9], MEMENTO [2], TGV-KV [3], AirCache [4]), we believe our work highlights risks that arise with recycling or maintaining the KV-cache. The attack requires white-box access, we did not attack any deployed system, and our targets are illustrative. On a positive note, since the attack poisons the KV cache, recomputing it without the adversarial image removes the effect (Table 2). This aligns with other defences that constrain access to untrusted data, such as CaMeL [10]. We will also release our code to support research on defences.

AuthorsOpenReview ↗

Response to B639 (2/2)

References

[1] Xiao, Guangxuan, et al. "Efficient Streaming Language Models with Attention Sinks." ICLR, 2024.
[2] Kontonis, Vasilis, et al. "Memento: Teaching LLMs to Manage Their Own Context." arXiv:2604.09852, 2026.
[3] Liu, Jizhihui, et al. "TGV-KV: Text-Grounded KV Eviction for Vision-Language Models." ICML, 2026.
[4] Huang, Kai, et al. "AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inference." ICCV, 2025.
[5] Zheng, Lianmin, et al. "SGLang: Efficient Execution of Structured Language Model Programs." NeurIPS, 2024.
[6] Penaloza, Emiliano, et al. "KV-streams for efficient compaction in agentic reinforcement learning." arXiv preprint arXiv:2609.35750.
[7] Zhang, Zhenyu, et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." NeurIPS, 2023.
[8] Li, Yuhong, et al. "SnapKV: LLM Knows What You Are Looking for Before Generation." NeurIPS, 2024.
[9] Chen, Liang, et al. "An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models." ECCV, 2024.
[10] Debenedetti, Edoardo, et al. "Defeating Prompt Injections by Design." IEEE SaTML, 2026.

Reviewer_KCDvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an adversarial attack that persists a payload in the model's KV-cache and activates it with a trigger prompt after a number of benign turns. It follows the Visual Memory Injection (VMI) setting, in which an adversarial image stays in context, the model behaves normally on benign turns, and a later trigger elicits an attacker-chosen response; the key difference is that the perturbation is optimized for the behaviour that remains after the input has been removed from attention, so that its influence can only be carried by the KV states of the tokens computed while the input was still visible.

The attack objective combines a grounding term, which keeps the anchor turn benign while the image is visible, with a target term computed at the trigger with the image masked. Since no token attends to the image at the trigger, the target loss has no direct gradient with respect to the input and can only be optimized through the retained cached states. The authors optimize image pixels or soft-prompt embeddings with ℓ∞\ell_\infty APGD, using a multi-pass forward pass that mimics each successive removal, and evaluate four removal settings that differ in when access to the image is lost and in whether the context is compacted while the summary's KV-cache is retained.

Experiments are run on open-weight language and vision-language models across multiple targets, held-out conversations and held-out context distributions. The attacks remain effective after the image is masked, stay effective on conversations substantially longer than those used during optimization, and survive compaction that keeps the summary cache, while benign turns stay largely intact. A cache-swap ablation and a linear probe in the residual stream indicate that the persistent influence is carried by the KV-cache.

Critical strengths

The paper explains why the attack fails. An attack trained in the usual way, with the image always visible, obtains SR∧=0%\mathrm{SR}_\wedge = 0\% once the image is masked at the trigger. The failure is therefore not caused by the absence of the image from the context, but by the absence of any training for survival. The authors then define an objective that can only act through the KV states written while the image was visible, so persistence becomes a property that can be trained for.

The removal settings support both attribution and causal evidence. Four removal settings decouple two independent choices. One concerns when access to the image is lost. The other concerns whether the context is compacted while the summary KV cache is retained. The attribution is causal. The evidence supports the claim that the retained states carry the influence.

Critical weaknesses

W1. The threat model is white box and per model. The claims are nevertheless stated without this scope, and no defense is proposed. Each perturbation is optimized with gradients from the model on which it is then evaluated. The Gemma experiment retrains the attack rather than transferring it. The ethics statement correctly notes that transfer was not tested. The abstract and the introduction, however, present persistence as a property of retained KV states without this qualification. The reader therefore cannot tell whether the finding is a general property of these architectures or a property of the specific weights. The authors should state the scope where the claims are made. Where feasible they should report a transfer probe in both directions without reoptimization. A negative result would also be informative, because it would bound the claim. The paper should additionally propose at least one testable defensive mechanism and state its assumptions. The conclusion currently restates the cache swap ablation rather than measuring a defense.

W2. The method is not positioned against other adversarial attacks. The only comparison in the paper is with VMI, which scores 0%0\% under masking, and between the base and the regularized recipes of the proposed method. The reader therefore has no reference point for how strong the attack is relative to other jailbreaks or injection attacks. A comparison inside the masked protocol is admittedly of limited value, because competing attacks require the input to remain in context. A comparison is nevertheless necessary when the input is visible.

W3. The attack is not evaluated under any defense. The paper should test the method under defensive mechanisms and report success and benign quality together. It is important for an assessment of the method to verify whether it can break defenses that are currently shown to be effective.

W4. The resolution of the adversarial images is not reported. Under a fixed budget of 8/255, the number of optimizable coordinates and the total perturbation scale with resolution. The authors should report the raw resolution of the image sets, the resolution passed to the vision encoder, and confirm that no resize occurs between optimization and evaluation.

Questions for authors
  1. All perturbations are optimized on the very model at which they are evaluated, and transfer is not reported. Do they remain effective on another model, for example Gemma-4-12B, without reoptimization?

  2. The only baseline drawn from prior work is the VMI objective, reported in a single setting. Why are other adversarial or jailbreak attacks not compared under the same protocol, and could the closest comparable one be added?

  3. The paper is motivated by defense but measures none. Does recomputation of the retained states restore benign behavior on the same conversations, and at what cost?

2 comments
AuthorsOpenReview ↗

Response to KCDv (1/2)

We thank reviewer KCDv for their review, and for highlighting the causal attribution. We address your concerns below (G1–G6 refer to the results found in the general response).

W1. We agree that the knowledge of the attacker should be clearly stated, and will add "white-box, per model" to the abstract, introduction and threat model. On whether persistence is a property of the architecture or of the specific weights, our results suggest both: 1) a persistent attack can be trained on every model we tried (Qwen3-8B, Qwen3-VL-8B, Gemma-4-12B with sliding-window attention in 40/48 layers, and the MoE Gemma-4-26B-A4B; G3), but 2) a given perturbation is specific to the weights it was trained on. It does not transfer between Gemma-4-12B and Gemma-4-26B-A4B in either direction (0% SR∧, vs. 85% and 65% on the source model), and partially transfers to a public LoRA fine-tune (G4). As a defence, one could recompute the retained states without the untrusted input (see W3 + Q3). This requires the defender to know which inputs are untrusted (or recomputes everything), though there are recent defences that isolate untrusted data, such as CaMeL [8], that explore this direction. We added a discussion on how the community could defend against persistent attacks to the paper.

W2 + Q2. Most other attacks are not suited to this threat model: we target a specific response (e.g. hijacking an agent) after the input is no longer accessible, whereas attacks like VMI [1], image hijacks [2] or indirect prompt injection [3] require the input to stay in context. The original VMI gets 0% under masking (Sec. 4.3). As a reference point with the input visible, the same P-VMI images reach 70% SR∧ after 16 turns (51% masked). On the ChartQA/ScreenQA conversations in G5, the validation-selected images reach 41.25% / 46.25% with the image visible after three/six turns, compared with 70.0% / 76.25% when masked. These are visibility controls using the same trained images, not separately trained visible attacks. Nevertheless, to compare existing jailbreaks, we are currently attempting to train a GCG attack using our masking objective; we will report our findings here if time permits, or otherwise in the final version of our paper.

W3 + Q3. In our KV-cache ablation (Table 2), rebuilding the cache from the same conversation text without the adversarial image never elicits the target, for images optimised under both $\texttt{mask@trigger}$ and $\texttt{mask@anchor}$. This requires additional computation, whose cost depends on the context being reprocessed. There is also a potential, task-dependent utility cost: retained KV states can preserve useful information that is absent from the summary text. MEMENTO [4], for example, reports lower reasoning accuracy when these states are recomputed without the preceding reasoning context. Thus, recomputation can remove the cached adversarial influence, but may also discard useful information. We do not claim that this utility loss occurs for every task or quantify it in our setting.

We also ran a preliminary check on how robust the trained attacks are to image transformations: a 2-pixel shift drops SR∧ from 90% to 0% in a small probe (5 images × 2 targets, 3 turns), so our attacks are not robust to transformations out of the box. However, adversarial examples can be made robust to input transformations, compression and even physical-world viewpoint changes by including those transformations in the optimisation [9, 10], and patch-based image hijacks against VLMs already survive noise and JPEG compression without being trained for it [2]. Likewise, optimising against an ensemble of surrogate models is a standard technique for black-box transfer, both for image classifiers [6] and for multimodal LLMs [11]. There is no theoretical reason to expect persistent KV-cache attacks to be exempt from these techniques.

Yet, we deliberately chose not to pursue transformation-invariant or cross-model transfer optimisation. The objective of this work is to establish whether adversarial influence can persist in retained state after the input is removed, rather than to engineer a robust, deployable exploit. Given the dual-use risks of this vulnerability, as also highlighted by the ethics concerns raised during review (see Reviewer B639), developing an out-of-the-box attack that transfers across models and survives standard preprocessing would provide clear misuse potential with little scientific benefit beyond what our proof-of-concept already demonstrates. We have added this reasoning to the revised paper.

AuthorsOpenReview ↗

Response to KCDv (2/2)

W4.

  • LMARKS and COCO images are 224×224.
  • Qwen3-VL-8B: the vision encoder receives 224×224 (dynamic resizing disabled), and optimisation and evaluation use the same pixel tensor, with no resize in between.
  • Gemma-4: the perturbation is applied to the processor's 768×768 input, i.e. ~11.8× as many coordinates under the same 8/255 budget.
  • New datasets: ChartQA and ScreenQA images are resized with their aspect ratio preserved and padded to 224×224 before optimisation.
  • SGLang takes PNGs, so it includes 8-bit quantisation.

We will add a resolution table to the appendix.

Q1. Following your review, we ran transfer probes without re-optimisation (G4). The attacks do not transfer between Gemma-4-12B and Gemma-4-26B-A4B in either direction, but partially transfer to a public LAION LoRA of Gemma-4-12B (54% SR∧ after 3 turns, 38% after 6, sampled triggers). We have not tested Qwen3-VL-8B → Gemma-4-12B; given the within-family result and prior work [5], we expect little to no transfer without optimising for it. Past work shows that transferability can be improved by optimising against model ensembles [6] and with input diversity [7], although transferable image jailbreaks between VLMs remain difficult [5]. We deliberately do not pursue this here, for the dual-use reasons given in W3 + Q3.

References

[1] Schlarmann, Christian, and Matthias Hein. "Visual Memory Injection Attacks for Multi-Turn Conversations." arXiv:2602.15927, 2026.
[2] Bailey, Luke, et al. "Image Hijacks: Adversarial Images can Control Generative Models at Runtime." ICML, 2024.
[3] Greshake, Kai, et al. "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection." ACM AISec Workshop, 2023.
[4] Kontonis, Vasilis, et al. "Memento: Teaching LLMs to Manage Their Own Context." arXiv:2604.09852, 2026.
[5] Schaeffer, Rylan, et al. "Failures to Find Transferable Image Jailbreaks Between Vision-Language Models." ICLR, 2025.
[6] Liu, Yanpei, et al. "Delving into Transferable Adversarial Examples and Black-box Attacks." ICLR, 2017.
[7] Xie, Cihang, et al. "Improving Transferability of Adversarial Examples with Input Diversity." CVPR, 2019.
[8] Debenedetti, Edoardo, et al. "Defeating Prompt Injections by Design." IEEE SaTML, 2026.
[9] Athalye, Anish, et al. "Synthesizing Robust Adversarial Examples." ICML, 2018.
[10] Brown, Tom B., et al. "Adversarial Patch." arXiv:1712.09665, 2017.
[11] Dong, Yinpeng, et al. "How Robust is Google's Bard to Adversarial Image Attacks?" arXiv:2309.11751, 2023.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

General Response

We thank the reviewers for their feedback. Since several concerns overlap, we summarise the new experiments here (G1–G6) and refer to them in the individual responses.

G1. Seed selection. To address reviewer B639’s comments, we re-exported the compaction results (Fig. 7, Table 7) so that one seed per image, target and removal setting is selected using only the validation scores, then kept fixed across all (k, j) points and evaluation conversations. In the submission, the seed was selected per (k, j) point on the evaluation conversations.

Removal settingSR∧, submittedSR∧, validation-selected [95% CI]Benign mean / min
$\texttt{mask@anchor}^{\mathsf{C}}$70.2%64.2% [51.7, 75.6]0.934 / 0.809
$\texttt{mask@postsummary}^{\mathsf{C}}$88.8%69.0% [62.0, 76.1]0.866 / 0.710

Validation-selected SR∧ by number of benign turns before (k) and after (j) compaction, shown as $\texttt{mask@anchor}^{\mathsf{C}}$ / $\texttt{mask@postsummary}^{\mathsf{C}}$:

j=0j=1j=2j=3j=4
k=171% / 78%69% / 78%64% / 74%59% / 70%46% / 58%
k=275% / 81%74% / 77%71% / 73%62% / 69%51% / 62%
k=372% / 79%75% / 74%71% / 71%66% / 62%52% / 55%
k=467% / 76%67% / 69%64% / 65%61% / 55%46% / 53%

Across all (k, j), the mean benign score stays between 0.91 and 0.96 for $\texttt{mask@anchor}^{\mathsf{C}}$ and between 0.84 and 0.89 for $\texttt{mask@postsummary}^{\mathsf{C}}$. Validation-only seed selection reduces mean SR∧ by 6.0 percentage points for $\texttt{mask@anchor}^{\mathsf{C}}$ and 19.8 points for $\texttt{mask@postsummary}^{\mathsf{C}}$. Overall, though we see a drop in performance, the corrected results still demonstrate persistence through compaction. We will update the affected figures.

G2. KV eviction in SGLang. To demonstrate our method working on realistic inference infrastructure, we added image support to the KV-Streams [1] fork of SGLang [2]. After the model generates a summary, the KV containing the image is evicted and generation continues from the retained system prompt KV and summary KV. We use our original attack PNGs, which were only optimised against attention masking using the Transformers library, on 6 LMARKS images × 2 targets × 3 seeds. SR∧ with the validation-selected seed, by number of benign turns before (k) and after (j) compaction:

j=0j=1j=2j=3j=4
k=181.2%81.2%70.8%66.7%60.4%
k=279.2%75.0%70.8%68.8%56.2%
k=375.0%75.0%70.8%68.8%47.9%
k=468.8%66.7%66.7%56.2%47.9%

The grid mean is 67.7% SR∧, compared with 68.3% for the original HF tensors on the same six images (69.0% over all 20 images; G1). These results show comparable persistence under physical eviction with a real inference framework, though there is some slight difference between image representations and trajectories between the evaluations.

G3. New model results: Gemma-4-26B-A4B. We trained the attack on Gemma-4-26B-A4B (MoE) with 20 LMARKS × 2 targets × 3 seeds, image masked at the trigger and observed that it works well. We report the best seed selected on validation data, and the mean over all seeds:

Benign turnsSR∧, validation-selectedSR∧, mean over seedsBenign, validation-selectedBenign, mean over seeds
097.5%95.0%––
195.0%90.8%0.9750.925
380.0%77.5%0.9750.924
662.5%70.8%0.9750.924

Note that validation uses a single 3-turn conversation, hence the selected seed can be worse on 6-turn conversations. Overall, these results demonstrate that this persistence phenomenon works and can be exploited on a variety of model sizes and architectures (e.g. MoE, sliding window attention) without significant changes to our attack setup.

AuthorsOpenReview ↗

General Response

G4. Attack Transfer. Without re-optimising the image on the target model, we evaluate it masked at the trigger:

Source → targetImages × targetsBenign turnsSR∧Source model on itself
Gemma-4-12B → Gemma-4-26B-A4B5 × 260%85%
Gemma-4-26B-A4B → Gemma-4-12B5 × 260%65%
Gemma-4-12B → LAION LoRA20 × 23 / 654% / 38%–

The attacks do not transfer between different models, but partially transfer to a public LoRA fine-tune of the source model. The LAION row uses sampled triggers at T=1. Prior work similarly finds that image jailbreaks rarely transfer between VLMs [3], although historically transferability has been improved by optimising against an ensemble of models [6] and with input diversity [7]. We deliberately do not optimise for transfer (see Reviewer KCDv, W3 + Q3). Nevertheless, the attack also remains relevant without transfer. Many companies and users self-host a small number of popular open-weight models, often as public fine-tunes, so an attacker can potentially optimise directly against the weights that are actually deployed.

G5. More diverse data: generated UIs, ChartQA and ScreenQA. To broaden image coverage and better mimic the visual inputs a coding agent might encounter when building or debugging interfaces, we evaluated Qwen3-VL-8B attacks on:

  • Generated UIs: five dashboards rendered programmatically at 224×224: a monthly-sales bar chart, weather forecast, project task board, website-traffic line chart and inventory chart. Each has a one-sentence grounding target, such as “This dashboard shows monthly sales in a blue bar chart.”
  • ChartQA [4]: ten charts from a dataset for question answering about charts, including visual comparisons and numerical reasoning. These provide structured visual content beyond the photographs used in our main experiments.
  • ScreenQA [5]: ten mobile-app screenshots from a dataset of questions about interface content. These introduce controls, labels and layouts resembling screenshots an agent could receive during interface development or debugging.

External images are resized with their aspect ratio preserved and padded to 224×224. We retain the same training conversations, two targets, and optimisation recipe. For ChartQA/ScreenQA, we train three seeds per image/target and select the highest-scoring seed on validation data, while the generated dashboards use only a single seed.

New test conversations. For each image, we use GPT-6 to generate two new six-turn conversations about its content and design. For example, a chart discussion moves from “What is being compared?” to “What conclusion is justified?”, while a screenshot discussion asks which control to use next and how its layout could be clearer. Questions were fixed before evaluation and excluded from training and validation. The model generates its own replies, with the attack trigger tested after three or six turns and the image masked only at the trigger.

SourceSeed selectionSR∧ stock, 3 / 6 turnsSR∧ party, 3 / 6 turnsBenign, 6 turns (stock / party)
Generated dashboards (5)Single seed70% / 60%90% / 100%0.97 / 0.92
ChartQA (10)Best of three on validation70% / 75%70% / 75%0.84 / 0.80
ScreenQA (10)Best of three on validation65% / 75%75% / 80%0.55 / 0.83

Across the twenty ScreenQA + ChartQA images, validation-selected SR∧ is 70.0% after three turns and 76.25% after six. Mean benign scores on the external images are 0.68 and 0.76, versus approximately 0.99 for clean conversations. Though these results do not constitute an end-to-end coding agent evaluation, these results demonstrate persistence using images much more relevant to a coding agent, and also on benign test conversations that are specific to that image while only training on the same generic training set as our original LMARKS results.

G6. Paper changes.

  1. State the white-box, per-model scope in the abstract, introduction and threat model.
  2. Motivate removal with context management (eviction, compaction) rather than safety filters.
  3. Add a defences and limitations section.
  4. Replace Fig. 7 and Table 7 with validation-selected versions.
  5. Add an appendix with G2–G5, the defence results and the image resolutions.
AuthorsOpenReview ↗

General Response

References
[1] Penaloza, Emiliano, et al. "KV-streams for efficient compaction in agentic reinforcement learning." arXiv preprint arXiv:2609.35750.
[2] Zheng, Lianmin, et al. "SGLang: Efficient Execution of Structured Language Model Programs." NeurIPS, 2024.
[3] Schaeffer, Rylan, et al. "Failures to Find Transferable Image Jailbreaks Between Vision-Language Models." ICLR, 2025.
[4] Masry, Ahmed, et al. "ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning." Findings of ACL, 2022.
[5] Hsiao, Yu-Chung, et al. "ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots." NAACL, 2025.
[6] Liu, Yanpei, et al. "Delving into Transferable Adversarial Examples and Black-box Attacks." ICLR, 2017.
[7] Xie, Cihang, et al. "Improving Transferability of Adversarial Examples with Input Diversity." CVPR, 2019.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.