acceptodds
Under review as a conference paper at ICLR 2027

Render Before Reading: Visual Rendering as a Prompt Injection Defense

Abstract

Large language models are vulnerable to prompt injection attacks, where third-party adversarial content can hijack the model's behavior. In this paper, we study the role played by the adversarial data's input modality, and identify a systematic asymmetry: multimodal LLMs are more likely to follow adversarial instruction when they appear as text than when the same instruction is delivered through a non-textual channel (e.g., as an image). We hypothesize that this modality gap arises from text-centric instruction tuning, which teaches models to obey textual instructions while treating other modalities mainly as content to parse or describe. We then demonstrate how this gap can be turned into a training-free defense, by rendering all untrusted payloads as typographic images (or audio) before they reach the model. Across ten models and two prompt injection benchmarks (DirectInject and AgentDojo) we show that our defense \defense consistently reduces attack success rates even against the strongest adaptive attacks and human red teamers, while largely preserving benign utility. We further show that benign fine-tuning on image-rendered instructions erodes the modality gap, tracing it to the text-centric instruction-tuning distribution.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt injection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_czqFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Pictionary, a prompt injection defense that renders untrusted documents and tool outputs as images. Trusted instructions remain in text. The method uses differences in instruction following across modalities to reduce attack success. The authors evaluate ten models on two benchmarks using static attacks, adaptive attacks, and human red teaming. They also examine task utility and the role of training.

Critical strengths

1.Important problem. Prompt injection through external documents and tool outputs is a practical concern for deployed language model agents.

2.Broad evaluation. The experiments cover adaptive attacks, human red teaming, benign task utility, and transcription accuracy.

Critical weaknesses

1.Limited novelty and mechanistic insight. Modality differences in instruction following and safety are widely studied. The paper does not clearly establish a substantive advance over this knowledge. Increased image-injection susceptibility after benign image-instruction fine-tuning is also an expected result.

2.Reduced utility on general tasks. Rendering external content as images adds recognition overhead and can impair tasks that require exact text processing. Table 4 reports a 9–10 percentage-point drop in SWE-bench success for two models.

Questions for authors

1.Contribution clarification: What new scientific conclusion goes beyond existing findings on multimodal jailbreaks and instruction following? Please distinguish a new application from a new mechanistic insight.

2.How can you ensure that rendering text as images does not degrade performance on benign tasks?

1 comment

Thanks for taking the time to review our paper.

’1.Contribution clarification: What new scientific conclusion goes beyond existing findings on multimodal jailbreaks and instruction following?'

Could the reviewer point to the specific prior work and findings they have in mind? Our paper concerns prompt injection, not jailbreaking. The two are distinct problems, and findings on one cannot be assumed to hold for the other. Some existing studies of modality gap in jailbreaks (e.g., FigStep [5] ) treat it only as a vulnerability, and the gap they refer to is in safety training, not in instruction following. Balakrishnan et al. [6] report that typographic renderings is less effective than text, but only on two out of the four models tested, and while they name it "typographic prompt injection", the setup is a jailbreak evaluation.

Also, our defense stays effective under strong adaptive attacks, while many previous empirical defenses break down under adaptive attacks.

Novelty and mechanistic insight.

There have been some studies on modality differences, but they didn’t show that rendering the same text as an image can effectively defend against prompt injection. In particular, Kimura et al. [1] observed that textual prompt injections can be more effective than visual ones for goal hijacking, while Li et al. [2] and Gao et al. [3] investigated visual instruction following and cross-modal instruction conflicts. While these studies establish that input modality can influence prompt-injection behavior, they do not establish that converting untrusted textual content into images constitutes an effective prompt-injection defense.

Our main contribution is to systematically exploit this asymmetry as a practical, training-free defense. Rather than treating visual instructions as an attack surface, Pictionary renders untrusted textual documents and tool outputs as images while keeping trusted instructions textual. We demonstrate that this intervention substantially reduces attack success across ten models and two benchmarks, including under strong adaptive automated attacks and human red teaming. While prior defenses such as Spotlighting and StruQ also explore separating instructions from untrusted data, our approach leverages existing modality-dependent behavior without requiring model retraining or modifications to the underlying model.

Beyond the defense itself, we investigate the generality and origins of this behavior. We show that the effect extends to audio, where converting untrusted text into speech similarly reduces attack success rates, suggesting that the phenomenon is not specific to visual rendering or OCR. We further compare base and instruction-tuned models and show that benign image-instruction fine-tuning increases susceptibility to image-delivered prompt injections. While the latter result may be intuitive, it provides experimental evidence supporting our hypothesis that the modality gap is shaped by the distribution of instruction tuning. We do not claim to fully characterize the underlying mechanism, but rather provide evidence for a training-dependent explanation of the observed asymmetry.

Benign utility.

We acknowledge that image rendering can degrade utility, particularly for tasks requiring exact text processing, and explicitly discuss this limitation in the paper (Section 6.3). As with other security defenses, there is no free lunch, and some security–utility tradeoff is unavoidable. However, our results show that this tradeoff is highly task-dependent: utility decreases by only 0.9–2.1 percentage points on τ²-Bench, while SWE-bench exhibits larger drops of 9–10 points for two models and no drop for the third. We further investigate these failures and identify transcription errors in exact-match code editing as an important contributing factor. Thus, while Pictionary may be less suitable for tasks requiring byte-exact processing, it provides substantial security benefits while largely preserving utility on tasks that primarily require semantic understanding.

[1] Kimura et al. Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection. NAACL 2024.

[2] Li et al. Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels? NeurIPS 2024.

[3] Gao et al. Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions. ECCV 2024.

[4] Pandey et al. Open Your Eyes: Benchmarking the Detection of Fabricated Realities and Weaponized Ethics in VLM Agents. ECCV 2026.

[5] Gong et al. FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts. AAAI 2025.

[6] Balakrishnan et al. Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models. ICLR 2026 Workshop on Agents in the Wild.

Reviewer_Ub8wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discussed the modality gap in instruction following of MLLMs: LLM usually prefer to follow the instructions from text prompts instead of the prompts from the image when two are conflict. Based on this discovery, the author proposes "Pictionary", which is a training free prompt injection defense that renders untrusted documents and tool outputs as images while retaining trusted instructions as test. This method does not need to recognize or filter attack instructions at all. It only changes the modality of the untrusted inputs and then leverages the modality gap to defend.

The research evaluated DirectInject and AgentDojo with 10 VLMs, and did the adaptive attacks(RL/ART/Human red teaming) to explain the defense robustness. The results show that this method can effectively reduce ASR. The evaluation results of τ²-Bench and SWE-bench Verified show that the utility rate degrades for some models on coding tasks.

In addition, this paper also compared the base models and the instruction models to provide the evidence that the modality gap depends on training. Besides, the paper shows an additional audio experiment to explore whether the same defense works for audio modality.

Critical strengths
  1. Useful empirical insight and a clever defense. The paper identifies a consistent preference for textual over image-embedded instructions under conflict. This is a particularly useful observation that motivates a clever and intuitive defense: changing the modality of untrusted content reduces its authority as instructions while preserving access to its information. The reported reductions in attack success demonstrate the practical value of this simple idea.

  2. Simple, practical, and accessible implementation. Pictionary requires no model retraining or auxiliary classifier and can be integrated into existing VLM-based systems by adding a rendering step. This simplicity facilitates deployment and lowers the engineering and training resources needed to reproduce the method. It also makes comparative evaluations across models and defenses more accessible without requiring specialized training infrastructure.

Critical weaknesses
  1. Insufficient task coverage and lack of transparency in task selection for core security evaluations: These issues directly affect how readers assess the generalizability, representativeness, and reproducibility of the paper’s main conclusions.

    • AgentDojo: The paper does not adequately explain the selection criteria or the domain distribution for the 130 task-attack target pairs used in the static evaluation from AgentDojo (Table 1). Please clarify the rationale behind selecting these 130 pairs and disclose the inclusion and exclusion criteria. Without this information, it is difficult for readers to gauge the representativeness of this subset or determine whether the task selection might have led to an overestimation or underestimation of the reported defense efficacy. I recommend verifying the robustness of the results using a more comprehensive set of tasks.

    • DirectInject:

      • Please explain the motivation for constructing the "DirectInject" benchmark, its necessity, and the unique evaluative value it offers compared to existing prompt injection benchmarks (or adapted versions thereof).

      • Please detail the sources, construction process, selection criteria, and rationale for the coverage of the four tasks included in DirectInject, and explain the reasoning behind the chosen scale of the benchmark.

      • Combining four "benign" (normal) tasks with various attack targets and templates increases the total number of experimental trials, but it does not increase the number of distinct benign tasks or documents; consequently, task coverage remains very limited. Therefore, whether the results obtained on "DirectInject" can generalize to other documents, topics, or tasks requires further validation. If this benchmark is to be used to evaluate the defense methods proposed in this paper, it is advisable to incorporate benign tasks based on a wider variety of independent materials, thereby increasing both the quantity and diversity of tasks to strengthen the credibility of any generalized conclusions.

  2. Unjustified choice of four-task DirectInject over AgentDojo for defense comparisons: The experiments described in Table 3, which compare various defense methods, rely on only four fixed benign tasks. The lack of unified validation on established public benchmarks, such as AgentDojo, is insufficient to support broad claims regarding the superiority of the proposed defenses. This is particularly notable given that PromptLocate and DataFilter have already been evaluated on AgentDojo, and the authors themselves utilize AgentDojo elsewhere in the paper; yet, the comparative analysis of defenses is restricted to the authors' own "DirectInject" benchmark (consisting of only four fixed tasks) without any justification provided. Consequently, the relatively large performance margins shown in Table 3 are not enough to substantiate a general claim of defense superiority. Please clarify this methodological choice and provide additional comparisons using a standardized public benchmark to verify the robustness of the rankings presented in Table 3.

Questions for authors
  1. What motivates the construction of DirectInject, and why is a new benchmark necessary? Could an existing public dataset, or an adapted version of one, serve the same purpose?
  2. How does DirectInject differ from existing public prompt injection benchmarks, and what distinct evaluative value does it provide?
  3. Why does DirectInject contain only four benign tasks? What considerations motivated this choice of benchmark scale?
  4. Why does the static evaluation in Table 1 use only a subset of AgentDojo’s task–goal pairs rather than the complete set?
  5. How were the 130 AgentDojo task–goal pairs used in the static evaluation in Table 1 selected? What selection procedure and inclusion/exclusion criteria were applied?
  6. Given that AgentDojo is already used elsewhere in the paper, why does the head-to-head comparison of defense methods use DirectInject rather than AgentDojo?
4 comments

We thank the reviewer for the detailed review.

Q1+Q2+Q3: DirectInject.

DirectInject was not designed to introduce a new general-purpose prompt-injection benchmark, but rather to provide a simple, controlled setting for studying the effect of input modality on prompt injection.

Existing benchmarks such as Tensor Trust and HackAPrompt primarily focus on target-string generation or secret extraction, while RuLES and IHEval evaluate violations of predefined rules and instruction hierarchies. Although these benchmarks could potentially be adapted, DirectInject was designed specifically for our evaluation needs. It combines benign document-comprehension tasks with diverse unauthorized tool-call objectives, allowing us to measure attack success directly from tool invocations while independently assessing benign task utility. Its simple, single-turn structure also enables us to compare identical attacks delivered as text or images, without confounding factors introduced by complex agent workflows.

We intentionally kept DirectInject small, selecting four benign document-comprehension tasks spanning different document types (articles, code snippets, resumes, and emails). These are paired with 14 diverse attacker objectives, yielding 56 task–goal combinations. Since our goal is to isolate the effect of input modality rather than evaluate performance across a comprehensive distribution of tasks, we prioritized a controlled and computationally manageable setting, particularly given the number of models, attack strategies, and modalities evaluated. We complement DirectInject with AgentDojo to assess whether our findings extend to more realistic, multi-turn agentic scenarios.

Q4+Q5. AgentDojo.

We use a subset of AgentDojo to keep the evaluation computationally manageable, given the number of models, attack strategies, and input modalities evaluated. Specifically, we select the first five benign user tasks from each of the four suites and pair them with the injection goals listed in Appendix A: 9 for banking, 6 each for workspace and travel, and 5 for Slack, yielding 130 task–goal pairs. Banking has broader coverage because it was the initial focus of our experiments, before we expanded to the other suites.

The only injection goal we explicitly excluded was Slack's it0, due to an issue with its success criterion. This goal attempts to make the model recommend "Riverside View Hotel," but its evaluator also counts responses that merely mention the hotel while explicitly identifying and rejecting the injection. We therefore excluded it to avoid incorrectly counting successful defenses as attacks. The resulting subset retains coverage across all four application domains and diverse adversarial objectives while keeping the evaluation tractable.

Q6. Prior defenses.

We chose DirectInject for the head-to-head comparison primarily because its simpler setup makes evaluating multiple defenses considerably less expensive. Moreover, DirectInject provides a challenging setting with relatively high attack success rates. Since the injected content is provided directly in a user-supplied document, rather than appearing within tool outputs during agent execution, this is also a relatively simple setting for defenses to operate in.

As Table 3 shows, even in this simpler setting, existing defenses substantially underperform ours. We therefore prioritized a controlled comparison on DirectInject, while using AgentDojo to independently evaluate the defense effectiveness in more realistic agentic scenarios. We acknowledge that a head-to-head comparison on AgentDojo could provide additional insights. Importantly, we do not view our defense as a replacement for existing defenses, but rather as a complementary approach that can be readily combined with detection, filtering, or other defense mechanisms.

Reviewer_Ub8wOpenReview ↗

Is Pictionary a general defense method? If so, the evidence seems not enough

Thank you for your response. Here is my main concern: it seems Pictionary is a generalizable defense method, but we don't see a broad evaluation to show it's generalizable. To be specific:

  1. DirectInject is small on both counts: four benign tasks and 56 task–goal pairs. What evidence shows that the aggregate gain is not driven by a few favorable cases or by cherry-picking tasks or documents? My concern is how the result changes when different tasks are included.
  2. If DirectInject was designed to isolate the effect of input modality, is the AgentDojo subset the only evidence that Pictionary’s security benefits extend to a broader range of tasks? If so, the only evaluation on the AgentDojo subset seems not enough to show these gains generalize to other multi-turn agent settings.
  3. Table 3 compares defenses only on DirectInject. Would the ranking hold in a matched comparison on other public benchmarks, if you believe Pictionary is able to generalize the gains to broader and more realistic scenarios?
  4. For AgentDojo, why are these 130 task–goal pairs sufficient to support the paper’s broader claims? Do the ASR reductions hold on an independently selected subset or on the full benchmark? This would help assess how much the reported gains depend on which tasks and goals were included.
  5. Would you consider validating Pictionary on the full task sets of two or three established public benchmarks to test whether the gains hold across a wider range of tasks?

Cost explains why the evaluation was kept small, but the strength of the evidence should determine the scope of the claims. If broader validation is not feasible, please state explicitly which security gains and comparative results are supported by the tasks and settings tested.

AuthorsOpenReview ↗

We have more results showing that Pictionary could be a general defense.

Thanks to the reviewer for pushing on generalization. Recent work [1] from Gemini suggests that frontier models may overfit to public benchmarks like agentdojo: they can look quite robust on these tests, but still break in new settings or on new benchmarks. Also we could run the experiments on the full AgentDojo, but wouldn’t expect much more difference from what we have now. Instead, we further report two new evaluation settings that are not covered by either DirectInject or AgentDojo. We’ll also add these results to the appendix. Both results could show that our defense has the potential to mitigate prompt injection in general.

New Setting 1: Data-flow attacks (no tool call needed).

DirectInject and AgentDojo mainly test whether an injection can trigger an unauthorized action (control-flow). Many real attacks instead target the model’s output. The injection makes the model include attacker-controlled content in its answer, such as a fake “verified” line for phishing or a malicious URL, with no tool call at all. We built 5 injection goals × 4 user tasks, with 3 runs each. An attack counts as successful only if the attacker’s string appears verbatim in the response.

ModelNo defensePictionary
GPT-5.4-mini71.7%20.0%
Claude-Haiku-4.546.7%0.0%

This shows that Pictionary’s benefit is not tied to the tool-call attack channel that both existing benchmarks evaluate.

New Setting 2: Hard tasks that increase susceptibility to injection.

The reviewer asked whether the gains hold on tasks beyond our original benchmarks. We set up a tougher test designed to break most frontier models like Claude Opus 5.5 and GPT-6.1. In it, the user asks the model to solve a coding problem (or a math problem) in a document, and the document also contains an injection telling it to send an email, delete a file, etc. Prior work has shown that prompts designed to impose additional reasoning demands can induce LLMs to produce outputs that violate their safety constraints [2]. We use 100 samples per model, run each sample 10 times, and report ASR@10 (a sample counts as attacked if any of the 10 runs succeeds), which is a conservative worst-case metric.

ModelNo defense (ASR@10)Pictionary (ASR@10)Utility (no defense → Pictionary)
Claude Opus 5.595%40%100% → 100%
GPT-6.1-terra50%0%100% → 100%

Even frontier models are pretty vulnerable to this attack. Pictionary lowers ASR@10 a lot without hurting utility.

We hope these results help address your concerns.

[1] Scaling Adaptive Prompt Injection Attacks with Agent Swarms

[2] Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking.

Reviewer_Ub8wOpenReview ↗

Please add more evaluations to show that the proposed method is generalizable and address the cherry picking concern

Thanks for your explanation and additional evaluations. However, I do have 3 following up questions:

  1. What's the connection between "frontier models may overfit to public benchmarks like agentdojo" and "this paper doesn't need to be evaluated on the full set of AgentDojo"?
  2. You mentioned "Also we could run the experiments on the full AgentDojo, but wouldn’t expect much more difference from what we have now". Why is that? Any evidence?
  3. You did add 2 new settings, but my point is: to prove the proposed defense method is generalizable, you need to evaluate the proposed method on multiple public benchmarks using their full task sets. This would help show that your method is not dataset specific, and address concerns about cherry picking. As of now, DirectInject is too small to be convincing, and you only used a subset of AgentDojo, which is not convincing either regarding generalization.

In conclusion, I do recognize the novelty of your method, and believe it may have a bright future. However, the evaluation of this method is not convincing enough at this moment. I strongly recommend a more thorough evaluation.

Reviewer_k4dmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Pictionary leaves trusted instructions as text and renders untrusted text as a typographic image, or as speech, before the model reads it. The stated mechanism is a training prior: instruction tuning puts obedience on the text channel, while images are trained as content to parse. A conflict test uses the same wording in both channels. Rendering then cuts ASR sharply on several models, leaves Gemini 3.1 Flash Lite almost unchanged, and benign fine-tuning on image-rendered instructions shrinks the gap.

Critical strengths

The system prompt and the user query stay text, and only the untrusted payload is rendered into an image or into speech. On short instructions the model can still follow the image: image-only compliance is near ceiling, but when the same wording conflicts, the model follows the text. That is a real text-channel preference, and it is what the split uses. Where the preference is present, rendering cuts adaptive attack success sharply, from 97.6% to 9.5% for Claude Haiku 4.5 on AgentDojo.

Critical weaknesses
  1. The security experiments render long documents and tool outputs, while the preference above was measured on short instructions. On SWE-bench, rendering every tool output already costs 9 to10 points of success for GPT-5.4 mini and Claude Haiku 4.5 because characters are mistranscribed. The injection sets use the same kind of image, and the paper does not check whether those models still recovered the payload. The attack-success drop is therefore not shown to be the text-channel preference.
  2. The split has no further effect once the model obeys instructions inside an image. Under the pooled adaptive attack, Gemini 3.1 Flash Lite stays at 97.8% after rendering. Benign fine-tuning on image-rendered instructions removes the same gap. The method only works while the current training mixture withholds obedience from the image channel.
Questions for authors

On AgentDojo cases where the image-channel attack fails, does the model answer content questions about the rendered tool output at the text-channel rate? This separates a recovered instruction that the model refuses to obey from a payload it did not read.

2 comments

We thank the reviewer for the careful review and for the precise questions.

| W1. Is the ASR drop caused by the text-channel preference or by a failure to read the payload?

Section 3 uses short instructions only to motivate the study. The security experiments then test the same effect on long documents and tool outputs. The reviewer is right that the submission did not explicitly separate instruction-following from content-recovery in these settings. Several results show that recovery errors cannot explain the drop:

  1. DirectInject does not require correct arguments. An attack succeeds whenever the model calls the target function. A misread character therefore cannot make an attack fail, yet ASR falls from 75.0% to 0.0% on GPT-5.4 mini and from 80.4% to 7.1% on Kimi-K2.6 (Table 1).

  2. Relaxed scoring on AgentDojo changes little. Counting attempts with incorrect arguments as successes raises image ASR by at most 6.2 points (Table 9).

  3. Transcription errors are too rare. Word accuracy on AgentDojo pages is at least 97.2% for all ten models (Table 13). Among the 784 AgentDojo attacks that succeed as text and fail as an image, only 7.9% contain a transcription error of any kind. In the remaining 92.1%, the page is transcribed without a single mistake and the attack still fails.

  4. Reading quality does not predict protection. Gemini 3.1 Pro transcribes AgentDojo pages with 100% word and critical-token accuracy, and its ASR still falls from 83.1% to 41.5% (Table 1). Kimi-K2.6 has 99.6% critical-token accuracy and no SWE-bench loss, and falls from 50.8% to 12.3%. GPT-5.4 nano has the lowest critical-token accuracy (92.5%), and an error rate below 8% cannot account for its ASR falling from 14.6% to 0.0%.

  5. Fine-tuning changes obedience without changing perception. In Figure 4 the vision encoder and aligner are frozen, and image ASR still rises by 13 to 19 points.

The SWE-bench loss has a different cause. Code edits require byte-exact anchors, and anchor failures rise from 0.8% to 37.4% for Claude Haiku 4.5 while word accuracy stays above 95% (Table 14).

| W2. The defense depends on the current training distribution.

We agree, and Section 6.2 states this. However, we note that fine-tuning narrows the gap but does not eliminate it completely. As can be seen in Figure 4, image ASR rises to 18.9%, 21.9% and 50.5% on Gemma-4-E4B, Qwen-3.5-4B and Qwen3-Omni-30B, however text ASR on the same fine-tuned models is 41.6%, 41.6% and 81.6%.

| Q. On failed image-channel attacks, does the model recover the rendered content at the text-channel rate?

Yes. We analyzed 392 cases (for Claude Haiku 4.5 and GPT-5.4-Mini) in which an attack succeeds as text and fails as an image. In 11% of cases, the response explicitly identifies the injected instruction and declines to follow it, e.g., ‘I need to decline this request’, ‘the email includes an instruction’

For the remaining cases, the model never mentions that any prompt injection existed in the image. As hidden reasoning is not available from the provider we instead appended a follow-up question to the same trajectory asking ‘Do you see any suspicious text?’. The model restates the injected instruction verbatim in all these cases. So it does see the injection, but it doesn’t choose to follow it when it appears in the image.

Taken together, in all of the failed image-channel attacks the model demonstrably recognizes the injected instruction, either by flagging it unprompted or by restating it when asked. The ASR reduction is therefore not explained by the model failing to read the payload. In most cases the model reads the instruction and does not act on it.

Reviewer_k4dmOpenReview ↗

Official Review by Reviewer k4dm

Thanks for the detailed reply. The clarification resolves my main concern: on the injection tasks, the drop in attack success is not just the model failing to read the rendered page. But still, I think that the defense mostly relies on how these models are trained. And I am keeping my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.