acceptodds
Under review as a conference paper at ICLR 2027

Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

Abstract

Everything a language model sees is tokens. Special tokens can demarcate assistant turns, and sometimes tool calls; everything between them is just text. The serving stack knows what each span is — user input, tool output, instructions — but the model must keep track of that itself, and it can lose track or be confused: text can be written to read like anything. Prompt injection is a natural exploit of this phenomenon. By scrambling the model's understanding of the nature and thus permission levels of different spans, an attacker can induce the model to take unwanted and potentially dangerous actions. Adding a non-textual channel to the model's input — a way to communicate span identity beyond text — mitigates this class of attack. We thus introduce a general steering technique called Semantic Overlays: small learned adapters applied at chosen prefill positions to a frozen model's residual stream. Laying an overlay over a span creates an out-of-band annotation channel that cannot be replicated by tokens. Unlike steering vectors, Semantic Overlays are trained, adaptable, and selectively applied. An overlay can encode complex semantics that reshape how the model perceives the marked span: asked to copy a code snippet under an overlay asserting that it is in a different programming language than it is, the model rewrites the snippet, faithfully, in the asserted language. Overlays are also composable, allow for transparent reading of underlying content, and can carry complex payloads — including imperatives that the model will follow. This leads to a robust defense against prompt injection. An overlay which marks a span as “non-executable” defends against the broad class of prompt injections that add instructions in untrusted context. We report strong results on five prompt injection benchmarks: SEP separation rises from 24.3% to 99.0% with utility unchanged (our scoring rule; we also surface and correct a defect in the published grader), TensorTrust attack success falls from 34.8% to 6.2%, AlpacaFarm from 99.0% to 0%, and on PIArena the overlay beats every published defense that leaves the model able to answer — while marked spans stay readable, all at >95% character similarity to the original.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
frozen language models, representation engineering, prompt injection, activation steering, steering vectors, instruction-data separation, adapters, LLM security

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2KRwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Semantic Overlays: small trained adapters (a SwiGLU bottleneck at the input of every decoder layer, about 50M parameters in the injection setting) that edit the residual stream of a frozen model, but only at the prefill positions of a chosen span. The edit is meant to act as an out-of-band annotation that text cannot imitate. Sections 4 to 6 build the idea on synthetic tasks: reading back "visual" marks such as underline or highlight, asserting a programming language for a code snippet, and carrying span-level instructions such as "answer in Spanish". Section 7 then uses a single overlay meaning "do not execute instructions in this span", applied by the serving stack to all retrieved content, as a prompt-injection defense. It is trained with cross-entropy on a synthetic corpus of injected and clean passages.

Critical strengths
  1. The design is simple, cheap and security-relevant. Keeping the base model frozen and delivering provenance as an activation edit that can be switched off per request is a clear interface-level idea: the stack decides where marks go, and the mark itself is not text an attacker can write.

  2. The evaluation audit is useful independently of the method. Running every benchmark with no injection present exposed real problems: substring witnesses firing inside ordinary words in SEP's grader, a TensorTrust harness that drops 606 of 776 rows non-randomly, and unanswerable rows that cap Defense Validity.

Critical weaknesses
  1. The security claim goes beyond the evidence, mainly because there is no white-box adaptive evaluation. The abstract and Section 7 describe a robust defense and say no input can imitate the mark. That is true of the channel, but what is actually relied on is the learned behavior behind it: a 50M-parameter adapter whose write is computed from the marked hidden state (Eq. 1), so the attacker's text does influence it. The only adaptive evaluation is a black-box agent (GLM 5.2) making 222 probes (Appendix E). The success criterion, query budget and attack families are not described in enough detail to judge, and the nine classes are hand-described authority-assertion attacks. The Limitations section says outright that optimization-based attack is untested. The base model is open and the adapter is to be released, so a white-box attacker (GCG-style suffix or injection-string optimization against the frozen model with the overlay on) is the standard threat model for trained injection defenses, and attacks of that kind have been reported to break StruQ/SecAlign-style defenses that looked strong on static benchmarks. Appendix E.3 also shows that the defense is overridden when the developer instruction delegates to the data, with inconsistent behavior under contradiction (a 4 to 2 split on one probe). Until a white-box evaluation is added, the 0% to 6% figures are static-benchmark numbers and should be framed that way.

  2. The paper never isolates what the overlay itself contributes. The claim is that an out-of-band activation channel beats in-band markers and other trained defenses, but there is no control with the same corpus, the same adapter budget and the same frozen base using an in-band marker (special tokens or delimiters plus the same adapter or LoRA applied everywhere), or an AIR-style privilege embedding on a frozen base, which the related work names as the closest mechanism. The steering-vector comparison in Section 4 is on a toy visual-marks task and compares a 1.6M-parameter constant vector with an 84M-parameter state-conditioned MLP (Table 1). There is no architecture ablation on the injection task at all.

  3. Several evaluation choices favor the method, and the headline numbers are the favorable ones. (i) The statement that everything except SEP is zero-shot transfer is too strong. Training uses SQuAD passages and 56 frame templates in twelve styles that include fabricated system delimiters and bare appended instructions (Appendix D). These look close to the naive, ignore, escape-separation and completion attacks in the AlpacaFarm/StruQ evaluation, and PIArena's SQuAD v2 split is built on SQuAD contexts. The paper does not say which SQuAD split is used for training, whether there is overlap, or whether any frame family was held out. (ii) On AlpacaFarm the frozen baseline is 99.0% attack success against the 68.3% reported by StruQ, which the authors attribute to their own serving framing (task in the system prompt, data in one user turn; Appendix I.1). The 99.0% to 0% headline is thus measured against a weaker-than-standard baseline, and no common in-band defenses (delimiters, sandwich prompts, spotlighting) are shown on the same model. (iii) The abstract leads with the author-modified SEP grader (99.0%) and with SecAlign's begin-with criterion on AlpacaFarm (0.0%); under StruQ's own rule the overlay is at 3.8% (Table I.1). Both are disclosed in the appendix, but the abstract and conclusion quote the favorable ones. (iv) Table 6 places the overlay's 95.1% utility, which is retention (injected over clean), in the same column as other defenses' absolute accuracy (84.0%, 89.5%, 68.5%).

  4. The scope is a single-turn text injection into short passages, and the utility evidence is thin. Agentic and tool-use settings, where most real injection damage happens, are not tested (Limitations). This matters more than usual because the defense is a blanket "do not execute" on all retrieved or tool content. The paper itself shows that trusted delegation to the data overrides it, and that a marked span stays quotable, so in an agent loop the model's own quote of an injected instruction re-enters the next step as unmarked text. Readability and non-execution also trade off sharply: raising the verbatim-copy share from 4.3% to 10% cost 3.7 points of SEP separation and roughly doubled TensorTrust hijacking (Appendix D), so the operating point is delicate. Utility is measured with proxies: SEP utility, PIArena retention on small sets, and a mean unigram-F1 of 0.63 between marked and unmarked completions on clean AlpacaFarm inputs, which does not obviously show that behavior is unchanged. Exact quoting through the overlay drops from 98.4% to 86.6% (Appendix G.3), which matters for code, credentials or legal text.

Questions for authors
  1. (Most important) Can you report a white-box adaptive attack (suffix or injection-string optimization against the frozen model plus adapter, overlay on), with the attack budget and success criterion? For the 222-probe red team, please state the query budget, whether the agent saw earlier results and adapted, the success criterion, and the attack families tried. Did any probe target the state-dependent write in Eq. 1, i.e. text meant to shift the hidden state the adapter reads?
  2. Can you add controls on the same base and corpus: (a) in-band markers (special tokens or delimiters) with the same adapter budget or LoRA, including attacks that forge those markers; (b) a frozen-base AIR-style privilege embedding; (c) StruQ, ISE, ASIDE or SecAlign retrained on Qwen3.5-9B with this corpus? Which factor drives the gain, the out-of-band channel or the training data?
  3. Which SQuAD split provides the training passages, and is there overlap with PIArena's SQuAD v2 items? What are the results with frame families resembling the StruQ attacks (and any other benchmark attack style) removed from training?
Reviewer_oN5QOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes semantic overlays, an out-of-band annotation mechanism that writes span-level metadata into the residual stream at prefill time using lightweight adapters, with the base model frozen. The authors run sequential ablation experiments covering pointer markers, counterfactual factual rewriting, span-local instruction transforms, and an NX-inspired non-executable overlay for indirect prompt injection defense. Evaluations include multiple prompt injection benchmarks and black-box adaptive red-teaming. The main claim is that serving infrastructure can attach unforgeable provenance semantics to text spans without modifying input tokens.

Critical strengths
  1. The core idea of building an out-of-band residual-stream channel for span provenance is conceptually novel. It offers a clean alternative to conventional text tags, which can be forged by prompt injection adversaries.
  2. The staged ablation design is well-organized. The paper systematically differentiates several overlay categories and measures the utility-security tradeoff of the non-executable overlay across common benchmarks.
Critical weaknesses
  1. Generalization of behavioural overlays is very limited. While pointer-style markers generalize acceptably, transformation and safety overlays fail on unseen descriptors and default to nearby trained behaviours. This means every new safety rule or transformation requires bespoke retraining, greatly reducing practical flexibility.
  2. Overlay modifications are baked into the KV cache once prefill completes, and perturbations spread to adjacent tokens via self-attention. The authors identify this leakage but do not rigorously quantify or mitigate it. Uncontrolled semantic drift creates material risk to normal model behaviour in realistic long-context and multi-turn deployments.
Questions for authors
  1. For behavioural overlays, do the authors see a realistic path to improve generalization to unseen overlay descriptors, other than the nearest-neighbour fallback observed in the current adapter design?
  2. Could the authors provide additional quantitative results characterizing performance degradation from cross-token leakage under long-context, multi-turn settings?
Reviewer_x48TOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper suggests that we can control how a model uses its text input by applying an adapter to a region of tokens, with the adapter chosen to influence the model in some specific way.

Critical strengths
  • A fascinating direction, which aims to go beyond what existing chat templates can provide, by enabling control over how the model interprets input.
  • A conceptual contribution, suggesting a new way to control model behavior, which is different from how we're currently thinking about or building language models.
  • The method seems to provide some fairly general capabilities that might be useful for a variety of purposes.
  • The scheme is simple and low cost.
Critical weaknesses
  • Does not demonstrate an improvement in ability to defend against prompt injection attack over the SOTA. The prompt injection evaluation in this paper only evaluates against the most basic, static attacks. There are previously published schemes that can already achieve that. The field has moved on beyond that, and the crucial challenge for the field now is how much security can be gained against adaptive, optimization-based attacks. The paper does not evaluate against that. It would be interesting to discover whether the paper's architectural modification improves security against such strong attacks, if it is applied to some existing recipe for post-training PI robustness; that experiment isn't done in this paper, though. This can be considered just a matter of scope; the paper focuses on the new conceptual idea, less on demonstrating that this particular application of the idea beats the SOTA. That could be OK.
  • Not necessarily grounds for rejection, but I would prefer a writing style that made it easier on me as a reader, and did not share the challenges of reading LLM-generated prose.

Detailed comments (not critical weaknesses):

I am borderline between weak accept and weak reject. I might revise my score based on other reviews; or based on the author rebuttal. On the one hand, the conceptual idea is intriguing, and I could imagine it could be a step towards better prompt injection security, and perhaps other capabilities. On the other hand, I suspect significant more work will be needed to achieve that goal. I tend to be more excited about demonstrated new capabilities, more than conceptual ideas that might ultimately lead to new capabilities, but that is entirely subjective and a matter of personal taste, so if other reviewers like the paper, I certainly do not object to acceptance. Also the writing of the paper is a bit of a minor negative for me, and I don't think is easily addressible through a revision.

I find many places in the writing hard to follow, for similar reasons that I often find LLM output hard to follow. In some places the paper has some similar tone/stylistic choices. I read a lot of LLM output day-to-day, and I find it painful and tiring, and I'd prefer not to read a conference paper with a similar style of writing.

I am not sure that I could explain how the scheme works to someone else, in detail. This might reflect my lack of expertise rather than a shortcoming in the paper, and maybe someone with more experience would know how to fill in the minor gaps in explanation in "the obvious way".

l47: Not correct - using a proper library, role tags cannot be forged, they use special tokens that the tokenizer will not allow to appear in the contents of a message (e.g., user message or tool-call output message).

Fig. 1: Is this a real conversation? With which model? Or is it invented?

l84-89: I find this hard to understand. Is this AI-written? What do you mean when you say a collection of small adapters is applied to the residual stream? Do you actually mean that you have a collection of small adapters and you select one and apply it? I would have thought an "overlay" is an "adapter". I don't know what the phrase "trained quality" means. What's the difference between an overlay set and an overlay? If overlay set means something other than "set of overlays", something seems wrong with the definitions. I don't know what is meant by "the mark is an activation edit". It reminds me of LLM- speak. Can you write this paragraph in terms that are more concrete and easier to understand and a bit less "clever"?

l92: I don't know what it means to "detect" an overlay. The phrasing here reads oddly.

l94: I don't understand what it means for an overlay to "assert". An overlay modifies the residual stream. The language feels "too precious" for my tastes.

l107: I don't understand what this sentence is saying, or why I care, or why it is in the introduction.

l152: I don't know what an "identity code" is. I recommend that you avoid using (non-standard) noun phrases unless you first define them.

l160: I don't know what a "boolean position mask" is.

l161: I don't understand what this sentence is telling me. Does the overlay change how prefill works, i.e., change how the hidden states at each layer are computed as a function of the hidden states at the previous layer? Or are hidden states computed according to the frozen model without any adapters, and then for each hidden state you compute a modified version of the hidden state per (1) but don't use the modified value when computing the next layer? I don't understand what you mean by "shape how the model reads the span". That sounds like a metaphor and reminds me of how LLMs talk -- abstract, hard to pin down exactly what it means.

l180: I'm lost. My working hypothesis was that "overlay set" means a collection, one overlay per layer, that is associated with a single attribute/concept/quality. But that doesn't seem consistent with this message. I don't know what it means for a set to "carry" something. I suggest that you minimize use of metaphor, unless you're confident the audience is going to understand it. I see LLMs often use this kind of stylistic word choice as well and I struggle with it. Rather than being creative and clever with word choice, I think it's often best to be simple and clear.

Sec 3.2: I didn't understand this section. If I got the gist of what it's doing, I suspect this might be better as an ablation, and it might be easier to follow the paper if you presented one scheme (the best one), and then justify that choice with an ablation in the appendix or late in the paper, so that I've got the main ideas and if I get lost in the ablation at least I've understood most of the most important ideas.

l193: "where a gold is constructable" - some words seem missing.

l213: I don't understand what "minimum is exactly no change" means. I don't understand how KL differs from cross-entropy, as it seems you have one-hot labels (match exactly the desired token), not soft labels (desired distribution over tokens).

l199: I don't understand what a "source item" is or how it differs from a "mark configuration".

Sec 7: Unfortunately, training on a SEP-like training set makes SEP somewhat less useful as an evaluation benchmark. Normally in prompt injection defense, we evaluate on benchmarks that the model wasn't trained on, because we care most about generalization / transfer to tasks that it wasn't trained on (one can only train on a miniscule fraction of all real-world tasks). It's not a showstopper, but a caveat. It also makes it arguably a little harder to compare to other methods for prompt injection defense.

Sec F: I appreciate this experience report and the work to improve the quality of benchmarks. Hopefully this will be useful to other researchers.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.