acceptodds
Under review as a conference paper at ICLR 2027

Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection

Abstract

Prompt injection against LLM agents becomes much stronger when the injected instruction is wrapped in the model's own chat template. A forged template marker such as <|im_start|> can reach the model either as a single reserved control token or as a sequence of ordinary subword tokens. The two decode to exactly the same text, and because tokenization runs on the server, the defender rather than the attacker decides which one the model receives. We use this to measure how much of the injected instruction's authority comes from the reserved token's learned representation. Encoding the forged markers as subwords, with the text held fixed and a control for the extra tokens this adds, lowers attack success on the InjecAgent benchmark by 39 to 66 percentage points on three of four open-weight families, and the gap carries over to multi-turn agent tasks in AgentDojo. On Qwen3-8B the gap is 8 points, because without reserved ids the model still recognises the forged turn from its text by reasoning; suppressing the reasoning block widens the gap to 50. The authority sits in the single learned vector at the marker position: the mean of the marker's subword vectors does not reproduce it, the vector of the nearest ordinary token restores the attack on Llama-3.1, and an adaptive attacker who searches for non-reserved markers finds such embedding neighbours on three of four families. In every base and instruction-tuned pair we test, instruction tuning strengthens the model's preference for reserved markers. The standard mitigation, a tokenizer option that encodes special tokens as ordinary subwords, applies only to tokens a configuration declares special, so in 33 of 67 distinct tokenizer configurations, covering 255 of the 400 most-downloaded chat models on Hugging Face, it leaves intact the tool-protocol tokens through which agents read untrusted tool output, and the gap persists on that channel.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt injection, LLM agents, LLM security, chat templates, special tokens, tokenization, instruction hierarchy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SLV6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on prompt injection attacks, specifically ones that includes chat template tokens (e.g. to mark conversation turns) in an injected payload, so a model/agent would be more likely to follow the injected instructions. Prior work has shown that this works (Chen et al., 2026).

At the same time, authors pose that the same text can be encoded differently by the same tokenizer: a string corresponding to a reserved token can be encoded as the one designated token, or could be broken down into different smaller tokens.

The paper studies the effect of these encoding differences on the success of the corresponding attack (while the text remains the exact same).

They find that:

  • breaking down the encoding to different subwords leads to lower ASR. This suggests that the reserved tokens, and their learned representations, rather than the exact string, carry most of the attack.
  • This effect (the "attack authority") lives in the token's learned embedding. While an average of the subword vectors do not stand in for the reserved vector, for Llama3.1 the nearest ordinary token in embedding space can restore attack success and an adaptive attacker can find exactly these kinds of tokens too.
  • Instruction tuned models have a stronger preference for reserved tokens.
  • Current mitigations based on tokens still have gaps, as the standard defenses split the tokens explicitly marked as special, but leave tool call related tokens untouched.
Critical strengths
  • The paper studies the success of prompt injection attacks, which remains an important and unresolved problem.
  • Extensive discussion of related work.
  • The insights around whether attack authority comes from bytes or specific tokens, and how this differs, gives us insights in how and why particular attacks and defenses succeed.
  • The conditions mentioned in Table 1 represent a thoughtful controlled setup to measure the impact of tokenization on attack success.
  • Extensive evaluation across 4 models and 2 benchmarks.
  • Addition of several, more secondary experiments that add to the insights.
Critical weaknesses
  • While the paper's findings are in itself interesting and well executed, I believe it remains a relatively niche research question in prompt injection security. It studies one particular kind of attack (one that tries to mimic parts of a chat template), and provides thorough insights into its mechanics and possible defenses. Yet, as this is not per se the strongest attack (e.g., the paper itself shows how adaptive attacks can find ways around the reserved tokens) the practical impact of these insights on improving defenses against prompt injection might remain limited.
  • The differences across models are quite striking, and it would be great to have a better understanding/intuition why this is the case.
  • The experiment with base models and comparing logits might not be that insightful, as many so many things change between both model versions and a logit for a tool call of the base model is likely not that meaningful.
  • It would be nice to also compare the token level defenses discussed in the paper to other system or model based defenses.
Questions for authors

Mostly see weaknesses, but maybe most pertinent:

  • Could authors provide more intuition, or possibly, conduct experiments to better understand the differences across models?
  • In the fourth contribution mentioned in the intro, the argument that current defenses are missing tool-call-related tokens, isn't that easy to solve with the insights of this paper? By just also splitting the tool related tokens. Or am I missing something?
Reviewer_q1FLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a question for prompt injection in LLM agents: when an attacker forges a chat-template marker inside untrusted tool output, how much of the attack’s effectiveness comes from the marker’s text versus from its reserved token ID and learned input representation? The authors construct byte-identical prompts that differ only in whether forged markers are encoded as reserved control tokens or as ordinary subwords, introduce a token-count-matched control, and define an “identity gap” ∆. Across InjecAgent and AgentDojo, they show large gaps on Llama-3.1, GLM-4.5, and Seed-OSS-36B, a smaller gap on Qwen3-8B. They further provide evidence that the effect is carried by the input vector at the marker position, that instruction tuning increases the preference for reserved markers, and that the standard Hugging Face special-token mitigation leaves tool-protocol tokens uncovered in many deployed tokenizer configurations.

Critical strengths
  1. The experimental contrast is conceptually clean and directly addresses an important confound in prior template-injection work: text and token IDs are normally varied together.
  2. The coverage census of tokenizer configurations is a practically useful contribution, and the finding that tool-protocol tokens are often outside the standard mitigation is relevant to deployed agents.
Critical weaknesses
  1. The defense implications are limited. The paper shows that the standard mitigation is incomplete and that an adaptive attacker can recover much of the attack. The source-aware encoder is described in the appendix but is not evaluated against the adaptive attacker.
  2. The evaluation is only conducted with small models. Larger models are typically more robust to prompt injection attacks. The results on larger models (e.g., of 235B parameters) should be added.
Questions for authors

Please reply to the weaknesses.

Reviewer_Kmo1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how the reserved chat-template-related token representations contribute to the success of chat-template-based prompt injection for LLM agents. Through comparing byte-identical prompts as reserved tokens (embeddings) or ordinary subword tokens, the paper shows substantial gaps in attack success rates across diverse models and demonstrates the effects of embedding representations by comparing base and instruction fine-tuned LLMs. The paper also verifies the tokenizer-side mitigation strategies.

Critical strengths
  • The paper presents an interesting experimental framework to isloate the effect of reserve token embeddings from textual differences.
  • The embedding replacement experiments shows valuable insights to verify the main cause of chat-template based prompt injection attacks.
Critical weaknesses
  • The core observations seem substantially overlap with ChatInject (Chang et al., ICLR 2026). For example, a previous paper has already demonstrated the effectiveness of chat-template-based prompt injection and its robustness to template perturbations. Although the fixed-byte comparison in this paper offers a more controlled analysis, the paper does not sufficiently clarify what fundamentally new insights are gained beyond the findings of ChatInject, particularly its perturbation experiments.

  • The comparison between base and instruction-tuned models uses logit margins between attacker-selected and tools. However, since base models may not possess meaningful tool-use capabilities (the raw pre-trained models' capability are not enough), I'm concerned that these logits may not represent comparable tool-selection preferences across checkpoints. Consequently, the observed differences could reflect improved familiarity with tool-calling formats rather than increased authority assigned to reserved tokens.

  • The tokenizer-side mitigation in the paper largely shares the same objective as conventional special-token stripping introduced in the previous works such as ChatInject. Although the proposed source-aware encoding preserves the original text, the paper does not clearly establish its practical advantages over simpler alternatives like parsing.

  • The paper interprets the effects of reserved token embeddings as the main evidence of learned instruction authority. However, I think the experiments in the paper do not clearly distinguish whether these representations cause models to assign higher priority to injected instructions or merely improve their recognition of conversational boundaries.

Questions for authors
  • In Sec. 4.4, the paper compared base and instruction-tuned models using logit margins between attacker-selected and legitimate tools. However, base models may not have meaningful tool-use capabilities, making it unclear whether these logits represent comparable tool-selection preferences across checkpoints. How do authors distinguish the increased reliance on reserved token representations from the effects of learning tool-calling formats during instruction tuning?

  • The proposed tokenizer-side mitigation seems conceptually similar to conventional special-token stripping or sanitization, which was also discussed in the previous work ChatInject. Could the authors clarify the practical advantages of your approach over these simpler alternatives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.