- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper focuses on prompt injection attacks, specifically ones that includes chat template tokens (e.g. to mark conversation turns) in an injected payload, so a model/agent would be more likely to follow the injected instructions. Prior work has shown that this works (Chen et al., 2026).
At the same time, authors pose that the same text can be encoded differently by the same tokenizer: a string corresponding to a reserved token can be encoded as the one designated token, or could be broken down into different smaller tokens.
The paper studies the effect of these encoding differences on the success of the corresponding attack (while the text remains the exact same).
They find that:
- breaking down the encoding to different subwords leads to lower ASR. This suggests that the reserved tokens, and their learned representations, rather than the exact string, carry most of the attack.
- This effect (the "attack authority") lives in the token's learned embedding. While an average of the subword vectors do not stand in for the reserved vector, for Llama3.1 the nearest ordinary token in embedding space can restore attack success and an adaptive attacker can find exactly these kinds of tokens too.
- Instruction tuned models have a stronger preference for reserved tokens.
- Current mitigations based on tokens still have gaps, as the standard defenses split the tokens explicitly marked as special, but leave tool call related tokens untouched.
Critical strengths
- The paper studies the success of prompt injection attacks, which remains an important and unresolved problem.
- Extensive discussion of related work.
- The insights around whether attack authority comes from bytes or specific tokens, and how this differs, gives us insights in how and why particular attacks and defenses succeed.
- The conditions mentioned in Table 1 represent a thoughtful controlled setup to measure the impact of tokenization on attack success.
- Extensive evaluation across 4 models and 2 benchmarks.
- Addition of several, more secondary experiments that add to the insights.
Critical weaknesses
- While the paper's findings are in itself interesting and well executed, I believe it remains a relatively niche research question in prompt injection security. It studies one particular kind of attack (one that tries to mimic parts of a chat template), and provides thorough insights into its mechanics and possible defenses. Yet, as this is not per se the strongest attack (e.g., the paper itself shows how adaptive attacks can find ways around the reserved tokens) the practical impact of these insights on improving defenses against prompt injection might remain limited.
- The differences across models are quite striking, and it would be great to have a better understanding/intuition why this is the case.
- The experiment with base models and comparing logits might not be that insightful, as many so many things change between both model versions and a logit for a tool call of the base model is likely not that meaningful.
- It would be nice to also compare the token level defenses discussed in the paper to other system or model based defenses.
Questions for authors
Mostly see weaknesses, but maybe most pertinent:
- Could authors provide more intuition, or possibly, conduct experiments to better understand the differences across models?
- In the fourth contribution mentioned in the intro, the argument that current defenses are missing tool-call-related tokens, isn't that easy to solve with the insights of this paper? By just also splitting the tool related tokens. Or am I missing something?