- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies adversarial tokenization attacks from a mechanistic interpretability point of view and how to perform unsupervised fine-tuning to help defend against these attacks.
Critical strengths
- The mechanistic interpretation analysis is quite interesting.
- The fact that fine-tuning over different tokenizations enhances defense even for non-adversarial tokenization attacks is surprising.
Critical weaknesses
- Many sections of the paper seem to have been either completely generated or heavily edited by AI. These parts of the paper are very hard to follow (and I state some examples of that below), and I would strongly recommend a human rewrite these sections.
- I would appreciate at least a paragraph explaining the nomenclature used in the mechanistic interpretability section. It is hard to follow what exactly is being done for someone who is an outsider to this community.
Questions for authors
- What is in line 281?
- Line 426: to be precise, adversarial tokenization should not delete any readout, but rather the readout will be absent on an adversarially tokenized tokenization versus the canonical. This is an important distinction, because "restoring" the readout means that some capacity of the model will be transported from some other region of the space to this particular readout. It would be interesting to see whether these consistency trained models degrade their performance the more retokenizations are "repaired". In other words, is there not a trade-off between robustness to adversarial tokenization and downstream semantic performance?
- Line 431: As someone who is not familiar with mechanistic interpretability, I would appreciate a paragraph clarifying nomenclature: for example, what is a probe, what are the assumptions behind this probe, what is a topic, and what is topic decodability?
- The entire paragraph in 436-440 is very confusing. What do the authors mean by "the feature is absent rather than rotated out of linear reach?" What is "the attack's own token inflation?" What is "ordinary depth?" What is meant by "rare ones?" rare tokenization lengths? What is meant by depth here?
- Figure 3: cracked cells ? Do the authors mean something like ?
2 comments
Q4. Paragraph at PDF l. 436–440.
What do the authors mean by "the feature is absent rather than rotated out of linear reach?" What is "the attack's own token inflation?" What is "ordinary depth?" What is meant by "rare ones?" [...] What is meant by depth here?
If a feature of the text (i.e. harmful/benign) is represented in the model's activations, it could be linearly decodable or non-linearly decodable. A linear probe and a two-layer probe give the same accuracy on attacked prompts (0.32), so the drop is not merely an limitation of the linear readout. We call the "depth" to be the distance of the tokenization from the canonical one. Essentially, it counts how many times the tokens are split apart into sub-tokens. "The attack's own token inflation" denotes other perturbations that share "the same depth as the attack." This was poorly written. We have now revised the paragraph. On re-evaluation we also found that its random control was not properly matched to the attack's depth. With a correct match, random retokenizations as deep as the attack lower the safety probe to chance (0.54), and the attack lowers it further (0.32). The paragraph now correctly states that part of the suppression is an effect of depth.
Q5. Fig. 3 "cracked cells".
Figure 3: cracked cells ? Do the authors mean something like ?
We meant that the number of prompts in each outcome group ranges from 5 to 36 across variants. The caption now says this in words, and also explains the split: each variant’s own-attack bar is divided into prompts where the attack succeeded and prompts where it failed.
Thank you for pointing out the prose issues and the unclear explanation of the interpretability methods.
W1. Prose reads as AI-generated.
We agree that the prose needed work. In this revision we rewrote the introduction, related work and setup sections, and rewrote the results and probing sections around the re-evaluated numbers, with shorter paragraphs and defined terms. We will edit the remaining sections to the same standard for the final version.
W2 / Q3. Nomenclature paragraph.
A probe is a linear classifier (logistic regression). A safety probe takes in the model's activations when it sees a canonically tokenized prompt and predicts whether the prompt is benign or harmful. A topic probe takes the same input and predicts the topic (cyber/malware, fraud/theft, misinformation/deception, weapons/explosives, violence/self-harm, miscellaneous). The probes are trained with canonical inputs only, and then we examine how they behave when the model receives non-canonical inputs. We say the topic is decodable when the classifier is correct (that is, above 0.5 for safety (binary) and 0.35 for topic, since the largest topic class makes up 35% of held-out prompts). Linear probes are a standard tool in interpretability; see Arditi et al. (2024), Zou et al. (2023a). If a property of the text, such as a high-level idea, is identifiable from the residual stream activations by a linear probe, it is taken to mean that the information is present in the model's representations. If it is not linearly decodable, it may still be present nonlinearly, so we additionally fit a two-layer model as a control. The control finds the same result: the safety readout is absent under the attack. Linear decodability does not necessarily mean the model uses that representation. To test whether the model does use the safety representation, we employ the technique of activation steering, where we push the activations along the direction that separates harmful from benign prompts (according to the probe) and observe that the refusal behavior responds as predicted. We fleshed out the interpretability tools in the revised related work section, added a "setup" paragraph to the probing section, and will continue to work to make the definitions and results clearer as we edit prose.
Q1. "What is in line 281?.
Alpha is the share of encodings that receive gradient; we use 0.25, so of the 8 samples encodings k=2 receive gradient. We now define alpha alongside CVaR, thank you for catching that.
Q2. "Deletes" vs "absent"; capacity trade-off.
Agreed, and we have changed the wording. The model is unchanged by the attack, so nothing is deleted. Under an adversarial tokenization, the safety readout is absent from the activations, where it is present under the canonical one. We now say “suppresses” or “is absent” throughout.
On the robustness/performance trade-off. As known from the adversarial literature, there is indeed a trade-off between robustness and downstream performance. However, we want to highlight that our method doesn't aim to mitigate an effect of general semantical adversarial attacks, where we suppose the trade-off to be more severe, but to specify the behavior os the model on OOD tokenizations it has not encountered, which does not affect semantics in any way, unlike prior consistency training approaches (Irpan et al., 2025). Even so, every fine-tuned arm loses some benchmark accuracy. On OLMo-2-7B the consistency arms lose 0.012 to 0.042 on MMLU, 0.033 to 0.047 on GSM8K and 0.046 to 0.124 on IFEval, and the two smaller models lose more (see General §1.4).