acceptodds
Under review as a conference paper at ICLR 2027

Stochasticity in Tokenization Improves Robustness

Abstract

The widespread adoption of large language models (LLMs) has increased concerns about their robustness. Vulnerabilities to perturbations of the input tokenization indicate that models trained with a deterministic canonical tokenization can be brittle to adversarial attacks. Recent studies suggest that stochastic tokenization can deliver internal representations that are less sensitive to perturbations. In this paper, we analyse how stochastic tokenizations affect robustness to adversarial attacks and random perturbations. We systematically study this over a range of learning regimes (pre-training, supervised fine-tuning, and in-context learning), datasets, and model architectures. We show that pre-training and fine-tuning with uniformly sampled stochastic tokenizations improve robustness to random and adversarial perturbations. Evaluating on uniformly sampled non-canonical tokenizations reduces the accuracy of a canonically trained Llama-1b model by 29.8%. We find that training with stochastic tokenization preserves accuracy without increasing inference cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, tokenization, robustness, stochastic tokenization, finetuning, pretraining

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Yq2aOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the question, "Can stochastic tokenization improve robustness?". The paper shows that the previous approach StochasTok can improve both the nomral test accuracy and the robustness in pre-training, fine-tuning, and ICL. The paper then present two approaches StochasTok-Uni and Uniform-k to improve the uniformity and support of the sampling approach in StochasTok. It then shows the Uniform-k and StochasTok-Uni has better performance than StochasTok.

Critical strengths
  • The paper proposed two approaches to improve the uniformity and the support of StochasTok and empirically show that the theoretic improvement can translate to empirical improvement in the accuracy under uniform stochasity and the worst-case adversarial accuracy.
  • The paper is well-written.
  • The paper conduct extensive study on different configurations and on different benchmarks.
Critical weaknesses
  • The paper can be improved by showing the generalization across model sizes. Especially for Figure 2a, the test accuracy drops when the StochasTok is used, but it can also be the model size limits the learning of the robustness behavior, scale up the model size can dive deep into this.
  • Table 1, the results on CSQA and HellaSWAG do not show improvement at all. Why this happened and how do these two benchmarks differ from others?
Questions for authors
  1. Line 246-247, the notion P(.∣x;α)P(. \mid x; \alpha) is not defined, especially how is the α\alpha used in this conditional probability.
  2. In Eq 2, the f(v) is not the output label because it is hard to compute a distance. so is the output space Y the probability or the logits?
  3. I don't see the usefulness of the section 6.1 at all, unless the paper can show that the bound is tighter than the previous bounds derived from the ϵ\epsilon-ball or L0, L2, Linf adversarail settings, i.e., the stochastic tokenization can have a different bound.
3 comments
AuthorsOpenReview ↗

Answer to Reviewer Yq2a (1/2)

We thank the reviewer for the positive assessment and the detailed questions. We appreciate that the reviewer recognized our extensive study and the clarity of the paper. Below, we address the concerns in detail. All corresponding changes are highlighted in the revised manuscript.


Generalization across model sizes

We provide a model-size ablation for Llama-3 with 1B, 3B, and 8B parameters (now Table 3 in Sec. 6), fine-tuned on LanguageGame and evaluated under canonical and adversarial tokenization. We present the results again in Table R5 below.

Table R5 (Model size). Canonical / adversarial accuracy (%) on LanguageGame for Llama-3 models of different sizes (αfine=0.5\alpha_\text{fine} = 0.5).

Fine-tuning1B3B8B
Canonical95.4 / 4.098.2 / 39.099.4 / 60.2
StochasTok95.6 / 59.698.4 / 87.499.0 / 92.6
UniK96.8 / 80.698.6 / 91.699.4 / 95.8

Canonically fine-tuned models become less vulnerable with scale, but remain brittle even at 8B (60.2% adversarial accuracy). Stochastic tokenization during fine-tuning improves adversarial robustness at all three model sizes, with the largest absolute gains for the smaller, initially more brittle models.


Table 2 (Table 1 in the original submission): CSQA and HellaSwag

The limited gains are not specific to CSQA and HellaSwag. In ICL, the stochastic context tokenization only results in clear improvements for LanguageGame. On the general benchmarks, the gains are small. Under uniformly sampled queries (now Table 16, Sec. I.3), the best stochastic context recovers +11.5 pp on LanguageGame, but only +0.6 to +3.8 pp on the general benchmarks, with CSQA (+1.1) and HellaSwag (+0.8) at the lower end.

With fine-tuning, however, stochastic tokenization also improves robustness on those two datasets (Tables R3–R4 below): it reduces the drop in accuracy from canonical tokenization to uniformly sampled tokenization from 12.1 to 5.8–9.3 pp on CSQA and from 4.8 to 0.4–1.4 pp on HellaSwag. We attribute the difference in robustness to the varying exposure to non-canonical tokenizations: in our fine-tuning experiments, the weights are updated using 9,741 (CSQA) and 39,905 (HellaSwag) training samples, covering a wider space of possible non-canonical tokenizations. In our ICL setting, the model is frozen and only sees K=10K = 10 examples. We now discuss this in Sec. 4.3.

Table R3 (CSQA). MCQ accuracy (%) on CSQA using Llama-8B:

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
None71.464.962.258.2
Canonical81.677.173.469.5
StochasTok 0.583.178.376.173.8
StochasTok 1.080.879.076.675.0
UniK 0.583.380.276.574.7
UniK 1.081.078.776.673.8

Table R4 (HellaSwag). MCQ accuracy (%) on HellaSwag using Llama-8B:

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
None52.049.947.846.3
Canonical59.458.456.954.6
StochasTok 0.559.559.458.758.1
StochasTok 1.059.259.158.858.4
UniK 0.559.859.459.358.6
UniK 1.058.958.859.158.5

Q1: Definition of P(⋅∣x;α)P(\cdot \mid x; \alpha)

P(⋅∣x;α)P(\cdot \mid x; \alpha) is the distribution over tokenizations v∈TV(x)v \in \mathcal{T}_\mathcal{V}(x) induced by StochasTok with split budget N=α⋅∣vc∣N = \alpha \cdot |v^c|, where α\alpha is the normalized number of splits per canonical token (Eq. 3, Sec. D.1). αICL\alpha_\text{ICL} and αeval\alpha_\text{eval} denote this rate for the context examples and for the query, respectively.

Example: Let x=x = "the revolution" with vc=(v^c = (the, revolution)), and assume the valid splits the → (th, e) and revolution → (rev, olution) or (re, volution). For α=0.5\alpha = 0.5, we split the canonical token sequence N=1N = 1 times. StochasTok selects a splittable token uniformly at random, and among the valid tokenizations a pair again uniformly at random:

vvP(v∣x;α=0.5)P(v \mid x; \alpha{=}0.5)
(th, e, revolution)1/2
(the, rev, olution)1/4
(the, re, volution)1/4

A uniform scheme would assign 1/3 to each. This illustrates the sampling bias discussed in Sec. 5, which StochasTok-uni and UniK remove. We now define P(⋅∣x;α)P(\cdot \mid x; \alpha) explicitly in Sec. 2.


Q2: Output space in Eq. 2

f=A∘Ef = A \circ E maps a token sequence through the embedding and transformer block to the final linear head WoutW_\text{out}, so f(v)∈Rmf(v) \in \mathbb{R}^m is the logit vector (Sec. G, Eq. 13). The predicted label is its argmax. We now state this in Sec. 6.1.

AuthorsOpenReview ↗

Answer to Reviewer Yq2a (2/2)

Usefulness of Section 6.1:

Our aim with the theorem was to formalize how local sensitivity over the tokenization neighbourhood affects adversarial risk. The adversarial risk (Eq. 2) is the largest change in the model’s output that an attacker can cause by choosing a tokenization within the neighbourhood. If the model’s representations change little between neighbouring tokenizations, this change, and hence the adversarial risk, is small. Theorem 6.1 bounds this sensitivity from above by the model’s Lipschitz constants. The bound itself does not distinguish between training schemes.

To assess the sensitivity, we compared the normalised distance between hidden representations of different tokenizations of the same word (Fig. 22). If the normalised distance is smaller, the hidden representations are closer and which tokenization is used affects the hidden representation of a word less. We empirically found that fine-tuning with stochastic tokenization reduces the normalised distance compared to canonical tokenization. We therefore conclude that stochastic fine-tuning reduces the model’s sensitivity to the tokenization, the quantity that controls adversarial risk in Theorem 6.1.

The worst case tokenization is measured directly by the adversarial tokenization attacks: stochastic fine-tuning reduces the adversarial risk (Fig. 6), and methods covering more of the neighbourhood achieve higher adversarial robustness (Fig. 16). We have revised Sec. 6.1 accordingly.

We hope our answer addresses the reviewer’s concerns and are happy to answer further questions.

Reviewer_Yq2aOpenReview ↗

Response

I thank authors for their responses, edits to the paper, and additional experiment results. I will retain my score.

Reviewer_AUKZOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate whether introducing stochastic tokenization during training can improve the robustness of LLMs to non-canonical tokenizations. They first systematically evaluate the existing method StochasTok across three regimes—pre-training, supervised fine-tuning, and in-context learning—and find that introducing stochastic tokenization during fine-tuning yields the largest robustness gains against randomly sampled non-canonical tokenizations, followed by pre-training, whereas the gains from ICL are the weakest and largely limited to subword-level tasks. The authors then point out that StochasTok's sampling distribution is biased and has incomplete support, and propose StochasTok-Uni, which samples uniformly conditioned on the per-token split counts, and UNIFORM-K, which samples uniformly at the sequence level and can split across canonical token boundaries. In fine-tuning experiments, the proposed methods generally outperform StochasTok under both uniform random perturbations and greedy adversarial tokenization attacks. In the ICL setting, however, the differences among the sampling methods are not pronounced.

Critical strengths
  1. The paper clearly demonstrates the brittleness of canonically trained LLMs to non-canonical tokenizations, and shows that stochastic tokenization during fine-tuning is an effective and practical mitigation that preserves canonical accuracy without additional inference cost.
  2. The authors provide an insightful analysis of the limitations of StochasTok, and propose principled sampling schemes with improved uniformity and support coverage.
  3. The empirical evaluation is comprehensive, covering different training regimes, model scales, and both random and adversarial tokenization perturbations.
Critical weaknesses

The central fine-tuning claims are validated only on synthetic subword-level tasks. The paper does not evaluate whether stochastic tokenization during supervised fine-tuning improves robustness on general-purpose benchmarks or more realistic downstream tasks; its general-benchmark results come from pre-training or frozen-model in-context learning rather than stochastic fine-tuning. This limits confidence in how broadly the reported robustness benefits generalize.

Questions for authors

Could the authors supplement experiments to verify whether stochastic tokenization used in supervised fine-tuning improves robustness?

2 comments
AuthorsOpenReview ↗

Answer to Reviewer AUKZ

We thank the reviewer for the positive assessment and the suggestion to evaluate fine-tuning on general-purpose tasks, which we address below and in more detail in the revised manuscript.


Fine-tuning beyond synthetic subword-level tasks

Following the reviewer’s suggestion, we fine-tune Llama-8B with canonical and stochastic tokenization on general-purpose and realistic downstream tasks, and evaluate under canonical and stochastic tokenizations:

  • Free-form math generation (Tables R1–R2): MAWPS and GSM8K with chain-of-thought. Under uniformly sampled tokenizations, canonical fine-tuning loses about 16 percentage points on both tasks. Fine-tuning with StochasTok/UniK at α=0.5\alpha = 0.5 roughly halves this drop, while canonical accuracy changes by at most 1.7 pp.
  • General-purpose MCQ benchmarks (Tables R3–R4): CSQA and HellaSwag. Stochastic fine-tuning narrows the drop in accuracy from canonical-to-uniform evaluation from 12.1 pp to 5.8–9.3 pp (CSQA) and from 4.8 pp to 0.4–1.4 pp (HellaSwag).

Across all four datasets, fine-tuning with stochastic tokenization improves robustness to randomly sampled non-canonical tokenizations without a meaningful loss in canonical accuracy for α≤0.5\alpha\leq 0.5, consistent with our results on the subword-level tasks. We added these results in more detail to the paper (Sec. 4.2, Table 1, detailed results in Sec. I.2.3).

We hope this addresses the reviewer’s concern and are happy to answer further questions.

Table R1 (MAWPS). Exact-match accuracy (%) of free-form answers on MAWPS, using Llama-8B.

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
Canonical88.575.370.072.2
StochasTok 0.187.982.577.976.3
StochasTok 0.586.882.879.378.4
StochasTok 1.082.580.577.677.9
UniK 0.187.680.574.973.0
UniK 0.587.382.378.578.5
UniK 1.084.583.481.080.1

Table R2 (GSM8K). Exact-match accuracy (%) of free-form answers on GSM8K with chain-of-thought (first 500 test questions), using Llama-8B:

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
Canonical63.457.051.047.0
StochasTok 0.564.862.759.756.1
UniK 0.562.861.960.456.1

Table R3 (CSQA). MCQ accuracy (%) on CSQA using Llama-8B:

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
None71.464.962.258.2
Canonical81.677.173.469.5
StochasTok 0.583.178.376.173.8
StochasTok 1.080.879.076.675.0
UniK 0.583.380.276.574.7
UniK 1.081.078.776.673.8

Table R4 (HellaSwag). MCQ accuracy (%) on HellaSwag using Llama-8B:

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
None52.049.947.846.3
Canonical59.458.456.954.6
StochasTok 0.559.559.458.758.1
StochasTok 1.059.259.158.858.4
UniK 0.559.859.459.358.6
UniK 1.058.958.859.158.5

Dataset links:

Reviewer_AUKZOpenReview ↗

Response to authors

I thank the authors for their responses and supplementary experiments, and I will retain my score.

Reviewer_T3gUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies whether training with stochastic tokenizations improves the robustness of LLMs to random and adversarial non-canonical tokenizations. Authors evaluate StochasTok [1] in pre-training, fine-tuning and in-context learning, showing that it improves robustness to random retokenizations on subword-level tasks such as LanguageGame and CUTE. Authors identify that StochasTok samples tokenizations from a biased distribution with incomplete support, and propose two alternatives: StochasTok-UNI, which samples uniformly given per-token split counts, and Uniform-K, which samples uniformly among tokenizations with kk additional tokens. When fine-tuning Llama-1b on LanguageGame, Uniform-K reduces the accuracy drop under adversarial tokenization [2] from 87.9 to 25.2 percentage points without decreasing canonical accuracy. Additionally, authors provide an upper bound on the adversarial risk of a 1-layer transformer under retokenizations with bounded edit distance.

[1] Sims et al., StochasTok: Improving Fine-Grained Subword Understanding in LLMs, ICLR 2026

[2] Geh et al., Adversarial Tokenization, ACL 2025

Critical strengths
  • Systematic study of stochastic tokenization for robustness: Authors evaluate pre-training, fine-tuning and in-context learning across several models and datasets. They show that canonically fine-tuned models are brittle to non-canonical tokenizations, e.g., Llama-1b loses 29.8% accuracy under uniformly sampled tokenizations, and that fine-tuning with stochastic tokenizations removes most of this drop without hurting canonical accuracy or increasing inference cost. This is useful evidence for practitioners.

  • Identifying the limitations of StochasTok: Authors show that StochasTok [1] samples from a biased distribution with incomplete support, and that sampling uniformly improves robustness to adversarial tokenization, e.g., Uniform-K reduces the accuracy drop of Llama-1b on LanguageGame from 33.0 (StochasTok) to 25.2 percentage points with αfine=1\alpha_{\text{fine}} = 1.

  • Clear and illustrative figures and paper design. The authors put great care into having colurful figures, making the name and color of each LLM consistent accross the text and overall in making the paper have a good appearance. That really helps grasp the paper.

[1] Sims et al., StochasTok: Improving Fine-Grained Subword Understanding in LLMs, ICLR 2026

Critical weaknesses
  • Unrealistic threat model: The only adversarial attack considered is adversarial tokenization [2], where the attacker chooses how a fixed input text is tokenized. In practice, an attacker is not able to choose how the input is tokenized, e.g., when interacting with a model through an API. The attacker can only change the input text, which is then canonically tokenized. Authors do not evaluate attacks that change the input text, neither character-level attacks such as Charmer [3], nor token-level attacks such as GCG [4] or TextFooler [5]. Since these attacks produce canonically tokenized inputs, it is unclear whether training with stochastic tokenizations brings any robustness against them, which is the most relevant question for the significance of this work. Therefore, the claim that stochastic tokenization improves robustness is only supported for a perturbation model that is hard to exploit in practice. Additionally, the evaluation is restricted to multiple-choice question answering with base models, while adversarial tokenization was proposed to break the safety alignment of chat models [2].

  • Limited novelty with respect to StochasTok [1]: The main contribution is evaluating the robustness of an existing method [1] on the datasets proposed in [1]. The proposed sampling schemes build on existing tools: Uniform-K samples from the Multi-valued Decision Diagrams of [2, 6], and the non-uniformity of stochastic tokenizers and the benefits of uniform sampling were already studied for BPE-dropout in [7]. Lastly, the robustness bound in Theorem 6.1 does not depend on the tokenizations used during training, so it does not explain why training with stochastic tokenizations improves robustness.

[1] Sims et al., StochasTok: Improving Fine-Grained Subword Understanding in LLMs, ICLR 2026

[2] Geh et al., Adversarial Tokenization, ACL 2025

[3] Abad Rocamora et al., Revisiting Character-level Adversarial Attacks for Language Models, ICML 2024

[4] Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models, arXiv 2023

[5] Jin et al., Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment, AAAI 2020

[6] Geh et al., Where is the Signal in Tokenization Space?, EMNLP 2024

[7] Cognetta et al., Distributional Properties of Subword Regularization, EMNLP 2024

Questions for authors
  • How do models fine-tuned canonically, with StochasTok and with Uniform-K compare under attacks that change the input text, such as Charmer [3], GCG [4] or TextFooler [5]? Showing that stochastic tokenization improves robustness against these attacks would improve my assessment.

  • Clarification: How does Theorem 6.1 relate to the tokenization distribution used during training? As stated, γ\gamma only depends on the weights, the embedding dimension and kk.

[2] Geh et al., Adversarial Tokenization, ACL 2025

[3] Abad Rocamora et al., Revisiting Character-level Adversarial Attacks for Language Models, ICML 2024

[4] Zou et al., Universal and Transferable Adversarial Attacks on Aligned Language Models, arXiv 2023

[5] Jin et al., Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment, AAAI 2020

7 comments
AuthorsOpenReview ↗

Answer to Reviewer T3gU (1/3)

We thank the reviewer for their detailed review! We are pleased that the reviewer found our study to be systematic and our presentation to be clear. We agree that the wording of our original manuscript suggested general (adversarial) robustness. Thus, we have revised it to present the scope of our study more accurately. Below, we respond to each individual comment in detail.


Threat model:

  • Clarify scope: We revised the abstract, introduction, conclusion, and limitations to clarify that we study robustness to non-canonical tokenizations, where the input string remains unchanged and only its tokenization differs. Attacks that change the text (e.g., Charmer, GCG, TextFooler) define a different perturbation set: character edits produce a different string x′≠xx'\neq x, whose tokenizations are not part of TV(x)\mathcal{T}_\mathcal{V}(x), the set of tokenizations of xx that our models are trained with. Thus, stochastic tokenization is not designed as a defense against such edits and its evaluation is out of scope for our study. How robustness to tokenizations interacts with robustness to text edits is, however, an interesting direction for future work and now discussed in our limitations.

  • Practical relevance: Robustness to different tokenizations can be viewed as a consistency requirement: all tokenizations in TV(x)\mathcal{T}_\mathcal{V}(x) represent the same string xx, and predictions should not change. Importantly, such non-canonical tokenizations can arise in practice:

    • Open-weight models are increasingly deployed and routinely run with token-level access, and common inference stacks (e.g., vLLM, llama.cpp) accept token IDs as inputs, unlike text-only APIs. This makes it possible to input different (possibly adversarial) tokenizations without changing the underlying text.
    • Without an attacker: LLMs have been shown to generate the same output string with different tokenizations themselves [8]. For German outputs, 6–22% (Llama-8B) and 5–29% (Qwen-7B) of sampled generations are non-canonical across translation, spell checking, rephrasing and math tasks [8, Table 1]. Such token sequences may be passed between components (e.g., autoregressive generation, or in multi-turn or agent loops that keep the token history), where downstream behavior should be consistent across tokenizations of the same string.
  • Safety alignment: We use the attack of [2] to find worst-case tokenizations to study the tokenization robustness of models trained with different tokenization schemes. However, we do not study the safety alignment of chat models, as attacked in [2]. Such an evaluation is of practical interest, but requires integrating stochastic training in the SFT/preference/safety post-training pipeline, which is an interesting future direction. We discuss this in the limitations.


Beyond token-sensitive MCQ:

We perform additional fine-tuning experiments to test whether performance degradation of canonically trained models also extends to free-form generation.

  • We fine-tune Llama-8B on two math datasets: (i) MAWPS and (ii) GSM8K with chain-of-thought. We analyze the free-form generation robustness given stochastic tokenizations during evaluation (Tables R1-R2 in comment below). The findings are consistent with our MCQ evaluations: Under uniformly sampled tokenizations, canonical fine-tuning loses about 16 percentage points on both tasks. StochasTok/UniK (α=0.5\alpha=0.5) fine-tuning roughly halves the degradation, while keeping canonical accuracy drop within 1.7 pp.
  • We also add general-purpose benchmarks (CSQA, HellaSwag, Tables R3–R4 in comment below), where stochastic fine-tuning narrows the drop in accuracy from canonical-to-uniform evaluation from 12.1 pp to 5.8–9.3 pp (CSQA) and from 4.8 pp to 0.4–1.4 pp (HellaSwag).

We include the results in our revised paper (Sec. 4.2, Table 1; detailed results in Sec. I.2.3).

AuthorsOpenReview ↗

Answer to Reviewer T3gU (2/3)


Novelty:

  • Relative to StochasTok [1]. StochasTok was proposed to improve subword understanding and was mainly applied during pre-training. Its robustness evidence is limited to multi-digit addition, qualitative generations and an embedding-space analysis. Our contributions are (i) a systematic study of robustness to non-canonical tokenizations for randomly sampled and worst-case tokenizations across pre-training, fine-tuning, and ICL, and (ii) showing that StochasTok's bias and limited support matter most under adversarial tokenization attacks (Fig. 6). In this rebuttal, we also present new experiments (Tables R1-R4, discussed above; revised paper Sec. 4.2, Table 1, detailed results in Sec. I.2.3), where our evaluation extends beyond the datasets in [1].
  • Relative to [7]. Cognetta et al. show that uniform sampling improves translation quality for BPE-dropout. Instead, we study robustness to non-canonical tokenizations. Our ablation (now Sec. I.2.4) separates the effects of uniformity and support: Increasing uniformity on a fixed (incomplete) support provides little gain, while increasing support does. We also compare directly to BPE-dropout (Fig. 21).
  • Relative to [2]. UniK uses the MRMDD construction of Geh et al. [2], and we do not claim that the construction is novel. We adapt it to sample a fixed number of added tokens, matching StochasTok’s split budget. Our contribution is using such controlled sampling distributions during training to study how support and uniformity affect tokenization robustness.

Theorem:

We agree that γ\gamma depends on the training distribution only via the learnt weights. Thus, the theorem alone does not explain why stochastic training is helpful. Its purpose is to show that local sensitivity across the tokenization neighbourhood controls the risk of adversarial tokenization. Whether a stochastic training scheme reduces this sensitivity is assessed empirically. Our support ablation experiments (Figure 16) show that methods covering a larger portion of the tokenization neighbourhood achieve greater robustness against adversarial tokenization. Furthermore, our layer-wise sensitivity analysis in Figure 22 shows smaller mean normalised distances between the hidden representations of canonical and non-canonical tokenizations following stochastic training, particularly in deeper layers, which is consistent with reduced local sensitivity. We have revised Section 6.1 and the conclusion accordingly.

We hope this clarifies the scope and contribution of the paper, and we are happy to answer further questions.

AuthorsOpenReview ↗

Results - Answer to Reviewer T3gU (3/3)

Table R1 (MAWPS). Exact-match accuracy (%) of free-form answers on MAWPS, using Llama-8B.

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
Canonical88.575.370.072.2
StochasTok 0.187.982.577.976.3
StochasTok 0.586.882.879.378.4
StochasTok 1.082.580.577.677.9
UniK 0.187.680.574.973.0
UniK 0.587.382.378.578.5
UniK 1.084.583.481.080.1

Table R2 (GSM8K). Exact-match accuracy (%) of free-form answers on GSM8K with chain-of-thought (first 500 test questions), using Llama-8B:

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
Canonical63.457.051.047.0
StochasTok 0.564.862.759.756.1
UniK 0.562.861.960.456.1

Table R3 (CSQA). MCQ accuracy (%) on CSQA using Llama-8B:

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
None71.464.962.258.2
Canonical81.677.173.469.5
StochasTok 0.583.178.376.173.8
StochasTok 1.080.879.076.675.0
UniK 0.583.380.276.574.7
UniK 1.081.078.776.673.8

Table R4 (HellaSwag). MCQ accuracy (%) on HellaSwag using Llama-8B:

Fine-tuningCanonicalStochasTok 0.5StochasTok 1.0Uni
None52.049.947.846.3
Canonical59.458.456.954.6
StochasTok 0.559.559.458.758.1
StochasTok 1.059.259.158.858.4
UniK 0.559.859.459.358.6
UniK 1.058.958.859.158.5

Dataset links:

[1] Sims et al., StochasTok: Improving Fine-Grained Subword Understanding in LLMs, ICLR 2026

[2] Geh et al., Adversarial Tokenization, ACL 2025

[7] Cognetta et al., Distributional Properties of Subword Regularization, EMNLP 2024

[8] Chatzi, Ivi, et al. "Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service." arXiv preprint arXiv:2506.06446 (2025).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank all reviewers for their detailed and constructive feedback. We appreciate that the reviewers recognized the systematic study of stochastic tokenization (T3gU, AUKZ, Yq2a), the analysis of StochasTok’s limitations and the proposed uniform sampling schemes with larger support (T3gU, AUKZ, Yq2a), and the clear presentation (T3gU, Yq2a).

We address all comments individually and have updated the manuscript accordingly. The main changes are:

  • Clarified scope: we revised our manuscript to clarify that we study robustness to non-canonical tokenizations of a fixed input string, not general adversarial robustness.
  • New experiments (Sec. 4.2, Table 1; full results in App. I.2.3): fine-tuning on general-purpose benchmarks (CSQA, HellaSwag) and free-form math generation (MAWPS, GSM8K with chain-of-thought, GSM-Symbolic).
  • Model-size ablation (1B–8B) moved to the main text (Sec. 6, Table 3).
  • Clarifications: definition of P(⋅∣x;α)P(\cdot \mid x; \alpha), and the role of Theorem 6.1 (Sec. 6.1).

All changes are highlighted in red in the revised manuscript, with margin notes indicating the reviewer whose comment they address.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.