acceptodds
Desk rejected at ICLR 2027

GlitchPatch: Repairing Glitch Tokens in Frozen Language Models via Local Retokenization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Glitch tokens are anomalous vocabulary entries that can cause large language models (LLMs) to produce outputs inconsistent with their inputs. Existing repair methods require access to model internals, making them impractical for frozen checkpoints. We investigate whether glitch tokens can be repaired outside the model by optimizing the input tokenization. An empirical study on BPE merge-rule deletion reveals that (1) deleting a glitch token's merge rule can fix a substantial fraction of failures, yet disrupting normal tokens sharing intermediate merge nodes causes the overall glitch rate to rise, and (2) different decomposition granularities yield non-monotonic fix rates while collateral damage on normal tokens grows monotonically. Motivated by these findings, we propose **GlitchPatch**, a repair framework for frozen language models based on local retokenization, consisting of two stages: the *offline stage* uses Behavioral Path Optimization (BPO) to find the behaviorally optimal replacement token sequence for each glitch token and compiles validated replacements into a rule table; the *online stage* substitutes only the IDs of matched glitch tokens in the canonical token sequence, with no modification to model parameters or internal states. Experiments on ten models spanning six tokenizer families show that **GlitchPatch** achieves an 85.10% mean fix rate, outperforming the strongest baseline by 14.37 percentage points, and reduces the average glitch rate from 14.88% to 2.27%. **GlitchPatch** achieves a 0.00% RR in full-vocabulary evaluation and leaves rule-unmatched inputs unchanged by design. We further evaluate the practical impact of repair from the perspectives of time cost, language understanding, and capability, supporting its deployment feasibility. We hope this work provides a practical option for improving tokenizer reliability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Glitch Tokens, Tokenization, Frozen Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_NtfNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors researched a patching method of glitch tokens in LLMs. The existing methods require access to the model internal states, a requirement that is hard to meet in certain deployment scenarios. They find that simply deleting glitch-token merge rules helps repair but introduces new errors, and fix rates vary by path while collateral damage grows with depth. Based on these findings, the propose GLITCHPATCH. They used BPO to find an optimal replacement token sequence for each glitch token offline, and replace matched glitch-token IDs at inference time using replacement sequences precomputed offline.

Critical strengths
  1. The paper addresses a novel and practically valuable problem. It investigates how glitch tokens can be repaired in restricted-access settings where model parameters and internal states cannot be modified. By intervening at the tokenization level, the proposed approach provides a lightweight patch that can be deployed without retraining or altering the model.
  2. The local replacement mechanism effectively avoids disturbing normal tokens. GLITCHPATCH replaces only token IDs that match known glitch tokens in the canonical token sequence, while preserving all other token IDs. As a result, inputs that do not trigger a rule remain exactly unchanged, substantially reducing the risk of collateral damage.
  3. The method achieves substantial improvements under the paper’s primary repair evaluation. It consistently outperforms the strongest baseline in fix rate while markedly reducing the overall glitch rate. These results show that local retokenization can be highly effective for the targeted glitch-token failures.
  4. The repair results are validated across a diverse range of models and tokenizers. The full-vocabulary evaluation spans multiple model and tokenizer families, suggesting that the observed repair effect is not confined to a particular model configuration.
Critical weaknesses
  1. The tests used to identify glitch tokens do not directly imply safety risks. The authors define glitch tokens solely based on failures in the repetition, spelling, and length tasks. However, failing these tasks does not necessarily mean that a token can be exploited to bypass safety mechanisms or induce unsafe outputs. The paper also does not evaluate the unsafe rate or attack success rate before and after repair, and therefore cannot demonstrate that GLITCHPATCH reduces the safety risks identified in prior work.

  2. The optimization objective for replacement paths is aligned with the primary evaluation objective, making it difficult to establish a general capability improvement. BPO’s path selection, utility evaluation, and final FR evaluation are all based on the repetition, spelling, and length tasks. Although the prompt templates differ, they still belong to the same task family. The results mainly demonstrate that the selected paths can pass these optimized tests, but do not yet establish that they truly restore the token’s semantic representation or generalize to more general tasks. The limited OOD experiments in Appendix H provide some supplementary evidence, but their coverage in terms of models, sample size, and evaluation methodology remains limited. In addition, the paper does not compare against a simple baseline that locally decomposes only glitch tokens into character/byte tokens. Prior work shows that even full-text character-level tokenization has a limited impact on the performance of several large models (Understanding the Ability of LLMs to Handle Character-Level Perturbation), and since glitch tokens typically account for only a small fraction of the text, the length cost of applying this decomposition locally is also relatively limited. Therefore, the additional benefit of BPO remains unclear.

  3. The general-capability evaluation does not effectively isolate the samples on which the patch actually takes effect. On MMLU-Pro, ARC, GSM8K, and HumanEval, the paper reports only overall score changes, without specifying how many sequences in each dataset contain a glitch token or how many actually trigger a repair rule. Sequences that do not contain a glitch token are, by construction, unchanged by the patch. Therefore, the small changes in the overall scores are largely expected and cannot effectively evaluate the patch’s impact on the inputs that are actually modified.

Questions for authors
  1. Could the authors evaluate whether GLITCHPATCH reduces the incidence of unsafe behavior, or other practically meaningful failures such as the misunderstanding of user input illustrated in Figure 1?
  2. Could the authors compare BPO against a simple local baseline that directly decomposes each glitch token into character/byte-level tokens while leaving the rest of the canonical tokenization unchanged, and quantify how much improvement BPO provides over this baseline?
  3. Could the authors clarify what proportion of the instances in the general-capability benchmarks contain a glitch token and actually trigger an accepted repair rule, or directly report the performance on those triggered instances?
4 comments
AuthorsOpenReview ↗

A Big Thank You and Our Response to Reviewer NtfN

Thank you so much for the careful and insightful review! The comments have helped us make the paper noticeably stronger, and we have addressed each of them. All changes are marked in blue in the revised PDF. Our point-by-point responses are as follows.


To W1 and Q1: Thank you for the suggestion. We have added a safety evaluation in §6.2 (new paragraph) and Appendix I (new Table I1):

  • Setup: following the protocol of [1], we construct 1,000 harmful prompts per model, each containing a glitch token sampled uniformly from the detected set, judge outputs with GPT-5.6-Terra (93% agreement with humans on 100 items); all methods use the same prompts.
  • Results: the original models produce unsafe outputs on 23.4% of these prompts; GlitchPatch lowers this to 6.1%, the lowest of all methods on every model.
  • The Figure 1 type of failure is quantified by the Entity QA and Verbatim Copy tasks (Table 3), where GlitchPatch gains +25.6 and +26.2 pp over the original.

Table I1. Unsafe-output rate (%) on 1,000 glitch-containing prompts per model.

ModelOriginalGlitchProberGlitchEditGlitchCleanerGlitchPatch
Mistral-7B30.928.312.721.49.2
Qwen3.8-27B22.119.67.614.14.9
Llama-3.1-70B17.315.86.211.24.1
Mean23.421.28.815.66.1

[1] Tang et al., USENIX Security 2026. https://dl.acm.org/doi/10.5555/3857392.3857500


To W2 and Q2: Thank you for the suggestions. We have scaled up the OOD evaluation of Appendix H.2 (now also §6.2, Table 3) and added the simple local baseline in §6.5 (new Table 7), as follows:

  • On the OOD evaluation: we have scaled the three downstream tasks from 500 to 2,000 uniformly sampled glitch tokens per model and from three to five model families. GlitchPatch raises mean accuracy from 45.2% to 67.8% (+22.6 pp), with gains on every task and model, suggesting that the selected paths restore the tokens' behavior beyond the detection tasks.
  • On the simple local baseline: we evaluate five fixed local rules: local-char/byte with and without the length budget, and local depth-1/2/3 (suggested by Reviewer 1MP3), all under the same settings as GlitchPatch. No fixed rule exceeds 53.89% FR, versus 85.00–89.00% for GlitchPatch: strong models do tolerate character-level tokenization, but the best path is model-specific and non-monotonic in depth (Finding 2), so the gain comes from behavioral path selection.

Table 3 (new, §6.2). Downstream accuracy (%), mean over five models (per-model results in Table H2).

TaskOriginalGlitchProberGlitchEditGlitchCleanerGlitchPatch
Entity QA43.745.251.458.469.3
Verbatim Copy39.541.347.053.865.7
Code Identifier52.654.057.361.368.3
Overall mean45.246.851.957.967.8

Table 7. Fixed local decompositions vs. BPO, FR (%); all under budget B = 8 (†without).

MethodLlama3.1-8B FRQwen3.8 FRGemma4-31B FR
Local-Depth138.3841.8643.53
Local-Depth235.0745.0344.21
Local-Depth339.2139.6941.81
Local-Char/Byte31.8435.3836.01
Local-Char/Byte (no budget)†47.5851.2953.89
GlitchPatch85.0087.5089.00

To W3 and Q3: Thank you for the suggestion. We have added the triggered-subset analysis in Appendix F.1 (new Table F2):

  • Setup: per model and benchmark, we count the instances that contain a glitch token and those that trigger a rule, and report accuracy on the triggered subset with gain / loss counts.
  • Results: 1.9% of instances contain a glitch token and 1.5% trigger a rule. On the 489 triggered instances, GlitchPatch changes 7 outcomes (+5 / −2; acc. 66.1% → 66.7%). The gains are instances whose answer hinged on the token, the losses borderline items that flip under the small input change; for the remaining 482 the outcome is unchanged either way. Where answers do hinge on the token (the Appendix H.2 tasks), GlitchPatch gains +22.6 pp. The repair thus changes outcomes where glitch tokens matter and otherwise leaves them unchanged.

Table F2. Glitch-containing instances and triggered-subset accuracy, aggregated over the ten models.

BenchmarkContainTriggerOrig. acc. (%)GlitchPatch acc. (%)Gain / Loss
MMLU-Pro268 (2.7%)221 (2.2%)57.957.9+2 / −2
ARC142 (1.4%)118 (1.2%)75.477.1+2 / 0
GSM8K61 (0.6%)49 (0.5%)73.575.5+1 / 0
HumanEval124 (7.6%)101 (6.2%)69.369.30 / 0
All595 (1.9%)489 (1.5%)66.166.7+5 / −2

We hope these clarifications address the concerns and would be grateful if the reviewer would consider updating the assessment. We would be happy to answer any further questions during the discussion period. Thank you again!

AuthorsOpenReview ↗

Gentle follow-up on our response

Dear Reviewer NtfN,

Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.

We understand that this is a very busy period, and we apologize for the additional message. As the discussion period is ending soon, we would be very grateful if you could let us know whether our response has addressed your concerns.

If anything remains unclear, we would be more than happy to provide further clarification. If you find that your concerns have been resolved, we would deeply appreciate it if you would kindly consider updating your assessment.

Thank you again for your time and consideration.

Best regards,

The Authors

I appreciate the authors' response and supplementary experiments, which have clarified my concerns. And I will raise my score from 2 to 3. Best of luck with the submission!

AuthorsOpenReview ↗

Thank You to Reviewer NtfN

Dear Reviewer NtfN,

Thank you so much for reading our response and for raising your score. We are glad that your concerns have been clarified, and we sincerely appreciate your constructive suggestions, which have made the paper stronger. Thank you again for your thoughtful review and kind wishes! :)

Best regards,

The Authors

Reviewer_Zcm7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to solve a problem of "glitch tokens" where certain tokens cause LLMs to give surprisingly incorrect answers. The authors aim to solve this by finding such glitch tokens and finding an alternative to tokenizing them. Experiments show this method solves most of the glitch tokens (85%)

Critical strengths
  1. The paper is well written and easy to follow.
  2. the method is easy to implement in open-source models
Critical weaknesses
  1. Is this a problem on modern LLMs like gpt-6-astra, opus-5.5, kimi k3, glm 5.3? I know that the method cannot work on some of the closed source models I mentioned but it is possible to evaluate and see if they have any glitch tokens in the first place. If modern LLMs do not have glitch tokens, I am not sure the problem is well motivated.

  2. "For each model, every vocabulary token is tested on three GlitchQuiz tasks (repetition, spelling, length)." this is very limited. The glitch token should be evaluated in more realistic contexts beyond these three toy tasks. In the later experiments, where it is evaluated on much harder tasks, the improvement is not very big.

  3. The "glitch" tokens are identified using glitchquiz and also evaluated on whether they are solved using glitchquiz. This is not rigorous.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

A Big Thank You and Our Response to Reviewer Zcm7

Thank you so much for the careful and insightful review! The comments have helped us make the paper noticeably stronger, and we have addressed each of them. All changes are marked in blue in the revised PDF. Our point-by-point responses are as follows.


To W1: Thank you for raising this point, which we agree is important for the motivation. Glitch tokens are still present in current models, including the four mentioned. We have updated §1, and we would like to offer three pieces of evidence:

  • Most of the ten models we evaluate were released in 2026 (e.g., Qwen3.8, Gemma-4, GLM-4.7), with 11.86–18.42% glitch tokens in their vocabularies (Table A2).
  • Beyond the recent academic work reviewed in §2, public reports of the same phenomenon continue for modern LLMs: input strings that trigger anomalous output on Claude Opus 5 [1], glitch tokens in GPT-5 [2], token-drift repetition loops on GPT-6 [3], and anomalous tokens on DeepSeek-V3/R1 [4].
  • We also tested the four models named. (1) kimi k3 and glm 5.3 (public tokenizers): we randomly sampled 5,000 vocabulary entries and ran the three detection tasks; 9.26% (kimi k3) and 12.82% (glm 5.3) fail at least one task. (2) gpt-6-astra and opus-5.5 (tokenizers not public): we ran the same three tasks on 5,000 randomly sampled strings of glitch tokens detected on our ten models; although most of these strings may be re-tokenized into ordinary token sequences by the closed models' own tokenizers, 5.82% (gpt-6-astra) and 4.06% (opus-5.5) still fail at least one task, versus 0.28% / 0.12% on 5,000 control strings of ordinary tokens, so the phenomenon persists there as well.

[1] https://www.lesswrong.com/posts/ZSBse2fyftgHiJ3Kq

[2] https://www.lesswrong.com/posts/iY9584TRhqrzawhZg

[3] https://community.openai.com/t/1408624

[4] https://outsidetext.substack.com/p/anomalous-tokens-in-deepseek-v3-and


To W2 and W3: Thank you for these comments. Both concerns relate to how detection and evaluation are connected, so we clarify them separately: how glitch tokens are detected and how our proposed repair is evaluated.

  • How glitch tokens are detected (§2.2, §5, Appendix A). The three-task detection criterion follows prior work (§2.2) and is applied to every vocabulary entry, not a sample. It is deliberately a low bar: a token that fails even these basic tasks is unlikely to behave correctly in harder contexts, and indeed 70.7% of the detected tokens fail at least one downstream task (Appendix H.2). This step only identifies which tokens need repair. Our main contribution is the repair, whose conclusions hold under our sensitivity analysis of the detection tasks (Appendix A.6).
  • How the proposed repair is evaluated. We evaluate GlitchPatch along four dimensions: detection-level repair, realistic tasks on glitch-containing inputs, safety, and general-capability preservation. The first three ask whether the repair works; the last asks whether it is reliable to deploy. GlitchQuiz is thus only one of the four, and the most basic:
    • Detection-level repair (§6.1, Tables 1–2): as the most basic requirement, we check whether the identified glitch tokens pass the detection tasks after repair. This is the natural first check for three reasons: 1) a repaired token must at least pass the tasks that flagged it; 2) prior repair methods use this criterion; and 3) our repair never uses the detection prompts (§4.1.2). GlitchPatch fixes 80.50–91.04% of glitch tokens, surpassing all existing methods.
    • Realistic tasks on glitch-containing inputs (§6.2; Appendix H.2): we then test whether repaired tokens behave correctly in realistic contexts on three downstream tasks (Entity QA, Verbatim Copy, Code Identifier) and five model families. GlitchPatch improves accuracy by +22.6 pp over the original model and +15.6 pp over the three baselines on average; the loss on affected text (Appendix G) and contextual regression (Appendix H.1) agree.
    • Safety (§6.2; new Appendix I): we further measure whether glitch tokens still induce unsafe outputs after repair, following prior work. GlitchPatch reduces the unsafe rate from 23.4% to 6.1%, the lowest among all methods.
    • General-capability preservation (§6.2): finally, we check that the repair does not affect normal inputs on four benchmarks; scores change by 0 to +0.025 pp. This is the intended outcome: by Proposition 2, inputs without a glitch token are unchanged, so Δ ≈ 0 means no collateral damage. If the “improvement is not very big” impression comes from this, we would like to note that it measures preservation rather than improvement.
    • Overall, GlitchPatch repairs most glitch tokens, carries over to realistic tasks and safety, and leaves ordinary inputs untouched.

We hope these clarifications address the concerns and would be grateful if the reviewer would consider updating the assessment. We would be happy to answer any further questions during the discussion period. Thank you again!

AuthorsOpenReview ↗

Gentle follow-up on our response

Dear Reviewer Zcm7,

Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.

We understand that this is a very busy period, and we apologize for the additional message. As the discussion period is ending soon, we would be very grateful if you could let us know whether our response has addressed your concerns.

If anything remains unclear, we would be more than happy to provide further clarification. If you find that your concerns have been resolved, we would deeply appreciate it if you would kindly consider updating your assessment.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_1MP3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript proposes a method at the tokenization level to solve the problem of glitch tokens. Particularly, this method uses Behavioral Path Optimization to find the best replacement of glitch tokens at the offline stage, and replace the glitch tokens at the online stage. The experiment result shows that this method has better performance than the existing three methods over 10 LLMs.

Critical strengths
  1. It is natural to solve the glitch token problem at the tokenization level without access to the model internals.
  2. The solution is simple and at low cost.
Critical weaknesses

Although I generally think it has good performance, I am concerned about the evaluation on how good it is:

  1. Wide definition on glitch token. LLMs have problems on count letters, even for well-trained tokens [1], and one of the glitch token’s standards: length (whether models can correctly count characters in tokens), is a similar task. This may cause a higher estimation of the number of glitch tokens, resulting in overestimating on repairing performance.
  2. The choice of main result tables. The result in H.2 seems a better choice of main result table, but requires a larger scale on settings. Here are the reasons: 1. Glitch tokens may not really "glitch". The original models already reach 46.6% mean accuracy on inputs containing detected glitch tokens, suggesting many detected glitch tokens do not cause failures in context. 2. The performance gain between GlitchPatch and the original model is more valuable to estimate the performance of GlitchPatch.

[1] Fu, Tairan, et al. "When Do Large Language Models (LLMs) Struggle to Count Letters?." ACM Transactions on Intelligent Systems and Technology 17.5 (2026): 1-17.

Questions for authors
  1. Section 3 reports the result of depth-1 with global deletion. Could you report the result of applying the depth-1 with local deletion, i.e., replacing each glitch-token ID with its two merge-tree children at the online stage? This can avoid the problem in finding 1, and the comparison with GlitchPatch can show the advantage of Path Search via BPO.

Things to improve the paper that did not impact the score:

  1. Tables D7 -> Table D7 (Appendix D.4)
  2. Qwen3.8-27B(Qwen Team, 2026) -> Qwen3.8-27B (Qwen Team, 2026) (Line 117)
4 comments
AuthorsOpenReview ↗

A Big Thank You and Our Response to Reviewer 1MP3

Thank you so much for the careful and insightful review! The comments have helped us make the paper noticeably stronger, and we have addressed each of them. All changes are marked in blue in the revised PDF. Our point-by-point responses are as follows.


To W1: Thank you for raising this point. We have added a sensitivity analysis of the glitch-token definition in Appendix A.6 (new Table A3) and would like to clarify the following:

  • The three-task definition follows prior work (§2.2, lines 136–141) and is applied identically to all four repair methods, so it affects the absolute counts but not the comparison of repair effectiveness across methods. Moreover, at least 80–88% of each vocabulary (the normal tokens) passes the length task, suggesting that length failures are largely token-specific rather than a general counting limitation.
  • Appendix A.6 evaluates all methods under three definitions: (A) repetition only, (B) repetition or spelling, and (C) any of the three tasks (the paper's definition). Tokens that fail only the length task make up 15.05–29.49% of the detected set. GlitchPatch's mean FR is 82.2% / 84.0% / 85.1% under A / B / C, and the ranking of the four methods is unchanged under each definition, so the conclusions do not hinge on the length task.

Table A3 (summary). Mean FR (%) over ten models (per-model results in the revised PDF).

DefinitionGlitchProberGlitchEditGlitchCleanerGlitchPatch (Ours)
A: repetition only22.956.168.382.2
B: repetition ∪ spelling23.857.369.584.0
C: any of the three (paper)24.658.470.785.1

To W2: Thank you for this suggestion. We address the two points as follows:

  • On the 46.6% accuracy: this figure is the mean accuracy per task, not the fraction of tokens that behave normally. A token correct in one task may still fail another, and our detection criterion is likewise failure on any of the three tasks. Measured the same way, i.e., failing at least one of the three downstream tasks, 70.7% of the sampled glitch tokens fail under the original models (67.4–74.6% across models). These results therefore support our detection criterion, and the detected tokens still cause failures in downstream contexts.
  • On the main results: we have moved the downstream evaluation of H.2 into the main text (§6.2, new Table 3; full table in Appendix H.2, Table H2) and, to make it a stronger main result, scaled it from 500 to 2,000 uniformly sampled glitch tokens per model and from three to five model families. GlitchPatch raises mean accuracy from 45.2% to 67.8% (+22.6 pp over the original, +9.9 pp over GlitchCleaner), with consistent gains on every task and model.

Table 3 (new, §6.2). Downstream accuracy (%) on glitch-containing inputs, mean over five models (per-model results in Appendix H.2, Table H2).

TaskOriginalGlitchProberGlitchEditGlitchCleanerGlitchPatch (Ours)
Entity QA43.745.251.458.469.3
Verbatim Copy39.541.347.053.865.7
Code Identifier52.654.057.361.368.3
Overall mean45.246.851.957.967.8

To Q1: Thank you for the suggestion. We have added the local depth-1/2/3 and character/byte-level decomposition baselines (the latter suggested by Reviewer NtfN) in §6.5 (new Table 7):

  • Setup: each glitch-token ID is replaced online by its merge-tree children at the given depth (char/byte: single characters or bytes), with all other IDs unchanged and under the same length budget B = 8 as GlitchPatch (unless marked).
  • Results: local depth-1 reaches an FR close to the global depth-1 rule in §3 but without the collateral regression; deeper rules do not help monotonically, and char/byte-level decomposition fixes only 31.84–36.01% because it exceeds the budget for most glitch tokens (47.58–53.89% without the budget), whereas GlitchPatch fixes 85.00–89.00%.
  • The gap between any fixed rule and BPO is the quantitative counterpart of Finding 2: which path works is token- and model-specific, so a fixed rule cannot exploit it, whereas BPO does.

Table 7. Fixed local decompositions versus BPO (FR, %; †without the budget).

MethodLlama3.1-8B FRQwen3.8 FRGemma4-31B FR
Local-Depth138.3841.8643.53
Local-Depth235.0745.0344.21
Local-Depth339.2139.6941.81
Local-Char/Byte31.8435.3836.01
Local-Char/Byte (no budget)†47.5851.2953.89
GlitchPatch (Ours)85.0087.5089.00

To minor points: Both fixed. Thank you!


We hope these clarifications address the concerns and would be grateful if the reviewer would consider updating the assessment. We would be happy to answer any further questions during the discussion period. Thank you again!

AuthorsOpenReview ↗

Gentle follow-up on our response

Dear Reviewer 1MP3,

Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.

We understand that this is a very busy period, and we apologize for the additional message. As the discussion period is ending soon, we would be very grateful if you could let us know whether our response has addressed your concerns.

If anything remains unclear, we would be more than happy to provide further clarification. If you find that your concerns have been resolved, we would deeply appreciate it if you would kindly consider updating your assessment.

Thank you again for your time and consideration.

Best regards,

The Authors

Thanks for the updates. These updates partly answer my concerns. However, the new Table A3 indicates that a wide range of "glitch tokens" just fail on one task. Therefore, (i) some "glitch tokens" may be not so glitch. (ii) some real glitch tokens may cannot be detected by these three tasks.
SInce the score is already 3, I will keep my score. Good luck.

AuthorsOpenReview ↗

A Big Thank You for the Follow-up

Dear Reviewer 1MP3,

Thank you so much for taking the time to read our response and for the thoughtful follow-up! We are glad the updates were helpful, and we would like to add two brief clarifications on the points raised.

(i) Some detected tokens fail only one task. This is what Appendix A.6 was added to examine. The three-task criterion is the shared definition of prior work (§2.2), so we kept it for comparability, and Table A3 shows that under the stricter definitions (repetition only; repetition or spelling) the ranking of all four repair methods and GlitchPatch's fix rate (82.2–85.1%) are unchanged. Moreover, 70.7% of the tokens detected under the broad definition fail at least one realistic downstream task (Appendix H.2), so most of them are problematic in practice, not only on the detection tasks.

(ii) Some real glitch tokens may escape the three tasks. We agree that detection recall is an open problem for the field, and we do not claim to solve it: detection is the input to our method, not its contribution. GlitchPatch repairs whatever set a detector provides, and any token not detected is simply left unchanged (Proposition 2), so missed tokens cause no regression and can be repaired incrementally once a better detector flags them. The downstream, safety and general-capability evaluations (§6.2, Appendices F–I) assess the repair independently of the detection tasks.

In short, both points concern the detection criterion, which we adopt from prior work, rather than the repair method the paper proposes. Thank you again for your careful review and for helping us make the paper clearer!

Best regards,

The Authors

Reviewer_JDYYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

GlitchPatch repairs problematic vocabulary entries in frozen language models through local input retokenization. It builds a graph of byte-preserving segmentations using normal tokens, ranks candidates with a cheap proxy, and validates selected paths with behavioral probes and separate calibration. At inference, only targeted token IDs are replaced. On ten models, the paper reports an average 85.10% fix rate, compared with 70.73% for the strongest reported baseline, and no measured regression on isolated normal-token probes. Additional ablations and contextual evaluations examine repairability, general-task behavior, and deployment cost.

Critical strengths

S1. Practical locality The method changes only selected input IDs and preserves decoded content, making the intervention easy to delimit. The guarantee is correctly framed around unchanged effective inputs rather than arbitrary downstream semantic preservation.

S2. Broad and consistent repair evidence Full-vocabulary experiments across ten models and six tokenizer families show substantial improvements over several repair baselines. The top-K and component ablations explain important engineering choices.

S3. Checks beyond isolated probes The manuscript includes held-out calibration, general benchmark checks, neighboring-span tests, and contextual entity/copy/identifier tasks. These are useful steps toward assessing collateral effects, although their scope remains limited.

Critical weaknesses

W1. Loss values use different prediction units Appendix G compares token-averaged cross-entropy before and after retokenization and interprets a mean 13.47% reduction as improved understanding. Both the token events and the denominator change, so these averages are not directly comparable. The passages also contain targets sampled from successfully repaired tokens. A common-byte normalization and/or likelihood on an identical untouched continuation would provide a more interpretable test; the current result should not establish general language understanding improvement.

W2. Acceptance criteria need reconciliation Section 4 and Algorithm 1 require candidates to pass all three behavioral dimensions, while the utility contains a regression penalty and Appendix D reports sensitivity to κ. If every selection probe must pass, the corresponding regression count should already be zero. If passing means a weaker aggregate criterion or uses a separate probe set, the manuscript should state that rule precisely and distinguish selection, validation, and calibration.

W3. Deployment overhead and collateral regressions need clearer denominators The mean 0.065% online overhead is difficult to assess without input/output lengths, trigger frequency, batch conditions, and a separation of host processing, prefill, and decoding. A longer replaced prefix can also increase decoding work. Likewise, near-zero net accuracy changes on neighboring spans can hide offsetting per-example gains and regressions. Triggered-only latency and per-example regression counts would make these claims more actionable.

Questions for authors

Q1. Does the natural-language benefit persist when evaluated on identical continuation tokens or an explicitly justified byte-normalized objective, including targets not selected for successful repair?

Q2. What exactly constitutes passing each behavioral dimension, which probe set determines this, and when can the κ penalty change an accepted candidate?

Q3. Can the runtime table include triggered-only and overall latency under specified prompt length, output length, batch size, and trigger density, together with per-example contextual regression counts?

2 comments
AuthorsOpenReview ↗

A Big Thank You and Our Response to Reviewer JDYY

Thank you so much for the careful and insightful review! The comments have helped us make the paper noticeably stronger, and we have addressed each of them. All changes are marked in blue in the revised PDF. Our point-by-point responses are as follows.


To W1 and Q1: Thank you for the suggestion. Accordingly, we have revised Appendix G to better evaluate the natural-language benefit with two comparable objectives: (a) byte-normalized NLL, the total NLL of the passage divided by its byte length, so the denominator is identical under both tokenizations; and (b) identical-continuation NLL, the NLL on the 32 tokens immediately following the target span. Both are evaluated on all ten models and all four methods. For each model, 1,000 targets are sampled uniformly from all detected glitch tokens (not from repaired tokens only), and GPT-5.6-Terra generates a natural passage of about 200 tokens containing each target. The results show that GlitchPatch yields the largest reduction under both objectives on every model, and that the continuation following a repaired token becomes measurably more predictable. We have also revised the related statements in the abstract, §1, and §6.2 accordingly. Below we provide the results on three representative models; the full ten-model results are in Tables G1 and G2 of the revised PDF.

Table G1. Relative reduction (%) in byte-normalized NLL on passages containing glitch tokens (full ten-model results in Table G1 of the revised PDF).

ModelGlitchProberGlitchEditGlitchCleanerGlitchPatch (Ours)
Mistral-7B1.73.45.08.7
Qwen3.8-27B1.43.05.48.2
Llama-3.1-70B1.12.84.46.3
Mean (10 models)1.43.34.98.4

Table G2. Relative reduction (%) in NLL on the identical 32-token continuation following the target span; positive values mean the continuation becomes more predictable after repair (full ten-model results in Table G2 of the revised PDF).

ModelGlitchProberGlitchEditGlitchCleanerGlitchPatch (Ours)
Mistral-7B0.61.52.34.1
Qwen3.8-27B0.51.22.53.9
Llama-3.1-70B0.31.11.93.1
Mean (10 models)0.51.32.23.8

To W2 and Q2: Thank you for raising this point. The hard gate is the three check tests, not the selection probes; we have clarified Algorithm 1 step (f) accordingly. We have rewritten the "Full-sequence Evaluation" paragraph in §4.1.2, added Table D0 in Appendix D.1 summarizing all probe sets, and reported when κ actually changes the selected candidate in Appendix D.4. The three stages are distinguished as follows:

  • Selection: Selection probes P_s (12 probes, four per dimension) are used only to compute U(g,p). Partial regression on P_s is allowed and penalized through κ, which is why N_reg can be non-zero.
  • Validation: Check tests (3 probes, one per dimension; templates distinct from both the selection probes and GlitchQuiz) are the hard gate: a candidate must pass all three or is discarded.
  • Calibration: Calibration probes P_c (4 probes) confirm only the selected p*.
  • κ therefore re-ranks candidates that have all passed the check tests, and can change the accepted candidate only when at least two candidates pass the check tests and differ in their regression count on P_s. On Qwen3.8-27B, 61.3% of targets have at least two such candidates; among them, moving from κ=1 to κ=2 changes the selected path for 7.4% and the final repair outcome for 1.2% of targets (FR 87.18% vs. 87.50%).

To W3 and Q3: Thank you for the suggestion. We have added the requested details for both deployment overhead and contextual regression in the revised PDF:

  • For deployment overhead, Appendix E now states the measurement conditions behind Table E1 (512-token prompts, 128 generated tokens, batch size 1, trigger density 10%, i.e., one request in ten contains a rule-matched glitch token) and breaks the overhead into host-side matching, prefill and decoding (new Table E4). On triggered requests only, the latency increase is 0.33% / 0.71% / 0.53% for Mistral-7B / Qwen3.8-27B / Llama-3.1-70B, of which prefill accounts for 0.18–0.47% and decoding (longer KV cache) for 0.14–0.23%; at 1% trigger density the overall increase is 0.011–0.023%.
  • For contextual regression, Table H1 now reports per-example counts in addition to the net change: on the 500 sentences per model, GlitchPatch changes the outcome of +1/−2 (Mistral-7B), +3/−2 (Llama-3.1-70B) and +2/−3 (Qwen3.8-27B) examples, the fewest regressions of the four methods on every model.

We hope these clarifications address the concerns and would be grateful if the reviewer would consider updating the assessment. We would be happy to answer any further questions during the discussion period. Thank you again!

AuthorsOpenReview ↗

Gentle follow-up on our response

Dear Reviewer JDYY,

Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.

We understand that this is a very busy period, and we apologize for the additional message. As the discussion period is ending soon, we would be very grateful if you could let us know whether our response has addressed your concerns.

If anything remains unclear, we would be more than happy to provide further clarification. If you find that your concerns have been resolved, we would deeply appreciate it if you would kindly consider updating your assessment.

Thank you again for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions for All Reviewers and ACs

Dear Area Chairs and Reviewers,

We thank all reviewers for their careful and constructive reviews and for recognizing the problem as "novel and practically valuable" (NtfN), the method's "practical locality" (JDYY), its simplicity and low cost (1MP3), and its ease of implementation (Zcm7).

We posted point-by-point responses to each reviewer and uploaded a revised PDF with all changes marked in blue. As the discussion period is drawing to a close, we summarize here the main changes and where they can be found so that the ACs and reviewers can verify them at a glance.

1. Summary of revisions

Detailed responses can be found in our replies to the individual reviewers. W/Q refer to the weakness and question numbers in each review.

1.1 New experiments

#What is addedRaised inIn PDF
E1Safety evaluation (unsafe-output rate before / after repair)NtfN W1, Q1§6.2, App. I
E2Downstream evaluation scaled to 2,000 tokens × 5 model families1MP3 W2; NtfN W2; Zcm7 W2Table 3, App. H.2
E3Fixed local-decomposition baselines (depth-1/2/3, char/byte)1MP3 Q1; NtfN W2, Q2Table 7
E4Byte-normalized and identical-continuation NLLJDYY W1, Q1Fig. 6(b), App. G
E5Sensitivity to the glitch-token definition1MP3 W1App. A.6

1.2 Analysis and clarifications

#What is added or changedRaised inIn PDF
A1Triggered-subset accuracy on general-capability benchmarksNtfN W3, Q3App. F.1
A2Latency decomposition and per-example regression countsJDYY W3, Q3App. E.4, H.1
A3Glitch tokens in current LLMs, incl. the four models namedZcm7 W1§1; reply to Zcm7
A4Selection / check-test / calibration probe sets distinguishedJDYY W2, Q2§4.1.2, Table D0

2. What is new in the revised PDF at a glance

  • Beyond the detection tasks: downstream accuracy +22.6 pp, unsafe-output rate 23.4% → 6.1%, language-modeling loss −8.4%; general-capability scores unchanged by construction.
  • Baselines: BPO outperforms all five fixed local-decomposition rules by >30 pp.
  • Rigor: probe sets, acceptance rule, latency conditions, and glitch-token definition made explicit.

None of the changes alters the paper's core findings; they extend the evidence to safety and realistic tasks and clarify its scope.

We hope the revised paper addresses all raised concerns and would be sincerely grateful if the reviewers could consider our responses and the new results and reassess the paper in light of the revisions. We remain available to answer any further questions during the remaining discussion period.

Thank you again for your time and for helping us improve the paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.