- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors researched a patching method of glitch tokens in LLMs. The existing methods require access to the model internal states, a requirement that is hard to meet in certain deployment scenarios. They find that simply deleting glitch-token merge rules helps repair but introduces new errors, and fix rates vary by path while collateral damage grows with depth. Based on these findings, the propose GLITCHPATCH. They used BPO to find an optimal replacement token sequence for each glitch token offline, and replace matched glitch-token IDs at inference time using replacement sequences precomputed offline.
Critical strengths
- The paper addresses a novel and practically valuable problem. It investigates how glitch tokens can be repaired in restricted-access settings where model parameters and internal states cannot be modified. By intervening at the tokenization level, the proposed approach provides a lightweight patch that can be deployed without retraining or altering the model.
- The local replacement mechanism effectively avoids disturbing normal tokens. GLITCHPATCH replaces only token IDs that match known glitch tokens in the canonical token sequence, while preserving all other token IDs. As a result, inputs that do not trigger a rule remain exactly unchanged, substantially reducing the risk of collateral damage.
- The method achieves substantial improvements under the paper’s primary repair evaluation. It consistently outperforms the strongest baseline in fix rate while markedly reducing the overall glitch rate. These results show that local retokenization can be highly effective for the targeted glitch-token failures.
- The repair results are validated across a diverse range of models and tokenizers. The full-vocabulary evaluation spans multiple model and tokenizer families, suggesting that the observed repair effect is not confined to a particular model configuration.
Critical weaknesses
-
The tests used to identify glitch tokens do not directly imply safety risks. The authors define glitch tokens solely based on failures in the repetition, spelling, and length tasks. However, failing these tasks does not necessarily mean that a token can be exploited to bypass safety mechanisms or induce unsafe outputs. The paper also does not evaluate the unsafe rate or attack success rate before and after repair, and therefore cannot demonstrate that GLITCHPATCH reduces the safety risks identified in prior work.
-
The optimization objective for replacement paths is aligned with the primary evaluation objective, making it difficult to establish a general capability improvement. BPO’s path selection, utility evaluation, and final FR evaluation are all based on the repetition, spelling, and length tasks. Although the prompt templates differ, they still belong to the same task family. The results mainly demonstrate that the selected paths can pass these optimized tests, but do not yet establish that they truly restore the token’s semantic representation or generalize to more general tasks. The limited OOD experiments in Appendix H provide some supplementary evidence, but their coverage in terms of models, sample size, and evaluation methodology remains limited. In addition, the paper does not compare against a simple baseline that locally decomposes only glitch tokens into character/byte tokens. Prior work shows that even full-text character-level tokenization has a limited impact on the performance of several large models (Understanding the Ability of LLMs to Handle Character-Level Perturbation), and since glitch tokens typically account for only a small fraction of the text, the length cost of applying this decomposition locally is also relatively limited. Therefore, the additional benefit of BPO remains unclear.
-
The general-capability evaluation does not effectively isolate the samples on which the patch actually takes effect. On MMLU-Pro, ARC, GSM8K, and HumanEval, the paper reports only overall score changes, without specifying how many sequences in each dataset contain a glitch token or how many actually trigger a repair rule. Sequences that do not contain a glitch token are, by construction, unchanged by the patch. Therefore, the small changes in the overall scores are largely expected and cannot effectively evaluate the patch’s impact on the inputs that are actually modified.
Questions for authors
- Could the authors evaluate whether GLITCHPATCH reduces the incidence of unsafe behavior, or other practically meaningful failures such as the misunderstanding of user input illustrated in Figure 1?
- Could the authors compare BPO against a simple local baseline that directly decomposes each glitch token into character/byte-level tokens while leaving the rest of the canonical tokenization unchanged, and quantify how much improvement BPO provides over this baseline?
- Could the authors clarify what proportion of the instances in the general-capability benchmarks contain a glitch token and actually trigger an accepted repair rule, or directly report the performance on those triggered instances?
4 comments
Gentle follow-up on our response
Dear Reviewer NtfN,
Thank you so much for the time and care you have devoted to reviewing our paper. Your constructive comments have been very helpful in improving our work.
We understand that this is a very busy period, and we apologize for the additional message. As the discussion period is ending soon, we would be very grateful if you could let us know whether our response has addressed your concerns.
If anything remains unclear, we would be more than happy to provide further clarification. If you find that your concerns have been resolved, we would deeply appreciate it if you would kindly consider updating your assessment.
Thank you again for your time and consideration.
Best regards,
The Authors
I appreciate the authors' response and supplementary experiments, which have clarified my concerns. And I will raise my score from 2 to 3. Best of luck with the submission!
Thank You to Reviewer NtfN
Dear Reviewer NtfN,
Thank you so much for reading our response and for raising your score. We are glad that your concerns have been clarified, and we sincerely appreciate your constructive suggestions, which have made the paper stronger. Thank you again for your thoughtful review and kind wishes! :)
Best regards,
The Authors
A Big Thank You and Our Response to Reviewer NtfN
Thank you so much for the careful and insightful review! The comments have helped us make the paper noticeably stronger, and we have addressed each of them. All changes are marked in blue in the revised PDF. Our point-by-point responses are as follows.
To W1 and Q1: Thank you for the suggestion. We have added a safety evaluation in §6.2 (new paragraph) and Appendix I (new Table I1):
Table I1. Unsafe-output rate (%) on 1,000 glitch-containing prompts per model.
[1] Tang et al., USENIX Security 2026. https://dl.acm.org/doi/10.5555/3857392.3857500
To W2 and Q2: Thank you for the suggestions. We have scaled up the OOD evaluation of Appendix H.2 (now also §6.2, Table 3) and added the simple local baseline in §6.5 (new Table 7), as follows:
Table 3 (new, §6.2). Downstream accuracy (%), mean over five models (per-model results in Table H2).
Table 7. Fixed local decompositions vs. BPO, FR (%); all under budget B = 8 (†without).
To W3 and Q3: Thank you for the suggestion. We have added the triggered-subset analysis in Appendix F.1 (new Table F2):
Table F2. Glitch-containing instances and triggered-subset accuracy, aggregated over the ten models.
We hope these clarifications address the concerns and would be grateful if the reviewer would consider updating the assessment. We would be happy to answer any further questions during the discussion period. Thank you again!