acceptodds
Under review as a conference paper at ICLR 2027

Echoing Is Not All You Need: Semantically Guided Repair of Glitch Tokens

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across tasks. However, they can fail to use certain tokens, known as glitch tokens, even in simple repetition tasks. Existing methods can restore token generation; however, this does not ensure semantic recovery. Our key insight is that models can retain semantic understanding of glitch tokens, yet naive repairs may corrupt this understanding. Therefore, we target functional repair, aiming to restore correct use of these tokens in context. To this end, we propose GlitchPatch, a semantically guided framework combining unified closed-form output correction with selective semantic learning. This training-free correction calibrates one output-head row per glitch token using a few contexts, leaving all other parameters unchanged. For tokens failing semantic assessment, we show that reliable repair requires prior semantic learning, achievable through simple reference-based training of a shared adapter. With repairs applied jointly across the vocabulary, GlitchPatch achieves functional repair rates of 86.5% on 16,788 Qwen-7B-Chat targets and 78.6% on 4,351 DeepSeek-LLM-7B-Chat targets, compared with 16.2% and 24.6%, respectively, for GlitchCleaner. Complete-repair evaluations additionally cover Llama-2-7b-chat-hf, Qwen3.5-9B, and Qwen3-32B. On Qwen-7B-Chat, accuracy changes remain negligible on MMLU, QQP, and GSM8K when numeric-answer extraction is used. Low repair cost and modest measured inference overhead support vocabulary-wide deployment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
glitch tokens, large language models, model editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_muiBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses glitch tokens, which are vocabulary items a model cannot reproduce even in a simple repetition task. It argues that earlier repairs such as GlitchCleaner do not restore the model's understanding of the token, since the model still cannot generate the glitched token. The authors propose GlitchPatch, which targets "functional repair": the token can be reproduced and the model can still say what it means.

The method has two parts. First, it trains a Glitch cleaner style lora adapter to fix the model's understanding of these tokens. Then, it trains one output-head row per glitch token so it passes 5 copy prompts. The adapter is switched on only when the input contains such a token. On Qwen-7B-Chat, GlitchPatch reaches 86.5% functional repair on 16,788 tokens, versus 16.2% for GlitchCleaner.

Critical strengths
  • The method works, and the gains are large and consistent. The ablation is informative: on semantic-deficient tokens, combined finetuning is much better than row-only or lora-only. General-task accuracy is essentially unchanged.
  • The paper is well written and easy to follow overall, and the figures and tables are clear.
Critical weaknesses
  • The semantic LoRA with a gate on glitch-token inputs follows the same general recipe as GlitchCleaner. The main new ingredient is the closed-form edit of one output-head row per token. This is a neat, cheap idea, and the ablation shows it matters on repetition prompts. But the paper does not demonstrate that this is a fix necessary and effective for LLMs in practical use cases (non repetition/copy prompts).
  • The straightforward baseline is to jointly tune LoRA adapters and the specific output rows, then apply the adapter everywhere. This is a simple functional repair that GlitchPatch should be compared to.
  • The paper is generally easy to read, but the definition of output success is harder to follow than it needs to be. A concrete example, such as the one in the appendix, would explain it faster, and the formal notation could move to the appendix. The same applies to Definitions 1 and 2, which restate simple ideas in notation and slow down the reader.
  • Output success is a narrow definition. Being able to reproduce a token under a direct copy prompt does not mean the model can use it correctly in context without such prompting.
Questions for authors

Could a better measure be used, for example using the token in a sentence, completing text containing it, or downstream tasks with natural text containing the token?

I am looking for evidence that GlitchPatch is necessary and effective for generating glitched tokens in practical settings (outside repetition prompts).

2 comments

Response to Reviewer muiB

Thank you for the suggestions. We added contextual tasks and executable examples, implemented the joint-training baseline, and revised the presentation of the definitions.

W1. Practical relevance of the repair

The semantic LoRA with a gate on glitch-token inputs follows the same general recipe as GlitchCleaner. The main new ingredient is the closed-form edit of one output-head row per token. This is a neat, cheap idea, and the ablation shows it matters on repetition prompts. But the paper does not demonstrate that this is a fix necessary and effective for LLMs in practical use cases (non repetition/copy prompts).

Response. We address practical relevance with executable tasks outside copying prompts, then clarify how GP differs from GlitchCleaner. In the Qwen7B API tasks, we compare base model with GP on identifiers whose correct use affects program behavior. For an assembly-trademark task, base model generates AssemblyProduct, whereas GP uses the required token "AssemblyTrademark"; compilation and reflection verify the correct attribute and value. For a container-layout task, GP uses "addContainerGap" with the required bounds and passes compilation and layout-size checks. These examples connect token repair to concrete tasks outside repetition.

GP directly calibrates output rows for targets that pass semantic diagnosis. For deficient targets, it first performs reference-supervised semantic learning and then calibrates the rows in the adapted model state. This semantic supervision differs from GC's repetition objective, where the prompts for training and testing are identical.

Related failures also occur in modern models. https://www.minimax.io/blog/sparse-token-forgetting describes the Chinese-name case, and https://fi-le.net/oss/ records failures to interpret particular Chinese tokens. These cases motivate jointly examining output accessibility and semantic behavior.

W2. Joint LoRA and output-row training

The straightforward baseline is to jointly tune LoRA adapters and the specific output rows, then apply the adapter everywhere. This is a simple functional repair that GlitchPatch should be compared to.

Response. We implemented the requested joint LoRA and output-row baseline with an always-on adapter, and compared target repair, training cost, and instruction following.

Following GP’s separation of semantic and output repair, we trained the joint baseline on semantic data followed by copy data. This baseline achieves 7.30% functional repair, compared with 86.20% for GP.

To explore its potential under overly favorable supervision, we also evaluated a mixed joint variant that combines copy and semantic supervision at every optimization step. Using the same training examples, this variant achieves 93.22% functional repair. Both joint variants optimize 69,971,968 parameters, approximately 58 times the 1,208,320 parameters in GP’s semantic adapter. The mixed variant requires 3.77 times GP’s recorded gradient-training time (6 h 2 min versus 1 h 36 min). Its IFEval strict prompt accuracy is 16.45%, compared with 32.72% for GP. Thus, GP achieves high functional repair with substantially fewer trainable parameters, less gradient-training time, and better retained instruction following.

The repaired-target results are:

MethodTime costIFEval strict prompt accuracyFunctional repair
Base modelN/A35.86%0
GP1 h 36 min32.72%86.20%
Joint (sequential)5 h 57 min16.45%7.30%
Joint (mixed)6h2min16.45%93.22%

W3. Examples before formal definitions

The paper is generally easy to read, but the definition of output success is harder to follow than it needs to be. A concrete example, such as the one in the appendix, would explain it faster, and the formal notation could move to the appendix. The same applies to Definitions 1 and 2, which restate simple ideas in notation and slow down the reader.

Response. We have revised the main text to place a concrete example before the definitions and moved nonessential formal notation to the appendix. The example now introduces the criteria before their detailed notation.

W4. Correct use beyond direct copying

Output success is a narrow definition. Being able to reproduce a token under a direct copy prompt does not mean the model can use it correctly in context without such prompting.

Response. We evaluate correct token use beyond direct copying with sentence-generation and cloze tasks. We identified cases in which base model cannot use the target correctly. GP recovers 45/50 of these base model failures (90%) without training on these contexts.

Q1. Sentence completion and natural-text evaluation

Could a better measure be used, for example using the token in a sentence, completing text containing it, or downstream tasks with natural text containing the token?

Response. We adopted the suggested sentence-generation and cloze tasks to evaluate correct token use in context. On 50 tasks that base model answers incorrectly, GP correctly completes 45 (90%). Evaluation checks whether the response meets the task requirements and uses the target appropriately. These results provide the proposed contextual measure alongside the original copying and semantic checks.

Q2. Evidence outside repetition prompts

I am looking for evidence that GlitchPatch is necessary and effective for generating glitched tokens in practical settings (outside repetition prompts).

Response. Two executable Qwen7B tasks show how GP recovers correct token use outside repetition prompts:

  1. AssemblyTrademark: the required API name is absent from the input. Base model substitutes AssemblyProduct; GP generates the correct trademark attribute and passes compilation and reflection checks.
  2. addContainerGap: the API name is supplied, but the model must satisfy new layout constraints. GP uses the correct operation with the required bounds and passes compilation and size checks, whereas base model uses addGap.

The MiniMax name case (it failed to produce a Chinese name 马嘉祺) and the gpt-oss token-interpretation failures provide further motivation. GlitchSplit recovers the name's surface form without producing a correct identity answer. This illustrates why repair should be evaluated for both output accessibility and semantic use. Our contextual tasks and executable examples directly examine these two aspects.

Reviewer_dM3AOpenReview ↗
Rating
3(−1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel method called GlitchPatch to address the problem of glitch tokens. It classifies glitch tokens as semantic-preserved or semantic-deficient using LLM-as-a-judge, and repairs them by modifying the corresponding output-head rows. For semantic-deficient tokens, the method combines semantic learning through LoRA with closed-form output-head correction. Experimental results show that the proposed method improves the functional repair rate over a previous method by a large margin. Further analysis shows that the output correction and semantic adaptation function complementarily.

Critical strengths
  1. The proposed method is clever and sophisticated, yet simple and computationally efficient. The motivation is clear and easy to understand.
  2. Strong performance gain. Table 1 clearly shows the effectiveness of the proposed method across five LLMs of different sizes. The ablation study supports the claims presented in this study.
Critical weaknesses
  1. For semantic-deficient tokens, training and evaluation use the same tokens and references, although the question wording differs. Consequently, it is not clear whether the semantic recovery reported in this paper reflects memorization of the reference answers or a more generalizable understanding of the tokens.
  2. In the classification of semantic preservation and semantic deficiency, LLM-as-a-Judge was used, but its agreement between the two trials was not reported. Reporting the agreement between the two judgments would help establish the reliability of the semantic assessment, which is central to the proposed routing strategy and evaluation.
Questions for authors
  1. The training and evaluation phases for semantic-deficient tokens use the same reference, but do the authors observe a similar semantic recovery even when the evaluation does not depend on the references?
  2. What is the agreement rate between the two LLM-as-a-Judge evaluations? How sensitive are the SP/SD classification and the resulting repair performance to the evaluation criteria?
3 comments

Response to Reviewer dM3A

Thank you for your positive assessment of the method. We added contextual-use and no-reference evaluations, recovered the original judgment pairs, and examined the sensitivity of GlichPatch (GP) 's comparison with GlichCleaner(GC).

W1. Contextual evidence beyond reference-answer fitting

For semantic-deficient tokens, training and evaluation use the same tokens and references, although the question wording differs. Consequently, it is not clear whether the semantic recovery reported in this paper reflects memorization of the reference answers or a more generalizable understanding of the tokens.

Response. We test whether semantic recovery extends beyond reference-answer fitting through contextual-use tasks and evaluation without references. We added sentence-generation and cloze tasks in which correct target use depends on a new sentence or situation. We identify contextual failures of the Qwen7B base model and evaluate whether GP recovers those failures. On 50 tasks that the base model fails, GP correctly solves 45 (90%). These contexts are not used for repair training.

Apart from the added tasks that provide behavioral evidence beyond answering a reworded reference question, we also kept the generated explanations fixed and evaluated them without supplying the canonical reference to the judge; the comparison is reported in Q1.

W2. Agreement between the two judgments

In the classification of semantic preservation and semantic deficiency, LLM-as-a-Judge was used, but its agreement between the two trials was not reported. Reporting the agreement between the two judgments would help establish the reliability of the semantic assessment, which is central to the proposed routing strategy and evaluation.

Response. We assess evaluator reliability through judgment agreement and the sensitivity of the repair comparison. We determined agreement for base model routing assessment and the GP semantic assessment, using all 16,788 Qwen7B targets. The two judgments agree in 91.56% and 91.97% of cases, respectively. Thus, both stages show agreement above 90%:

Assessment, N=16,788Pass/passPass/failFail/passFail/failAgreementCohen's κ
Base routing5,17675466310,19591.56%0.815
GP post-repair14,66671463477491.97%0.491

To assess the effect on functional repair, we also examined score variation across judgment rules. The estimated maximum change is approximately ±4 percentage points. In practice, we examined that directly routing all controversial tokens to SP or SD yields a change of 4.52 percentage points on Qwen7B model.

Q1. Semantic recovery without references

The training and evaluation phases for semantic-deficient tokens use the same reference, but do the authors observe a similar semantic recovery even when the evaluation does not depend on the references?

Response. We do observe a similar semantic recovery on evaluation that doesn't depend on the references. Concretely, We constructed a context question set that gives no direct reference, and a new semantic recovery test where the semantic judge receives only the exact token and explanation.

We added sentence-generation and cloze tasks to test whether GlitchPatch (GP) uses target tokens correctly in a given sentence or situation. Because some glitch tokens do not support a clear contextual task, we selected 50 targets and constructed a benchmark that the Qwen7B base model answers incorrectly. These tasks require the target to fit a particular sentence or situation, and their contexts are not used in repair training. GP recovers 45/50 base model failures (90%), suggesting that our method is robust in the extended contextual evaluation.

We assessed semantic recovery with and without references while keeping the generated responses fixed. Without the reference, the judge received only the exact token and the generated explanation. On the 200-target routing-disputed subset, the SD repair increases semantic passes from 9 for base model to 185. The table compares the results with and without references.

Repair pathW/o referenceW/ reference
Base model98
SD repair path185172

Q2. Sensitivity of classification and repair performance

What is the agreement rate between the two LLM-as-a-Judge evaluations? How sensitive are the SP/SD classification and the resulting repair performance to the evaluation criteria?

Response. We examine both changes in the SP/SD assignments and changes in the functional-repair comparison on Qwen7B model. OR requires at least one judgment to pass; AND requires both judgments to pass. The OR rule assigns 6,593 targets to SP and 10,195 to SD. The AND rule changes the split to 5,176 SP and 11,612 SD. Thus, the change is confined to 1,417 targets (8.44%); the remaining assignments agree.

GP's functional-repair advantage over GC is preserved under each of the four scoring summaries applied to the original fixed responses:

Scoring ruleGPGC
First judgment82.89%14.62%
Second judgment82.49%14.49%
OR86.48%16.23%
AND78.90%12.89%

Plus, we directly compare assigning all 1,417 disagreement tokens to SP versus SD, while keeping the remaining assignments unchanged. Functional repair across all 16,788 targets reaches 86.22% and 81.70%, respectively, a modest difference of 4.53 percentage points. These results suggest that variations in SP/SD assignments have only a slight impact on GP’s overall functional-repair performance.

Thank you for the response, and for reporting the agreement rate. However, my concern about memorization is only partially resolved. The evaluation without references uses the same generated explanations, and the contextual tasks include only cases where the base model fails. In addition, after reading the other reviews, I find the regression on ordinary tokens and the limited evidence outside copy prompts problematic. For these reasons, I am lowering my rating from 4 to 3, although I still think the core idea is valuable.

Reviewer_QnB1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

GlitchPatch distinguishes output accessibility from semantic accessibility when repairing glitch tokens. It diagnoses tokens using reference-based semantic assessments, trains a shared semantic adapter for deficient tokens, and applies a closed-form correction to output-head rows. The repaired rows are installed jointly, with semantic adaptation enabled for selected requests. Experiments evaluate output occurrence, reference-based semantic performance, and their combined functional-repair score, with component comparisons and additional controls for competition and collateral effects.

Critical strengths
  1. Separating reproducibility of a token from its semantic behavior is a useful formulation. The output-row correction is clearly specified, and its closed-form solution follows from the stated regularized objective. The component experiments support the complementarity of semantic adaptation and output correction.

  2. The main experiments install the repaired rows together, and the appendix examines competition and collateral failures. This goes beyond isolated single-token calibration and provides useful evidence about practical deployment constraints, even though the remaining reliability concerns are substantial.

Critical weaknesses
  1. The functional-repair score combines occurrence of the target string in one response with a reference-based semantic answer to another question. For SD tokens, training and semantic evaluation share token identities and canonical references with different question wording (Appendix A.4). This is legitimate fitting of known repair targets, but it leaves contextual use largely untested. In the Qwen3-32B assessment, 468 of 3,309 semantic judgments change for identical base-model answers under different batching (Appendix I), which also weakens confidence in the routing labels. These are changed evaluator judgments, not demonstrated changes in the base model's understanding. The semantic evidence is insufficient for the central claim of reliable functional repair.

  2. Output reliability and preservation of ordinary inputs remain unresolved. The occurrence criterion accepts repetition and extra content despite the copying instruction. More directly, Appendix I reports that full repair converts 11 previously successful copies into failures among 64 ordinary-word responses, with the semantic adapter inactive. This small control cannot estimate a general regression rate, but it shows a material side effect that nearly unchanged MMLU accuracy does not rule out. Strict copying with the full row bank and ordinary-input preservation need more weight in the main evaluation.

Questions for authors
  1. Focused validation of the main claim: On a small independently checked contextual subset, how often does repair recover correct token use beyond answering paraphrases of the training reference question?

  2. Focused validation of output reliability: What are the strict single-copy rates with the complete repaired row bank, and do the ordinary-word regressions persist on a larger control set? Please distinguish these results from isolated row-calibration diagnostics.

6 comments

Response to Reviewer QnB1

W1. Contextual use beyond reference questions

The functional-repair score combines occurrence of the target string in one response with a reference-based semantic answer to another question. For SD tokens, training and semantic evaluation share token identities and canonical references with different question wording (Appendix A.4). This is legitimate fitting of known repair targets, but it leaves contextual use largely untested. In the Qwen3-32B assessment, 468 of 3,309 semantic judgments change for identical base-model answers under different batching (Appendix I), which also weakens confidence in the routing labels. These are changed evaluator judgments, not demonstrated changes in the base model's understanding. The semantic evidence is insufficient for the central claim of reliable functional repair.

Response. We address contextual token use and evaluator disagreement separately. We added sentence-generation and cloze tasks to test whether GlitchPatch (GP) uses target tokens correctly in a given sentence or situation. Because some glitch tokens do not support a clear contextual task, we selected 50 targets and constructed a benchmark that the Qwen7B base model answers incorrectly. These tasks require the target to fit a particular sentence or situation, and their contexts are not used in repair training. GP recovers 45/50 base model failures (90%), suggesting that our method is robust in the extended contextual evaluation.

For the fluctuation concern on the 32B assessment, inspection of the 468 changed cases identified both comparatively clear judgment inconsistencies and cases involving interpretation boundaries, context-dependent fragments, or meanings requiring further verification.

CategoryCountProportion
Clear-cut cases13528.85%
Ambiguous cases33371.15%

Besides, on Qwen7B, we directly compare assigning all 1,417 disagreement tokens to SP versus SD, while keeping the remaining assignments unchanged. Functional repair across all 16,788 targets reaches 86.22% and 81.70%, respectively, a modest difference of 4.53 percentage points. For the main comparisons on Qwen7B, GP remains ahead of GC when functional repair is computed from the first judgment, second judgment, OR (at least one judgment passes), or AND (both judgments pass). So we are safe to arrive at the conclusion that our central claim of functional repair remains reliable.

W2. Repetition and ordinary-input preservation

Output reliability and preservation of ordinary inputs remain unresolved. The occurrence criterion accepts repetition and extra content despite the copying instruction. More directly, Appendix I reports that full repair converts 11 previously successful copies into failures among 64 ordinary-word responses, with the semantic adapter inactive. This small control cannot estimate a general regression rate, but it shows a material side effect that nearly unchanged MMLU accuracy does not rule out. Strict copying with the full row bank and ordinary-input preservation need more weight in the main evaluation.

Response. We examined repetition and ordinary-input preservation separately. For repetition, the unmodified Qwen7B model achieves strict single-copy success rates of 18.89% and 15.66% on Prompts 0 and 1, respectively, on tokens outside our glitch-candidate list. Thus, exact single-copy behavior is unreliable even on this ordinary-token control. We evaluate strict copying on the repair targets separately in Q2.

For ordinary-input preservation, we examined how output-row repair affects ordinary generation. The Appendix I control consists of 16 fixed ordinary words, which are deliberately selected as stress test candidate, so their error rate are higher than generalized experiments. In a separate Qwen32B vocabulary-wide control, the full target string no longer appeared after repair for 2,148 of the 85,943 unrepaired tokens that the base model had successfully copied (2.50%).

Q1. Recovery on contextual base model failures

Focused validation of the main claim: On a small independently checked contextual subset, how often does repair recover correct token use beyond answering paraphrases of the training reference question?

Response. On 50 contextual tasks that base model answers incorrectly, GP correctly answers 45 (90%). The tasks include sentence generation and cloze completion, with task correctness and appropriate target use evaluated in the same response. These results directly test whether repair enables answers that base model could not produce in new contexts.

Q2. Single-copy output and larger ordinary-word controls

Focused validation of output reliability: What are the strict single-copy rates with the complete repaired row bank, and do the ordinary-word regressions persist on a larger control set? Please distinguish these results from isolated row-calibration diagnostics.

Response. We evaluated complete-bank copying and ordinary-word regressions separately. For strict single-copying, on the same 16,788 Qwen7B repair targets, GP achieves strict single-copy success rates of 52.36% and 72.12% on Prompts 0 and 1, compared with 18.38% and 3.10% for GC. These results show that GP improves strict single-copy performance over GC and base model.

For ordinary-word preservation, we measured whether previously successful outputs remain correct after repair on Qwen32B model. In the vocabulary-wide control, 2,148 of 85,943 previously successful unrepaired tokens regressed (2.50%) under the target-occurrence criterion.

As a follow-up to your concern about ordinary-input preservation in W2 and Q2, we have now completed an additional evaluation covering the full tokenizer vocabulary. The Qwen32B base model successfully copies 85,943 targets in the main unrepaired-token cohort under our occurrence criterion; 2,148 of these become failures after repair, yielding a regression rate of 2.50%. We apologize for the delayed update, as this full-vocabulary evaluation required substantial computation.

Comments on the paper

2 comments

Dear reviewers,

Please check your assigned papers for author responses. If the authors have addressed any of your questions or concerns, please leave a brief comment for them indicating which points have been resolved and which, if any, remain open. Even if your assessment is unchanged, a short follow-up is helpful.

Best, AC

Dear Reviewers,

The Author-Reviewer Discussion period is coming to an end soon. Please actively participate in the rebuttal discussion with the authors after carefully reading the authors' responses.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.