acceptodds
Under review as a conference paper at ICLR 2027

A DEAFENING SILENCE: CATASTROPHIC FORGETTING LIVES IN THE OUTPUT EMBEDDINGS OF TOKENS THE DATA NEVER SPEAKS

Abstract

Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using often-inaccessible original data. In this data-free regime, we analyze where forgetting occurs and why. Systematic parameter freezing across five settings up to 1.4B reveals that forgetting concentrates selectively in the output embeddings of tokens rarely seen in the new corpus, whereas the same sqrt(v-hat) band of the body is inert and new learning resides elsewhere. This localization is governed by the vocabulary deficiency of the corpus rather than the training mode, allowing pre-retraining risk ranking from token counts alone within a fixed base model. Mechanistically, absent tokens receive persistent one-sided softmax gradients that Adam’s second-moment (sqrt(v-hat)) normalization amplifies into full-sized updates. We therefore propose an intervention: raising Adam’s epsilon exclusively for the output projection during training. Across eight settings spanning 160M to 12B parameters and four model families, this removes 39.4% to 67.9% of forgetting across all seven stable configurations without degrading target learning or requiring per-model tuning. The defense combines additively or better with replay (79.8% on Qwen/Korean) and rescues released-head LoRA from a 23-fold forgetting surge. Because post-hoc editing of the drifted rows recovers under 5% of forgetting, the intervention must operate during training. Our findings indicate that a single-line optimizer adjustment may serve as the primary defense against catastrophic forgetting where the corpus starves the vocabulary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Catastrophic forgetting, Continual pretraining, Continual learning, Large language models, Fine-tuning, Adam optimizer, Output projection, Replay-free

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_z9AvOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies data-free continual pre-training and fine-tuning (Pythia 160M-12B, Qwen2.5-0.5B, TinyLlama-1.1B, OLMo-2-1B; Korean, math, code, Alpaca, KoAlpaca; 40M or 12.6M tokens; three seeds for references and the Table 5 intervention except two seeds for the Pythia-12B and OLMo baselines (Table 5 caption, line 378); two seeds per freezing arm except Qwen, Table A3 lines 648-653) and asks where forgetting, defined as the mean increase in held-out cross-entropy on old domains (Eq. 3), is located. Freezing one (site, v^\sqrt{\hat{v}} band) parameter group at a time (Section 4.1) shows that freezing the lowest-v^\sqrt{\hat{v}} band of the output embeddings removes 31.3-51.0% of forgetting at five settings with no learning cost, while the same band of the body holds a negligible share (-1.3% to +2.7%) and mass-matched body freezing 0.4-12.3% (Table 2); learning lives in the upper body bands. The site is governed by vocabulary deficiency, the share of old-domain evaluation tokens under-exposed (c(t)<1,000c(t) < 1{,}000) in the new corpus (Table 3), and the mechanism offered is one-sided softmax gradients on absent tokens (Eq. 1) amplified by Adam's normalisation (Eq. 2). The intervention raises AdamW's ϵ\epsilon to 1×10−41 \times 10^{-4} on the output projection only; Table 5 reports 39.4-67.9% forgetting reduction at seven of eight settings with learning change +0.4% to +10.9%, additive composition with replay (Table 6), a 91% rescue of released-head LoRA (Section 5.3), and that post-hoc row edits recover under 5% (Table 7). Appendix A.2 (Table A10) shows a 10×10\times lower embedding learning rate traces the same frontier (67.5% vs 66.7%). The localisation grid is a genuine and well-executed finding and the intervention is compared at the same learning rate within each setting (Table 5; paired across the rate grid in Table A11); the claims are weakened because forgetting is measured only as loss, the diagnostic needs an old-domain corpus the data-free setting does not grant, the remedy is empirically a per-tensor learning-rate reduction already known from the rare-token literature, and one setting is bimodal across seeds while one is excluded from the summary.

Critical strengths
  • Originality: The causal-share grid is a real contribution. Partitioning three sites by three v^\sqrt{\hat{v}} bands, freezing one group at a time with fixed baseline masks and mass-matched body controls (Section 4.1, lines 206-215), gives a necessity test rather than a correlational drift metric (two seeds per freezing arm except Qwen, Table A3), and the result is clean: the bottom output band holds 31.3-51.0% of forgetting at five settings with +0.1% to +5.2% learning change, the same band of the input embeddings -1.5% to +1.4%, and mass-matched body freezing 0.4-12.3% (Table 2, lines 235-247). The split of forgetting (bottom output band) from learning (upper body bands, 8.8-40.7% of learning, line 249) is the kind of localisation the field has been missing, and the vocabulary-deficiency rank association (Table 3) suggests it can be anticipated before training, with the caveat that each Table 3 row changes both the selected learning rate and the evaluation pool (lines 216-224, 264-269; Appendix A.1 lines 628-634), so the ordering is a within-grid association and does not isolate deficiency as the cause.
  • Significance: If the intervention's effect survives the controls requested below, a parameter-group ϵ\epsilon change is the cheapest forgetting defence proposed for data-free adaptation: zero compute or memory overhead, no token lists, no replay data (lines 328-329), composing additively with replay (79.8% on Qwen/Korean, Table 6) and rescuing released-head LoRA from a 23-fold forgetting surge (lines 419-421).
Critical weaknesses
  • Soundness: Forgetting is measured only as held-out loss, the diagnostic needs data the setting excludes, and stability is uneven. Every reduction in the eight-setting main grid is a change in cross-entropy in nats (Eqs. 3-4). The paper does report two capability-based results (Appendix A.5 lines 786-797, Table A12): 83.6% reduction at Pythia-6.9B against 73.9% loss-based on the same pair, and 49.0% vs 49.1% at Pythia-1.4B, but at ϵ=1×10−5\epsilon = 1 \times 10^{-5} on math with 1% replay, which the authors say "predates the grid" and is "not data-free Korean" (lines 470-472). Those two cells are encouraging and reduce the weight of this concern, but they do not cover the data-free Korean grid on which every headline number rests. At 6.9B and 12B the baseline forgetting is about 0.1 nats (Table 5), so a 60-65% reduction is 0.06 nats whose effect on any task is unmeasured; the title's "catastrophic forgetting" is a loss-drift claim. The pre-retraining diagnostic, vocabulary deficiency, is defined on old-domain evaluation positions (lines 119-123) sampled from DoldD_{old} for Pythia and OLMo or from proxies for closed models (lines 181-186), and the deployment rule (lines 343-348) again needs the share of old-domain positions; under the paper's own zero-access-to-DoldD_{old} assumption (lines 112-115) this statistic cannot be computed from the new corpus alone, so "risk ranking from token counts alone" (abstract) requires a representative old-domain sample the data-free framing does not grant. Finally the Pythia-410M arm is bimodal: across six seeds reduction spans 4.6-46.6% and learning change -32.9% to +16.8% (lines 283-286; Table 5 footnote 32.7% and -7.0% at six seeds vs 43.8% and +10.9% at three), and Pythia-160M collapses at its peak rate and is excluded from the summary range while remaining in the "eight settings" count (lines 253-260, 356-357).
  • Significance: The remedy is, on the paper's own evidence, interchangeable with a per-tensor learning-rate reduction, and its mechanism (arresting under-exposed rows) is the one rare-token gating and freezing already use, so the contribution is the diagnosis more than the instrument. Appendix A.2 states that freezing rows with c(t)<1,000c(t) < 1{,}000, per-step gating and output-scoped ϵ\epsilon "agree closely" (lines 334-338), and Table A10 shows scaling the embedding learning rate by 0.10 gives 67.5% reduction at +0.7% learning versus 66.7% at +2.2% for ϵ=5×10−5\epsilon = 5 \times 10^{-5}. Table A10 is a single Qwen/Korean campaign, so the equivalence is an empirical trade-off match in one setting, not a theorem. Section 6 acknowledges that Yu et al. 2022's rare-token gating "matches output ϵ\epsilon" (line 461), and Kwok et al. 2024 freeze rare-token rows for continual ASR; neither proposed the specific optimizer setting, but both act on the same rows by the same logic (lines 334-338 list freezing and gating as equivalents). The paper's framing ("a single-line optimizer adjustment may serve as the primary defense," abstract; "identify Adam as the amplifier and ϵ\epsilon as the brake," lines 105-106) presents the instrument as the discovery. The same-rate comparisons also show the intervention improving target learning by +5.4% to +10.9% (Table 5) and +10.0% / +18.4% (Table 3, Qwen math/code), which a pure brake on absent-token rows should not do; the paper does not analyse why, and without that analysis the "no learning cost" claim is an observation rather than an understood property. What survives is the localisation (strength above), which is valuable but is not what the abstract and title lead with.
Questions for authors

Questions that could change my assessment

  1. Please add a downstream evaluation (e.g. the Pythia evaluation suite or a small set of old-domain tasks such as LAMBADA, HellaSwag and a code or math task) for the reference and intervention arms at the eight settings of Table 5, with three seeds. Score-change criterion: if the intervention preserves old-task accuracy relative to the reference by a margin consistent with the loss reductions, Soundness rises to 3 and I would raise the Rating to 4; if the loss reductions do not translate into accuracy, the claims must be restated as loss drift and the Rating falls to 2.
  2. Please compare against a reference tuned over a two-rate grid (body rate ×\times output-embedding rate), i.e. the best per-tensor learning-rate baseline, and against whole-output-embedding freezing and MoFO in the Section 5.3 frontier, at the same body rate as the intervention. Score-change criterion: if output-scoped ϵ\epsilon still gives a non-trivial reduction at zero learning cost against the best two-rate reference, Informativeness stays at 3 and the Rating stays at 3; if the two-rate reference matches it, the contribution must be reframed as the localisation plus a per-tensor rate rule, and the Rating falls to 2 unless question 1 succeeds.
  3. Report Table 5 with six seeds for every arm, include Pythia-160M at its tuned rate in the summary range, give the per-seed distribution for 410M, and explain the +5% to +18% target-learning improvements at the same rate. Score-change criterion: a range that includes all eight settings at six seeds with the bimodal arm resolved and the learning gain explained would raise Soundness to 3.

Clarifications and minor points (do not affect my assessment)

  1. State in the abstract that the localisation and the remedy apply to high vocabulary-deficiency injections; under in-distribution tuning forgetting is in the MLPs (Table 4) and body-scoped ϵ\epsilon is free at only one of two cells (Section 5.5). The title claims the general case.
  2. State early that vocabulary deficiency requires an old-domain evaluation sample or proxy, and what to do when none exists.
  3. Table 3's caption says rows use different evaluation pools and are "comparable with neither," but the text reads cross-row trends from the same table; either match EE across rows or drop the cross-row prose.
  4. Table A11 (same-rate pairing and rate invariance) and Table A10 (equivalence to learning-rate scaling) belong in the main text; they answer the first question any reader asks about the protocol.
  5. State the seed counts of the freezing grid (two per arm except Qwen, Table A3) in Section 4.1 rather than only in the appendix caption.
  6. Release the corpora splits, rate grids per arm and seed lists; the intervention is one line, but the grid is not reproducible from the text.
5 comments
AuthorsOpenReview ↗

Response 1

(Edited 2026-11-15 22:02)

Thank you for the time and care you put into this review. We also thank you for seeing the causal-share grid as a real contribution. Your three questions each came with a clear criterion, and we followed them. Responses 1 to 3 answer them in order.

Question 1 and Weakness (Soundness, first part)

  • Question 1: Please add a downstream evaluation (e.g. the Pythia evaluation suite or a small set of old-domain tasks such as LAMBADA, HellaSwag and a code or math task) for the reference and intervention arms at the eight settings of Table 5, with three seeds. Score-change criterion: if the intervention preserves old-task accuracy relative to the reference by a margin consistent with the loss reductions, Soundness rises to 3 and I would raise the Rating to 4; if the loss reductions do not translate into accuracy, the claims must be restated as loss drift and the Rating falls to 2.
  • Weakness (Soundness, first part): At 6.9B and 12B the baseline forgetting is about 0.1 nats (Table 5), so a 60-65% reduction is 0.06 nats whose effect on any task is unmeasured; the title's "catastrophic forgetting" is a loss-drift claim.

Thank you for this request. We ran the downstream evaluation at all eight settings, with six paired seeds. The ε run keeps more old-task accuracy than the reference at every setting, and the gain is largest where the reference loses most.

What we changed. We now evaluate every reference and ε run of Table 6 zero-shot, in FP32, with the LM Evaluation Harness (v0.4.12). The evaluation runs inside the training process. We used six paired seeds instead of the three requested. The old-domain tasks are LAMBADA, HellaSwag, PIQA, ARC-Easy and WinoGrande. KoBEST measures the new domain. GSM8K and HumanEval are also reported (Sections 3.3 and 5.2, Table 7, Tables A24 and A25).

Evidence. On the five-task English mean, the ε run scores above the reference at all eight settings, by 0.7 to 4.3 points (Table 7 in the general response). At 6.9B, the base model scores 65.9, the reference 64.2 and the ε run 65.5. At 12B, the scores are 68.5, 66.0 and 67.5. The gain is largest on LAMBADA (Tables A24 and A25).

Tables A24 and A25, LAMBADA rows. LAMBADA accuracy (%). Zero-shot, FP32, mean over six paired seeds. Pythia-160M is at lr 3e-5, because training at its peak lr 3e-4 collapses (Table A16).

Settingbaseref.εε − ref.
Pythia-160M35.410.717.4+6.7
Pythia-410M51.627.032.2+5.2
Qwen2.5-0.5B52.527.637.6+10.0
TinyLlama-1.1B49.041.444.2+2.8
Pythia-1.4B61.738.046.9+8.9
OLMo-2-1B65.130.041.9+11.9
Pythia-6.9B67.160.566.7+6.2
Pythia-12B70.763.770.2+6.5

LAMBADA is the task where the reference loses most: 6.6 to 35.1 points against the base model. The ε run gets back 2.8 to 11.9 points of this at every setting. At 6.9B and 12B, it ends within 0.5 points of the base model.

Outside LAMBADA, the ε run also gains on HellaSwag, PIQA and ARC-Easy at Qwen2.5-0.5B and OLMo-2-1B, whose references lose the most. The largest of these gains is 5.9 points (Qwen, ARC-Easy). In the remaining cells, the two runs differ by at most 1.6 points. Across the forty cells (eight settings × five tasks), the more accuracy the reference loses on a task, the more the ε run gets back: the correlation between the two is 0.87 (Section 5.2). The KoBEST mean stays within 0.5 points of the reference at every setting, so the gain on English does not reduce accuracy on Korean. GSM8K and HumanEval are near zero for most base models at these scales (Tables A24 and A25), so they cannot show a difference. The exception is HumanEval at Qwen2.5-0.5B: 28.7 (base), 10.3 (reference) and 14.9 (ε run).

On your criterion: where the reference loses at least 3 points, the ε run gets back 35% of the drop on average, while it removes 39.0–67.8% of the loss increase (Table 6). The accuracy gain is smaller than the loss reduction. But on the five-task English mean it is positive at all eight settings, and across the forty cells it grows with the size of the drop. At 6.9B and 12B, the 0.06-nat difference is visible in accuracy: the ε run gets back 1.3 of the 1.7 points the reference loses on the English mean at 6.9B, and 1.5 of 2.5 at 12B. At these two models, forgetting also grows with longer training, and the reduction grows with it (Table A6, Section 5.2).

Table A6, selected columns. Runs at lr 3e-5 scheduled to 160M tokens, two paired seeds per model, evaluated at four checkpoints. The 40M row is a checkpoint of these runs, not the 40M-scheduled runs of Table 6.

Tokens6.9B ref. forgetting (nats)6.9B reduction (%)12B ref. forgetting (nats)12B reduction (%)
40M0.23553.10.22554.5
80M0.30258.70.29059.5
120M0.30961.50.29563.6
160M0.30662.00.28963.4
AuthorsOpenReview ↗

Response 2

(Edited 2026-11-15 22:02: table and section numbers updated to match the revised PDF.)

Question 2 and Weakness (Significance)

  • Question 2: Please compare against a reference tuned over a two-rate grid (body rate × output-embedding rate), i.e. the best per-tensor learning-rate baseline, and against whole-output-embedding freezing and MoFO in the Section 5.3 frontier, at the same body rate as the intervention. Score-change criterion: if output-scoped ϵ still gives a non-trivial reduction at zero learning cost against the best two-rate reference, Informativeness stays at 3 and the Rating stays at 3; if the two-rate reference matches it, the contribution must be reframed as the localisation plus a per-tensor rate rule, and the Rating falls to 2 unless question 1 succeeds.
  • Weakness (Significance): The remedy is, on the paper's own evidence, interchangeable with a per-tensor learning-rate reduction, and its mechanism (arresting under-exposed rows) is the one rare-token gating and freezing already use, so the contribution is the diagnosis more than the instrument.

Thank you for this request. We ran the two-rate grid. The best output-embedding learning rate matches ε on forgetting, so we accept the reframing you suggest. The reason is in the update rule. ε multiplies each step by √v̂ / (√v̂ + ε). Within the output embeddings, this lowers the step of rows with small √v̂, such as the rows of rare tokens, by orders of magnitude, while rows with √v̂ above 1e-4 keep at least half of it. ε therefore acts as a lower learning rate that each parameter sets by itself from its √v̂ (Section 5.1).

What we changed. We ran the grid at Qwen2.5-0.5B and Pythia-1.4B on Korean, each at two body learning rates (lr 1e-4 and 3e-5). The output-embedding learning rate was scaled by 1, 0.3, 0.1, 0.03 and 0 (frozen output embeddings). We also ran MoFO. Each condition has three seeds (Section 5.1, Table A22). Section 5.1 now says that output-scoped ε gives the result of the best output-embedding learning rate, with no second learning rate to tune. The abstract and the Introduction now say the same. Related Work now states that freezing the output layer (Gu & Feng, 2020) and freezing rare-token rows act on the same rows. What the paper adds is which rows carry the forgetting, and why.

Evidence. Each lower output-embedding learning rate removes more forgetting. At ×0.03 it removes 44.9–71.8%, against 45.3–69.5% for ε (Table A22). Freezing the output embeddings removes nearly the same forgetting as ε, within 2.5 points. At Pythia, whose output embeddings are not tied to its input embeddings, the learning change is also nearly the same. Qwen's output embeddings are tied to its input embeddings. At Qwen, freezing loses 4.3% of target learning at body lr 3e-5, while ε loses none. MoFO removes 51.5–58.6% but loses 3.6–13.6% of target learning.

Table A22. Korean, three seeds per condition. Each cell: share of reference forgetting removed (%), with the learning change (%) in parentheses. Output-embedding lr ×k: output-embedding learning rate scaled by k relative to the body learning rate. ×0 freezes the whole output embedding.

Qwen2.5-0.5B, body lr 1e-4Qwen2.5-0.5B, body lr 3e-5Pythia-1.4B, body lr 1e-4Pythia-1.4B, body lr 3e-5
Ref. forget / learn (nats)0.614 / 0.3760.136 / 0.3200.463 / 0.3310.107 / 0.287
Output-scoped ε67.7 (+5.8)69.5 (+0.4)45.3 (+2.3)52.0 (+0.6)
Output-embedding lr ×0.354.4 (+3.7)57.4 (+0.6)32.6 (+1.5)38.3 (+0.5)
Output-embedding lr ×0.165.8 (+4.6)69.6 (−0.5)40.9 (+2.0)50.0 (+0.5)
Output-embedding lr ×0.0369.0 (+3.8)71.8 (−2.8)44.9 (+2.1)52.7 (+0.5)
Output-embedding lr ×0 (frozen)69.2 (+0.9)72.0 (−4.3)46.4 (+2.1)52.9 (+0.4)
MoFO58.6 (−3.6)51.5 (−13.6)56.1 (−4.4)53.3 (−11.2)

Over 160M tokens, freezing and ε still remove nearly the same forgetting at the four models tested (Table A7). At Qwen, the frozen tied embeddings have a new-domain loss 0.018 nats higher at the end of training, in every seed, and the gap grows during training (Appendix A.5).

AuthorsOpenReview ↗

Response 3

(Edited 2026-11-15 22:02: table and section numbers updated to match the revised PDF.)

Question 3 (six seeds, Pythia-160M, per-seed values) and Weakness (Soundness, last part)

  • Question 3: Report Table 5 with six seeds for every arm, include Pythia-160M at its tuned rate in the summary range, give the per-seed distribution for 410M, and explain the +5% to +18% target-learning improvements at the same rate. Score-change criterion: a range that includes all eight settings at six seeds with the bimodal arm resolved and the learning gain explained would raise Soundness to 3.
  • Weakness (Soundness, last part): Finally the Pythia-410M arm is bimodal: across six seeds reduction spans 4.6-46.6% and learning change -32.9% to +16.8% (lines 283-286; Table 5 footnote 32.7% and -7.0% at six seeds vs 43.8% and +10.9% at three), and Pythia-160M collapses at its peak rate and is excluded from the summary range while remaining in the "eight settings" count (lines 253-260, 356-357).

Thank you for this request. All eight settings are now at six seeds, Pythia-160M is in the reduction range, and the 410M seeds are no longer bimodal.

What we changed. We reran every run of Table 6 at six paired seeds (seeds 100–105) with one pipeline, including Pythia-6.9B and 12B. Every row now reports n and a 95% paired-bootstrap confidence interval. Pythia-160M is reported at its tuned learning rate (lr 3e-5) in Table 6, because training at its peak lr 3e-4 collapses (Table A16). The abstract now gives the reduction range over all eight settings and states that target learning is not degraded at seven of them. The per-seed values for 410M are in Appendix A.3 (Table A2). We also corrected the TinyLlama row (item 5(a) of the general response).

Evidence. Item 2 of the general response (part 1) shows the rerun Table 6. Over all eight settings at six seeds, the reduction is 39.0–67.8%, including Pythia-160M at its tuned learning rate (56.5% [49.5, 62.6]). The learning change is −0.2% to +5.7% at the other seven settings, and none of their intervals lies entirely below zero. At Pythia-160M it is −6.6% [−11.5, −1.6], so the paper does not call that setting free of a loss in target learning. At these seeds, Pythia-410M is not bimodal: its six seeds span 31.2–46.7% (Table A2). The seed with the smallest reduction is also the one that learns least.

Table A2, two rows. Reduction (%) per seed, with the learning change (%) in parentheses.

Settingseed 100101102103104105
Pythia-160M (lr 3e-5)61.5 (+3.9)58.8 (−6.1)52.8 (−12.5)41.3 (−15.6)67.6 (−3.0)57.0 (−6.6)
Pythia-410M36.6 (−2.6)46.7 (+13.8)46.2 (+0.6)31.2 (−9.4)41.2 (−3.2)32.3 (−0.1)

Question 3 (learning gain)

  • Question 3: explain the +5% to +18% target-learning improvements at the same rate.

Thank you. The gain does not mean that the ε run learns Korean better. Without ε, the reference pushes down the output rows of tokens that are rare in the new data. This is the drift that causes forgetting. Some of these rare tokens are Korean, so the reference also predicts them worse. ε stops this drift, and the learning change counts what the reference loses as a gain for ε.

What we changed. After training, we split the tokens of the Korean validation text into three groups by how often each token occurs in the first 40M tokens of the Korean training pool: absent (c(t) = 0), rare (1 to 999 times) and frequent (1,000 times or more). We then measured the loss of each run on each group separately (Table A3).

Evidence. In the rerun Table 6, the learning gains are at most +5.7%, smaller than in the submission. Rare tokens are a small part of the Korean validation text (Positions column). On frequent tokens, the two runs reach the same loss at every setting except Pythia-160M. The ε run is better only on rare tokens, and most on tokens absent from those 40M tokens (c(t) = 0). Across models, the gap is largest at Qwen and OLMo, whose tokenizers produce the most rare tokens in Korean.

Table A3, selected columns. Six paired seeds. Loss is cross-entropy per token (nats). Ref. − ε is the reference's loss minus the ε run's loss; a positive value means the ε run predicts these tokens better.

SettingPositions: c(t) = 0 / 1–999 / ≥ 1,000Ref. − ε, c(t) = 0Ref. − ε, 1 ≤ c(t) < 1,000Ref. − ε, c(t) ≥ 1,000
Pythia-160M (lr 3e-5)132 / 7,861 / 401,207+1.086+0.040−0.051
Pythia-410M132 / 7,861 / 401,207+0.797+0.257−0.007
Qwen2.5-0.5B402 / 18,462 / 390,336+2.058+0.4340.000
TinyLlama-1.1B87 / 6,509 / 402,604+0.529+0.1220.000
Pythia-1.4B132 / 7,861 / 401,207+0.971+0.295+0.001
OLMo-2-1B331 / 11,638 / 397,231+1.586+0.5130.000
Pythia-6.9B132 / 7,861 / 401,207+0.343+0.0950.000
Pythia-12B132 / 7,861 / 401,207+0.358+0.120−0.001
Reviewer_fiUNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes an interesting idea on what causes forgetting with continual learning in LLMs: authors attribute this to the absence of certain tokens from the model vocabulary in the retraining dataset, which causes non-zero gradient updates during back propagation from the cross entropy loss, degrading stored information in the network. Authors propose to mitigate this phenomenon by dampening updates to the output projection matrix by raising the \epsilon parameter, and show that they are able to reduce forgetting during training, with minimal impact on learnability with new tasks.

Critical strengths
  1. Soundness

    • The idea that a key cause for catastrophic forgetting is that certain tokens from the old dataset are infrequent during retraining, which in turn affects their unembedding matrix parameters sounds plausible, and the authors present promising results to support this in Table 2.
    • The proposed intervention of increasing the \epsilon parameter to mitigate catastrophic forgetting is reasonable, and incurs no additional cost.
  2. Experiments and Results

    • Experiments are fairly detailed, spanning a variety of model sizes and families.
    • The presented mitigation results are promising and show significant reduction in model forgetting, with minimal impact on learnability of new tasks.
Critical weaknesses
  1. Presentation:

    • Several portions of this draft are authored in an inscrutable manner, and required substantial effort to parse.
    • Authors use a number of terms without a clear definition: "bottom output band", free point, etc.
    • Line 240 states: "The low-sqrt(v^hat) body band holds negligible share", but authors don't include a clear reference to this result.
    • Section 4.3, paragraph "Empirical evidence: travel distance across ϵ" is quite unclear.
  2. Novelty:

    • A few prior works cited here have already explored freezing the output layers (e.g., Gu & Feng, 2020) to prevent forgetting in models.
Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response 1

(Edited 2026-11-15 22:02: table and section numbers updated to match the revised PDF.)

Thank you for the time you put into this review. We also thank you for finding the explanation plausible and the experiments detailed.

Weakness 1 (Presentation), point 1

  • Several portions of this draft are authored in an inscrutable manner, and required substantial effort to parse. Authors use a number of terms without a clear definition: "bottom output band", free point, etc.

Thank you for this point, and we apologize for the extra effort it caused. To fit the page limit, we shortened the text by giving names to several quantities, and this made the paper harder to read. We now keep only a few names, define each one before it is used, and write the others out in plain words.

What we changed. Section 3.1 ("Notation and terminology") keeps four names and defines each of them: vocabulary deficiency, causal share, √v̂ bands and free reduction (also marked in Figure 2). Names that only labelled a simple idea are now written out. For example, "bottom output band" is now "the output embeddings at √v̂ < 1e-6", "under-exposed tokens" is now "tokens seen fewer than 1,000 times in the new corpus", and "free point" is dropped in favour of "free reduction". We also rewrote the paragraphs that put several results into one sentence. Each now states what was measured, what was observed and what we infer from it.

Evidence. The definitions in Section 3.1:

  • Vocabulary deficiency: the fraction of token positions in an old-domain evaluation set whose token occurs fewer than 1,000 times in the new corpus. When the original data is not available, a general-domain set replaces it.
  • Causal share: the share of forgetting that is prevented when a group of parameters is kept at its base values during training.
  • √v̂ bands: groups of parameters by the size of Adam's second moment, for example √v̂ < 1e-6.
  • Free reduction: a reduction in forgetting while target learning stays within the seed variation of the reference.

Weakness 1 (Presentation), point 2

  • Line 240 states: "The low-sqrt(v^hat) body band holds negligible share", but authors don't include a clear reference to this result.

Thank you. We added the reference.

What we changed. The sentence now cites Figure 2 and Table A17, which give the freezing results for each site and band. The freezing grid that supports this sentence was rerun at six paired seeds, and the range in the sentence is recomputed from it.

Evidence. Freezing the lowest √v̂ band of the body removes −4.2% to +0.5% of forgetting. The same band of the output embeddings removes 29.2–50.6% (Table A17).

Table A17, two rows. Six paired seeds, the five settings other than Pythia-160M. Share of forgetting removed by freezing the band √v̂ < 1e-6 (%), with the learning change (%) in parentheses.

SitePythia-410MPythia-1.4BTinyLlamaOLMoQwen
Output embeddings29.2 (−3.0)34.5 (+1.7)37.3 (+0.4)31.5 (+1.8)50.6 (+4.5)
Body−4.2 (−6.1)−0.4 (+0.1)0.5 (0.0)−1.4 (+0.1)0.2 (−0.1)

Weakness 1 (Presentation), point 3

  • Section 4.3, paragraph "Empirical evidence: travel distance across ϵ" is quite unclear.

Thank you. We rewrote this paragraph.

What we changed. The paragraph is now titled "Empirical validation: state distribution and travel distances". It is written in this order: what was measured, what was observed, and what we infer from it. Every value in it is taken from the tables it cites (Tables A28 and A29).

Evidence. The paragraph now has four steps. The values are from Tables A28 and A29, both rerun at six seeds (100–105) with the same pipeline as Table 6.

  1. Input embeddings of absent tokens receive no gradient and stay at √v̂ = 0 (34.6–51.6% of input parameters). The softmax gives every output row a non-zero second moment, and most of these are in the lowest band (73.1–85.6% below √v̂ = 1e-6).

  2. Over the first 120 steps at the default ε = 1e-8, displacement is of the same order in all bands, although the gradients differ by several orders of magnitude. In the lowest band, the updates point in one consistent direction (directional consistency 0.811–0.880).

  3. Raising ε to 1e-4 reduces displacement in the lowest band (√v̂ < 1e-6) by 54.9–146.2×, and in the highest band (√v̂ > 1e-4) by only 1.4–1.6×.

  4. The default ε = 1e-8 is too small to change this. Adam divides each update by √v̂ + ε, and the median √v̂ of the output embeddings is above 1e-8 in every setting (1.1e-8 to 7.9e-8).

AuthorsOpenReview ↗

Response 2

(Edited 2026-11-15 22:02: table and section numbers updated to match the revised PDF.)

Weakness 2 (Novelty)

  • A few prior works cited here have already explored freezing the output layers (e.g., Gu & Feng, 2020) to prevent forgetting in models.

Thank you for this point. We agree that freezing the output layer is prior work, and Related Work now says so directly.

What we changed. Related Work states that freezing the output layer as a module was done before (Gu & Feng, 2020). What the paper adds is which rows of that layer carry the forgetting, and why. The output rows of tokens that are rare or absent in the new data receive small one-sided gradients from the softmax. Adam's second-moment normalization turns these into full-sized parameter updates. We also compared ε with a frozen output layer (Section 5.1, Table A22). Section 5.1 now says that output-scoped ε matches the best output-embedding learning rate, including freezing, with no second learning rate to tune.

Evidence. Freezing the output embeddings and output-scoped ε remove nearly the same forgetting, within 2.5 points (Table A22). They differ in target learning only when the output embeddings are tied to the input embeddings. At Qwen, freezing loses 4.3% of target learning at body lr 3e-5, while ε loses none. Over 160M tokens, the frozen tied embeddings have a new-domain loss 0.018 nats higher at the end of training, in every seed (Appendix A.5). The freezing grid (Section 4.1, Table A17 in Response 1) shows where in the output layer the forgetting is. Freezing the bottom √v̂ band of the output embeddings removes 29.2–50.6% of forgetting. The same band of the body removes almost none.

Table A22, selected rows. Korean, three seeds per condition. Share of reference forgetting removed (%), with the learning change (%) in parentheses. Qwen's output embeddings are tied to its input embeddings.

Model (body lr)Output-scoped εFrozen output embeddings
Pythia-1.4B (1e-4)45.3 (+2.3)46.4 (+2.1)
Pythia-1.4B (3e-5)52.0 (+0.6)52.9 (+0.4)
Qwen2.5-0.5B (1e-4)67.7 (+5.8)69.2 (+0.9)
Qwen2.5-0.5B (3e-5)69.5 (+0.4)72.0 (−4.3)

Ethics flag

  • Flag For Ethics Review: Yes

Thank you. We explained our use of LLMs in an earlier comment, and we will add it to the paper.

What we changed. As described in our earlier comment on LLM usage, the LLM usage statement in the revised paper will name the models and what they were used for. The study uses only public corpora and public model checkpoints.

Evidence. All corpora are public and are listed with their identifiers in Appendix A.1 (Table A1): FineWeb-2 Korean, FineWeb-Edu, OpenWebMath, CodeParrot-clean, OpenOrca, Alpaca and KoAlpaca. All base models are public checkpoints (Pythia, Qwen2.5, TinyLlama, OLMo-2).

To Reviewer fiUN. We sincerely thank you for the time and care you have given to our paper. Your comments led us to define each term before it is used and to point each claim to the table that supports it. They also led us to state clearly what is prior work and what the paper adds. We hope these two responses address your concerns. If any point remains unclear, or if further evidence would help, we would be glad to provide it during the discussion.

Reviewer_kRufOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies where forgetting happens when a pretrained model is trained further on new data without replay. It freezes one group of parameters at a time, defined by site and by the size of Adam's second-moment estimate sqrt(v_hat), and measures how much forgetting disappears. On replay-free Korean data, the output-embedding coordinates with sqrt(v_hat) < 1e-6 account for 31 to 51% of the forgetting, while the same band in the input embedding and in the model body accounts for almost none (Tables 2 and A3). The proposed explanation is that the output rows of tokens that never appear in the new data receive small, one-sided softmax gradients, which Adam normalizes into full-size updates. The proposed fix is to set Adam's epsilon to 1e-4 for the output projection only. The paper reports 39.4 to 67.9% less loss-based forgetting with unchanged target loss in seven settings from 410M to 12B parameters, using 40M training tokens (Table 5). Math, code and instruction data (Alpaca, KoAlpaca) are tested as additional domains on three small models (Tables 3 and 4).

Critical strengths
  1. A careful causal localization with good controls. The paper compares freezes of matched size (Table 2: body 0.4 to 12.3% vs output 31 to 51%), freezes random rows as a control (App. A.2: 12.1% vs 42.7%), ablates weight decay (Table A7), shows three measurement methods that agree (Table A8), and shows that the same epsilon change on the input embedding does nothing (Table A18: -0.5%). Negative results are reported openly (Tables 7 and A16, App. A.3), and all reductions I recomputed from the raw losses in Tables 5, 6, A15 and A19 match.

  2. A useful practical message for one specific scenario. When the new corpus leaves much of the vocabulary unseen, the output rows of the unseen tokens are a major source of forgetting, which points to a simple rule: give the LM head its own optimizer treatment in that case.

Critical weaknesses
  1. The setup is very specific, and the conclusions are stated far more generally than the evidence supports. All main results (the localization in Tables 2 and A3, the headline reductions in Table 5, and the scaling to 12B) come from one scenario: training an English-pretrained model on Korean, with no replay, for 40M tokens. This is an extreme change of language and vocabulary, not general domain adaptation. The paper's own additional domains show how special this case is. Without the fix, forgetting on Korean is 0.51 to 0.66 nats, against 0.03 to 0.06 for math and 0.08 to 0.18 for code at the same model sizes (Table 3), so outside the language switch there is little forgetting to remove. When the new data stays in English and covers most of the vocabulary (Alpaca), output epsilon removes only 5.6 to 14.6% of forgetting and the forgetting sits mainly in the MLPs (Table 4). Even on KoAlpaca, freezing attention removes more forgetting than output epsilon at 410M (42.7% vs 29.8%), which the paper itself says "does not settle" the comparison. Yet the title and abstract claim that catastrophic forgetting in continual pretraining and fine-tuning in general "lives in the output embeddings". I think "forgetting under complete domain or language change" would describe what is actually shown.

This also matters for whether the proposed mechanism explains forgetting in more common settings. Other work suggests forgetting can occur even without any vocabulary or distribution gap. For example, in Thinking Machines Lab's on-policy distillation blog post, fine-tuning a model on samples generated by the same model still degrades it, which points to the optimization itself (for example batch composition and noise) rather than unseen tokens. If forgetting appears even when the training data comes from the model's own distribution, the "tokens the data never speaks" explanation cannot be the general cause, and the paper should test it in such a control setting. It is also not clear how much of the measured effect depends on how forgetting is defined here, as loss on a held-out pool of old-domain text, which a language switch is especially likely to inflate.

  1. The effect size and the evidence on general capabilities are overstated, and the fix performs the same as simpler alternatives.

(a) At six seeds Pythia-410M gives a 32.7% reduction (Table A7), yet the 39.4% lower end of the headline range uses the three-seed figure of 43.8%. The same configuration also reads 43.5% (Table A11) and 39.2% (Table A7), with a seed range of 4.6 to 46.6% (Sec. 4.2), and OLMo reads 39.4%, 40.7% and 34.2% in different tables. No confidence intervals are given.

(b) The only downstream check (Table A12) uses a different setting (math data, 1% replay, epsilon 1e-5) with one evaluation per arm. Its untouched Pythia-6.9B LAMBADA score is 61.05, while the Pythia paper reports 67.3 +/- 0.7, and without LAMBADA the capability-based reduction falls from 83.7% to 55.1%. The main Korean experiments have no downstream evaluation at all.

(c) In the paper's own controls, a 0.1x embedding learning rate gives 67.5% reduction (Table A10), and freezing the head on TinyLlama gives 47.0% against 47.2% for epsilon (Table A17). Kwok et al. (2024, arXiv 2407.03645) already freeze unused token embeddings. The epsilon value is also a threshold on the absolute size of sqrt(v_hat), which changes with width, batch size and learning rate, but the sweep covers only 160M to 1.4B (Table A5) and batch size is never reported.

This matters because the headline number and the claim that the fix preserves general capabilities are what readers will take away.

Other issues:

  • The risk ranking in Table 3 is confounded with learning rate: Korean runs use 1e-4, math and code 3e-5, and Qwen math 1e-5 (App. A.1).
  • Freezing the upper body bands removes more forgetting than freezing the output band in 4 of 5 settings (Table A3), which conflicts with "disjoint regions".
  • KL regularization is only tried at beta >= 0.1, where every value loses about 95% of learning (Tables A6, A18). MoFo is cited but not run.
  • One cited preprint (arXiv 2601.18699) claims weight-level analyses of closed models, which cannot be credible evidence.
  • Datasets, batch size and schedule are not fully described, and no code link is given.
Questions for authors
  1. [Clarification, affects my score] Does the mechanism hold outside a language switch? In particular, how much forgetting occurs, and how much does output epsilon remove, when the model is fine-tuned on its own generated samples or on same-language data that covers most of the vocabulary?

  2. [Clarification, affects my score] Would you be willing to scope the title and claims to forgetting under complete domain or language change, given that forgetting on math and code is about 3 to 24 times smaller than on Korean (Table 3)?

  3. [Clarification] Why does the headline range use the three-seed Pythia-410M result rather than the six-seed one? Please report means over all seeds with confidence intervals.

  4. [Clarification] Why is the untouched Pythia-6.9B LAMBADA score about 6 points below the published value, and can you run a downstream evaluation of the main Korean setting with epsilon 1e-4?

  5. [Clarification] Is there any setting where output epsilon clearly beats a 0.1x head learning rate or freezing rare-token rows?

8 comments
AuthorsOpenReview ↗

Response 1

(Edited 2026-11-15 22:02: table and section numbers updated to match the revised PDF.)

Thank you for the time and care you put into this review. We also thank you for noting the causal localization and its controls, and the practical message for corpora that leave much of the vocabulary unseen. Responses 1 and 3 answer the two questions marked as affecting your score.

Weakness 1 (first part) and Question 2

  • Weakness 1 (first part): The setup is very specific, and the conclusions are stated far more generally than the evidence supports. All main results (the localization in Tables 2 and A3, the headline reductions in Table 5, and the scaling to 12B) come from one scenario: training an English-pretrained model on Korean, with no replay, for 40M tokens. [...] Without the fix, forgetting on Korean is 0.51 to 0.66 nats, against 0.03 to 0.06 for math and 0.08 to 0.18 for code at the same model sizes (Table 3), so outside the language switch there is little forgetting to remove. [...] Yet the title and abstract claim that catastrophic forgetting in continual pretraining and fine-tuning in general "lives in the output embeddings". I think "forgetting under complete domain or language change" would describe what is actually shown.
  • Question 2: Would you be willing to scope the title and claims to forgetting under complete domain or language change, given that forgetting on math and code is about 3 to 24 times smaller than on Korean (Table 3)?

Thank you for this question. The parts of Weakness 1 marked [...] are answered in Responses 2 and 3. We agree that our original claims were too broad, and we have narrowed the claims. We also plan to narrow the title.

What we changed. We limited the claims to vocabulary-deficient adaptation, where the new corpus leaves much of the base vocabulary unseen. The Abstract, Introduction, Sections 4.2 and 6 (Limitations), and the Conclusion are revised. They now state that when the new data mostly uses familiar tokens, a comparable or larger share of the forgetting is in the transformer body. We chose vocabulary deficiency rather than a change of language or domain as the boundary, for two reasons. Forgetting happens even without a language switch. The share of forgetting in the output embeddings changes with deficiency. The title is now "A Deafening Silence: Under Vocabulary-Deficient Adaptation, Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks".

Evidence. Three sets of runs support this boundary.

First, forgetting does not require a new language. In the revised Table 2, math and code forget 0.12 to 0.39 nats. The 3 to 24 times ratio in the submitted Table 3 came from rows trained at different learning rates. With one learning rate per model, Korean forgets 1.4 to 3.1 times as much as math and code at the Pythia models.

Second, without a new language (Table A23 in Response 3), output-scoped ε removes 38.9–45.6% of forgetting on English web text (39.4% deficiency, measured on the evaluation domains of each row as in Table A23). On the model's own generated samples (21.5% deficiency), it removes only 8.3–14.8%.

Third, in instruction tuning (Table 3 in Response 2), we compare OpenOrca, an English instruction dataset (43% deficiency), with KoAlpaca, a Korean one (82% deficiency). OpenOrca replaces Alpaca in the revised Table 3, and Response 2 explains why. Moving from OpenOrca to KoAlpaca raises the share that output-scoped ε removes at every model. On KoAlpaca, MLP freezing and body-scoped ε lose target learning at every model.

Table 2. One learning rate per model, six paired seeds. Forgetting is scored on the four evaluation domains minus the injected one. Reduction: share of reference forgetting removed by output-scoped ε = 1e-4. n/q: not quotable (Table 2 caption).

Base model (lr)InjectedVocab. deficiency (%)Ref. forgetting (nats)Reduction (%) [95% CI]Learning change (%)
Pythia-410M (1e-4)math33.90.20924.5 [21.5, 27.5]+3.4
code55.90.39237.2 [30.5, 42.1]+10.3
Korean63.20.64536.5 [32.2, 40.6]−0.2
Pythia-1.4B (1e-4)math33.90.23621.1 [19.9, 22.6]+9.4
code55.90.36137.9 [36.4, 39.4]+13.7
Korean63.20.50841.3 [40.7, 42.0]+2.2
Qwen2.5-0.5B (3e-5)math50.60.11768.6 [67.7, 69.7]n/q
code52.50.17772.3 [69.6, 75.5]+14.4
Korean61.30.14863.6 [62.9, 64.2]+0.3

Table 2 also shows that deficiency does not predict the size of forgetting across models. The rank correlation between its deficiency and reference-forgetting columns is 0.57 over the nine cells (Appendix A.2). At Qwen, code has lower deficiency than Korean (52.5% against 61.3%) but forgets more (0.177 against 0.148 nats). We therefore use deficiency only to state where our results apply.

AuthorsOpenReview ↗

Response 2

(Edited 2026-11-15 22:02: table and section numbers updated to match the revised PDF.)

Weakness 1 (second part)

  • When the new data stays in English and covers most of the vocabulary (Alpaca), output epsilon removes only 5.6 to 14.6% of forgetting and the forgetting sits mainly in the MLPs (Table 4). Even on KoAlpaca, freezing attention removes more forgetting than output epsilon at 410M (42.7% vs 29.8%), which the paper itself says "does not settle" the comparison.

Thank you for this point. We agree that on English instruction data the output embeddings carry a small share of the forgetting. This is one reason why we narrowed the claims (Response 1).

What we changed. We reran Table 3 at one learning rate per model. During this rerun, we found a problem with Alpaca. The 4.2M-token Alpaca corpus is repeated three times within the 12.6M-token budget. At the KoAlpaca learning rate, the Pythia-1.4B and Qwen references overfit: their held-out Alpaca loss rises after the first evaluation (Table A27, Appendix A.7). At the same learning rates, the KoAlpaca loss keeps falling at every model. At lower learning rates Alpaca does not overfit, and the output share stays small (4.8% at Pythia-1.4B and 6.1% at Qwen; Table A8). We therefore replaced Alpaca in Table 3 with OpenOrca (43% deficiency, no example repeated). It is compared with KoAlpaca (82% deficiency) at six paired seeds and matched learning rates. We withdrew the earlier statement that forgetting on English instruction data is in the MLPs. The instruction-tuning claim in Section 4.2 and Appendix A.6 is narrowed.

Evidence. The revised Table 3 agrees with your observation on low-deficiency data. On OpenOrca, body-scoped ε removes more forgetting than output-scoped ε at both Pythia models. On KoAlpaca, output-scoped ε removes more than on OpenOrca at every model. On OpenOrca, MLP freezing loses no target learning at Pythia-1.4B and Qwen, and body-scoped ε loses none at Qwen. On KoAlpaca, the same methods lose 15.9–26.3% of target learning.

Table 3. Instruction tuning, one learning rate per model (lr: Pythia-410M 3e-5, Pythia-1.4B 1e-4, Qwen2.5-0.5B 3e-5), six paired seeds. Each cell: share of reference forgetting removed (%), with the change in target learning (%) in parentheses. Qwen's output embeddings are tied to its input embeddings (n/a).

410M OpenOrca410M KoAlpaca1.4B OpenOrca1.4B KoAlpacaQwen OpenOrcaQwen KoAlpaca
Ref. forgetting (nats)0.0980.1740.2560.4600.0680.104
Attention freeze15.7 (−6.3)46.2 (+5.7)20.4 (+2.0)32.6 (+0.0)11.6 (+3.5)15.4 (−0.2)
MLP freeze21.3 (−18.0)18.1 (−13.9)33.2 (+1.3)48.4 (−15.9)30.2 (+2.7)26.3 (−26.3)
Input-embedding freeze−4.3 (−1.6)9.2 (+9.6)0.7 (+0.3)−0.6 (+0.1)n/an/a
Output-embedding freeze22.2 (−2.7)34.8 (+3.9)23.0 (+2.6)37.1 (+0.6)43.0 (+6.6)46.6 (−4.0)
Output-scoped ε22.9 (−0.8)32.4 (+6.9)20.7 (+2.5)34.2 (+0.5)38.9 (+5.9)43.7 (−1.5)
Body-scoped ε30.2 (−17.8)46.6 (−16.7)48.3 (−1.4)60.1 (−15.2)34.8 (+7.3)29.3 (−15.9)

On the attention column, the rerun confirms your observation at 410M. On KoAlpaca, freezing attention removes 46.2% of forgetting, against 32.4% for output-scoped ε. At 1.4B the two are close (32.6% and 34.2%), and at Qwen ε removes more (43.7% against 15.4%). Neither method removes more at every model. We therefore no longer claim that, in instruction tuning, more of the forgetting is in the output embeddings than in attention. Our claim is narrower. When the corpus brings new vocabulary, output-scoped ε removes a larger share of the forgetting (20.7–38.9% on OpenOrca against 32.4–43.7% on KoAlpaca), with learning changes of −1.5% to +6.9% on KoAlpaca.

AuthorsOpenReview ↗

Response 3

(Edited 2026-11-15 22:02: table and section numbers updated to match the revised PDF.)

Weakness 1 (third part) and Question 1

  • Weakness 1 (third part): Other work suggests forgetting can occur even without any vocabulary or distribution gap. For example, in Thinking Machines Lab's on-policy distillation blog post, fine-tuning a model on samples generated by the same model still degrades it, which points to the optimization itself (for example batch composition and noise) rather than unseen tokens. [...] It is also not clear how much of the measured effect depends on how forgetting is defined here, as loss on a held-out pool of old-domain text, which a language switch is especially likely to inflate.
  • Question 1: Does the mechanism hold outside a language switch? In particular, how much forgetting occurs, and how much does output epsilon remove, when the model is fine-tuned on its own generated samples or on same-language data that covers most of the vocabulary?

Thank you for suggesting these controls. We ran both of them. Forgetting occurs in both, and output-scoped ε removes much less of it on the model's own samples.

What we changed. The two controls are reported in Section 4.2 and Table A23. Pythia-410M and Pythia-1.4B were trained at lr 1e-4 for 40M tokens, with three seeds, on two kinds of data:

  • (i) the English split of our pool, with forgetting measured on code and math;
  • (ii) 40M tokens of each model's own unconditional samples, with forgetting measured on English, code and math.

Evidence. Forgetting occurs in both controls (Table A23): the reference forgets 0.42–0.46 nats when trained on English and 0.11–0.20 nats when trained on its own samples. Unseen tokens are therefore not the only cause of forgetting, and Section 4.2 now says so. Output-scoped ε still removes 38.9–45.6% of forgetting on English. On self-generated text, whose vocabulary is the base model's own, it removes only 8.3–14.8%.

Table A23, selected columns. lr 1e-4, 40M tokens, three seeds, output-scoped ε = 1e-4. In all four rows, the reference's new-domain loss rises by 0.01–0.17 nats during training, so only reductions are reported.

Base modelNew dataForgetting measured onVocab. deficiency (%)Ref. forgetting (nats)Reduction (%) [95% CI]
Pythia-410MEnglishcode, math39.40.46145.6 [40.3, 50.4]
Pythia-410MSelf-generatedEnglish, code, math21.50.10814.8 [14.0, 16.1]
Pythia-1.4BEnglishcode, math39.40.41938.9 [36.7, 41.0]
Pythia-1.4BSelf-generatedEnglish, code, math21.50.2018.3 [8.3, 8.4]

On the loss-based measure, every reference and ε run of Table 6 is now also evaluated on downstream benchmarks (Table 7, Tables A24 and A25, Response 5). The ε run scores 0.7 to 4.3 points above the reference on the five-task English mean, at all eight settings. The reduction is therefore not only a change in loss.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response(1/2)

(Edited 2026-11-15 22:02: table and section numbers updated to match the revised PDF.)

We apologize for posting our responses late. The reviews asked for many new experiments, and their results could change the main claims of the paper. While rerunning, we also found that the outputs and some analysis scripts of several appendix experiments in the submission were not kept. We are rerunning these experiments from the start at six seeds (100–105), with the same pipeline as Table 6. Their numbers may change slightly. If a claim changes, we will say so in this thread. Unless noted otherwise, the tables we quote in these responses are already rerun.

We thank the reviewers for their careful reading and helpful comments. Changes shared across the reviews are summarized here, in two comments. Each item first quotes what the reviewers asked. Each point is also answered in the individual responses. All changes in the revised PDF are marked in color. Tables in our responses are copied from the revised paper and keep its table numbers.

1. Scope

  • Reviewer kRuf (Weakness 1): The setup is very specific, and the conclusions are stated far more generally than the evidence supports.
  • Reviewer z9Av (Minor 1): State in the abstract that the localisation and the remedy apply to high vocabulary-deficiency injections

We now limit the claims to vocabulary-deficient adaptation, where the new corpus leaves much of the base vocabulary unseen. The abstract, introduction, Sections 4.2 and 6 (Limitations), and the Conclusion are revised. We also plan to narrow the title, for example to "A Deafening Silence: Under Vocabulary-Deficient Adaptation, Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks". Forgetting also occurs without a new language (Table 2 for math and code, Table A23 for English and self-generated text). Unseen tokens are therefore not the only source of forgetting. Vocabulary deficiency changes how much of the forgetting is in the output embeddings. On data with low deficiency, a comparable or larger share is in the transformer body (Table 3).

2. Downstream evaluation

  • Reviewer kRuf (Question 4): can you run a downstream evaluation of the main Korean setting with epsilon 1e-4?
  • Reviewer z9Av (Question 1): Please add a downstream evaluation (...) for the reference and intervention arms at the eight settings of Table 5, with three seeds.

Every reference and ε run of Table 6 is now evaluated zero-shot in FP32 at six paired seeds. The tasks are the five English tasks of Table 7, KoBEST, GSM8K and HumanEval. On the five-task English mean, the ε run scores 0.7 to 4.3 points above the reference at all eight settings (Table 7). The largest gains are on LAMBADA. The KoBEST mean stays within 0.5 points of the reference (Tables A24 and A25).

Tables 6 and 7, combined. Korean, 40M tokens, output-scoped ε = 1e-4, six paired seeds. Each value is the mean over the six seeds. Brackets give the 95% confidence interval (lower bound, upper bound). Forgetting is measured on English and code. English mean: zero-shot FP32 accuracy (%) averaged over LAMBADA, HellaSwag, PIQA, ARC-Easy and WinoGrande. Pythia-160M is at lr 3e-5, because training at its peak lr 3e-4 collapses (Table A16).

SettingRef. forgetting (nats)Reduction (%) [95% CI]Learning change (%) [95% CI]English mean: base / ref. / ε
Pythia-160M1.05956.5 [49.5, 62.6]−6.6 [−11.5, −1.6]43.7 / 37.0 / 38.5
Pythia-410M0.62939.0 [34.2, 43.9]−0.2 [−5.1, +6.0]51.7 / 44.7 / 45.8
Qwen2.5-0.5B0.61667.8 [67.6, 68.1]+5.7 [+5.6, +5.8]57.9 / 48.6 / 52.9
TinyLlama-1.1B0.35447.8 [46.3, 49.1]+0.5 [+0.4, +0.5]53.4 / 51.9 / 52.6
Pythia-1.4B0.46544.8 [44.4, 45.3]+2.2 [+2.1, +2.4]59.2 / 51.4 / 53.6
OLMo-2-1B1.16540.6 [39.4, 42.1]+2.2 [+2.0, +2.3]69.5 / 53.3 / 57.2
Pythia-6.9B0.11160.0 [59.4, 60.6]+0.6 [+0.5, +0.7]65.9 / 64.2 / 65.5
Pythia-12B0.10065.2 [64.3, 66.1]+0.7 [+0.5, +0.9]68.5 / 66.0 / 67.5
AuthorsOpenReview ↗

General Response(2/2)

(Edited 2026-11-15 22:02: table and section numbers updated)

This is the second part of the general response.

3. Seeds and confidence intervals

  • Reviewer kRuf (Question 3): Why does the headline range use the three-seed Pythia-410M result rather than the six-seed one? Please report means over all seeds with confidence intervals.
  • Reviewer kRuf (Other issues): The risk ranking in Table 3 is confounded with learning rate
  • Reviewer z9Av (Question 3): Report Table 5 with six seeds for every arm
  • Reviewer z9Av (Minor 5): State the seed counts of the freezing grid (two per arm except Qwen, Table A3) in Section 4.1

We reran every run of Table 6 at six paired seeds with one pipeline, including Pythia-6.9B and 12B. Table 6 now reports n and 95% confidence intervals. Tables 2 and 3 and the freezing grid (Tables 1, A15 and A17) were also rerun at six seeds. Tables 2 and 3 now use one learning rate per model. Over the seven models other than Pythia-160M, output-scoped ε reduces forgetting by 39.0–67.8%, with learning changes of −0.2% to +5.7% (Table 6 in part 1).

4. Simpler alternatives

  • Reviewer kRuf (Question 5): Is there any setting where output epsilon clearly beats a 0.1x head learning rate or freezing rare-token rows?
  • Reviewer z9Av (Question 2): Please compare against a reference tuned over a two-rate grid (body rate × output-embedding rate), i.e. the best per-tensor learning-rate baseline, and against whole-output-embedding freezing and MoFO
  • Reviewer fiUN (Weakness 2): A few prior works cited here have already explored freezing the output layers (e.g., Gu & Feng, 2020) to prevent forgetting in models.

A low enough learning rate for the output embeddings removes nearly the same forgetting as output-scoped ε. So does freezing them (Table A22 below). We no longer claim that ε does better than such a learning rate. The paper now says that ε gives the result of the best output-embedding learning rate, with no second learning rate to tune. ε differs from freezing only when the output embeddings are tied to the input embeddings. At Qwen2.5-0.5B, freezing learns less than ε at both body learning rates: +0.9% against +5.8% at lr 1e-4, and −4.3% against +0.4% at lr 3e-5. On prior work on freezing, our response to Reviewer fiUN (Response 2) explains what the paper adds.

Table A22. Korean, three seeds per condition. Each cell: share of reference forgetting removed (%), with the learning change (%) in parentheses. Output-embedding lr ×k: output-embedding learning rate scaled by k relative to the body learning rate. ×0 freezes the whole output embedding.

Qwen2.5-0.5B, body lr 1e-4Qwen2.5-0.5B, body lr 3e-5Pythia-1.4B, body lr 1e-4Pythia-1.4B, body lr 3e-5
Ref. forget / learn (nats)0.614 / 0.3760.136 / 0.3200.463 / 0.3310.107 / 0.287
Output-scoped ε67.7 (+5.8)69.5 (+0.4)45.3 (+2.3)52.0 (+0.6)
Output-embedding lr ×0.354.4 (+3.7)57.4 (+0.6)32.6 (+1.5)38.3 (+0.5)
Output-embedding lr ×0.165.8 (+4.6)69.6 (−0.5)40.9 (+2.0)50.0 (+0.5)
Output-embedding lr ×0.0369.0 (+3.8)71.8 (−2.8)44.9 (+2.1)52.7 (+0.5)
Output-embedding lr ×0 (frozen)69.2 (+0.9)72.0 (−4.3)46.4 (+2.1)52.9 (+0.4)
MoFO58.6 (−3.6)51.5 (−13.6)56.1 (−4.4)53.3 (−11.2)

5. Errors found while rerunning

  • Reviewer kRuf (Question 4): Why is the untouched Pythia-6.9B LAMBADA score about 6 points below the published value

We found five errors while rerunning and corrected them. Item (d) answers this question.

  • (a) The submitted TinyLlama row of Table 6 came from runs with ε raised on both embeddings. It is replaced by output-scoped ε runs.
  • (b) The freezing grid recomputed its band masks at every step, not once at the start as the text said. Section 4.1 is corrected.
  • (c) We could not find the data behind the Spearman correlation of 0.60 in the submission. We recomputed it: 0.57 over the nine cells of Table 2 and 0.96 within the Pythia models.
  • (d) The LAMBADA score of 61.05 for Pythia-6.9B came from evaluating in pure bf16. In FP32 the score is 67.1. The published value is 67.3.
  • (e) At one learning rate per model, Alpaca is repeated three times within the 12.6M-token budget, and the Pythia-1.4B and Qwen references overfit. Table 3 therefore uses OpenOrca, and Alpaca is kept in Appendix A.7.

6. Reproducibility

  • Reviewer kRuf (Other issues): Datasets, batch size and schedule are not fully described, and no code link is given.
  • Reviewer z9Av (Minor 6): Release the corpora splits, rate grids per arm and seed lists

Section 3.3 gives the full training setup, including the batch layout and schedule. Appendix A.1 lists each corpus with its identifier and how its splits are cut. The code, the data splits and the per-run configurations with seeds will be released at https://anonymous.4open.science/r/Deafening-silence-7939/ (anonymized) before the camera-ready version.

AuthorsOpenReview ↗

Summary of revisions in the updated PDF

Summary of revisions

We thank Reviewers kRuf, fiUN and z9Av for their careful reading. Their comments led to the reruns, the new experiments and the rewording summarized below, and the paper is much stronger for it. We have uploaded the revised PDF. Table numbers in the revised PDF differ from the submission; the map is in our replies to each reviewer.

What changed in the main claims

  • Where the forgetting is. The claim is unchanged, but we now say that a large share of the forgetting, not most of it, is in the output embeddings of rarely used tokens. All freezing grids were rerun at six paired seeds with 95% CIs. Freezing those rows removes 29.2–50.6% of the forgetting at five settings at no cost in target learning; freezing the same number of body parameters removes at most 12.5%, and only at a cost in learning.
  • The intervention. Output-scoped ε now covers all eight settings, 160M to 12B, including Pythia-160M, which the submission had excluded as unstable. It removes 39.0–67.8% of the forgetting, keeps target learning at seven of the eight settings, and raises accuracy on five English benchmarks at all eight. One ε value is used everywhere.
  • Reading the risk from the data. We narrowed this claim. Vocabulary deficiency ranks the forgetting only within one base model (Pythia); it does not compare models, and it does not predict the size of the forgetting. This is now stated in Section 4.2 and in the limitations.
  • What forgetting is measured on. Forgetting is measured on public text of the old kind, not on the pre-training data. A new experiment compares the two where the pre-training data is public (Pile, SlimPajama, OLMo-mix): the order of the results is the same, and the reduction on Korean is 3–11 points lower on the pre-training text. We state this as a limitation.
  • The body. When the new data uses familiar vocabulary, a comparable or larger share of the forgetting is in the transformer body, and we have no free reduction there. The conclusion no longer suggests that an optimizer change could be the primary defense against forgetting.

What we withdrew or replaced

  • The Alpaca column of the instruction-tuning table. Alpaca is 4.2M tokens, so the 12.6M-token budget repeats it three times and the reference overfits at the learning rates of that table. We replaced it with OpenOrca, which is not repeated, and document the overfitting in the appendix. Alpaca at lower learning rates remains in the appendix.
  • The reading that freezing the whole output module on KoAlpaca removes nothing (−3.7% at Pythia-410M). At six seeds it removes 34.8%, on par with ε. We withdraw that reading. As Reviewer kRuf observed, freezing attention removes more than ε at 410M on KoAlpaca; it removes less at 1.4B and Qwen, and we no longer claim that the output embeddings hold more of the forgetting than attention in instruction tuning.
  • Numbers updated by the six-seed reruns: replay plus ε at Qwen 79.8% → 78.5%; post-hoc editing recovers at most 7.0% rather than under 5%; in the learning-rate grid, Pythia-6.9B at lr 3e-4 goes from 47.0% to −35.5% with a CI that includes zero.

What we added

  • Learning-rate grid at all eight settings, six seeds in every cell, and the equivalence of raising ε and lowering the output-embedding learning rate (Section 5.1).
  • Downstream accuracy on five English benchmarks for every main-result run, plus Korean benchmarks, GSM8K and HumanEval in the appendix.
  • Forgetting on three old-domain sets: our public splits, public text of another kind, and the models' own pre-training text where public.
  • Mixing 1–3% of public Wikipedia and Python code into the new data: it removes 22–44% of the forgetting by itself and 73–78% together with ε, at a learning cost of at most 1.2%.
  • Long training to 160M tokens at Pythia-6.9B and 12B, and frozen output embeddings against output-scoped ε at 160M tokens at four models.
  • Controls without a new language (English continued pre-training and self-generated text), a two-rate grid with MoFO, ε sensitivity at six models, and replay at a second model (OLMo-2-1B).

Wording and structure

The terms we use are now defined in one place (Section 3.1) and used consistently; every learning rate, √v̂ value and ε is labelled. Sections 4 and 5 are reorganized into where and why, the method and its equivalents, results, and boundaries. Section 6 is limitations only. Figure 1 is updated to the six-seed values.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.