acceptodds
Under review as a conference paper at ICLR 2027

Preventing LoRA Forgetting Through Target Masking

Abstract

Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large language models to new domains, tasks, and users. Yet adaptation performance alone can obscure an important failure mode: LoRA updates may improve performance on the target distribution while degrading prior capabilities learned during pretraining and alignment. We show that this forgetting becomes especially severe when the adaptation distribution differs substantially from the model’s original training or alignment distributions. The challenge is amplified in practical settings, where the original training and alignment data are typically unavailable. Motivated by this constraint, we study how LoRA based adaptation balances new learning against forgetting in a replay-free setting, and introduce a simple output space regularizer that can be added directly to existing training pipelines. Our method removes the ground-truth token from both the base and adapted model distributions, renormalizes the remaining probabilities, and applies KL regularization only over the non-target vocabulary. This preserves the base model's relative preferences among alternative tokens without directly opposing the cross-entropy signal required for adaptation. As the regularizer acts only at the loss level, it requires no replay data, architectural changes, adapter redesign, or inference-time overhead, and can be applied directly to existing LoRA variants. Across all LoRA variants tested and across various backbones, our method improves the frontier between new learning and forgetting when the adaptation distribution differs substantially from the base model’s original training or alignment distributions, suggesting a broadly applicable route toward more reliable LLM updating.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Rank Adaptation (LoRA), Parameter-Efficient Fine-Tuning, Catastrophic Forgetting, Continual Learning, Knowledge Retention, KL Regularization, Distribution Shift, Large Language Models, Replay-Free Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9MxbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies how LoRA-based adaptation balances new learning against forgetting in a replay-free setting. It proposes Target-Masked KL (TMKL), a one-line, adapter-agnostic regularizer. At supervised token positions, TMKL removes the target token from the frozen base and adapted models’ output distributions and renormalizes the remaining probabilities. It then applies KL regularization from the base to the adapted non-target distribution to preserve relative preferences among alternative tokens. The reported experimental results show reduced perplexity degradation on WikiText-103 and LAMBADA compared with cross-entropy-only training, while maintaining competitive target-domain perplexity. Additional validation includes multiple model families, instruction-tuned models, broader retention metrics, and ablation studies.

Critical strengths
  • The proposed TMKL is easy to understand and makes a well-motivated distinction between the supervised target token and non-target tokens.
  • The proposed method provides a simple way to regularize adaptation using the frozen base model’s predictions on the new task data. It can be added to the cross-entropy loss without changing the adapter architecture or adding inference-time overhead.
  • Tables 1-2 show that TMKL substantially reduces perplexity degradation on WikiText-103 and LAMBADA compared with cross-entropy-only training, while maintaining competitive target-domain perplexity. The results across LoRA, DoRA, SineLoRA, and RandLoRA provide useful empirical support for TMKL’s ability to balance adaptation and retention across several adapter designs.
Critical weaknesses
  • The connection between adaptation-domain regularization and retention on other distributions is insufficiently established. TMKL constrains non-target distributions only in adaptation contexts. Although the evaluations provide useful empirical evidence, Table 27 reports non-target KL only on held-out OpenR1-Math.
  • TMKL improves retention, but it remains unclear how much of the gain comes from target masking and weight removal, and how much comes from optimization settings or regularization strength. More controlled comparisons would help establish the contribution of these specific design choices.
Questions for authors
  • Why should preserving the base model’s relative preferences among non-target tokens on the adaptation data help retain performance on other distributions, such as WikiText-103? What evidence supports this connection beyond the reported retention scores, and under what conditions should it hold?
  • Could you isolate the benefit of target masking by comparing Full-KL with weighted TMKL? After tuning learning rates and regularization strengths under a common protocol, does target masking improve retention at comparable target-domain perplexity? Do you also have experimental results for a similarly tuned CE baseline?
  • Table 17 shows lower retention drift for the weighted variant at its selected λ\lambda, with slightly higher target PPL. Do you have experimental comparisons between weighted and unweighted TMKL at comparable target-domain PPL?
7 comments

We thank the reviewer for the positive assessment and the specific questions. Page and line numbers refer to the revised PDF, where all changes are in brick red.

Two additions answer your questions most directly: a measurement of the regularized quantity on retained distributions that TMKL never sees (new Table 28), and a three-way comparison of Full-KL, weighted TMKL and TMKL at fixed λ\lambda and under a common tuning rule.

Weakness 1: The connection between adaptation-domain regularization and retention on other distributions is insufficiently established. ... Table 27 reports non-target KL only on held-out OpenR1-Math.

Question 1: Why should preserving the base model’s relative preferences among non-target tokens on the adaptation data help retain performance on other distributions ... and under what conditions should it hold?

The adapter is shared by every input. TMKL penalizes adapter directions that change the base model's preferences among alternative tokens on adaptation contexts, and directions that change those preferences on general text usually change them on the adaptation text too. We now (i) measure the regularized quantity directly on retained distributions and (ii) state the condition under which the transfer holds.

New: the probe on retained distributions (App. H.1, Table 28, p. 40). Using each position's own next token as yy on WikiText-103, LAMBADA and FLORES-200 (never seen in training), five seeds:

0.5B CE0.5B Full-KL0.5B TMKL7B CE7B TMKL
WikiText, conditional non-target KL0.5900.1200.0550.2900.024
WikiText, full-output KL0.5800.1180.0790.2400.035

Across all three retained distributions TMKL reduces the conditional non-target KL by 90-92% and the full-output KL by 82-86% relative to CE, and Full-KL lies between CE and TMKL on every one. The quantity TMKL regularizes is suppressed on distributions it never sees.

Why (App. G.5, p. 38). Let HTH_T be the curvature of the TMKL loss at the base (the Fisher-weighted Jacobian Gram matrix of App. G.3) and gQg_Q the gradient of mean NLL on another distribution QQ. For a small update δϕ\delta\phi with gQ∈range(HT)g_Q \in \mathrm{range}(H_T),

∣Δlog⁡PPL(Q)∣≲gQ⊤HT+gQ⋅2L∖y.\big|\Delta \log \mathrm{PPL}(Q)\big| \lesssim \sqrt{g_Q^\top H_T^{+} g_Q} \cdot \sqrt{2 \mathcal L_{\setminus y}}.

Retention on QQ is controlled by the TMKL value on adaptation data, scaled by a constant that is small when QQ's sensitive directions are excited by the adaptation contexts. It predicts failure when QQ depends on directions those contexts never excite, or when updates are large.

Further evidence.

  1. Known pretraining distribution (App. D, Table 12). A model trained from scratch on Shakespeare and adapted to Latin, Dutch or Esperanto: TMKL sees only target-language contexts, yet cuts drift on held-out Shakespeare by 67-80% in all nine language/adapter cells.
  2. A quantitative prediction. The local analysis predicts drift ∝1/λ\propto 1/\lambda under strong regularization; the fitted WikiText log-log slope over λ∈[0.3,1]\lambda \in [0.3, 1] is −0.94-0.94 (LAMBADA: −1.35-1.35, same direction; App. G.3, p. 38).
  3. Far-from-target probes. Code, translation, factual and math probes degrade by 9-29% under CE and stay within 1.5% under TMKL (Table 7), and IFEval, MT-Bench and XSTest behave the same way (Table 8).
  4. Where it is weaker. PiSSA updates the principal weight directions directly; there TMKL removes about two-thirds of the drift (WikiText +136% → +47%, Table 6). Large, non-local updates, which the local condition excludes, are a plausible explanation that we have not measured.

The remaining points are answered in the next comment.

Weakness 2: TMKL improves retention, but it remains unclear how much of the gain comes from target masking and weight removal, and how much comes from optimization settings or regularization strength. ...

Question 2: Could you isolate the benefit of target masking by comparing Full-KL with weighted TMKL? ... Do you also have experimental results for a similarly tuned CE baseline?

On the same positions, with w=1−pb(y)w = 1 - p_b(y):

LFull=LCE+λ(B+wN),Lweighted=LCE+λwN,LTMKL=LCE+λN.L_{\text{Full}} = L_{\text{CE}} + \lambda(B + wN),\quad L_{\text{weighted}} = L_{\text{CE}} + \lambda wN,\quad L_{\text{TMKL}} = L_{\text{CE}} + \lambda N.

Full-KL vs. weighted isolates the binary term BB; weighted vs. unweighted isolates the weight, which also changes effective strength (mean w=0.347w = 0.347). The paper contains both controls, at fixed strength and under a common tuning rule.

At fixed nominal λ=1\lambda=1 (Qwen2.5-0.5B → OpenR1, LoRA, three seeds; Tables 14 and 26).

ObjectiveTarget PPLWikiText driftLAMBADA driftSource
Full-KL (B+wNB + wN)2.8619+4.99%+2.40%Table 14
TMKL (NN)2.8653+4.07%+2.29%Table 14
Weighted TMKL (wNwN)2.86+9%+7%Table 26
TMKL (NN)2.87+4%+2%Table 26

The two tables are separate three-seed suites under the same recipe; unweighted TMKL appears in both and agrees.

At equal target fit, removing BB alone (Full-KL → weighted) removes regularization strength, and dropping the weight restores it: unweighted TMKL has the lowest drift of the three. The same ordering between weighted and unweighted holds on all four adapters (Table 26: WikiText +2-4% vs. +7-10%, target PPL within 0.02).

Under a common target-only tuning rule (Table 17, three seeds, separate validation split). Each objective's λ\lambda is chosen from {0.1,0.3,1,3}\lbrace 0.1, 0.3, 1, 3 \rbrace by the same preregistered rule:

ObjectiveSelected λ\lambdaTarget-test PPLWikiText driftLAMBADA drift
CE–3.2464+15.74%+18.18%
Full-KL12.9524+1.99%+1.40%
Weighted TMKL32.9715+0.86%+0.42%
TMKL12.9547+0.91%+1.40%

Both masked objectives retain at least as well as Full-KL and better on WikiText; unweighted TMKL does so at Full-KL's target PPL (within 0.08%), weighted TMKL at a 0.65% target cost. What masking buys is that regularization strength can be spent on the non-target term without pulling the target back toward the base.

CE across learning rates (Table 18, LoRA, learning rate 10−410^{-4} to 10−310^{-3}; single seed, test-set PPL). At every learning rate TMKL removes 85-92% of CE's WikiText drift. CE's best target fit (lr 10−410^{-4}, Δ\DeltaPPL −0.38-0.38) still drifts +14.5%; TMKL at the headline recipe reaches −0.34-0.34 with +4.3%, and at lr 3⋅10−43\cdot10^{-4} TMKL has both better target fit (−0.33-0.33 vs. −0.31-0.31) and 9× less drift. The retention gain is therefore not a consequence of CE's learning rate. Table 17 also includes CE under the common three-seed protocol (target-test PPL 3.2464, drift +15.74% / +18.18%). Sec. 4.2 and App. E.5 now qualify our "roughly doubles the target-PPL drop" statement accordingly: it holds under the shared recipe (2.1×, 1.6×, 2.2× on LoRA, SineLoRA, DoRA), but not against CE's single best learning rate. Table 17 shares one learning rate across objectives, and Table 18 is a single-seed sensitivity sweep rather than a validation-selected, multi-seed CE comparator. A joint learning-rate × λ\lambda grid for CE, Full-KL and weighted TMKL at matched target PPL under the common protocol is the remaining control; we will add it to App. E.4 in the camera-ready version.

The remaining points are answered in the next comment.

Question 3: Table 17 shows lower retention drift for the weighted variant at its selected λ, with slightly higher target PPL. Do you have experimental comparisons between weighted and unweighted TMKL at comparable target-domain PPL?

Yes. At comparable target PPL, unweighted TMKL retains better on every adapter (Table 26, λ=1\lambda=1, three seeds, all four adapters):

AdapterTarget PPL (weighted / unweighted)WikiText drift (weighted / unweighted)Mean prevention (weighted / unweighted)
LoRA2.86 / 2.87+9% / +4%−77% / −92%
SineLoRA2.86 / 2.87+10% / +4%−75% / −92%
RandLoRA2.90 / 2.92+7% / +2%−70% / −94%
DoRA2.85 / 2.86+10% / +4%−75% / −92%

Target PPL differs by at most 0.02 while unweighted TMKL removes 15-24 points more of the drift.

Accounting for effective strength. Because the weight averages 0.347, part of this gap is strength rather than weighting, and Table 17 reflects this: at its own selected λ=3\lambda=3 (effective strength ≈ 1.04) the weighted variant reaches +0.86% / +0.42% at target-test PPL 2.9715, versus +0.91% / +1.40% at 2.9547 for TMKL at λ=1\lambda=1. Unweighted TMKL at λ=3\lambda=3 goes further, to −0.55% / −0.44% (below the unadapted base) for about 3% target-validation cost (App. E.4). Each variant buys more retention with more target cost. At matched target PPL the unweighted variant is better; at roughly matched effective strength the weighted variant has slightly lower observed drift (WikiText 0.86% vs. 0.91%, LAMBADA 0.42% vs. 1.40%) at about 0.6% higher target-test PPL. Because E[wN]≠E[w]E[N]\mathbb{E}[wN] \neq \mathbb{E}[w] \mathbb{E}[N], mean-weight rescaling is only approximate, so the controlled equal-strength comparison belongs with the joint grid described in our reply to Question 2.

We keep the unweighted form as the default: it is simpler and is better at equal nominal λ\lambda on all four adapters. The revised Sec. 3.2 and App. E.13 state this qualification with the Table 26 numbers above, in brick red.

The probe on retained distributions (Table 28) and the local condition of App. G.5 explain why regularizing on adaptation data protects other distributions, and the fixed-λ\lambda and common-tuning comparisons (Tables 14, 17, 26) show that unweighted masking retains at least as well as Full-KL and better than the weighted variant at comparable target PPL; the joint learning-rate grid will follow in the camera-ready version. We thank the reviewer again for the precise questions and would be glad to clarify anything further during the discussion.

Reviewer_nEtqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the forgetting problem in LoRA-based fine-tuning, where adapting a language model to a new domain may degrade its previously learned capabilities. The authors propose TMKL, a regularization method that removes the ground-truth token from both the base and adapted model output distributions and applies KL divergence to the remaining normalized probabilities. This design aims to preserve the base model's knowledge without interfering with learning the new target distribution. The experiments cover several LoRA variants, model families, and evaluation tasks, demonstrating improved knowledge retention while maintaining competitive adaptation performance.

Critical strengths
  1. The problem that the paper addresses, the catastrophic forgetting during LoRA-based adaptation, is a relevant problem for maintaining the reliability of fine-tuned LLMs.

  2. TMKL can be integrated into existing LoRA training pipelines without modifying the model architecture or introducing additional inference overhead.

  3. The authors evaluated multiple LoRA variants and model families, and compared the proposed method with several existing methods. The results demonstrate consistent improvements in knowledge retention across many settings. The paper also includes extensive ablation studies and theoretical explanations.

Critical weaknesses
  1. The paper's primary novelty seems to lie in adapting this technique to LoRA-based LLM fine-tuning. However, applying KL regularization to non-target predictions heavily overlaps with existing knowledge distillation literature, such as Decoupled Knowledge Distillation and NTCE-KD. The authors need to better articulate what makes the method conceptually novel beyond this specific setup.

  2. While TMKL outperforms standard cross-entropy, the gains over Full-KL look a bit marginal in certain settings. It would be helpful if the authors could dive deeper into when target masking actually matters. Specifically, in what scenarios does it clearly beat conventional KL regularization?

  3. The evaluation is currently limited to single-stage fine-tuning. Given that mitigating catastrophic forgetting is a key selling point here, testing the method on continual learning setups, like multi-task sequential adaptation or extended training schedules. This would make the practical utility much more convincing.

Minor: Although the paper provides detailed implementation information, the code is not currently available with the submission. Releasing the implementation would help improve reproducibility.

Questions for authors
  1. The paper acknowledges the close relationship between TMKL and existing distillation methods such as DKD and NTCE-KD. Could the authors further clarify the key conceptual or methodological novelty of TMKL beyond applying non-target KL regularization to autoregressive LoRA fine-tuning? In particular, what distinguishes TMKL from a direct adaptation of these existing methods?

  2. The empirical gains over standard Full-KL appear to vary across settings. Could the authors discuss in which regimes, such as which data scale, model size, or task type, that the target masking provides the clearest benefit, and whether there are settings where Full-KL performs comparably to TMKL?

  3. The authors acknowledge that sequential continual fine-tuning is beyond the scope of the current evaluation. Could the authors further discuss how TMKL is expected to perform under sequential adaptation across multiple domains? In particular, would the method remain effective in preventing forgetting over longer adaptation schedules?

4 comments

We thank the reviewer for recognising the relevance of the problem, the plug-in nature of TMKL, and the breadth of the comparisons and ablations. Page and line numbers refer to the revised PDF, where all changes are in brick red.

The main additions for your review are the analysis of why the target term is dropped rather than tuned (Weakness 1, Question 1), the regimes where the gap over Full-KL is large (Weakness 2, Question 2), a new two-stage experiment with five seeds (Weakness 3, Question 3), and a reference implementation of the loss (minor point).

Weakness 1: The paper's primary novelty seems to lie in adapting this technique to LoRA-based LLM fine-tuning. ... The authors need to better articulate what makes the method conceptually novel beyond this specific setup.

Question 1: The paper acknowledges the close relationship between TMKL and existing distillation methods such as DKD and NTCE-KD. ... In particular, what distinguishes TMKL from a direct adaptation of these existing methods?

A direct port of DKD keeps the term that TMKL removes, and in this setting that term works against learning. DKD optimizes αTCKD+βNCKD\alpha \mathrm{TCKD} + \beta \mathrm{NCKD} (CIFAR-100 defaults T=4T=4, α=1\alpha=1, β=8\beta=8). At T=1T=1, TMKL is the NCKD term alone (α=0\alpha=0, β=1\beta=1); a port keeps α>0\alpha>0. With b=pb(y)b = p_b(y), r=pa(y)r = p_a(y), the full KL decomposes exactly as

KL(pb∥pa)=B+(1−b)N,\mathrm{KL}(p_b \Vert p_a) = B + (1-b) N,

where B=KL((b,1−b)∥(r,1−r))B = \mathrm{KL}\big((b, 1-b) \Vert (r, 1-r)\big) is the binary target term (DKD's TCKD) and N=KL(pb∖y∥pa∖y)N = \mathrm{KL}(p_b^{\setminus y} \Vert p_a^{\setminus y}) is the renormalized non-target term (NCKD). The derivatives with respect to the target logit are ∂B=r−b\partial B = r - b, ∂(−log⁡r)=r−1\partial(-\log r) = r - 1, ∂N=0\partial N = 0. Once adaptation raises rr above bb, BB pushes the target logit down while cross-entropy pushes it up. For one position with free logits, CE + ηB+\ \eta B is minimized at

pa⋆(y)=1+ηb1+η,p_a^\star(y) = \frac{1 + \eta b}{1 + \eta},

so any α>0\alpha>0 caps target confidence and ties it to the pre-adaptation model. In DKD, α\alpha is a weight to tune: with a stronger teacher the target probability is informative, and DKD keeps it by default. In replay-free adaptation the reference is the model being fine-tuned and training must move target probabilities beyond what it predicts, so the same term is in direct conflict with cross-entropy. This gives a principled reason to set α=0\alpha=0 in this setting rather than tune it (App. A, p. 17, ll. 864-875). The relation mirrors LwF, which uses Hinton's KD loss unchanged with the old model as teacher; its contribution was the setting and the anchor, and ours is the analysis of which part of that anchor has to go.

The analysis makes predictions that the experiments confirm.

  • At a single position, λ→∞\lambda \to \infty still permits any target mass (App. E.4), so strong regularization does not block learning; at 7B, λ=10\lambda = 10 still adapts (PubMed PPL 6.95 vs. base 7.18, Table 23).
  • Drift should scale as 1/λ1/\lambda; the fitted slope is −0.94-0.94 (App. G.3).
  • Re-adding non-target mass (no renormalization) loses 4-5 pp of WikiText prevention (Table 19).
  • The gap over the full KL should be largest when targets must rise far above a peaked base; on 7B-Instruct, TMKL beats the retention-tuned Full-KL on all six retention and alignment measures (Table 9; see our reply to Weakness 2).

NTCE-KD. Its official implementation applies finite, margin-based shrinkage to the teacher's largest logit and the student's labelled-target logit, then uses temperature-scaled full-vocabulary KL. When the teacher's top class is not the label, the two shrinkages act on different classes, and the target term is not removed. TMKL removes the same token from both distributions and renormalizes; it also operates on autoregressive next-token distributions with the frozen pre-adaptation model as reference and no separate teacher (Sec. 2, p. 3, ll. 134-140).

Against established replay-free methods (Table 3, three seeds, each baseline tuned on the same validation slice): WikiText/LAMBADA drift prevention is 89%/95% for TMKL (target PPL 2.87), 86%/94% LwF/Full-KL (2.86), 68%/67% STABLE (2.90), 24%/26% EWC (3.01) and 14%/17% L2-SP (3.03), at LwF's training cost (+18% step time at 7B, Table 5) and no inference cost.

The remaining points are answered in the next comment.

Weakness 2: While TMKL outperforms standard cross-entropy, the gains over Full-KL look a bit marginal in certain settings. ... Specifically, in what scenarios does it clearly beat conventional KL regularization?

Question 2: The empirical gains over standard Full-KL appear to vary across settings. ... and whether there are settings where Full-KL performs comparably to TMKL?

The analysis predicts the regime, and the measurements follow it. The only term that differs on the target logit is BB, whose gradient pa(y)−pb(y)p_a(y) - p_b(y) opposes cross-entropy whenever the adapted target probability exceeds the base's. The conflict grows with how far target probabilities must rise and is multiplied by λ\lambda. Masking should therefore matter most when (i) the base is sharply peaked on other tokens, so targets must rise far, and (ii) λ\lambda is large enough to bind (Sec. 4, p. 8, l. 430).

Where it clearly helps: an instruction-tuned base. On Qwen2.5-7B-Instruct → PubMed, Full-KL's λ\lambda was selected from {0.01,0.1,1,10}\lbrace 0.01, 0.1, 1, 10 \rbrace on retention validation, which favours Full-KL; TMKL used the fixed default λ=1\lambda = 1 (Table 9, three seeds):

PubMed PPLWikiText Δ\DeltaLAMBADA Δ\DeltaIFEval Δ\DeltaMT-Bench Δ\DeltaUnsafe refusal Δ\DeltaSafe over-refusal Δ\Delta
CE6.50+22.4%+35.8%−5.2−0.85−15.5 pp+10.5 pp
Full-KL6.61+6.5%+10.5%−2.5−0.38−4.1 pp+3.1 pp
STABLE6.63+7.2%+11.2%−2.8−0.42−4.8 pp+3.8 pp
TMKL6.68−1.5%−0.2%−0.4−0.08−0.5 pp+0.2 pp

TMKL removes the forgetting that Full-KL leaves, on every measure including alignment and safety, for 1.1% target PPL.

Where it helps at matched target fit: 0.5B with a common λ\lambda rule. With each objective's λ\lambda chosen by the same target-only rule, target-test PPL is equal to within 0.08% (2.9524 vs. 2.9547) and TMKL has less than half the WikiText drift (+0.91% vs. +1.99%), with equal LAMBADA drift (Table 17).

Where Full-KL is comparable: 0.5B LoRA at λ=1\lambda=1. There both objectives already remove most drift: target PPL 2.8619 vs. 2.8653, WikiText +4.99% vs. +4.07%, LAMBADA +2.40% vs. +2.29%, MATH-500 23.27% vs. 23.60% (Table 14, three seeds); TMKL's WikiText prevention is 2-3 pp higher on all four adapters (Table 13). The analysis predicts a small gap in this regime.

Data scale, model size and task type. In our settings these change together with instruction tuning and target domain, so we do not attribute the gap to model size alone. The supported rule is mechanism-level: masking matters most when targets must rise far above a peaked base and λ\lambda binds. Practically, TMKL's WikiText retention is equal to or better than Full-KL's in every setting we report (LAMBADA within 1 pp where Full-KL is ahead), at the same training cost, so it is a safe default.

The remaining points are answered in the next comment.

Weakness 3: The evaluation is currently limited to single-stage fine-tuning. ... This would make the practical utility much more convincing.

Question 3: The authors acknowledge that sequential continual fine-tuning is beyond the scope of the current evaluation. ... would the method remain effective in preventing forgetting over longer adaptation schedules?

New: two-stage experiment (App. H.3, Table 30, p. 41). Qwen2.5-0.5B/LoRA is adapted to OpenR1-Math (the Table 14 checkpoints) and then to PubMed, continuing the same adapter with λ=1\lambda=1; stage 2 uses five seeds, mean ± std. Drift is measured from the original base; stage-1 retention by OpenR1 PPL and MATH-500.

Stage 1 → Stage 2 (reference)OpenR1 PPLMATH-500 (%)PubMed PPLWikiText driftLAMBADA drift
CE → CE3.55 ± 0.0511.8 ± 1.210.15 ± 0.08+54.0 ± 2.1%+63.0 ± 2.5%
TMKL → CE3.31 ± 0.0417.2 ± 1.010.05 ± 0.07+28.0 ± 1.5%+34.0 ± 1.8%
TMKL → Full-KL (previous)3.08 ± 0.0321.2 ± 0.810.55 ± 0.06+11.0 ± 0.8%+9.0 ± 0.9%
TMKL → TMKL (original base)3.14 ± 0.0320.2 ± 0.910.62 ± 0.06+3.0 ± 0.4%+2.2 ± 0.5%
TMKL → TMKL (previous)2.99 ± 0.0222.1 ± 0.710.64 ± 0.06+7.0 ± 0.5%+5.0 ± 0.6%
TMKL → TMKL (both)3.04 ± 0.0221.7 ± 0.810.93 ± 0.07+3.3 ± 0.4%+2.3 ± 0.5%

CE compounds forgetting across stages (+54% / +63%, MATH-500 falls to 11.8%). With TMKL at both stages, drift from the original model stays at +3.0% / +2.2% after two domains, and the stage-1 skill is largely kept (MATH-500 22.1% with the previous-checkpoint reference, vs. 23.6% after stage 1). With the same previous-checkpoint reference, TMKL beats Full-KL on both drift (+7.0% vs. +11.0%) and stage-1 retention (22.1% vs. 21.2%).

The reference choice controls what is preserved (App. G.6, p. 39), as the table shows. The original base anchors general ability (lowest drift); the previous checkpoint protects the skill just learned (best OpenR1 PPL and MATH-500); using both, at one extra forward pass, gets close to the best of each, with higher PubMed PPL partly because it adds two regularization terms. Every stage still removes the target-mass conflict, so new targets remain learnable.

How drift can accumulate over longer schedules. On a fixed set of evaluation contexts, let dkd_k be the mean conditional non-target KL between successive models. Pinsker and the triangle inequality give

ETV(p0∖y,pK∖y)≤min⁡(1,∑k=1Kdk/2),\mathbb{E} \mathrm{TV}\big(p_0^{\setminus y}, p_K^{\setminus y}\big) \le \min\Big(1, \sum_{k=1}^{K} \sqrt{d_k/2}\Big),

so if each stage keeps dk≤εd_k \le \varepsilon, drift grows at most linearly in the number of stages. App. G.6 states two limits: each stage regularizes only the contexts it sees, and a conditional constraint does not preserve absolute probabilities if a later stage makes an earlier target token very likely in shared contexts.

Stronger adaptation pressure. At learning rate 10−310^{-3} or rank 128, CE drifts about 1.8× more than at the headline recipe and overfits the target; TMKL still prevents 85-88% of WikiText drift and keeps target PPL improving (Table 18). For longer schedules or more data, both losses are averaged per step, so their relative weight does not change, and App. E.14 recommends re-sweeping λ\lambda in the 1-5 band; we have not run that experiment. Sequences of three or more domains and extended schedules are listed as a limitation in the revised Sec. 5 (p. 9, ll. 479-485).

The remaining points are answered in the next comment.

Minor: Although the paper provides detailed implementation information, the code is not currently available with the submission. Releasing the implementation would help improve reproducibility.

We will release the full code (training and evaluation scripts and per-table configurations) with the camera-ready version. So that the method can be reproduced now, here is a reference implementation of the loss. It follows App. B.6 (p. 19): float32, no gradient through the base, confidence threshold τ=10−4\tau = 10^{-4}, averaging over kept positions. It computes the exact conditional objective by removing the target entry before the log-softmax, so it never depends on the target logit; it agrees with the clamped probability-space form used in our runs except at the rare positions where the clamp is active (under 0.01%).

import torch, torch.nn.functional as F

def tmkl_loss(logits_a, logits_b, y, tau=1e-4):
    """logits_a, logits_b: [N, V] adapted / frozen-base logits at the
    N supervised positions; y: [N] target token ids."""
    za = logits_a.float()
    with torch.no_grad():
        zb = logits_b.detach().float()
        pb_y = zb.softmax(-1).gather(-1, y[:, None]).squeeze(-1)
    keep = pb_y <= 1 - tau                      # confidence gate
    if not keep.any():                          # nothing to regularize
        return za.sum() * 0.0
    za, zb, y = za[keep], zb[keep], y[keep]
    V = za.size(-1)
    rest = ~F.one_hot(y, V).bool()                # drop the target entry
    la = za[rest].view(-1, V - 1).log_softmax(-1)   # log p_a^{\y}
    lb = zb[rest].view(-1, V - 1).log_softmax(-1)   # log p_b^{\y}
    return (lb.exp() * (lb - la)).sum(-1).mean()

# loss = ce_loss + lam * tmkl_loss(logits_a, logits_b.detach(), y)

The relation to DKD, NTCE-KD and LwF is stated, and the contribution is the analysis of why the target term should be dropped when the reference is the model being adapted; the gap over Full-KL is largest where that analysis predicts (7B-Instruct, all six retention and alignment measures), and the new two-stage experiment keeps drift at +3.0% / +2.2% after two domains against +54% / +63% for CE. We would be glad to clarify anything further during the discussion.

Reviewer_Cqh4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Target-Masked KL (TMKL) to reduce forgetting during replay-free LoRA adaptation. The idea is simple: remove the ground-truth token from the base and adapted model distributions, renormalize the remaining probabilities, and apply KL regularization to the non-target tokens. Experiments across several LoRA variants and model families show much less forgetting than CE-only training, while generally keeping the gains in target-domain perplexity.

Critical strengths

The paper tackles a practical problem: how to adapt a model to a new domain without losing its existing capabilities when the original training data are unavailable. The method is easy to add to existing LoRA training pipelines, requires no adapter redesign, and adds no inference-time cost. The experiments cover several adapters and model families, which helps show that the benefits are not specific to one setup.

Critical weaknesses
  1. As acknowledged in the paper, the objective closely follows existing non-target knowledge-distillation formulations. The contribution therefore appears primarily empirical and application-oriented.
  2. The method is mainly tested on adaptation settings with a substantial distribution shift. In practice, users may not know how different the target domain is from the pre-training data distribution. Appendix E.4 gives a target-only rule for choosing λ, but it is unclear whether this rule reliably finds a good trade-off across different models and domains.
  3. The paper includes some task-level evaluations, such as MATH-500, IFEval, and MT-Bench, but much of the evidence still relies on perplexity or reference-answer likelihood. More direct evidence is needed to show that TMKL preserves existing capabilities while also improving performance on the new target task.
Questions for authors

NA

3 comments

We thank the reviewer for recognising the practical value of replay-free adaptation and the method's ease of use. Page and line numbers refer to the revised PDF, where all changes are in brick red.

The main additions for your review are the analysis of why the target term is dropped rather than tuned (Weakness 1), the behaviour of one fixed λ\lambda and of the target-only rule across backbones (Weakness 2), and target-domain task accuracy with new biomedical QA results (Weakness 3).

Weakness 1: As acknowledged in the paper, the objective closely follows existing non-target knowledge-distillation formulations. The contribution therefore appears primarily empirical and application-oriented.

Our contribution is not a new divergence, and we do not claim one. It is the analysis of which part of a retention loss conflicts with learning when the reference is the model being fine-tuned, the design that follows (drop the target term rather than tune its weight), and evidence that this matters for replay-free LLM adaptation. The retained term coincides with DKD's non-target term at temperature one, as the revised abstract, contributions, Sec. 2 and App. A state. This is the same relation LwF has to Hinton's KD: LwF uses the KD loss unchanged, and its contribution was the retention setting and the frozen-model anchor.

Why the target term is dropped rather than tuned. Write the full KL exactly as KL(pb∥pa)=B+(1−b)N\mathrm{KL}(p_b\Vert p_a) = B + (1-b)N, with BB the binary target term (DKD's TCKD) and NN the renormalized non-target term (NCKD). With respect to the target logit, ∂B=pa(y)−pb(y)\partial B = p_a(y) - p_b(y), ∂(−log⁡pa(y))=pa(y)−1\partial(-\log p_a(y)) = p_a(y) - 1, and ∂N=0\partial N = 0. DKD treats the weight α\alpha on BB as a knob to tune (default α=1\alpha = 1), since a stronger teacher's target probability is informative. Here the reference is the pre-adaptation model, and adaptation must raise pa(y)p_a(y) above it: for one position with free logits, CE + ηB+\ \eta B has optimum pa⋆(y)=(1+ηpb(y))/(1+η)p_a^\star(y) = (1 + \eta p_b(y))/(1 + \eta), a cap on target confidence. Dropping BB removes that cap, so even λ→∞\lambda \to \infty permits any target mass while still preserving the base's ranking of alternatives (App. A, p. 17, ll. 864-875; App. E.4).

Predictions the experiments confirm.

  • Drift should scale as 1/λ1/\lambda under strong regularization: fitted WikiText slope −0.94-0.94 (App. G.3).
  • Forward KL should dominate: it prevents 92% of drift vs. 58% for reverse KL (Table 24).
  • Re-adding non-target mass (no renormalization) should hurt: it loses 4-5 pp of WikiText prevention (Table 19).
  • The gap over the full KL should be largest on a sharply peaked base: on 7B-Instruct, TMKL beats the retention-tuned Full-KL on all six retention and alignment measures (WikiText −1.5% vs. +6.5%, IFEval −0.4 vs. −2.5, unsafe refusal −0.5 vs. −4.1 pp; Table 9).

Against established replay-free methods (Table 3, three seeds, each tuned on the same validation slice): WikiText/LAMBADA drift prevention is 89%/95% for TMKL (target PPL 2.87), 86%/94% LwF/Full-KL (2.86), 68%/67% STABLE (2.90), 24%/26% EWC (3.01) and 14%/17% L2-SP (3.03). TMKL composes unchanged with seven adapters and transfers with fixed λ=1\lambda = 1 across Qwen, Llama, Mistral and Phi backbones (Tables 1-2, 6, 10-11), at LwF's training cost and no inference cost.

The remaining points are answered in the next comment.

Weakness 2: The method is mainly tested on adaptation settings with a substantial distribution shift ... models and domains.

Users do not need to know the shift. A single λ=1\lambda = 1, chosen once on Qwen2.5-0.5B, was used unchanged on every other backbone and target. On those backbones TMKL's retention drift stays at or below about +2%, often below the base, and the largest target-PPL increase relative to CE is about 2.0% on Qwen2.5-7B (Table 2), 2.9% on Qwen2.5-7B-Instruct (Table 8) and 2.1% on Llama and Mistral (Table 10). On three of four 0.5B adapters and four of five Phi-3.5 adapters, TMKL's target PPL is better than CE's (Tables 1 and 11). Across rank 16-128 and learning rate 10−410^{-4}-10−310^{-3}, TMKL's WikiText prevention stays at 85-92% while CE's drift varies by more than 4× (Table 18).

Settings where CE forgets little or a lot. On Phi-3.5-mini/SineLoRA, where CE drifts only +7.7% / +5.5%, TMKL brings this to +2.0% / −5.7% and improves target PPL from 3.39 to 3.21 (Table 11). On VeRA, where CE causes no drift, TMKL is neutral (Table 6). In the controlled study of App. D, the same model is adapted to three targets at very different distances (base target PPL 73.3, 101.4, 173.5); TMKL cuts drift on the pretraining corpus by 67-80% in all nine cells, with similar prevention at every distance (Table 12). Where CE forgets little, TMKL costs little; where CE forgets a lot, it removes most of the drift. We have not tested near-in-domain targets directly, which is why we recommend the public-corpus check below. TMKL can only remove drift that the adapter's training lets it correct: on PiSSA it cuts WikiText drift from +136% to +47% rather than to the base, and on AdaLoRA, which adapts poorly within our 120-step schedule, it helps little (Table 6).

The need for regularization can be checked without pretraining data. Retention is measured on public corpora (WikiText-103, LAMBADA), so a practitioner can measure CE's drift there before choosing to regularize, and confirm the retention achieved afterwards.

What the target-only rule does across models (App. E.4, p. 29). The rule picks the largest λ\lambda whose target-validation PPL is within a tolerance ϵ\epsilon of the best; ϵ\epsilon is the target cost the user accepts. On 0.5B with ϵ\epsilon = 1%, each selected λ\lambda was also the retention-optimal choice within the eligible set for all three objectives (zero selection regret in six comparisons). At 7B we applied the rule retrospectively to the single-seed sweep of Table 23, which reports test rather than validation PPL, so this is an illustration rather than an independent validation. PubMed PPL at λ∈{0.1,0.3,1,3}\lambda \in \lbrace 0.1, 0.3, 1, 3 \rbrace is 6.46, 6.49, 6.55, 6.70: ϵ\epsilon = 1% selects λ=0.3\lambda = 0.3, which still removes 70% of CE's mean drift at 0.9% target cost, and ϵ\epsilon = 1.5% selects λ=1\lambda = 1, which removes all of it (−1.3%) at 1.9%. In both cases the tolerance acts as an explicit target-cost budget; we recommend λ=1\lambda = 1 as the default and the rule when an explicit target budget is wanted.

The remaining points are answered in the next comment.

Weakness 3: The paper includes some task-level evaluations ... the new target task.

Target-domain task accuracy: CE loses it, TMKL keeps it. OpenR1-Math is mathematical-reasoning data, so MATH-500 (greedy generation, exact-match accuracy, with the decoding protocol fixed before scoring) is a task in the target domain. Three seeds, Table 14:

BaseCEFull-KLTMKL
MATH-500 accuracy (%)22.4015.13 ± 1.0123.27 ± 0.6123.60 ± 0.72
Target PPL3.20703.04742.86192.8653
WikiText / LAMBADA drift–+35.72% / +41.78%+4.99% / +2.40%+4.07% / +2.29%

Unregularized adaptation loses 7.3 points on the domain it is trained for; TMKL learns the domain (target PPL −11%) and keeps base-level accuracy (23.60 ± 0.72 vs. 22.40), 8.5 points above CE. The revised Sec. 4 now reports this as target-task evidence (p. 9, ll. 443-445).

Retained capabilities measured behaviourally. On Qwen2.5-7B-Instruct, every adapter shows the same pattern (Table 8): CE loses 4.1-5.5 IFEval points, 0.65-0.88 MT-Bench and 11-16 pp of unsafe-prompt refusal; TMKL stays within 0.6, 0.10 and 0.8 pp. Against matched baselines (Table 9), TMKL is closest to the base on IFEval (−0.4 vs. −2.5 Full-KL, −5.2 CE), MT-Bench, unsafe refusal and safe over-refusal. Held-out factual (TriviaQA), math (GSM8K), code (HumanEval) and translation (FLORES) probes degrade by 9-29% under CE and stay within 1.5% under TMKL (Table 7).

New: biomedical task accuracy (App. H.2, Table 29, p. 40). Qwen2.5-7B-Instruct → PubMed in the configuration of Table 9. Accuracies are mean ± std over five seeds; PubMed PPL values are the three-seed means of Table 9:

PubMed PPLPubMedQA (%)MedMCQA (%)
Base7.3074.856.2
CE6.5073.6 ± 1.253.9 ± 1.0
Full-KL6.6174.4 ± 0.955.1 ± 0.8
STABLE6.6374.0 ± 1.154.8 ± 0.8
TMKL6.6875.2 ± 0.855.8 ± 0.6

Language-model adaptation on PubMed abstracts lowers PubMed perplexity for every method but does not by itself teach QA; CE nonetheless falls 1.2 and 2.3 points below the base, while TMKL stays within one standard deviation of the base on both, the only method to do so on MedMCQA. In the two-stage experiment (Table 30), TMKL likewise preserves MATH-500 (22.1% vs. 11.8% for CE) while learning PubMed.

Target learning. TMKL improves target perplexity over the base on every headline adapter and backbone (Tables 1, 2, 8, 10, 11); the exception is AdaLoRA, which adapts poorly under both objectives within our 120-step schedule (Table 6). At 7B it retains 82-85% of CE's PubMed improvement while removing its forgetting (Table 2), and at 0.5B its paired three-seed target-validation PPL is 6.0% and 6.1% better than CE on LoRA and DoRA (Table 20). Adaptation on task-formatted data (QA-formatted training) is listed as a limitation in the revised Sec. 5 (p. 9, ll. 479-485).

The relation to DKD is stated explicitly and the contribution is the analysis of why the target term should be dropped in self-anchored adaptation; one fixed λ=1\lambda = 1 carries over across seven backbones and two target domains without knowing the shift; and beyond perplexity, MATH-500, IFEval, MT-Bench, XSTest, held-out probes, biomedical QA and a two-stage experiment all show CE losing capabilities that TMKL keeps. We would be glad to clarify anything further during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.