- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies how LoRA-based adaptation balances new learning against forgetting in a replay-free setting. It proposes Target-Masked KL (TMKL), a one-line, adapter-agnostic regularizer. At supervised token positions, TMKL removes the target token from the frozen base and adapted models’ output distributions and renormalizes the remaining probabilities. It then applies KL regularization from the base to the adapted non-target distribution to preserve relative preferences among alternative tokens. The reported experimental results show reduced perplexity degradation on WikiText-103 and LAMBADA compared with cross-entropy-only training, while maintaining competitive target-domain perplexity. Additional validation includes multiple model families, instruction-tuned models, broader retention metrics, and ablation studies.
Critical strengths
- The proposed TMKL is easy to understand and makes a well-motivated distinction between the supervised target token and non-target tokens.
- The proposed method provides a simple way to regularize adaptation using the frozen base model’s predictions on the new task data. It can be added to the cross-entropy loss without changing the adapter architecture or adding inference-time overhead.
- Tables 1-2 show that TMKL substantially reduces perplexity degradation on WikiText-103 and LAMBADA compared with cross-entropy-only training, while maintaining competitive target-domain perplexity. The results across LoRA, DoRA, SineLoRA, and RandLoRA provide useful empirical support for TMKL’s ability to balance adaptation and retention across several adapter designs.
Critical weaknesses
- The connection between adaptation-domain regularization and retention on other distributions is insufficiently established. TMKL constrains non-target distributions only in adaptation contexts. Although the evaluations provide useful empirical evidence, Table 27 reports non-target KL only on held-out OpenR1-Math.
- TMKL improves retention, but it remains unclear how much of the gain comes from target masking and weight removal, and how much comes from optimization settings or regularization strength. More controlled comparisons would help establish the contribution of these specific design choices.
Questions for authors
- Why should preserving the base model’s relative preferences among non-target tokens on the adaptation data help retain performance on other distributions, such as WikiText-103? What evidence supports this connection beyond the reported retention scores, and under what conditions should it hold?
- Could you isolate the benefit of target masking by comparing Full-KL with weighted TMKL? After tuning learning rates and regularization strengths under a common protocol, does target masking improve retention at comparable target-domain perplexity? Do you also have experimental results for a similarly tuned CE baseline?
- Table 17 shows lower retention drift for the weighted variant at its selected , with slightly higher target PPL. Do you have experimental comparisons between weighted and unweighted TMKL at comparable target-domain PPL?
7 comments
Weakness 2: TMKL improves retention, but it remains unclear how much of the gain comes from target masking and weight removal, and how much comes from optimization settings or regularization strength. ...
Question 2: Could you isolate the benefit of target masking by comparing Full-KL with weighted TMKL? ... Do you also have experimental results for a similarly tuned CE baseline?
On the same positions, with :
Full-KL vs. weighted isolates the binary term ; weighted vs. unweighted isolates the weight, which also changes effective strength (mean ). The paper contains both controls, at fixed strength and under a common tuning rule.
At fixed nominal (Qwen2.5-0.5B → OpenR1, LoRA, three seeds; Tables 14 and 26).
| Objective | Target PPL | WikiText drift | LAMBADA drift | Source |
|---|---|---|---|---|
| Full-KL () | 2.8619 | +4.99% | +2.40% | Table 14 |
| TMKL () | 2.8653 | +4.07% | +2.29% | Table 14 |
| Weighted TMKL () | 2.86 | +9% | +7% | Table 26 |
| TMKL () | 2.87 | +4% | +2% | Table 26 |
The two tables are separate three-seed suites under the same recipe; unweighted TMKL appears in both and agrees.
At equal target fit, removing alone (Full-KL → weighted) removes regularization strength, and dropping the weight restores it: unweighted TMKL has the lowest drift of the three. The same ordering between weighted and unweighted holds on all four adapters (Table 26: WikiText +2-4% vs. +7-10%, target PPL within 0.02).
Under a common target-only tuning rule (Table 17, three seeds, separate validation split). Each objective's is chosen from by the same preregistered rule:
| Objective | Selected | Target-test PPL | WikiText drift | LAMBADA drift |
|---|---|---|---|---|
| CE | – | 3.2464 | +15.74% | +18.18% |
| Full-KL | 1 | 2.9524 | +1.99% | +1.40% |
| Weighted TMKL | 3 | 2.9715 | +0.86% | +0.42% |
| TMKL | 1 | 2.9547 | +0.91% | +1.40% |
Both masked objectives retain at least as well as Full-KL and better on WikiText; unweighted TMKL does so at Full-KL's target PPL (within 0.08%), weighted TMKL at a 0.65% target cost. What masking buys is that regularization strength can be spent on the non-target term without pulling the target back toward the base.
CE across learning rates (Table 18, LoRA, learning rate to ; single seed, test-set PPL). At every learning rate TMKL removes 85-92% of CE's WikiText drift. CE's best target fit (lr , PPL ) still drifts +14.5%; TMKL at the headline recipe reaches with +4.3%, and at lr TMKL has both better target fit ( vs. ) and 9× less drift. The retention gain is therefore not a consequence of CE's learning rate. Table 17 also includes CE under the common three-seed protocol (target-test PPL 3.2464, drift +15.74% / +18.18%). Sec. 4.2 and App. E.5 now qualify our "roughly doubles the target-PPL drop" statement accordingly: it holds under the shared recipe (2.1×, 1.6×, 2.2× on LoRA, SineLoRA, DoRA), but not against CE's single best learning rate. Table 17 shares one learning rate across objectives, and Table 18 is a single-seed sensitivity sweep rather than a validation-selected, multi-seed CE comparator. A joint learning-rate × grid for CE, Full-KL and weighted TMKL at matched target PPL under the common protocol is the remaining control; we will add it to App. E.4 in the camera-ready version.
The remaining points are answered in the next comment.
Question 3: Table 17 shows lower retention drift for the weighted variant at its selected λ, with slightly higher target PPL. Do you have experimental comparisons between weighted and unweighted TMKL at comparable target-domain PPL?
Yes. At comparable target PPL, unweighted TMKL retains better on every adapter (Table 26, , three seeds, all four adapters):
| Adapter | Target PPL (weighted / unweighted) | WikiText drift (weighted / unweighted) | Mean prevention (weighted / unweighted) |
|---|---|---|---|
| LoRA | 2.86 / 2.87 | +9% / +4% | −77% / −92% |
| SineLoRA | 2.86 / 2.87 | +10% / +4% | −75% / −92% |
| RandLoRA | 2.90 / 2.92 | +7% / +2% | −70% / −94% |
| DoRA | 2.85 / 2.86 | +10% / +4% | −75% / −92% |
Target PPL differs by at most 0.02 while unweighted TMKL removes 15-24 points more of the drift.
Accounting for effective strength. Because the weight averages 0.347, part of this gap is strength rather than weighting, and Table 17 reflects this: at its own selected (effective strength ≈ 1.04) the weighted variant reaches +0.86% / +0.42% at target-test PPL 2.9715, versus +0.91% / +1.40% at 2.9547 for TMKL at . Unweighted TMKL at goes further, to −0.55% / −0.44% (below the unadapted base) for about 3% target-validation cost (App. E.4). Each variant buys more retention with more target cost. At matched target PPL the unweighted variant is better; at roughly matched effective strength the weighted variant has slightly lower observed drift (WikiText 0.86% vs. 0.91%, LAMBADA 0.42% vs. 1.40%) at about 0.6% higher target-test PPL. Because , mean-weight rescaling is only approximate, so the controlled equal-strength comparison belongs with the joint grid described in our reply to Question 2.
We keep the unweighted form as the default: it is simpler and is better at equal nominal on all four adapters. The revised Sec. 3.2 and App. E.13 state this qualification with the Table 26 numbers above, in brick red.
The probe on retained distributions (Table 28) and the local condition of App. G.5 explain why regularizing on adaptation data protects other distributions, and the fixed- and common-tuning comparisons (Tables 14, 17, 26) show that unweighted masking retains at least as well as Full-KL and better than the weighted variant at comparable target PPL; the joint learning-rate grid will follow in the camera-ready version. We thank the reviewer again for the precise questions and would be glad to clarify anything further during the discussion.
We thank the reviewer for the positive assessment and the specific questions. Page and line numbers refer to the revised PDF, where all changes are in brick red.
Two additions answer your questions most directly: a measurement of the regularized quantity on retained distributions that TMKL never sees (new Table 28), and a three-way comparison of Full-KL, weighted TMKL and TMKL at fixed λ and under a common tuning rule.
Weakness 1: The connection between adaptation-domain regularization and retention on other distributions is insufficiently established. ... Table 27 reports non-target KL only on held-out OpenR1-Math.
Question 1: Why should preserving the base model’s relative preferences among non-target tokens on the adaptation data help retain performance on other distributions ... and under what conditions should it hold?
The adapter is shared by every input. TMKL penalizes adapter directions that change the base model's preferences among alternative tokens on adaptation contexts, and directions that change those preferences on general text usually change them on the adaptation text too. We now (i) measure the regularized quantity directly on retained distributions and (ii) state the condition under which the transfer holds.
New: the probe on retained distributions (App. H.1, Table 28, p. 40). Using each position's own next token as y on WikiText-103, LAMBADA and FLORES-200 (never seen in training), five seeds:
Across all three retained distributions TMKL reduces the conditional non-target KL by 90-92% and the full-output KL by 82-86% relative to CE, and Full-KL lies between CE and TMKL on every one. The quantity TMKL regularizes is suppressed on distributions it never sees.
Why (App. G.5, p. 38). Let HT be the curvature of the TMKL loss at the base (the Fisher-weighted Jacobian Gram matrix of App. G.3) and gQ the gradient of mean NLL on another distribution Q. For a small update δϕ with gQ∈range(HT),
ΔlogPPL(Q)≲gQ⊤HT+gQ⋅2L∖y.
Retention on Q is controlled by the TMKL value on adaptation data, scaled by a constant that is small when Q's sensitive directions are excited by the adaptation contexts. It predicts failure when Q depends on directions those contexts never excite, or when updates are large.
Further evidence.
The remaining points are answered in the next comment.