acceptodds
Under review as a conference paper at ICLR 2027

Local Support Learning

Abstract

We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, catastrophic forgetting, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sEptOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Local Support Learning augments each fine-tuning phase's LoRA adapter with gates that restrict its application to activations resembling that phase's training data. Each gate compares a Gaussian mixture fitted to the new task against one fitted to a small generic reference corpus, with temporal smoothing of token-level decisions. Experiments fine-tune Qwen2.5 models on translation, chemistry, and cybersecurity and evaluate retention on GSM8K, HumanEval, and IFEval. The reported results show substantial retention gains while preserving new-task performance, including across three sequential phases.

Critical strengths
  1. The retention effect is substantial and supported by relevant ablations. Figures 4–5 show a large improvement over ordinary LoRA with comparable new-task performance.
  2. The paper examines robustness and practical cost. The hyperparameter sweeps suggest that the effect is not tied to a single learning rate or adapter rank.
Critical weaknesses
  1. The comparisons do not establish the benefit over simpler routing or equally informed retention methods. The single-phase baselines are LoRA, OP-LoRA, and LwF; the sequential comparison uses only LoRA. Although the per-matrix MLP ablation is useful, it does not answer whether a prompt-level domain router enabling an ordinary adapter would obtain similar results, or whether an existing routed-adapter approach would be competitive.
  2. Retention under overlapping distributions and the scope of the theory remain unresolved. The chosen tasks span different domains, but their separation in gate activation space is not established. The evaluation does not systematically vary that overlap or test reformatted task inputs and mixed-domain prompts. Such cases matter because a useful adapter may be rejected, or a prior-task input may activate a newer adapter.
Questions for authors
  1. How does LSL compare with a prompt-level domain router and a relevant routed-adapter method, under the same adapter capacity and reference-data access?
  2. How well do replay or LwF perform when allowed the same generic reference tokens used to fit the negative GMMs?
  3. Can you report gate activation rates and task scores for near-distribution paraphrases, overlapping tasks, and mixed-domain prompts? Does earlier adaptation provide useful forward transfer to a related later task?
4 comments
AuthorsOpenReview ↗

R3 Rebuttal (1/3)

Summary of additions

We thank the reviewer for taking the time to evaluate our work, as well as for their feedback, which helped improve our paper. Below is a summary of the main points addressed in the rebuttal:

  • (Q1, W1) Simpler routing alternatives. New results with single-router baselines: LSL exhibits stronger learning overall. All methods achieve strong retention. Added discussion regarding routing alternatives.
  • (Q1, W1) Alternative routed adapters. Under matched conditions, routed adapters reduce to our MLP-gate ablation, which retains substantially less than LSL (58.95 vs. 71.27).
  • (Q2) LwF and replay with matched reference data. New results showing LSL retains 3.4-6.5 points more than LwF and 14-17 more than replay.
  • (W2, Q3) Overlap tests and gate separation statistics. New results showing that LSL is robust in the requested overlap tests. Gates statitstics are measured and reported.

The full responses are added below. We hope these additions address the reviewers’ concerns, and we will be glad to answer any remaining questions during the discussion period.

.

(Q1, W1) The comparisons do not establish the benefit over simpler routing. How does LSL compare with a prompt-level domain router under the same adapter capacity and reference-data access?

We thank the reviewer for the question. Simpler routers face limitations that, once addressed, either make them more complex or degrade their performance:

  • Routing based on the prompt fixes the gating before any token is decoded. As the reviewer points out in W2, handling reformatted and mixed-domain inputs is important. For such inputs, a single decision per prompt cannot fit every part of the response: for example, when asked to write code that solves a chemistry problem, the chemistry adapter is useful for some tokens but not for others. LSL decides per token at each matrix, and GC2 (global comment) shows that its gates behave well in these cases.

  • A single router, applied per-token must make a decision before the adapters it controls are applied, leaving two options: It can sit at the embedding layer, controlling all adapters within a single pass. Alternatively, since routing should be based on deep, contextual representations, it can run a speculative forward pass per token up to an intermediate layer (the right decision is known only after processing several layers), followed by an additional forward pass with the chosen gating. This increases inference cost and adds a hyperparameter (the chosen layer for the router). In contrast, LSL's inference latency is only 1.21× that of the LoRA baseline (Fig. 8 (left), L393-394). We test both approaches on the 7B model with the same GMM-based gate, matching adapter capacity and reference data. The speculative model uses representations at layer 16 (out of 28) for the gating decision:

Chemistry:

MethodPretrain TasksNew Task
Base Model72.1431.80
Single router (embedding)70.8168.11
Single router (speculative, layer 16)71.4578.67
LSL71.2778.93

Igbo:

MethodPretrain TasksNew Task
Base Model72.1418.90
Single router (embedding)70.8330.18
Single router (speculative, layer 16)71.831.9
LSL71.4536.74

The simplest method, the embedding-layer router, retains comparably but learns substantially worse, since embedding features lack context. The speculative router at layer 16 recovers LSL's performance on chemistry, but not on Igbo translation (31.9 vs. 36.74), despite an additional partial forward pass per token. This may reflect either a single decision being too coarse for all matrices, or a task-dependent choice of router layer. In both cases, LSL avoids the issue by gating each matrix separately, within a single forward pass and without selecting a layer.

Two final remarks:

  • A separate token-level routing model is a discriminative classifier, whose behavior outside its training data is underspecified (Appendix F, L1053-1061). The fix requires a local-support objective like LSL's, so the router now faces the very problem it was meant to solve. It also adds a second model to the system.

  • Oracle bound. On all of our benchmarks, LSL already comes close to an oracle router with access to the domain label of every input (applying the base model on pretraining tasks and LoRA on the new task). Since LSL is already within 1.4% of the oracle on every benchmark, there is little room for any alternative router, including prompt-level ones, to improve upon its scores:

TaskRetention (vs. base)New task (vs. LoRA)
Chemistry98.8% (71.27 / 72.14)100.3% (78.93 / 78.67)
Igbo99.0% (71.45 / 72.14)98.8% (36.74 / 37.18)
Cybersecurity98.6% (71.15 / 72.14)100.2% (87.14 / 86.99)

We will include this discussion, as well as the additional results, in the updated manuscript.

AuthorsOpenReview ↗

R3 Rebuttal (2/3)

(Q1, W1) How does LSL compare with a relevant routed-adapter method, under the same adapter capacity and reference-data access? Although the per-matrix MLP ablation is useful, it does not answer whether an existing routed-adapter approach would be competitive. The comparisons do not establish the benefit over equally informed retention methods.

L759-762 discuss routed-adapter methods such as LoRA-MoE. The main problem with this approach is that it relies on a replay buffer. LoRA-MoE blends world-knowledge (closed-book QA) data by using a 3-million-sample fine-tuning mixture. For comparison, the Igbo translation dataset has about 50,000 samples, roughly 1/60 of that. In a multi-phase setting, such data would need to be blended into every phase.

Appendix F analyzes this approach as a classifier with a “buffer drawn from earlier tasks” (L1062-1065). It shows that full retention is achieved only if the replay buffer covers the entire pretraining distribution, and only if the model can be trained on that buffer in every phase. Both assumptions are infeasible at pretraining scale.

Lastly, when restricted to the same information as LSL (a small sample of generic reference data instead of a large replay mixture), a routed-adapter method reduces to a learned router that chooses, at each matrix, between the adapter and the frozen base weights. This is the configuration of our MLP-gate ablation (L423-427; Fig. 9, right), which we therefore use as the equally-informed routed-adapter baseline. It learns the new task well but retains substantially less (58.95 vs. 71.27 for LSL), as predicted by our analysis of classifiers trained against a generic buffer (L1058-1061).

.

(W1) The single-phase baselines are LoRA, OP-LoRA, and LwF; the sequential comparison uses only LoRA.

The multi-phase comparison extends the single-phase one. Since methods that fail to retain pretraining capabilities after one phase have no access to that data in later phases, their retention is highly unlikely to recover. As each multi-phase run is expensive, we selected LoRA as the reference baseline.

.

(W2) The chosen tasks span different domains, but their separation in gate activation space is not established. The evaluation does not systematically vary that overlap or test reformatted task inputs and mixed-domain prompts. Such cases matter because a useful adapter may be rejected, or a prior-task input may activate a newer adapter.

Overlap: We refer to response GC2 in the global comment, where we evaluate the overlap of reformatted task inputs and mixed-domain prompts, finding that LSL is robust to both failure modes. We thank the reviewers for this feedback and will add these experiments, as well as the discussion, to the manuscript.

Gate statistics: To quantify separation, we report the gate opening rate of each fine-tuning task on its own data and on the pretraining benchmarks. Gate opening rates are for the 7B model (first seed):

GateOwn taskGSM8KHumanEvalIFEval
Chemistry84.9%6.1%8.9%3.7%
Igbo84.6%1.1%1.8%5.0%
Cybersecurity94.3%6.3%6.1%19.3%

Each gate opens on 85-94% of its own task's tokens, compared with mostly 1-9% on pretraining tasks. The highest cross-opening rate, cybersecurity gates on IFEval (19.3%), reflects genuine overlap between two English instruction-following distributions. However, this has only a subtle effect on retention: after training on cybersecurity, LSL retains 98.6% of the base model's pretraining performance.

AuthorsOpenReview ↗

R3 Rebuttal (3/3)

(Q3) Can you report gate activation rates and task scores for near-distribution paraphrases, overlapping tasks, and mixed-domain prompts?

See response GC2 in the global comment, experiment (i) “Some gates open when they should be closed”.

.

(Q3) Does earlier adaptation provide useful forward transfer to a related later task?

This is an interesting question. Studying it requires a sequence of tasks designed to allow measurable forward transfer, which our current tasks do not. We leave this to future work.

.

(Q2) How well do replay or LwF perform when allowed the same generic reference tokens used to fit the negative GMMs?

We thank the reviewer for this suggestion. We add replay and LwF baselines that use the same 1M-token reference data as LSL's gate ("+ reference data"). Base Model, LoRA, LwF, and LSL results are taken from the paper.

Replay with the reference data does not retain pretraining capabilities well, lagging behind LSL by 14.0-17.2 points. This is expected, as the reference data was not sampled from the Qwen pretraining dataset, and, moreover, constitutes only about 0.0000067% of its size (L244). LwF improves with the reference data but still lags behind LSL by 3.4-6.5 points. We will add both baselines to the manuscript, providing an equally-informed comparison.

Igbo:

MethodPretrain TasksNew Task
Base Model72.1418.90
LoRA37.8237.18
LwF54.1537.00
LwF + reference data64.9236.98
Replay + reference data54.2436.89
LSL71.4536.74

Chemistry:

MethodPretrain TasksNew Task
Base Model72.1431.80
LoRA52.4778.67
LwF63.4577.44
LwF + reference data67.9179.32
Replay + reference data57.2975.89
LSL71.2778.93

Lastly, the strongest equally-informed baseline, LwF + reference data, already falls below LSL after the first phase (Igbo: -6.53 points, 64.92 vs. 71.45; chemistry: -3.36, 67.91 vs. 71.27). Same as in the response to W1, since later phases also lack pretraining data, we do not expect it to close this gap in the multi-phase setting.

.

Closing remark

We hope our rebuttal addresses the reviewer's concerns. If the concerns have been resolved, we would appreciate it if the reviewer would consider raising their score. If not, we will be glad to answer any remaining questions during the discussion period.

AuthorsOpenReview ↗

Gentle reminder: rebuttal period closing soon

Dear reviewer sEpt,

We thank you again for evaluating our work. As the rebuttal period will close soon, we would like to know whether you have any remaining concerns, which we will be happy to address. If not, we kindly ask you to consider increasing your rating.

Thank you,
The authors

Reviewer_xMQNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focuses on mitigating catastrophic forgetting by restricting model changes. Specifically, the authors focus on the input space of individual weight matrices and argue that, since parameter updates can affect the mappings of different inputs, updates should ideally be restricted to the support of the current task's data distribution. While this reasoning is intuitive, it would benefit from more convincing justification and analysis. The experiments demonstrate the effectiveness of LSL through comparisons with LoRA, LwF and OP-LoRA, but the limited breadth of the baselines somewhat weakens the overall strength of the empirical evidence. Overall, this paper requires substantial improvements in both its methodology and experiments.

Critical strengths
  1. The paper is well-structured and easy to follow.
  2. The paper addresses the important problem of preserving pretrained capabilities during continual fine-tuning.
  3. The paper proposes a simple and intuitive gating mechanism to mitigate catastrophic forgetting.
Critical weaknesses
  1. The central motivation of this paper is to restrict adapter updates to the local support of each task. However, the theoretical justification for extending this local-support argument to model-level capability preservation remains limited. In particular, local constraints at individual weight matrices do not necessarily guarantee the preservation of capabilities at the level of the entire model. The authors also acknowledge this limitation in their discussion, and further discussion or empirical validation would strengthen the evidence supporting this claim.

  2. My main concern is that the paper does not sufficiently characterize the failure cases and underlying assumptions of LSL (see Questions 3–5). In particular, the paper motivates LSL by linking changes in local input-output mappings to potential forgetting. A more detailed analysis or targeted experiments would help clarify when the proposed locality constraint effectively preserves capabilities and under what conditions it may fail.

  3. The experimental evaluation does not establish whether LSL is competitive with a broader range of relevant methods. I encourage the authors to compare LSL with the most applicable recent approaches. If such comparisons are infeasible, the authors should provide a more convincing justification for excluding these methods. As noted in Lines 299–300, the fact that a method was not originally designed to preserve pretrained capabilities is not, in itself, sufficient reason to exclude it from comparison.

Questions for authors
  1. Corresponding to W1, the authors should clarify whether LSL aims to preserve the model's input-output mapping or its performance on pretrained and previously learned tasks, and explain how the local-support argument supports the latter.

  2. Why change at the level of a single weight matrix is a natural retention objective in large-scale pre-trained model? (Line 080-081). I am curious why this objective is appropriate.

  3. How the method deals with the situation that the current task may overlap with regions used by previous or pretrained capabilities?

  4. Although the paper evaluates diverse finetuning domains, it remains unclear whether the GMM-based gate can reliably distinguish task-specific inputs when activation distributions overlap substantially.

  5. Since the same task may admit multiple solutions with different internal representations, why is the activation support of the observed data an appropriate basis for determining which adapter updates should take effect? Could alternative regions yield a better trade-off between stability and plasticity?

  6. The baseline comparison could be strengthened by including more recent and relevant methods, or by providing a more detailed discussion of closely related approaches [1-5].

[1] Orthogonal subspace learning for language model continual learning, EMNLP 2023

[2] Learn more, but bother less: parameter efficient continual learning, NeurIPS 2024

[3] Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning, ICLR 2026.

[4] Revisiting Weight Regularization for Low-Rank Continual Learning, ICLR 2026.

[5] Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning, ICML 2026.

6 comments
AuthorsOpenReview ↗

R2 Rebuttal (1/4)

Summary of additions

We thank the reviewer for their feedback and suggestions.
Below is a summary of the main points addressed in the rebuttal:

  • (W3, Q6) Additional baselines. New results: O-LoRA ([1]) on the multi-phase setup show that it degrades pretraining capabilities significantly already after the first phase (72.14 → 39.83). A discussion explains why all other suggested [2-5] methods share the same limitation or are covered by existing baselines.
  • (W1, Q1, Q2) Linking gate performance to model retention. Highlighted the relevant empirical results (Figure 10). Added a new token-level analysis showing which types of tokens activate the gates. Added a discussion that further motivates our approach.
  • (W2, Q3, Q4) Overlap, assumptions and failure-mode tests. The reviewer’s main concern. New results show LSL is robust on both tested failure modes. LSL’s assumptions are stated explicitly at the single-gate level (App. F) and are sharpened at the full-model level.

The full responses are added below. We hope these additions address the reviewers’ concerns, and we will be glad to answer any remaining questions during the discussion period.

.

(W3, Q6) The experiments demonstrate the effectiveness of LSL through comparisons with LoRA, LwF and OP-LoRA, but the limited breadth of the baselines somewhat weakens the overall strength of the empirical evidence. I encourage the authors to compare LSL with the most applicable recent approaches. If such comparisons are infeasible, the authors should provide a more convincing justification for excluding these methods [1-5]. As noted in L299-300, the fact that a method was not originally designed to preserve pretrained capabilities is not, in itself, sufficient reason to exclude it from comparison.

We evaluate O-LoRA [1] on the multi-phase finetuning setup (7B model, same phase order as Fig. 5, averaged over three seeds):

PhaseIgbo (task 1)Chemistry (task 2)Cyber (task 3)Pretraining tasks
Zero-shot19.1832.2280.2572.14
+Task 137.8833.6773.6939.83
+Task 236.6877.6776.8244.95
+Task 335.4974.6784.8141.87

Notice that O-LoRA severely degrades pretraining capabilities already after the first task (72.14 → 39.83), before its retention mechanism becomes active: since O-LoRA targets forgetting between fine-tuning phases, its orthogonality constraint applies only from the second task onwards. Like O-LoRA, methods [2], [4], and [5] compute their constraints from previous fine-tuning phases. In the first phase no constraint is active, so each behaves like plain LoRA and is prone to the same loss of pretraining capabilities observed for O-LoRA. We will update L299-300 to reference these results, and will cite the additional methods not already cited ([2], [4], and [5]).

[3] is discussed in L143-145. It belongs to the group of methods that constrain updates to subspaces orthogonal to the top singular vectors of the pretrained weights, for which OP-LoRA is our representative. In L319-321 we show this heuristic fails to prevent forgetting of pretraining capabilities, and in Appendix C (L829-845) we find that this is because forgetting is only weakly tied to the top-k singular-value subspace. Note that [3] itself reports GSM8K falling from 26.1 to 7.7 and BBH from 40.2 to 34.2.

Lastly, L139-152 and L758-774 discuss related work and potential baselines, covering a broad range of relevant approaches: methods that account only for interference between fine-tuning phases, weight-space heuristics, functional regularization, replay buffers, on-policy RL, and lifelong editing. For each group, we explain why it is either represented by one of our baselines or not applicable in our setting.

AuthorsOpenReview ↗

R2 Rebuttal (2/4)

(W1): Local constraints at individual weight matrices do not necessarily guarantee the preservation of capabilities at the level of the entire model. Further discussion or empirical validation would strengthen the evidence supporting this claim.

A formal whole-model characterization is highly challenging and is not provided for many widely adopted deep learning methods [a-d], including in this line of work (e.g., GPM [e]). Likewise, we do not provide one (L477-478). Instead, we provide a thorough empirical validation: our main forgetting results span a variety of tasks, phases, and model scales.

Specifically, Figure 10 provides the empirical evidence tying between the performance of the whole model (left) and the performance of the gates (right). For the paper’s default configuration (32 GMM components, Smoothing On), gates are wrong on tokens of pretraining tasks 5% of the time, and the whole model achieves almost perfect retention of 98.8%. For the tokens of the new task (chemistry instruction-finetuning), gates are wrong about 16% of the time, but the performance with respect to the LoRA baseline is almost identical - showing that LSL learned at full capacity.

To explain this we analyze the data in the new task and find that the 16% misclassification rate is mainly caused by simple english tokens: ones which the pretrained weights can handle without additional finetuning - a possible explanation to the high learning capacity. In contrast, specialized in-domain tokens like SMILES chemistry strings achieve a misclassification rate that is close to 0% - showing that the LSL gates capture the essence of the task faithfully.

FindingExample contextHigh-firing tokens (rate)Low-firing tokens (rate)
Gates fire on domain content, not generic phrasing (chemistry)Assuming absolute perfection, what is the projected yield for the chemical transformation represented by the SMILES code ...c1ccc(Cl)cc1.C1CNCCN1>>Clc1ccc...?>> (1.00), >C (1.00)what (0.00), please (0.00)
False fires on pretraining tasks concentrate on chemistry-related notation (GSM8K)What is the volume of the tank in cubic meters ...?ubic (0.35), Volume (0.22)·grocery (0.00), ·shopping (0.00)

Discussion: For a previous-task token, if all gates are correct (closed) then the output of the model will be identical to the base model. For a new task token, if all gates are correct (open), then the output will be identical to a LoRA trained model. Therefore, in theory, optimal gates lead to a precise model. In practice, we find that the hit rates are not optimal. However, as shown above, the model is able to tolerate imperfect hit rates.

We will include this additional analysis and discussion in the revised manuscript.

.

(Q2): Why change at the level of a single weight matrix is a natural retention objective in large-scale pre-trained model? (Line 080-081)?

Our motivation and analysis build on widely accepted per-layer analyses of forgetting, such as Gradient Projection Memory (GPM) [e] (L166-167, L197-204). Most of the methods suggested by the reviewer, e.g., O-LoRA [1] and Sculpting Subspaces [3], rely on the same per-layer view, restricting each layer's update to a subspace chosen to protect prior knowledge.

The main problem with previous approaches arises in large-scale pretraining settings, where estimating the subspace in which the pretraining distribution lives is infeasible: the pretraining data is too large, and often not even available. In our work, we explore an alternative approach: rather than estimating prior distributions, we guard against the worst-case prior distribution. The theoretical analysis in Appendix F shows that under this setting, a retention objective arises naturally: the optimal gate under the worst-case criterion restricts each update to the support of the new data (L984-992). This is exactly LSL's objective.

AuthorsOpenReview ↗

R2 Rebuttal (3/4)

(W2 - “main concern”): the paper does not sufficiently characterize the failure cases and underlying assumptions of LSL. In particular, the paper motivates LSL by linking changes in local input-output mappings to potential forgetting. A more detailed analysis or targeted experiments would help clarify when the proposed locality constraint effectively preserves capabilities and under what conditions it may fail.

Single-gate level:
Appendix F characterizes possible failure cases and underlying assumptions. In short, we rely on two assumptions:

  • The GMMs are L1-consistent estimators (L1038-1039), i.e., with enough data, the estimated density converges to the true one, which requires the GMM to be expressive enough.
  • Φneg, the negative GMM, provides a suitable input-dependent threshold (L245-252, L1048-1052).

Failure modes such as misspecification (e.g., a distribution that a Gaussian mixture cannot estimate consistently, L1083-1085) are also analyzed in Appendix F.

Full-model level:
One additional assumption follows from Figure 10 (see discussion in our response to W1):

  • The model can tolerate gate misfires and disagreements across matrices.

We support this assumption empirically. In practice, gates rarely agree on every token and every matrix, so all our post-training results (L306-323, Fig. 4; L339-347, Fig. 5) already reflect models with partially-agreeing gates: LSL retains near-full pretraining performance while matching LoRA on the new tasks. The token-level analysis in Figure 10 (see response to W1) further shows where the remaining gate errors occur.

Following the reviewer’s request, we investigate two additional failure modes with targeted experiments at the full-model level. Both can be found in the global comment (GC2). We find that in the tested cases, LSL is robust to the inspected failure modes: the model tolerates prior-task inputs that activate cross-domain adapters, and useful adapters are not rejected on reformatted inputs. We thank the reviewer for this feedback and will add these two experiments, as well as a discussion of overlap, to the manuscript.

.

(Q3): How does the method deal with the situation that the current task may overlap with regions used by previous or pretrained capabilities?

LSL lets newer data determine behavior where supports overlap (L981-983). In these regions, LSL acts like a standard LoRA adapter. We refer to our global comment response GC2, where we test this design choice and find that the model tolerates the overlap well (see GC2, (i) Some gates open when they should be closed).

.

(Q4): Although the paper evaluates diverse finetuning domains, it remains unclear whether the GMM-based gate can reliably distinguish task-specific inputs when activation distributions overlap substantially.

At the single-gate level, where the activation distributions of two tasks overlap, the overlapping inputs lie in both supports, so no gate can distinguish them by definition. LSL then lets the newer data determine behavior (L981-983).
The more relevant question is how the model as a whole handles partial overlap across gates. In our global comment response GC2 ((i) Some gates open when they should be closed), we show that the model with GMM-based gates tolerates such overlap well.

Reviewer_sJkFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper proposes restricting each adaptation phase’s adapter to activation regions occupied by that phase’s training data. A generative density gate, implemented using Gaussian mixtures and a generic reference corpus, controls when the adapter is active. The objective is to preserve both pretrained capabilities and knowledge acquired during subsequent adaptation phases. The introduction, method and conclusion follow this objective consistently. The paper’s strongest contribution is the combination of an interpretable retention principle with a practical implementation and informative ablations.

Critical strengths
  1. The paper clearly explains why small parameter updates can still affect inputs outside the current training distribution, then tests an actionable remedy.

  2. Comparisons between generative and discriminative gates, the alternative locality construction, and smoothing ablations help explain the mechanism rather than merely reporting final performance.

  3. Hyperparameters are selected using new-task validation rather than retention benchmarks. This makes the preservation results more credible.

  4. Although the components are familiar, their combination is well motivated. Limited architectural novelty would not, by itself, prevent acceptance given sufficiently convincing evidence.

Critical weaknesses
  1. The principal comparison gap concerns methods that already localize or route updates. Appendix A explains why several editing methods do not directly match this setting, but it does not establish an advantage over a carefully adapted routed-adapter baseline.

  2. The evaluation is also narrower than the general-purpose framing: model-size experiments remain within Qwen2.5, and the sequential study uses three tasks in a fixed order.

  3. Finally, the theory explicitly permits changing behavior where old and new supports overlap. A generic FineWeb reference is useful, but it does not provide the actual pretrained activation distribution. These qualifications matter when interpreting preservation beyond the selected benchmarks. Adapter storage and routing cost also grow with the number of phases.

Questions for authors
  1. How does LSL compare with a routed-adapter method under the same adapter capacity, reference data and inference budget?

  2. What happens on overlapping or contradictory domains, and across longer streams with several task orders?

  3. How sensitive is retention to the choice of generic reference corpus?

4 comments
AuthorsOpenReview ↗

R1 Rebuttal (Part 1/3)

Summary of additions

We thank the reviewer for evaluating our paper, and appreciate their feedback and suggestions, which improved our work. Below is a summary of the main points addressed in the rebuttal:

  • (W2, Q2) Additional task orders in multi-phase finetuning. New results over three task orders: LSL proves robust to all task orders tested.
  • (W3, Q2) Overlapping domains. New results showing LSL is robust in the requested overlap tests. Added a clarification for LSL’s explicit policy for overlap.
  • (Q3) Ablating the reference corpus. New results: a reference buffer sampled from SlimPajama-6B (instead of Fineweb-edu) causes negligible change in performance.
  • (W1, Q1, W3) Comparisons and complexity. Highlighted experiments with carefully adapted routing methods, and discussed LSL’s computational advantages and challenges.

The full responses are added below. We hope these additions address the reviewers’ concerns, and we will be glad to answer any remaining questions during the discussion period.

AuthorsOpenReview ↗

R1 Rebuttal (Part 2/3)

(W1, Q1) The principal comparison gap concerns methods that already localize or route updates. Appendix A explains why several editing methods do not directly match this setting, but it does not establish an advantage over a carefully adapted routed-adapter baseline. How does it compare under the same adapter capacity, reference data and inference budget?

Our gate ablations in L416-431 and Fig. 9 (right) provide carefully adapted versions of both localizing and routing methods.

GRACE: Union-of-Spheres ("LSL (UoS)") extends GRACE's localization mechanism to full-model, full-dataset fine-tuning. It retains pretraining capabilities nearly as well as LSL, confirming that locality drives retention, but requires a significantly larger amount of memory (5.67GB vs. 13.8MB), as well as a per-gate radius calibration (Appendix G, L1106-1117), making it less attractive in practice.

Routed adapters: the MLP gate ("MLP Gate") is a single-expert LoRA-MoE whose router chooses between the adapter and the frozen base weights at every matrix, a discriminative routing mechanism similar to the one used by WISE. With this gate the model learns the new task well but retains substantially less of its pretraining capabilities (58.95 vs. 71.27 for LSL), as predicted by our analysis of classifiers trained against a generic buffer (L1058-1061). We note that this gate is trained with the same adapter capacity and the same 1M-token generic reference set as LSL (L423-427). The two gates are also similar in size: LSL's gate (a JL projection followed by diagonal Gaussian components) is structurally close to the MLP's two layers, so their inference cost is comparable.

LSL thus improves on both adaptations: it slightly exceeds UoS's retention at a fraction of the memory, and substantially exceeds the MLP gate's retention under matched conditions.

.

(W2) model-size experiments remain within Qwen2.5, and the sequential study uses three tasks in a fixed order.

We report multi-phase finetuning in 3 different task orders, each averaged over 3 seeds.
On average across orders, LSL retains 98.47% of the base model's pretraining performance, compared with 52.32% for LoRA. On the fine-tuning tasks, LSL retains 99.81% of the acquired performance, compared with 96.09% for LoRA. We conclude that LSL is robust to task order.

Pretraining tasks:

OrderMethodBase ModelFinalRetention
Igbo → Chem → CyberLoRA71.5339.4655.17%
LSL71.5370.2398.18%
Cyber → Igbo → ChemLoRA71.5344.8562.71%
LSL71.5370.6498.76%
Chem → Cyber → IgboLoRA71.5327.9639.09%
LSL71.5370.4498.48%

Fine-tuning tasks:

OrderMethodInitial*FinalRetention
Igbo → Chem → CyberLoRA67.0564.6696.44%
LSL66.6966.98100.44%
Cyber → Igbo → ChemLoRA67.4666.1097.98%
LSL66.2865.9899.55%
Chem → Cyber → IgboLoRA67.4663.3293.86%
LSL66.5566.1899.44%

Note that:

  • Initial* is the average task score before any forgetting happens. For example, in Cyber → Igbo → Chem, Initial* averages the cyber score after phase 1, the Igbo score after phase 2, and the chemistry score after phase 3.

  • Final is the average task score after all three phases (averages the cyber score after phase 3, the Igbo score after phase 3, and the chemistry score after phase 3).

  • The three orders are chosen such that (i) each task appears exactly once in each position, and (ii) every ordered pair of tasks occurs, i.e., for any two tasks A and B, some order trains A before B. Thus, if training on B tends to cause forgetting of A, at least one order exposes it.

  • In the Igbo → Chem → Cyber order, LSL’s fine-tuning retention slightly exceeds 100%. This can happen when information from a later task is relevant to an earlier one. The change itself is small - we note this mainly for clarity.

Model family. our experiments use the Qwen2.5 family to enable a controlled scaling study across model sizes. Extending the evaluation to other model families, such as Llama, is a natural next step, which we leave to future work.

AuthorsOpenReview ↗

R1 Rebuttal (Part 3/3)

(Q2) What happens on overlapping or contradictory domains, and across longer streams with several task orders?

Overlap: We refer to GC1 and GC2 in the global comment. GC1 describes how LSL handles overlap at the level of a single gate: where new and prior data overlap or contradict, the newer data determines behavior. GC2 evaluates overlapping domains with targeted experiments at the whole-model level, finding that LSL remains robust in the tested scenarios. We thank the reviewer for this feedback and will add these experiments, as well as the discussion, to the manuscript.

Longer streams: in this work we focused on retaining pretraining capabilities, which we consider the most challenging case, since pretraining data is exceedingly large and typically unavailable. We evaluate up to three sequential finetuning phases across several task orders (see W2). Extending LSL to longer streams is a natural next step, which we leave to future work.

.

(W3) The theory explicitly permits changing behavior where old and new supports overlap. A generic FineWeb reference is useful, but it does not provide the actual pretrained activation distribution. These qualifications matter when interpreting preservation beyond the selected benchmarks.

Capability preservation: our evaluation benchmarks (GSM8K, HumanEval, IFEval) are held out from the reference data, a small random 1M-token sample used only to estimate the width of the pretrained distribution (L245-247). Retention on them thus shows that LSL preserves capabilities beyond this sample. Our results are consistent across three seeds and an alternative reference corpus (see response to Q3). Verifying every possible pretraining capability is infeasible. Therefore, we instead evaluate on three general, diverse, and widely used pretraining benchmarks.

Overlap: as noted in GC1, overlap is inherent to the data and the pretrained model. Since it arises regardless of the retention method, every method requires a policy for it, whether implicit or explicit. LSL lets newer data determine behavior where supports overlap (L981-983). At these regions, LSL acts like a standard LoRA adapter, while elsewhere it preserves the base model's behavior. In contrast, standard LoRA changes behavior in other regions as well.

.

(W3) Adapter storage and routing cost also grow with the number of phases.

We agree, and note this in our Limitations (L474-477). However, LSL offers a favorable trade-off: the cost of retaining pretrained knowledge is relatively low.

First, LSL's memory grows with the number of phases rather than the number of training tokens (Appendix B, L810-812), in contrast to replay buffers or even the UoS gate. Furthermore, LSL's additional storage per phase is negligible: about 6.9MB for the gates, compared with roughly 14GB for the base model weights. The gates even require an order of magnitude less memory than the weight adapters (77MB), which are already considered lightweight.

Second, the routing cost per phase is already modest, adding about 86ms per forward pass over a LoRA adapter in our unoptimized implementation (L393-397), and it leaves room for further optimization. For example, gates of different phases are independent, so they can be computed in parallel with a batched operation. More broadly, the overheads of popular deep learning algorithms such as attention, mixture-of-experts layers, and Adam were substantially reduced by dedicated implementations (e.g., FlashAttention, MegaBlocks, ZeRO). Similarly, we expect dedicated engineering to be beneficial to LSL.

.

(Q3) How sensitive is retention to the choice of generic reference corpus?

Sampling 1 million tokens from a different reference corpus (SlimPajama-6B) changes performance very little. The effect on retention is negligible (+0.22 points for Igbo finetuning, −0.06 for chemistry finetuning), showing that LSL is not sensitive to this choice:

Igbo:

MethodPretrain TasksNew Task
Base Model72.1418.90
LSL (Fineweb)71.4536.74
LSL (SlimPajama)71.6736.85

Chemistry:

MethodPretrain TasksNew Task
Base Model72.1431.80
LSL (Fineweb)71.2778.93
LSL (SlimPajama)71.2179.22

.

Closing remark

We hope our rebuttal addresses the reviewer's concerns. If the concerns have been resolved, we would appreciate it if the reviewer would consider raising their score. If not, we will be glad to answer any remaining questions during the discussion period.

AuthorsOpenReview ↗

Gentle reminder: rebuttal period closing soon

Dear reviewer sJkF,

We thank you again for your positive assessment and constructive feedback. As the rebuttal period will close soon, we would like to know whether our responses address your concerns, and would be happy to clarify any remaining points.

Thank you,
The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Comment

We thank the reviewers for their valuable feedback, as well as for their time evaluating our work.
Several reviewers raised questions about LSL's behavior under overlapping distributions. Since the questions are closely related, we address them jointly here. All other points are addressed in the individual responses.

.

(GC1) How does the method deal with the situation that the current task may overlap with regions used by previous or pretrained capabilities?

LSL handles overlap through an explicit policy, described below. GC2 tests its effect on the full model.

At the single-gate level, no gate can distinguish inputs that lie in the support of both the new and prior distributions. The amount of overlap is a function of the data and the pretrained model's representations, so a retention algorithm must have a policy for it. LSL's policy is to let the newer data determine behavior (L981-983), which allows updating existing knowledge as well as overriding pretrained behavior. Note that a standard LoRA adapter follows the same policy, but applies it to all inputs, whereas LSL applies it only on the support of the new data.

Since each matrix has its own gate and decisions are made per token, data in the intersection of several distributions may produce different gating patterns across matrices and tokens. GC2 (below) evaluates how the model handles such overlaps.

.

(GC2) How does the method behave under reformatted task inputs and mixed-domain prompts? What are potential failure modes?

We investigate two scenarios that have inherent overlap due to a mixed-domain content-style decomposition. The results are for the 7B model, averaged over three seeds, with one training epoch:

(i) Some gates open when they should be closed. The new task shares a prior capability's style but differs in content. We train LSL on chemistry questions reformatted as Python functions ("code-chemistry"), whose style overlaps with HumanEval but whose target behavior conflicts with it, and evaluate HumanEval retention. The control is LSL trained on standard chemistry (first row), which gives the retention without style overlap:

Fine-tuning dataNew TaskHumanEvalRetention (HumanEval)Gate Opening Rate (HumanEval)
Chemistry76.6762.6095.95%6.10%
Code-chemistry77.4462.2095.34%10.99%

The style change causes gate openings on HumanEval to nearly double (6.10% → 10.99%), yet HumanEval retention changes only slightly (95.95% → 95.34%). This further supports that the model can tolerate gate misfires and disagreements, even under non-trivial overlap. For reference, both LSL models learn the new task on par with LoRA (chemistry: 76.67 vs. 77.78; code-chemistry: 77.44 vs. 76.33).

(ii) Some gates are closed when they should be open. The immediate test is to check if the gates generalize to unseen test inputs from the new distribution. To be precise, gates do miss some in-distribution tokens (Figure 10), but we find that this has no significant effect on new-task performance: across all post-training experiments (L306-323, Fig. 4; L339-347, Fig. 5), LSL matches LoRA’s new-task scores on held-out test sets.

More advanced scenarios go beyond the new distribution: for example, training on standard chemistry and evaluating zero-shot on code-chemistry, a content-style combination never seen before. While the model is not explicitly trained on this format, we expect the relevant gates to open so that the model answers correctly despite the change in format.

MethodChemistryZero-shot Code-chemistryGate Opening Rate (Chemistry)Gate Opening Rate (Code-chemistry)
LoRA77.7872.22--
LSL76.6770.3384.39%84.50%

On code-chemistry, LSL scores 70.33 and LoRA scores 72.22. Both come close to their standard chemistry scores (76.67 and 77.78) and far exceed the base model (41.00), showing that generalization to the new style succeeds on most questions. LSL's gates remain open at their in-distribution rate (84.4% vs. 84.5%), indicating that LSL's drop on code-chemistry is not caused by gates wrongly closing, but by the same generalization difficulty that LoRA faces.

To conclude, in the tested scenarios, LSL is robust to both failure modes: the model tolerates prior-task inputs that activate cross-domain adapters, and useful adapters are not rejected on reformatted inputs. We thank the reviewers for this feedback and will add these two experiments, as well as a discussion of overlap, to the manuscript.

Area_Chair_SZETOpenReview ↗

Reminder: Author-reviewer discussion period ending soon

Dear Reviewers,

Thank you again for your time and effort in reviewing this paper. This is a friendly reminder that the author-reviewer discussion period is coming to a close.

If you haven't already, please check out the authors' rebuttal and engage in the discussion by leaving a comment. Even a brief acknowledgment confirming whether your questions were addressed would be very helpful for our final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.