- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Local Support Learning augments each fine-tuning phase's LoRA adapter with gates that restrict its application to activations resembling that phase's training data. Each gate compares a Gaussian mixture fitted to the new task against one fitted to a small generic reference corpus, with temporal smoothing of token-level decisions. Experiments fine-tune Qwen2.5 models on translation, chemistry, and cybersecurity and evaluate retention on GSM8K, HumanEval, and IFEval. The reported results show substantial retention gains while preserving new-task performance, including across three sequential phases.
Critical strengths
- The retention effect is substantial and supported by relevant ablations. Figures 4–5 show a large improvement over ordinary LoRA with comparable new-task performance.
- The paper examines robustness and practical cost. The hyperparameter sweeps suggest that the effect is not tied to a single learning rate or adapter rank.
Critical weaknesses
- The comparisons do not establish the benefit over simpler routing or equally informed retention methods. The single-phase baselines are LoRA, OP-LoRA, and LwF; the sequential comparison uses only LoRA. Although the per-matrix MLP ablation is useful, it does not answer whether a prompt-level domain router enabling an ordinary adapter would obtain similar results, or whether an existing routed-adapter approach would be competitive.
- Retention under overlapping distributions and the scope of the theory remain unresolved. The chosen tasks span different domains, but their separation in gate activation space is not established. The evaluation does not systematically vary that overlap or test reformatted task inputs and mixed-domain prompts. Such cases matter because a useful adapter may be rejected, or a prior-task input may activate a newer adapter.
Questions for authors
- How does LSL compare with a prompt-level domain router and a relevant routed-adapter method, under the same adapter capacity and reference-data access?
- How well do replay or LwF perform when allowed the same generic reference tokens used to fit the negative GMMs?
- Can you report gate activation rates and task scores for near-distribution paraphrases, overlapping tasks, and mixed-domain prompts? Does earlier adaptation provide useful forward transfer to a related later task?
4 comments
R3 Rebuttal (2/3)
(Q1, W1) How does LSL compare with a relevant routed-adapter method, under the same adapter capacity and reference-data access? Although the per-matrix MLP ablation is useful, it does not answer whether an existing routed-adapter approach would be competitive. The comparisons do not establish the benefit over equally informed retention methods.
L759-762 discuss routed-adapter methods such as LoRA-MoE. The main problem with this approach is that it relies on a replay buffer. LoRA-MoE blends world-knowledge (closed-book QA) data by using a 3-million-sample fine-tuning mixture. For comparison, the Igbo translation dataset has about 50,000 samples, roughly 1/60 of that. In a multi-phase setting, such data would need to be blended into every phase.
Appendix F analyzes this approach as a classifier with a “buffer drawn from earlier tasks” (L1062-1065). It shows that full retention is achieved only if the replay buffer covers the entire pretraining distribution, and only if the model can be trained on that buffer in every phase. Both assumptions are infeasible at pretraining scale.
Lastly, when restricted to the same information as LSL (a small sample of generic reference data instead of a large replay mixture), a routed-adapter method reduces to a learned router that chooses, at each matrix, between the adapter and the frozen base weights. This is the configuration of our MLP-gate ablation (L423-427; Fig. 9, right), which we therefore use as the equally-informed routed-adapter baseline. It learns the new task well but retains substantially less (58.95 vs. 71.27 for LSL), as predicted by our analysis of classifiers trained against a generic buffer (L1058-1061).
.
(W1) The single-phase baselines are LoRA, OP-LoRA, and LwF; the sequential comparison uses only LoRA.
The multi-phase comparison extends the single-phase one. Since methods that fail to retain pretraining capabilities after one phase have no access to that data in later phases, their retention is highly unlikely to recover. As each multi-phase run is expensive, we selected LoRA as the reference baseline.
.
(W2) The chosen tasks span different domains, but their separation in gate activation space is not established. The evaluation does not systematically vary that overlap or test reformatted task inputs and mixed-domain prompts. Such cases matter because a useful adapter may be rejected, or a prior-task input may activate a newer adapter.
Overlap: We refer to response GC2 in the global comment, where we evaluate the overlap of reformatted task inputs and mixed-domain prompts, finding that LSL is robust to both failure modes. We thank the reviewers for this feedback and will add these experiments, as well as the discussion, to the manuscript.
Gate statistics: To quantify separation, we report the gate opening rate of each fine-tuning task on its own data and on the pretraining benchmarks. Gate opening rates are for the 7B model (first seed):
| Gate | Own task | GSM8K | HumanEval | IFEval |
|---|---|---|---|---|
| Chemistry | 84.9% | 6.1% | 8.9% | 3.7% |
| Igbo | 84.6% | 1.1% | 1.8% | 5.0% |
| Cybersecurity | 94.3% | 6.3% | 6.1% | 19.3% |
Each gate opens on 85-94% of its own task's tokens, compared with mostly 1-9% on pretraining tasks. The highest cross-opening rate, cybersecurity gates on IFEval (19.3%), reflects genuine overlap between two English instruction-following distributions. However, this has only a subtle effect on retention: after training on cybersecurity, LSL retains 98.6% of the base model's pretraining performance.
R3 Rebuttal (3/3)
(Q3) Can you report gate activation rates and task scores for near-distribution paraphrases, overlapping tasks, and mixed-domain prompts?
See response GC2 in the global comment, experiment (i) “Some gates open when they should be closed”.
.
(Q3) Does earlier adaptation provide useful forward transfer to a related later task?
This is an interesting question. Studying it requires a sequence of tasks designed to allow measurable forward transfer, which our current tasks do not. We leave this to future work.
.
(Q2) How well do replay or LwF perform when allowed the same generic reference tokens used to fit the negative GMMs?
We thank the reviewer for this suggestion. We add replay and LwF baselines that use the same 1M-token reference data as LSL's gate ("+ reference data"). Base Model, LoRA, LwF, and LSL results are taken from the paper.
Replay with the reference data does not retain pretraining capabilities well, lagging behind LSL by 14.0-17.2 points. This is expected, as the reference data was not sampled from the Qwen pretraining dataset, and, moreover, constitutes only about 0.0000067% of its size (L244). LwF improves with the reference data but still lags behind LSL by 3.4-6.5 points. We will add both baselines to the manuscript, providing an equally-informed comparison.
Igbo:
| Method | Pretrain Tasks | New Task |
|---|---|---|
| Base Model | 72.14 | 18.90 |
| LoRA | 37.82 | 37.18 |
| LwF | 54.15 | 37.00 |
| LwF + reference data | 64.92 | 36.98 |
| Replay + reference data | 54.24 | 36.89 |
| LSL | 71.45 | 36.74 |
Chemistry:
| Method | Pretrain Tasks | New Task |
|---|---|---|
| Base Model | 72.14 | 31.80 |
| LoRA | 52.47 | 78.67 |
| LwF | 63.45 | 77.44 |
| LwF + reference data | 67.91 | 79.32 |
| Replay + reference data | 57.29 | 75.89 |
| LSL | 71.27 | 78.93 |
Lastly, the strongest equally-informed baseline, LwF + reference data, already falls below LSL after the first phase (Igbo: -6.53 points, 64.92 vs. 71.45; chemistry: -3.36, 67.91 vs. 71.27). Same as in the response to W1, since later phases also lack pretraining data, we do not expect it to close this gap in the multi-phase setting.
.
Closing remark
We hope our rebuttal addresses the reviewer's concerns. If the concerns have been resolved, we would appreciate it if the reviewer would consider raising their score. If not, we will be glad to answer any remaining questions during the discussion period.
Gentle reminder: rebuttal period closing soon
Dear reviewer sEpt,
We thank you again for evaluating our work. As the rebuttal period will close soon, we would like to know whether you have any remaining concerns, which we will be happy to address. If not, we kindly ask you to consider increasing your rating.
Thank you,
The authors
R3 Rebuttal (1/3)
We thank the reviewer for taking the time to evaluate our work, as well as for their feedback, which helped improve our paper. Below is a summary of the main points addressed in the rebuttal:
The full responses are added below. We hope these additions address the reviewers’ concerns, and we will be glad to answer any remaining questions during the discussion period.
.
We thank the reviewer for the question. Simpler routers face limitations that, once addressed, either make them more complex or degrade their performance:
Routing based on the prompt fixes the gating before any token is decoded. As the reviewer points out in W2, handling reformatted and mixed-domain inputs is important. For such inputs, a single decision per prompt cannot fit every part of the response: for example, when asked to write code that solves a chemistry problem, the chemistry adapter is useful for some tokens but not for others. LSL decides per token at each matrix, and GC2 (global comment) shows that its gates behave well in these cases.
A single router, applied per-token must make a decision before the adapters it controls are applied, leaving two options: It can sit at the embedding layer, controlling all adapters within a single pass. Alternatively, since routing should be based on deep, contextual representations, it can run a speculative forward pass per token up to an intermediate layer (the right decision is known only after processing several layers), followed by an additional forward pass with the chosen gating. This increases inference cost and adds a hyperparameter (the chosen layer for the router). In contrast, LSL's inference latency is only 1.21× that of the LoRA baseline (Fig. 8 (left), L393-394). We test both approaches on the 7B model with the same GMM-based gate, matching adapter capacity and reference data. The speculative model uses representations at layer 16 (out of 28) for the gating decision:
Chemistry:
Igbo:
The simplest method, the embedding-layer router, retains comparably but learns substantially worse, since embedding features lack context. The speculative router at layer 16 recovers LSL's performance on chemistry, but not on Igbo translation (31.9 vs. 36.74), despite an additional partial forward pass per token. This may reflect either a single decision being too coarse for all matrices, or a task-dependent choice of router layer. In both cases, LSL avoids the issue by gating each matrix separately, within a single forward pass and without selecting a layer.
Two final remarks:
A separate token-level routing model is a discriminative classifier, whose behavior outside its training data is underspecified (Appendix F, L1053-1061). The fix requires a local-support objective like LSL's, so the router now faces the very problem it was meant to solve. It also adds a second model to the system.
Oracle bound. On all of our benchmarks, LSL already comes close to an oracle router with access to the domain label of every input (applying the base model on pretraining tasks and LoRA on the new task). Since LSL is already within 1.4% of the oracle on every benchmark, there is little room for any alternative router, including prompt-level ones, to improve upon its scores:
We will include this discussion, as well as the additional results, in the updated manuscript.