acceptodds
Under review as a conference paper at ICLR 2027

Low Rank, Lasting Knowledge: Mitigating Catastrophic Forgetting in LoRA Fine-Tuning

Abstract

Low-Rank Adaptation (LoRA) adapts large language models to a target domain (TD) with few trainable parameters, but applying the adapter to every input can degrade performance on out-of-target-domain (OTD) data. The base model itself remains unchanged. We address this non-target interference through DG-LoRA, a sample-level dynamic gating method for the post-hoc conditional activation of a single frozen LoRA adapter. A lightweight multilayer perceptron computes a scalar gate value from an answer-independent routing sequence and uses it to interpolate the output logits of the base and LoRA-enabled models before softmax. This value stays fixed throughout candidate scoring and generation. After training the adapter, we freeze both predictive branches and train only the gate, combining domain classification, target-task and non-target preservation losses. These objectives account for domain membership as well as the effect of adaptation on predictions. Across the evaluated models and tasks, DG-LoRA performed comparably to standard LoRA on TD data while substantially reducing non-target interference on OTD data, including general-domain and other domain-specific tasks. OTD performance generally returned close to the base-model level. The gate added fewer than as many trainable parameters as the LoRA adapter.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation (LoRA), Catastrophic Forgetting, Dynamic Gating

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_asetOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DG-LoRA, a method to reduce the performance drop on OOD tasks caused by applying LoRA. After training a LoRA adapter, the method freezes both the base model and the adapter. Then it trains a small MLP gate to mix the output logits of these two parts. The gate has fewer than 10% as many trainable parameters as the LoRA adapter. The gate is trained using three losses: domain classification, target task prediction, and preservation on other tasks. Experimental results show that DG-LoRA largely keeps the performance on the target task while recovering performance on other tasks.

Critical strengths

Strong experimental results on preserving OOD capabilities compared with standard LoRA. The reported results show that DG-LoRA can recover performance on non-target tasks while largely maintaining the target-task gains of LoRA, as demonstrated in Table 2. This provides evidence that DG-LoRA can help preserve capabilities outside the target domain.

Critical weaknesses
  1. Most results are based on Llama models. Although the paper includes supplementary experiments with Qwen and Gemma models, these cover only a small number of settings. Moreover, the loss ablation is limited to Llama3.2-1B with BBH as the target task (Table 3), and the full objective shows only small improvements over domain classification alone. It therefore remains unclear whether the additional losses provide consistent benefits across models and target tasks. Broader ablations across model families and target tasks, together with evaluation on additional out-of-target-domain tasks such as MBPP, would strengthen the empirical evidence.

  2. Limited evidence beyond domain classification. The paper argues that domain membership and adapter utility are different. However, it is unclear whether the proposed gate can effectively account for this distinction. The target-task loss considers prediction quality only for target-domain samples. For non-target samples, both DC and NTP encourage the model to follow the base model, without directly rewarding adapter use when it is helpful. The authors acknowledge this limitation in Section 5, but the current experiments do not directly examine such cases.The reported results show that DG-LoRA can recover performance on non-target tasks while largely maintaining the target-task gains of LoRA.

Questions for authors
  1. In Table 1, the GSM8K accuracy of the same Llama3.2-3B-Instruct base model is 40.03 when GSM8K is the target domain, but 11.68 when it is a non-target domain. Could the authors explain this difference and clarify whether the same evaluation settings were used in both cases?

  2. The main experiments primarily compare DG-LoRA with the base model and standard LoRA, while the comparison with CLEAR is limited to the safety setting. Related methods such as SLIM and GainLoRA also use gating to reduce interference from LoRA adapters. I believe it would be helpful to include experimental comparisons with these baselines.

  3. Regarding Weakness 1, it would be helpful if the authors could extend their experiments to additional models and benchmarks.

2 comments

We thank the reviewer for raising these specific and important questions.

1.Regarding the discrepancy in Table 1

The 40.03% shown for the Llama-3.2-3B-Instruct base model when GSM8K is the target task is a transcription error in the table. In the original GSM8K-target experiments, the base model scored 11.55 ± 0.36% across three random seeds. In response to this issue, we carefully rechecked all the data in the table and found this to be the only error. We will correct it. In addition, Table 1 presents results for two target domains together with results for non-target domains, making the presentation insufficiently clear. We will therefore split it into two tables, one for each target domain.

2.Regarding additional baselines

We added MBPP as an evaluation task outside the target domain. The results are as follows:

ModelMethodGSM8K (Target), EM ↑MBPP (Out-of-domain), pass@1 ↑
Llama-3.2-1BBase4.32%26.80%
Llama-3.2-1BLoRA21.76%22.80%
Llama-3.2-1BDG-LoRA21.63%26.80%
Gemma-2-2B-itBase25.63%38.80%
Gemma-2-2B-itLoRA39.95%33.00%
Gemma-2-2B-itDG-LoRA40.03%38.80%

As the SLIM code has not been released to date, we are reproducing the method based on its paper. We have also added experiments with GainLoRA. First, using GainLoRA's publicly available code, we repeated the experiments in Table 1 in separate settings with MedMCQA and GSM8K each serving as the target domain. The results are as follows:

ModelMethodTarget-domain MedMCQA (GSM8K) ↑Out-of-domain GSM8K ↑HellaSwag ↑BBH ↑WikiText-2 ↓
Llama3.2-1BBase35.45 (4.32)4.32 (-)59.44 (-)24.47 (25.25)2.66 (2.66)
Llama3.2-1BLoRA37.97 (21.71)0.00 (-)26.85 (-)14.34 (21.71)16.49 (2.76)
Llama3.2-1BDG-LoRA37.93 (21.63)4.40 (-)59.44 (-)24.24 (25.34)2.66 (2.66)
Llama3.2-1BGainLoRA39.77 (13.98)2.25 (-)56.62 (-)24.99 (26.45)2.76 (2.67)

The results show that, as observed for standard LoRA relative to the base model, GainLoRA also exhibits reduced performance in other domains after target-domain fine-tuning. To further validate the effectiveness of our mechanism, we incorporated the DG-LoRA mechanism into GainLoRA. The results are as follows:

ModelMethodTarget-domain MedMCQA ↑GSM8K ↑HellaSwag ↑BBH ↑WikiText-2 ↓
Llama3.2-1BBase35.454.3259.4424.472.66
Llama3.2-1BLoRA37.970.0026.8514.3416.49
Llama3.2-1BDG-LoRA37.934.4059.4424.242.66
Llama3.2-1BGainLoRA39.772.2556.6224.992.76
Llama3.2-1BDG-GainLoRA39.714.3559.4424.252.72

In addition to the above experimental results, we are also extending our experiments to more models and benchmarks.

3.Regarding the objective function

Using Llama-3.2-1B with BBH and GSM8K as the target tasks in separate experiments, we extended the ablation in Table 3 to all seven nonempty combinations of DC, TT, and NTP. The results are as follows:

Table 1. BBH as the target task (Llama-3.2-1B)

DCTTNTPBBH ↑ARC-Challenge ↑Router-AUROC ↑Wiki ↓
✓××53.1742.3598.092.66
×✓×53.2236.4150.082.74
××✓19.4342.2423.042.66
✓✓×53.2242.4198.052.66
✓×✓53.2241.9297.052.66
×✓✓53.3142.7896.212.66
✓✓✓53.2242.7298.112.66

Table 2. GSM8K as the target task (Llama-3.2-1B)

DCTTNTPGSM8K ↑ARC-Challenge ↑Router-AUROC ↑Wiki ↓
✓××21.5341.8999.982.66
×✓×21.6336.9250.102.75
××✓4.4041.8926.142.66
✓✓×21.6141.9299.982.66
✓×✓21.6341.9299.972.66
×✓✓21.5641.8999.952.66
✓✓✓21.6341.8999.972.66

Each setting used three random seeds. Within each seed, the base model and the LoRA checkpoint were held fixed, and only the gate was retrained. In both sets of experiments, gate values were almost always close to 1 when TT alone was used, and almost always close to 0 when NTP alone was used. This is consistent with the scope of supervision provided by the two losses: TT optimizes answer likelihood only on target-domain samples, whereas NTP uses the base model's predictions as the reference, so setting g = 0 on non-target samples is sufficient to reduce the KL divergence to zero. Compared with DC alone, the full objective increased the BBH score from 53.17% to 53.22% and the GSM8K score from 21.53% to 21.63%.

Reviewer_asetOpenReview ↗

Response

Thank you for your detailed response. They have mostly addressed my concerns. I’ll raise my rating to weak acceptance.

Reviewer_TT2EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at a deployment problem that can happen with LoRAs. After a domain-specific adapter is attached to a network, performance will degrade if it is applied to every input and performance may get worse outside the target domain. The proposed method, DG-LoRA, keeps both the base model and a trained LoRA adapter frozen and trains a small MLP gate that reads mean-pooled base model hidden states of the prompt and outputs a single scalar gate value. This scalar interpolates between the base and LoRA output logits. It can turn the LoRA on and off, or choose an intermediate state.

The authors report that DG-LoRA roughly matches LoRA on the target domain while bringing out-of-domain performance back close to the base model, with a gate that is under 10 percent of the adapter's size.

Critical strengths

Deciding after the fact when to use a single frozen adapter, without touching the base model or the adapter, is a situation people may actually run into (depending on the particular use case), and the method is simple and cheap to train.

In the tested settings the method does what it claims: target-domain accuracy stays at the LoRA level and out-of-domain performance is close to the base model, across three model sizes and several target domains.

Critical weaknesses

The most obvious baseline is to just train the LoRA on both target and non-target data (for example with replay, or a KL penalty to the base model on the non-target data). The paper argues that retraining the adapter defeats the purpose of post-hoc deployment, but the gate training already uses the full target-domain training set plus a matched pool of general negatives. Anyone who has that data can just as easily train a better adapter, and at rank 8 that is cheap. Without this comparison it is unclear whether the gate buys anything over a less damaging adapter. The only baselines in the main results are the base model and plain LoRA. Maybe the gate is superior to this baseline because it handles help-out non-target domains better. Or maybe it doesn't. But if you have all this representative data (from non-target domains) available for training, why not use it to develop the LoRA?

  1. The gate is a single scalar per sequence, computed from the prompt and frozen for the whole generation. That seems unrealistic for many real deployments, where the input is built up over an interaction with a user, tools, or retrieved content, and where the right amount of adaptation can change partway through. The paper does not say whether the gate would be recomputed each turn, and it never evaluates multi-turn, agentic, or mixed-domain inputs, so a single prompt that needs both target and non-target skills can only get a compromise value.

Relatedly, the held-out evaluation is weaker than it looks. The non-target test domains are mostly different from the gate's negative training domains, but they are all far from the target (e.g. grade school math vs medical multiple choice), which a simple domain detector can easily separate. The appendix says MedMCQA runs were also evaluated on MedQA and PubMedQA, which would be the interesting near-domain test, but those results are not reported. ARC-Challenge and MMLU also appear both in the negative pool and in evaluation.

Questions for authors

How does DG-LoRA compare to a LoRA trained on the target data plus the same negative pool? What are the MedQA and PubMedQA results mentioned in the appendix? Are the ARC-Challenge and MMLU examples used for gate training disjoint from the ones used for evaluation?

Suggestion: A per-token or per-turn version of the gate, tested on multi-turn or mixed-domain inputs, would make the method much more convincing for real deployments. The more API-ish setting in which the user simply supposed an instruction and expects a simple interactive answer is a real use case, but most deployments don't look like this nowadays.

1 comment

We thank the Reviewer for the specific suggestions. We have added a Replay LoRA baseline trained by mixing MedMCQA target-domain examples with general non-target examples. Separately, we report near-domain results on MedQA and PubMedQA.

1. Replay LoRA baseline

To address the mixed-training baseline, in the Llama-3.2-1B MedMCQA setting, we combined 164,540 target-domain training examples with the same 6,726 non-target examples from the gate-training negative pool and trained Replay LoRA. Table 1 compares Base, the original LoRA, Replay LoRA, and DG-LoRA. All metrics except Wiki NLL are percentages for which higher is better; lower is better for Wiki NLL.

Table 1. Target-domain and out-of-domain results for Replay LoRA and DG-LoRA

ModelMethodMedMCQA ↑GSM8K ↑HellaSwag ↑BBH ↑Wiki NLL ↓
Llama-3.2-1BBase35.484.4059.4824.472.66
Llama-3.2-1BLoRA38.020.0026.8714.3416.49
Llama-3.2-1BReplay LoRA37.891.6741.8218.533.70
Llama-3.2-1BDG-LoRA37.984.4059.4824.242.66

Replay LoRA achieves 37.89% accuracy on MedMCQA, close to the original LoRA (38.02%) and DG-LoRA (37.98%). Compared with the original LoRA, Replay LoRA improves GSM8K from 0.00% to 1.67%, HellaSwag from 26.87% to 41.82%, and BBH from 14.34% to 18.53%, while reducing Wiki NLL from 16.49 to 3.70. These results indicate that adding these non-target examples mitigates some out-of-domain performance degradation, although Replay LoRA does not recover Base performance on the out-of-domain metrics evaluated here. DG-LoRA's out-of-domain results are closer to Base.

2. Near-domain results on MedQA and PubMedQA

Table 2 reports accuracies (%) for Base, LoRA, and DG-LoRA in the Llama-3.2-1B MedMCQA setting, evaluated on the target-domain task MedMCQA, the near-domain tasks MedQA and PubMedQA, and the out-of-domain task HellaSwag.

Table 2. Results on the MedMCQA target domain, near-domain tasks, and an out-of-domain task

ModelMethodMedMCQA (target)MedQA (near-domain)PubMedQA (near-domain)HellaSwag (out-of-domain)
Llama-3.2-1BBase35.4534.6456.5059.44
Llama-3.2-1BLoRA37.9732.2333.8026.85
Llama-3.2-1BDG-LoRA37.9333.8055.6359.44

On both MedQA and PubMedQA, DG-LoRA outperforms LoRA with the adapter always active and is close to Base. The HellaSwag results in Table 2 also show that DG-LoRA recovers the performance lost with the original LoRA on this out-of-domain task.

Note: Table 2 uses archived results from the original experiments, whereas the Base, LoRA, and DG-LoRA results in Table 1 were reevaluated for the Replay comparison. Thus, metrics shared by the two tables show small numerical differences, and direct comparisons between methods should be made within the same table.

3. ARC-Challenge and MMLU data splits

The original LoRA was trained only on MedMCQA target-domain data. The MMLU negative examples used for gate training came from auxiliary_train, and the ARC-Challenge negative examples came from the training split. For these two datasets, gate validation used the corresponding validation splits, while performance evaluation used the corresponding test splits. Thus, gate training did not directly use examples from the evaluation test splits.

4. Multi-turn interaction and dynamic gating

Thank you for this valuable suggestion. We agree that dynamically adjusting the gate per turn or per token, and evaluating it on multi-turn interactions, tool use, and mixed-domain inputs, would help improve the method's applicability in practical settings. Currently, the gate value is computed from the input before each model call and remains fixed during that generation. A subsequent call can recompute the gate value based on the updated context. Beyond this between-call recomputation, developing a dynamic strategy for multi-turn settings and per-token adjustment during generation would require redesigning the training procedure and evaluation setup and conducting systematic experiments. We have already begun work on the corresponding implementation.

Reviewer_19DoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies a deployment problem with LoRA. A task adapter trained on a target domain (TD) is normally applied to every input. On out-of-target-domain (OTD) inputs it can change the output distribution and perform worse than the unchanged base model. The paper proposes DG-LoRA, a post-hoc dynamic gating method for frozen LoRA adapters. A lightweight MLP computes a scalar gate value from the pooled base-model states of the answer-independent prompt. The gate value interpolates the base and LoRA logits before softmax and is shared across all candidate answers and generation steps. Experiments use three Llama models with MedMCQA, GSM8K, and HellaSwag as target domains, and compare against the base model and standard LoRA. DG-LoRA performs comparably to standard LoRA on TD data and generally recovers OTD performance to near the base-model level.

Critical strengths
  1. The paper targets a common deployment case where one adapter is attached to a model that also receives unrelated requests. DG-LoRA keeps the base model and the adapter frozen and only trains a small gate that has fewer than 10% as many parameters as the LoRA adapter. This makes the method cheap and low-risk to add to an existing adapter.

  2. Across three Llama models and three target domains, DG-LoRA stays within about 0.4 points of standard LoRA on the TD and brings OTD results close to the base model (Tables 1 and 2). These OTD benchmarks are not in the gate's negative pool, so the gate does not seem to simply memorize the negatives.

Critical weaknesses
  1. Several prior methods are closely related to DG-LoRA. DISeL [1] already studies when a single LoRA adapter should be active, using input-dependent gates that preserve pre-trained behavior on inputs outside the fine-tuning distribution. Engram Adapter [2] frames the same “when to adapt” question for domain specialization on a frozen backbone and also uses MedMCQA as a target domain. CLEAR [3] gates a single safety LoRA adapter on a frozen backbone, using a prompt-level MLP gate that is shared across decoding steps. SLIM [4] reduces forgetting by letting inputs bypass the LoRA adapter through identity experts. DISeL and Engram Adapter are not cited. Tables 1 and 2 only compare DG-LoRA with the base model and standard LoRA. Simple baselines, such as a thresholded domain classifier or a fixed interpolation coefficient, are also missing.

  2. The paper claims that domain membership differs from adapter utility, but the objective cannot capture this. The NTP loss uses the base model as the target, so for non-target inputs it is always minimized at g=0g = 0, even when the adapter would help. The TT loss is computed on the same samples used to train the adapter, where the adapter is expected to lower the NLL, so it pushes gg toward 1. Both losses therefore push the gate in the same direction as the DC loss. Table 3 supports this, as DC alone gives nearly the same results as all three losses. The evaluated domains are also easy to separate. Router-AUROC is about 98% in Table 3, and the TD and OTD tasks differ clearly in input format. Results on MedQA and PubMedQA, where positive transfer is most likely, are not reported. The results can therefore be explained by a simple domain classifier, and the claimed benefit of modeling adapter utility is not demonstrated.

  3. The reliability of the LoRA baseline is unclear. With the MedMCQA adapter, standard LoRA drops GSM8K to 0.00 for all three models. It also drops HellaSwag to about 26% for the 1B and 8B models, close to the 25% random-guess level of this four-choice task. For the same two models, Wiki NLL rises to 16.49 and 13.51. This degradation is unusually severe for a rank-8 adapter and may reflect format collapse rather than lost capability. This makes the size of the reported OTD gains hard to judge.

[1] Ali Zindari et al., Learning When to Adapt, arXiv 2026.

[2] Jiayu Hou and Lei Wang, When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization, Findings of EMNLP 2026.

[3] Chengxiao Wang et al., CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignment, arXiv 2026.

[4] Jiayi Han et al., SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture, NAACL 2025.

Questions for authors
  1. Is the gate applied in weight space as in Eq. (1), or to the output logits as in the abstract and Section 4.1? If it is logit interpolation, both branches run at every decoding step. What are the latency and memory costs compared with standard LoRA?

  2. Have the authors compared DG-LoRA with any gating or bypass method beyond the base model and standard LoRA in the main setting?

  3. What are the MedQA and PubMedQA results mentioned in Appendix A? What does the distribution of gate values look like on the TD and OTD test sets?

  4. How was the LoRA baseline trained and evaluated? On GSM8K and HellaSwag, does the MedMCQA adapter still produce valid outputs, or does it collapse to the MedMCQA answer format?

3 comments

We thank the reviewer for the comments. We rechecked implementation and item-level results and added experiments.

1. Gate and cost

Eq. (1) contradicts the abstract/Sec. 4.1: we mix logits before softmax:

zt=(1−gφ(r(x)))zb,t+gφ(r(x))zLoRA,t.z_t=(1-g_{\varphi}(r(x)))z_{b,t}+g_{\varphi}(r(x))z_{\mathrm{LoRA},t}.

z_t: mixed; z_{b,t}: Base; z_{LoRA,t}: LoRA logits for the same input/history. Gate g_{\varphi}(r(x)) uses r(x) and parameters \varphi, once per input and fixed during generation. We will correct Eq. (1)'s wrong.

Both branches run each step (shared Base weights, separate KV caches). On A100 40 GB we used Llama-3.2-3B-Instruct+MedMCQA; same MedQA/UltraChat requests, BF16, one request at a time, 128/384 input and 64 output tokens:

MetricStandard LoRADC-LoRA
Total generation time (median)~2.71 s2.81–2.85 s
Total generation time (p95)2.73–2.74 s2.84–2.88 s
Peak CUDA allocated memory6.07–6.13 GiB6.09–6.20 GiB

Each request uses Base/LoRA rows, one copy of base weights and adapter, one batched pass, separate KV caches; the mixed next token feeds both. Median time rises 3.8%–5.3%, peak allocated memory 0.4%–1.2% vs LoRA.

2. Related work

We will cite DISeL and Engram Adapter and discuss CLEAR and SLIM. DISeL jointly trains LoRA and an input gate; Engram uses conditional memory for specialization/retention; CLEAR jointly trains safety LoRA and a prompt gate; SLIM routes LoRA experts and an identity branch. We freeze Base/adapter, training only a prompt gate. SLIM code is not public; we are reproducing it from the paper. Comparisons:

ModelMethodTarget MedMCQA (GSM8K)↑OTD GSM8K↑HellaSwag ↑BBH ↑WikiText-2 ↓
Llama3.2-1BBase35.45 (4.32)4.32(–)59.44(–)24.47 (25.25)2.66 (2.66)
Llama3.2-1BLoRA37.97 (21.71)0.00(–)26.85(–)14.34 (21.71)16.49 (2.76)
Llama3.2-1BDG-LoRA37.93 (21.63)4.40(–)59.44(–)24.24 (25.34)2.66 (2.66)
Llama3.2-1BGainLoRA39.77 (13.98)2.25(–)56.62(–)24.99 (26.45)2.76 (2.67)

Like LoRA, target-tuned GainLoRA loses OTD performance vs Base. We tested DG-LoRA on GainLoRA:

ModelMethodTarget MedMCQA↑GSM8K ↑HellaSwag ↑BBH ↑WikiText-2 ↓
Llama3.2-1BBase35.454.3259.4424.472.66
Llama3.2-1BLoRA37.970.0026.8514.3416.49
Llama3.2-1BDG-LoRA37.934.4059.4424.242.66
Llama3.2-1BGainLoRA39.772.2556.6224.992.76
Llama3.2-1BDG-GainLoRA39.714.3559.4424.252.72

3. Objective and transfer

We thank the reviewer for the domain/utility distinction. All seven nonempty DC/TT/NTP combinations were tested on Llama-3.2-1B, targeting BBH/GSM8K separately:

Table 1. BBH target (Llama-3.2-1B)

DCTTNTPBBH ↑ARC-Challenge ↑Router-AUROC ↑Wiki ↓
✓××53.1742.3598.092.66
×✓×53.2236.4150.082.74
××✓19.4342.2423.042.66
✓✓×53.2242.4198.052.66
✓×✓53.2241.9297.052.66
×✓✓53.3142.7896.212.66
✓✓✓53.2242.7298.112.66

Table 2. GSM8K target (Llama-3.2-1B)

DCTTNTPGSM8K ↑ARC-Challenge ↑Router-AUROC ↑Wiki ↓
✓××21.5341.8999.982.66
×✓×21.6336.9250.102.75
××✓4.4041.8926.142.66
✓✓×21.6141.9299.982.66
✓×✓21.6341.9299.972.66
×✓✓21.5641.8999.952.66
✓✓✓21.6341.8999.972.66

Three seeds/setting; Base/LoRA fixed per seed, gate retrained. TT-only gates almost always ≈1; NTP-only ≈0. TT optimizes target likelihood; NTP uses Base, so KL=0 at g=0 on non-target inputs. Full vs DC: BBH 53.17→53.22%, GSM8K 21.53→21.63%. Target performance retained, OTD interference reduced in these settings.

Appendix A: Llama-3.2-3B-Instruct on MedMCQA:

MetricMedMCQA (target)MedQA (near-domain)PubMedQA (near-domain)HellaSwag (far OTD)
Base46.83%42.34%50.90%61.97%
LoRA50.20%46.45%62.27%59.93%
DG-LoRA50.13%47.11%59.07%61.97%
Mean gate0.98670.93470.6775<0.0001
Gate P050.99970.2017<0.0001<0.0001
Gate median1.00001.00000.9971<0.0001
Gate P951.00001.00001.0000<0.0001

MedMCQA/MedQA: medical MCQs; PubMedQA: abstract-based QA. Gates ≈1 on target/near data, ≈0 OTD.

4. LoRA baseline

Base frozen; LoRA trained only on MedMCQA (164,540 train, 18,282 validation from official train). Seven attention/FFN projections; r=8, alpha=16, dropout=.05; AdamW LR 2×10⁻⁴ for ≤3 epochs; lowest internal-validation per-token NLL checkpoint. Base/LoRA: identical evaluation questions, prompts, scoring; LoRA always on.

GSM8K test: numeric exact match. For every seed and each 1B/3B-Instruct/8B MedMCQA adapter, all 1,319 responses were A/B/C/D, hence scored zero.

HellaSwag: Base/always-on MedMCQA LoRA got the same context and four continuations; the lowest mean NLL candidate was selected. Table 1 item-level recheck found no LoRA run always chose one position for any model/seed.

Thank you for the detailed response and the additional experiments. However, the results do not resolve my concerns.

  1. The full ablation shows that DC alone matches the full objective (Table 1: BBH 53.17 vs. 53.22, Table 2: GSM8K 21.53 vs. 21.63). On PubMedQA, standard LoRA improves over the base model from 50.90 to 62.27, but DG-LoRA reaches only 59.07 (Appendix A). The gate therefore acts as a domain classifier. It also closes on some inputs where the adapter helps.

  2. As you confirm, the 0.00 on GSM8K arises because all 1,319 responses are A/B/C/D. This is a format collapse, not a loss of capability. GainLoRA keeps 56.62 on HellaSwag, against 26.85 for standard LoRA. The large OTD gains in Tables 1 and 2 are therefore mainly due to the collapse of this particular LoRA baseline

Thank you for your follow-up comments.

  1. The contribution of DC and the effect of gating on positive transfer.DG-LoRA aims to preserve the gains from fine-tuning on the target task while recovering as much of the decline in general-task performance as possible. We agree that DC alone nearly matches the full objective in these two ablations. On BBH (Table 1), accuracy is 53.17% with DC alone and 53.22% with the full objective. On GSM8K (Table 2), the corresponding accuracies are 21.53% and 21.63%. The full objective thus yields additional gains of 0.05 and 0.10 percentage points, respectively. On PubMedQA (Appendix A), standard LoRA improves accuracy from 50.90% for the base model to 62.27%. DG-LoRA reaches 59.07%, which is 8.17 percentage points above the base model and 3.20 percentage points below standard LoRA. MedMCQA and MedQA consist of multiple-choice medical exam questions, while PubMedQA requires answering research questions based on paper abstracts. This result is consistent with the limitation you identified. If the gate decides whether to activate the adapter primarily based on how similar the input is to the fine-tuning data, it may disable the adapter for some examples that would otherwise benefit from it.

  2. GSM8K format collapse and the effect of baseline choice on the magnitude of recovery. We agree that standard LoRA fine-tuned on MedMCQA outputs only A/B/C/D responses across all 1,319 GSM8K questions. The 0.00% accuracy reflects a collapse in response format under the current evaluation. We will use the same questions and model checkpoints, apply a common numerical answer requirement, and report both the proportion of valid numerical outputs and answer accuracy to determine how much of the performance gap remains after controlling for response format.

    We also agree that baseline choice substantially affects the measured recovery on HellaSwag. Standard LoRA achieves 26.85% accuracy, and GainLoRA achieves 56.62%. Adding a gate to the same frozen GainLoRA adapter raises HellaSwag accuracy from 56.62% to the base model’s 59.44%, with improvements across all three random seeds. MedMCQA accuracy remains essentially unchanged (39.77% → 39.71%). In a separate set of experiments with BBH as the fine-tuning target, ARC-Challenge accuracy increases from 36.41% with standard LoRA to 42.72%, close to the base model’s 42.24%. Both evaluations rank candidate answers by likelihood, so they provide evidence, independent of GSM8K’s response format, that gating can recover some performance on general tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.