- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DG-LoRA, a method to reduce the performance drop on OOD tasks caused by applying LoRA. After training a LoRA adapter, the method freezes both the base model and the adapter. Then it trains a small MLP gate to mix the output logits of these two parts. The gate has fewer than 10% as many trainable parameters as the LoRA adapter. The gate is trained using three losses: domain classification, target task prediction, and preservation on other tasks. Experimental results show that DG-LoRA largely keeps the performance on the target task while recovering performance on other tasks.
Critical strengths
Strong experimental results on preserving OOD capabilities compared with standard LoRA. The reported results show that DG-LoRA can recover performance on non-target tasks while largely maintaining the target-task gains of LoRA, as demonstrated in Table 2. This provides evidence that DG-LoRA can help preserve capabilities outside the target domain.
Critical weaknesses
-
Most results are based on Llama models. Although the paper includes supplementary experiments with Qwen and Gemma models, these cover only a small number of settings. Moreover, the loss ablation is limited to Llama3.2-1B with BBH as the target task (Table 3), and the full objective shows only small improvements over domain classification alone. It therefore remains unclear whether the additional losses provide consistent benefits across models and target tasks. Broader ablations across model families and target tasks, together with evaluation on additional out-of-target-domain tasks such as MBPP, would strengthen the empirical evidence.
-
Limited evidence beyond domain classification. The paper argues that domain membership and adapter utility are different. However, it is unclear whether the proposed gate can effectively account for this distinction. The target-task loss considers prediction quality only for target-domain samples. For non-target samples, both DC and NTP encourage the model to follow the base model, without directly rewarding adapter use when it is helpful. The authors acknowledge this limitation in Section 5, but the current experiments do not directly examine such cases.The reported results show that DG-LoRA can recover performance on non-target tasks while largely maintaining the target-task gains of LoRA.
Questions for authors
-
In Table 1, the GSM8K accuracy of the same Llama3.2-3B-Instruct base model is 40.03 when GSM8K is the target domain, but 11.68 when it is a non-target domain. Could the authors explain this difference and clarify whether the same evaluation settings were used in both cases?
-
The main experiments primarily compare DG-LoRA with the base model and standard LoRA, while the comparison with CLEAR is limited to the safety setting. Related methods such as SLIM and GainLoRA also use gating to reduce interference from LoRA adapters. I believe it would be helpful to include experimental comparisons with these baselines.
-
Regarding Weakness 1, it would be helpful if the authors could extend their experiments to additional models and benchmarks.
2 comments
Response
Thank you for your detailed response. They have mostly addressed my concerns. I’ll raise my rating to weak acceptance.
We thank the reviewer for raising these specific and important questions.
1.Regarding the discrepancy in Table 1
The 40.03% shown for the Llama-3.2-3B-Instruct base model when GSM8K is the target task is a transcription error in the table. In the original GSM8K-target experiments, the base model scored 11.55 ± 0.36% across three random seeds. In response to this issue, we carefully rechecked all the data in the table and found this to be the only error. We will correct it. In addition, Table 1 presents results for two target domains together with results for non-target domains, making the presentation insufficiently clear. We will therefore split it into two tables, one for each target domain.
2.Regarding additional baselines
We added MBPP as an evaluation task outside the target domain. The results are as follows:
As the SLIM code has not been released to date, we are reproducing the method based on its paper. We have also added experiments with GainLoRA. First, using GainLoRA's publicly available code, we repeated the experiments in Table 1 in separate settings with MedMCQA and GSM8K each serving as the target domain. The results are as follows:
The results show that, as observed for standard LoRA relative to the base model, GainLoRA also exhibits reduced performance in other domains after target-domain fine-tuning. To further validate the effectiveness of our mechanism, we incorporated the DG-LoRA mechanism into GainLoRA. The results are as follows:
In addition to the above experimental results, we are also extending our experiments to more models and benchmarks.
3.Regarding the objective function
Using Llama-3.2-1B with BBH and GSM8K as the target tasks in separate experiments, we extended the ablation in Table 3 to all seven nonempty combinations of DC, TT, and NTP. The results are as follows:
Table 1. BBH as the target task (Llama-3.2-1B)
Table 2. GSM8K as the target task (Llama-3.2-1B)
Each setting used three random seeds. Within each seed, the base model and the LoRA checkpoint were held fixed, and only the gate was retrained. In both sets of experiments, gate values were almost always close to 1 when TT alone was used, and almost always close to 0 when NTP alone was used. This is consistent with the scope of supervision provided by the two losses: TT optimizes answer likelihood only on target-domain samples, whereas NTP uses the base model's predictions as the reference, so setting g = 0 on non-target samples is sufficient to reduce the KL divergence to zero. Compared with DC alone, the full objective increased the BBH score from 53.17% to 53.22% and the GSM8K score from 21.53% to 21.63%.