- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DISeL, a LoRA variant that multiplies each rank-one component by an input-dependent sigmoid gate. The gate bias is initialized to a negative value, so that during the first forward pass of the model, it is identical to the base model. The authors experiment with RoBERTa on the GLUE benchmark, LLaMA-2-7B on GSM8K and Mistral-7B on HumanEval.
Critical strengths
The problem is well motivated, and the method is simple and easy to implement. The checkpoint-level retention curves (Figures 4 and 6) are more informative than the usual single-checkpoint numbers.
Critical weaknesses
-
The authors' main result uses outdated models, i.e., LLaMA-2-7B and Mistral-7B. Could the authors justify this choice, given that fine-tuning more modern models with the same parameter count would incur similar compute costs?
-
Similarly, the authors only evaluate the fine-tuned models on GSM8K and HumanEval. Two benchmarks on a single model each are not enough to support any claim made by the paper. Also, the authors should probably not claim “fine-tuned for mathematical reasoning and code generation” — this misrepresents the whole paper by making readers think there are experiments related to multiple math or coding benchmarks when, in reality, there are only two. Also, both of these benchmarks are outdated. Why did the authors not evaluate their models on new benchmarks such as AIME25 or similar modern evaluations?
-
In Figure 2, there is base model performance for retention accuracy, but there is no base model performance for fine-tuned accuracy. Without including the base model’s evaluation on the benchmark, it is impossible to know how much the fine-tuning allowed the models to learn.
-
The method novelty is limited. Other works, such as SMoRA [1] and Gated LoRA, have also proposed similar input-dependent gating of individual rank-one components. So the mechanism is not new. Additionally, the theoretical arguments are either not novel or unrealistic. Proposition 1 is a standard textbook result. Proposition 2 is unrealistic because “define the pre-training and fine-tuning inputs as x_pt = x and x_ft = ax” — this assumes that the input for both pre-training and fine-tuning only differ by a scalar.
Questions for authors
All questions are above in the weakness section.
2 comments
Thank you for engaging in the discussion. However, my concerns regarding weakness 1 and weakness 2 are not addressed. Also, my second weakness was not only that the authors did not explicitly state which dataset the models were fine-tuned on, but also, importantly, that two benchmarks on a single model each are not enough to support any claim made by the paper. Thus, I will maintain my score.
Thank you for the careful review.
Thank you for the suggestion. We are working on experiments with newer models and datasets, but our servers are currently heavily loaded, making it difficult to finish within the rebuttal period. We will include these results in the final version.
Thank you for pointing this out. You are right that our wording should be more precise about the scope of the evaluation. We will revise it to state explicitly that the models are fine-tuned on MetaMathQA and Magicoder, and evaluated on GSM8K and HumanEval, respectively.
Thank you for pointing out this omission. Under our evaluation setup, the base LLaMA-2-7B scores approximately 13% on GSM8K, and the base Mistral-7B scores approximately 3% on HumanEval. DISeL reaches about 60% and above 70%, respectively. Thus, the models learn substantially while maintaining strong retention. We will add these base-model task scores to Figure 2 so that both the task improvement and the retention change are directly visible.
Thank you for pointing out SMoRA. We missed this relevant reference and will add it. We agree that rank-wise input-dependent gating is not new. DISeL's contribution is its retention-focused design and the resulting adaptation-retention tradeoff. Unlike the tested ReLU Gated LoRA variant, DISeL learns its gates through a separate router rather than deriving them directly from the LoRA features. Unlike SMoRA, it does not use top-k selection or gate normalization, so all gates can become small together. Negative gate biases favor small corrections initially, and the router has its own higher learning rate. For clarity, exact equality with the base model at initialization comes from the zero-initialized LoRA factor, not negative gate biases alone.
Our new SMoRA comparison on Mistral/Magicoder gives DISeL higher reported task accuracy and retention at every tested rank. SMoRA's retention declines sharply with rank, while DISeL's remains stable. Full results and hyperparameters are in our response to Reviewer uGGJ (W1).
Proposition 1 uses standard conditional-expectation and Bayes calculations. Its purpose is to motivate the design, not to claim a new probability identity. In the regression setting, the best correction weights each component by the probability that it is needed. Several components can be needed together, so these weights need not sum to one. The practical affine sigmoid gates approximate the more general posterior scores. This is a design motivation, not a guarantee that DISeL learns the Bayes-optimal predictor in an LLM.
Proposition 2 deliberately studies a constructed case, not a model of how real pre-training and fine-tuning corpora differ. For a positively homogeneous correction, f(ax)=af(x). Therefore, making the correction small at x also restricts what it can do at ax, even when a substantial change is needed there. The proposition quantifies this restriction at the adapter level. It does not claim that real domains differ only by scaling or that this explains every observed failure. We will clarify this limited role as an illustrative failure case, rather than a general explanation of forgetting.
If we managed to address your concerns, we appreciate if you consider increasing your score.