acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Adapt

Abstract

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This input-agnostic approach creates an inevitable compromise between adapting to the fine-tuning distribution and preserving pre-trained behavior on inputs outside that distribution, contributing to catastrophic forgetting. We introduce DiAL (Dynamic input-Adaptive LoRA), which augments LoRA modules with lightweight input-dependent gates over individual rank-one components. The gating mechanism is designed to preserve the pre-trained model’s behavior by default, while training learns to activate selected components that reduce the fine-tuning loss. DiAL adds only a small number of parameters and preserves the low-rank structure. Across RoBERTa on GLUE, and Llama and Mistral models fine-tuned for mathematical reasoning and code generation, DiAL reduces forgetting relative to LoRA and related variants while maintaining competitive fine-tuning accuracy. In addition, the learned gate activations provide an interpretable diagnostic view of which layers and rank components are most activated during fine-tuning, giving insight into where task-specific adaptation is concentrated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Fine-tuning, Forgetting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9FU5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DISeL, a LoRA variant that multiplies each rank-one component by an input-dependent sigmoid gate. The gate bias is initialized to a negative value, so that during the first forward pass of the model, it is identical to the base model. The authors experiment with RoBERTa on the GLUE benchmark, LLaMA-2-7B on GSM8K and Mistral-7B on HumanEval.

Critical strengths

The problem is well motivated, and the method is simple and easy to implement. The checkpoint-level retention curves (Figures 4 and 6) are more informative than the usual single-checkpoint numbers.

Critical weaknesses
  • The authors' main result uses outdated models, i.e., LLaMA-2-7B and Mistral-7B. Could the authors justify this choice, given that fine-tuning more modern models with the same parameter count would incur similar compute costs?

  • Similarly, the authors only evaluate the fine-tuned models on GSM8K and HumanEval. Two benchmarks on a single model each are not enough to support any claim made by the paper. Also, the authors should probably not claim “fine-tuned for mathematical reasoning and code generation” — this misrepresents the whole paper by making readers think there are experiments related to multiple math or coding benchmarks when, in reality, there are only two. Also, both of these benchmarks are outdated. Why did the authors not evaluate their models on new benchmarks such as AIME25 or similar modern evaluations?

  • In Figure 2, there is base model performance for retention accuracy, but there is no base model performance for fine-tuned accuracy. Without including the base model’s evaluation on the benchmark, it is impossible to know how much the fine-tuning allowed the models to learn.

  • The method novelty is limited. Other works, such as SMoRA [1] and Gated LoRA, have also proposed similar input-dependent gating of individual rank-one components. So the mechanism is not new. Additionally, the theoretical arguments are either not novel or unrealistic. Proposition 1 is a standard textbook result. Proposition 2 is unrealistic because “define the pre-training and fine-tuning inputs as x_pt = x and x_ft = ax” — this assumes that the input for both pre-training and fine-tuning only differ by a scalar.

[1]https://arxiv.org/abs/2501.15103

Questions for authors

All questions are above in the weakness section.

2 comments

Thank you for the careful review.

W1

Thank you for the suggestion. We are working on experiments with newer models and datasets, but our servers are currently heavily loaded, making it difficult to finish within the rebuttal period. We will include these results in the final version.

W2

Thank you for pointing this out. You are right that our wording should be more precise about the scope of the evaluation. We will revise it to state explicitly that the models are fine-tuned on MetaMathQA and Magicoder, and evaluated on GSM8K and HumanEval, respectively.

W3

Thank you for pointing out this omission. Under our evaluation setup, the base LLaMA-2-7B scores approximately 13% on GSM8K, and the base Mistral-7B scores approximately 3% on HumanEval. DISeL reaches about 60% and above 70%, respectively. Thus, the models learn substantially while maintaining strong retention. We will add these base-model task scores to Figure 2 so that both the task improvement and the retention change are directly visible.

W4

Thank you for pointing out SMoRA. We missed this relevant reference and will add it. We agree that rank-wise input-dependent gating is not new. DISeL's contribution is its retention-focused design and the resulting adaptation-retention tradeoff. Unlike the tested ReLU Gated LoRA variant, DISeL learns its gates through a separate router rather than deriving them directly from the LoRA features. Unlike SMoRA, it does not use top-k selection or gate normalization, so all gates can become small together. Negative gate biases favor small corrections initially, and the router has its own higher learning rate. For clarity, exact equality with the base model at initialization comes from the zero-initialized LoRA factor, not negative gate biases alone.

Our new SMoRA comparison on Mistral/Magicoder gives DISeL higher reported task accuracy and retention at every tested rank. SMoRA's retention declines sharply with rank, while DISeL's remains stable. Full results and hyperparameters are in our response to Reviewer uGGJ (W1).

Proposition 1 uses standard conditional-expectation and Bayes calculations. Its purpose is to motivate the design, not to claim a new probability identity. In the regression setting, the best correction weights each component by the probability that it is needed. Several components can be needed together, so these weights need not sum to one. The practical affine sigmoid gates approximate the more general posterior scores. This is a design motivation, not a guarantee that DISeL learns the Bayes-optimal predictor in an LLM.

Proposition 2 deliberately studies a constructed case, not a model of how real pre-training and fine-tuning corpora differ. For a positively homogeneous correction, f(ax)=af(x)f(a\mathbf{x})=a f(\mathbf{x}). Therefore, making the correction small at x\mathbf{x} also restricts what it can do at axa\mathbf{x}, even when a substantial change is needed there. The proposition quantifies this restriction at the adapter level. It does not claim that real domains differ only by scaling or that this explains every observed failure. We will clarify this limited role as an illustrative failure case, rather than a general explanation of forgetting.

If we managed to address your concerns, we appreciate if you consider increasing your score.

Thank you for engaging in the discussion. However, my concerns regarding weakness 1 and weakness 2 are not addressed. Also, my second weakness was not only that the authors did not explicitly state which dataset the models were fine-tuned on, but also, importantly, that two benchmarks on a single model each are not enough to support any claim made by the paper. Thus, I will maintain my score.

Reviewer_6awsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper asserts that the main reason for catastrophic forgetting in PEFT is input-agnostic modification to the model parameters.
  • To counter this, the paper proposes DISeL, a method that learns to gate the individual ranks in LoRA using a learned, input dependent gating function.
  • The gating function is formulated as a logistic function g(x)=σ(Wgx+bg)g(x) = \sigma(W_gx + b_g) where σ\sigma is the sigmoid function, and x are the input features. bGb_G is initialized such that the output of g(x)g(x) is near 0.
  • WgW_g and adapter weights A,BA,B are assigned different learning rate, with WgW_g being traiend at a higher learning rate.
  • The method is evaluated on two setting:
    • Fine-tuning RoBERTa on GLUE, with perplexity n BookCorpusOpen, CC-News, and WIKIText-2 as the measure for catastrophic forgetting
    • Fine-tuning Llama 2 and Mistral (7B) on math reasoning and code generation tasks, with accuracy on commonsense, world knowledge, reading comprehension, and medical question answering as the evlauation for catastrophic forgetting.
Critical strengths
  • The method shows low forgetting and similar fine-tuning performance as standard adapters like LoRA, DoRA and AdaLoRA (as evidenced by performance on match and coding, section 4.2).
  • The interpretability analysis provides interesting insights into the rank gating behavior of layers (deeper layers have larger number of ope gates than shallow layers) and behavior across domains (e.g., math esibits larger number of gates that are ~1)
Critical weaknesses
  1. Misses a key citation SMoRA [1] which uses the exact same formulation as the gating function used by DISeL, with the differnce geing SMoRA uses a Softmax activation and topK selection over the ranks.
  2. The basic idea of introducing a gating function over ranks is similar to seminal work on adapters [2], where adapters (squeeze and expand MLPs with non-linearity in between) are inserted in series between attention/MLP block output and the residual
    • The gating function introduces an input dependent non-linearity, which means the adapters cannot be merged into the pretrained weights which is a key aspect of LoRA
    • Since DISeL loses this advantage, a natural question to ask is can the series adapters also exibit this behavior, and how does the fine-tuning performance compare with them?
    • If series adapters exibit the same behavior, the contribution of the paper is weakened.
  3. The paper uses very old models. It's hard to determine if the same findings will hold for more recent models like Qwen 3/Qwen 3.5.
  4. The motivation/reasoning behind the gate value analysis between odd and even token positions is not clear. In natural language sequences, odd vs. even token positions do not carry any meaning.

References

[1] Zhao et al., "Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-task Learning". ArXiv 2025, KDD 2026

[2] Houlsby et al., "Parameter-Efficient Transfer Learning for NLP ", ICML 2019

Questions for authors
  • Since the gating is input dependent, can the authors provide any analysis how which exact tokens the gate is open vs. closed? This analysis would provide more insights rather than simply analysing the gate values on a macro level.
2 comments

Thank you for the careful review. We address each concern below.

W1

Thank you for pointing out SMoRA [1]. We missed this relevant reference and will add it. We agree that the methods share the same rank-gated LoRA structure. SMoRA is designed for multi-task expert routing, while DISeL is designed specifically to preserve pre-trained capabilities during adaptation.

The key differences are in routing and initialization. SMoRA selects a fixed number of components and normalizes their weights. DISeL uses independent gates without top-k selection or normalization, so all gates can become small together and suppress the entire correction. Both methods initially produce zero correction through LoRA initialization. DISeL additionally initializes its gates near zero through negative biases and trains the router with a higher learning rate than the LoRA factors. These are retention-focused design choices, not a claim that rank-wise gating itself is new. Our comparison follows SMoRA's released sigmoid/top-k implementation, so the distinction is not simply sigmoid versus softmax.

In our Mistral-7B/Magicoder comparison, SMoRA's retention dropped sharply as rank increased, while DISeL's retention stayed stable and its task accuracy improved. DISeL had higher reported task accuracy and retention at every tested rank in this configuration. The full results and hyperparameters are provided in our response to Reviewer uGGJ (W1).

W2

The reviewer is correct that DISeL cannot generally be merged into a fixed weight matrix. It retains LoRA's low-rank parameterization, but not its mergeability. We will state this tradeoff explicitly.

Houlsby et al. [2] already introduce input-dependent nonlinear adapters and near-identity initialization. We do not claim either idea alone as new. Their serial adapters apply a bottleneck MLP after a sublayer. DISeL instead gates LoRA corrections inside individual linear projections, using a separate router to control how much each component contributes.

Our retention tests keep the same fine-tuned adapter enabled on both task-specific and general inputs. They do not rely on switching adapters or reverting to the base model. Serial adapters could also perform well in this setting. Our current results do not include that baseline and we try to include this comparison in the final version.

W3

Thank you for the suggestion. We agree that evaluating newer models would strengthen the paper. We are working on experiments with newer models and datasets, but our servers are currently heavily loaded, making it difficult to finish within the rebuttal period. We will include these results in the final version.

W4

Odd and even positions have no special linguistic meaning, and our analysis does not assume that they do. They simply split each response into two non-overlapping sets of tokens spread throughout the response. We average the gate values within each set. For each odd-token average, we select the most similar of the 50 even-token averages and check whether it comes from the same sample. All 50 samples come from the same domain.

The correct sample is identified in 50/50 math cases and 40/50 code cases, compared with 2% chance. This tests whether a sample has a consistent gate pattern across different tokens. It does not test whether odd and even positions play different roles. We will make this purpose clearer in Section 5.

Q1

Thank you for this helpful suggestion. We agree that showing gate values alongside individual tokens would provide a clearer picture of when the adapter is active. We are working on this token-level analysis and will share the results as soon as they are ready.

If we managed to address your concerns, we appreciate if you consider increasing your score.

Thank you for your rebuttal. My concern about comparison with SMoRA is addressed. However, the rest of my concerns remain unaddressed.

  • A comparison with Houlsby et al. style adapters is required and use of more recent models is necessary to evaluate the work. Hence, my concerns remain unaddressed.
  • The comparing the gate values at even and odd positions will not give a good estimate of the activation patterns, as the comparison is made between the individual samples of one set and a single sample chosen from the other set using the max cosine similarity. This can be an outlier.
  • Furthermore, this does not tell anything about the the activation pattern of the gates themselves, as the sigmoid values could just degenerate to a fixed random pattern for every sample very early in the training and not update later because of vanishing gradients of the sigmoid function (dead neuron problem).

For these reasons, I will maintain my score.

Reviewer_uGGJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DISeL (Dynamic Input-Sensitive LoRA), a parameter-efficient fine-tuning method that applies an independent, input-dependent sigmoid gate to each rank-one LoRA component, with negative gate-bias initialization to encourage conservative adaptation. The goal is to preserve pre-trained capabilities while learning new tasks. A Gaussian-mixture regression analysis motivates input-dependent component weighting, and a theoretical result identifies a limitation of positively homogeneous gates. Experiments on RoBERTa/GLUE, LLaMA 2-7B/MetaMathQA, and Mistral-7B/Magicoder evaluate target-task performance and retention against LoRA, DoRA, AdaLoRA, Gated LoRA, and full fine-tuning.

Critical strengths
  1. The paper addresses the practically important tradeoff between adapting to a new task and retaining pre-trained capabilities. In the authors' evaluations, DISeL achieves competitive target-task accuracy with stronger retention than several baselines, and the results remain relatively stable across adapter ranks and training checkpoints.

  2. The method is simple and reasonably motivated. The analysis of positively homogeneous gates offers a useful explanation for a possible adaptation-retention conflict, while the experiments include multiple backbone types, a three-seed RoBERTa study, and diagnostic analyses of gate activation across domains and layers.

Critical weaknesses
  1. The methodological distinction from prior input-dependent rank-wise LoRA methods remains limited. The authors acknowledge and compare Gated LoRA, but closely related approaches such as TopLoRA and SMoRA are not discussed. More importantly, the separate sigmoid router and near-closed initialization are not independently validated. Appendix III compares shared and per-rank sigmoid gating, yet the shared version remains competitive with 32 times fewer router parameters. Controlled ablations isolating the gating function, the separate router, and the bias initialization are needed to establish the novel contribution.

  2. The evidence does not fully distinguish learned input-selective adaptation from generally weaker effective updates. Figure 3 and Appendix VII show domain-dependent gate activations, which is useful evidence of input sensitivity. However, the near-closed initialization may also reduce the overall correction magnitude, and average gate values do not directly measure the actual output correction AG(x)Bx. Comparisons with static attenuation and update-norm-matched LoRA baselines, together with per-domain correction norms at comparable target-task accuracy, are needed. The theoretical propositions motivate the design but do not establish this mechanism in real language models.

  3. The rank-robustness claim lacks an analysis of effective adaptation capacity. The paper already reports both retention and target-task accuracy across ranks and acknowledges learning-rate effects; nevertheless, a larger nominal rank need not yield a larger effective update when many gates remain nearly closed. Effective rank and per-domain update magnitudes would clarify whether DISeL uses the added capacity or mainly restricts correction strength. Moreover, its roughly 60% GSM8K score is below the 66.5% reported for MetaMath-7B under different training and evaluation protocols. This is not a direct comparison, but it motivates controlled evaluation of retention at stronger, matched target-task performance.

Questions for authors

Please see the weaknesses.

2 comments

Thank you for the careful review.

W1

Thank you for pointing out SMoRA and TopLoRA. We missed these references and will add them. DISeL and SMoRA share the rank-gated LoRA structure, including a separate sigmoid router in SMoRA's released implementation. DISeL specifically targets retention, while SMoRA targets interference between tasks.

SMoRA selects a fixed number of components and normalizes their weights. DISeL uses neither top-k selection nor normalization, so all gates can become small together and suppress the entire correction.

Both methods start with zero correction through LoRA initialization. DISeL additionally uses negative gate biases to start with nearly closed gates, damping early corrections. We train the router with a higher learning rate than the LoRA factors. These choices target retention, although separate ablations are still needed to isolate their effects.

We evaluated SMoRA on Mistral-7B/Magicoder with k=r/8k=r/8 active components per token, gate weights normalized to sum to kk, and α=2r\alpha=2r (scaling α/k=16\alpha/k=16). The router and adapter share a learning rate of 2×10−42\times10^{-4} for ranks 16–128 and 10−410^{-4} for rank 256. The load-balancing bias update is 10−510^{-5} per microbatch.

Training uses three epochs, batch size 128, sequence limit 2048, and a cosine schedule with 2% warmup. We adapt all seven attention and MLP projections, with no dropout, gradient-norm clipping at 1.0, and weight decay of 0.01 for the adapter and zero for the router. The table compares the paper's DISeL results with SMoRA's best HumanEval checkpoint per rank, measuring retention at that same checkpoint. Scores are percentages, with retention averaged over 14 benchmarks.

RankDISeL HumanEvalSMoRA HumanEvalDISeL retentionSMoRA retention
1659.7654.2770.0565.09
3263.4151.2269.6261.35
6464.6352.4469.7755.78
12867.6847.5669.4050.36
25670.7354.2769.3944.47

DISeL has higher reported task accuracy and retention at every tested rank.

Appendix III compares one input-dependent gate per token and module with per-rank gates. At rank 32 on Mistral, per-rank gating improves HumanEval from 61.59% to 63.41% and retention from 69.36% to 69.62%. Shared gating remains a useful lower-cost alternative, with 32 times fewer router parameters, not a 32-fold reduction in total adapter size. Per-rank gating therefore helps but is not essential.

W2

DISeL retains pre-trained capabilities while adapting strongly. On Mistral, it exceeds 70% HumanEval accuracy while keeping retention within a fraction of a percentage point of the base model (Figure 2). Its retention benefit therefore does not require giving up strong adaptation.

The gates also change with the input. In Section 5, gate fingerprints identify individual samples with 100% accuracy on GSM8K and 80% on HumanEval, versus 2% chance. The router therefore does not apply one fixed scaling to every input.

We agree that our current results do not fully separate the benefit of input-dependent gating from smaller corrections. We will add the requested LoRA controls and per-domain correction measurements in the final version.

W3

Our rank-robustness claim concerns the accuracy-retention tradeoff, not full use of every component. On Mistral, increasing rank from 16 to 256 raises HumanEval accuracy from 59.76% to 70.73%, while retention stays around 69–70%.

Across 50 HumanEval solutions with Mistral, the average sum of gate values per token and adapted linear layer rises from 7.35 at rank 32 to 11.46 at rank 128 and 43.74 at rank 256. This shows greater gate activation, but does not measure how many independent directions contribute or how large the corrections are. Those measurements would clarify how the extra capacity is used.

The original MetaMath-7B study reports 66.5% GSM8K accuracy using full fine-tuning and zero-shot evaluation, while we use five-shot evaluation with strict answer matching. The gap from our roughly 60% therefore does not isolate DISeL's effect. Our comparisons use a common setup across methods, but testing retention at higher math accuracy would strengthen the evidence.

If we managed to address your concerns, we appreciate if you consider increasing your score.

Thanks for the additional experiments and explanations. The SMoRA comparison is helpful. I still have two concerns: the effect of the near-closed initialization has not been tested separately, and it’s not clear whether the retention gains mainly come from input-dependent gating or just smaller effective updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.