acceptodds
Under review as a conference paper at ICLR 2027

Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

Abstract

Low-rank adaptation (LoRA) constrains weight updates, yet their matrix-space fit does not by itself determine which features a nonlinear feed-forward network (FFN) transmits. We identify gate-selection mismatch: target and adapter scores can induce different transmission coefficients even when their weight-space discrepancy is small. To explain this mechanism, we introduce a multi-criteria decision-making interpretation of FFN evaluation and selection, then develop a homogeneity-based response framework for low-rank adaptation. It links relative gate response, channel aggregation, and the coupled FFN output, motivating persistent channel prioritization and regime-conditioned coordination. We instantiate these principles in NA-LoRA, using a temporally smoothed response-sensitive channel mask and gradient modulation coordinated across the gate, value, and output projections. The mask generalizes across FFN architectures, while the current macro rule specializes to SwiGLU's characteristic geometry. Language and vision-language experiments show improvements over vanilla LoRA and competitive results against strong PEFT variants; component ablations and nonlinear-state diagnostics support the proposed framework and design. Both training-only controls preserve LoRA's rank, forward parameterization, and inference cost.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Nonlinear Activations, Transformer Adaptation, Functional Approximation, Gradient Modulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_rFHROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the discrepancy between weight-space approximation and actual nonlinear feature transmission in low-rank adaptation. The authors identify that small weight residuals can cause disproportionate mismatches in feature selection because of nonlinear gating mechanisms. The mitigation is Nonlinearity-Aware LoRA, a method that applies training-time gradient modulation. This modulation uses a first (mirco) controller to mask channels based on historical responsiveness and a second (macro) controller to scale gradients based on the operative regimes of the activation gate. Experiments across natural language generation, understanding, and image classification demonstrate improvements over standard LoRA and other baselines without adding inference overhead.

Critical strengths

The paper provides a interesting perspective on low-rank adaptation by moving beyond purely weight-based or gradient-based criteria to explicitly account for the nonlinear activation dynamics. The formulation of the gate-selection mismatch offers a interesting theoretical justification for why standard matrix fitting can be suboptimal for feature transmission.

The method is implemented entirely during the backward pass via gradient modulation to ensure zero inference overhead over standard LoRA.

The empirical results across multiple architectures and domains validate the practical utility of this approach.

Critical weaknesses

A significant weakness of the paper is its conceptual overlap and incremental novelty relative to prior works interpreting neural activations through multi-criteria decision making. The theoretical grounding heavily relies on existing frameworks, notably FleS (Cai et al., "Toward Principled Flexible Scaling for Self-Gated Neural Activation", ICLR 2026) and IIEU (Cai, "IIEU: Rethinking Neural Feature Activation from Decision-Making", ICCV 2023), which already establish the score-weight-response paradigm for activations. While applying this to low-rank adaptation is new, the core interpretive lens is inherited rather than innovated.

The operational mechanics of the proposed method share significant overlap with existing parameter efficient techniques. To the authors' credit, the paper does cite related methods like AdaLoRA (Zhang et al., 2023), EVA (Paischer et al., 2025), and LoRA+ (Hayou et al., 2024), and it includes AdaLoRA and LoRA+ as empirical baselines. However, the conceptual distance to these approaches remains relatively small. Specifically, NA-LoRA’s micro controller generates a temporal mask based on an importance score (the activation derivative magnitude). This is mechanically very similar to AdaLoRA’s sensitivity-based scoring, but applied to gradient modulation instead of rank allocation. Similarly, the "macro controller" scales the gradients of the A and B factors based on layer activation statistics. While the authors argue their statistics uniquely capture FFN nonlinearity, the resulting operation effectively yields an adaptive, activation-informed version of the factor-specific gradient scaling introduced by LoRA+.

Given this overlap, the paper needs to more strongly justify why its specific FFN-derived statistics offer a fundamentally novel capability compared to these existing sensitivity and scaling mechanisms.

Questions for authors

How sensitive is NA-LoRA to the choice of hyperparameters for the gradient scale bounds, and could an adaptive or trained mechanism remove the need for this manual tuning?

Can you provide a more explicit comparison isolating the benefits of the proposed gradient masking versus a simpler activation-magnitude based or Fisher-information based gradient masking?

Given the reliance on previous multi-criteria decision making frameworks for neural networks (appendix A.1), what is the most significant new theoretical hurdle this paper overcomes that was not already addressed by FleS (Cai et al., "Toward Principled Flexible Scaling for Self-Gated Neural Activation", ICLR 2026) or IIEU (Cai, "IIEU: Rethinking Neural Feature Activation from Decision-Making", ICCV 2023)?

Reviewer_XGSHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NA-LoRA, a non-linearity-aware low-rank adaptation method, designed to address the mismatch in gating selection that exists in traditional LoRA when applied to non-linear feedforward networks. The method consists of two modules: the Micro Controller, which adjusts channel gradients based on response sensitivity, and the Macro Controller, which dynamically adjusts gradients across different projections based on gating states. The authors conducted experiments on language and vision tasks, and the results demonstrate that this method achieves performance improvements over traditional LoRA without incurring additional inference overhead.

Critical strengths

S1: This paper reexamines the low-rank adaptation problem from the perspective of nonlinear gating selection, rather than focusing solely on traditional weight space approximation. The related theoretical analysis provides a new perspective for understanding the impact of nonlinear FFNs on low-rank adaptation.

S2: The proposed method does not require any changes to the original LoRA network architecture and does not introduce additional inference overhead. Furthermore, experimental results across different models and tasks demonstrate the method’s effectiveness and practical value.

Critical weaknesses

W1: Eq. (3) shows that the functional error of the FFN is influenced by the gate value and output projection simultaneously, whereas Eq. (9) relies primarily on the derivative of the gating response for channel selection. Therefore, I think the authors should further explain whether this metric can accurately reflect the importance of the channels.

W2: The Macro Controller employs a heuristic gradient scaling strategy in Eq. (10). However, its connection to the theoretical analysis in Eqs. (6)–(7) is unclear and requires further justification.

W3: The microcontroller applies a channel mask only to the B matrix of LoRA, while the A matrix remains unchanged. Given that both matrices jointly determine the final weight updates. Since both matrices contribute to the final weight updates, why is the mask applied only to B?

W4: In Figures 3 and 4, the authors primarily illustrate changes in gate states, but I think larger state changes do not necessarily imply smaller selection errors. Therefore, I recommend that the authors conduct a comparison with a fully parameterized fine-tuned model to directly verify whether NA-LoRA reduces gate-selection mismatch.

W5: The authors only verify the overall contributions of the Micro and Macro Controllers in Table 4, but lack an analysis of their key internal designs. I recommend adding control experiments, e.g., random channel selection, removal of EMA smoothing, and fixed gradient scaling.

W6: I think some sensitivity experiments are lacking, e,.g., in Eqs. (9)–(10), the authors introduce several hyperparameters, but their sensitivity has not been evaluated. I also suggest reporting the actual training time and GPU memory usage.

Questions for authors

Some issues regarding the design of the methods and the experiments need to be clarified. Please refer to W1–W6.

Reviewer_hXyuOpenReview ↗
Rating
1(−1)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Authors propose NALora, which aims to account for the nonlinear behavior of the FFNs in this paper. Authors argue that conventional lora focuses on weight approximation, which might not preserve the mapping behavior of the model.

To figure out this discrepancy, authors introduce the gate selection mismatch to analyze how perturbations in gate scores influence feature and module output.

To bridge the gap, authors introduce macro level lora and channel level lora. Experiments on various models shoe improvements over basic lora and competitive results against recent methods.

Critical strengths

I find the central observation of this paper interesting and potentially insightful. To the best of my knowledge, this is among the first works to explicitly investigate LoRA from the perspective of function approximation, rather than focusing primarily on the rank constraints or parameter efficiency of low-rank updates. This perspective provides an alternative way to understand the representational limitations of conventional LoRA and motivates the consideration of nonlinear transformations within low-rank adaptation. In particular, the paper raises an interesting question about whether the effectiveness of parameter-efficient fine-tuning is limited not only by the rank of the adaptation matrices, but also by the class of functions that these updates can represent. I appreciate this attempt to provide a more fundamental understanding of LoRA's approximation capabilities, as it may offer useful insights for designing more expressive parameter-efficient adaptation methods.

Another strength of the proposed NA-LoRA method is its practical simplicity and efficiency. Despite introducing additional nonlinear modeling capabilities, the method largely preserves the lightweight parameterization that makes conventional LoRA attractive. Importantly, it does not substantially increase the number of trainable parameters or introduce significant additional inference overhead, making it potentially applicable to existing parameter-efficient fine-tuning pipelines. I consider this an important advantage, since improvements in adaptation performance are considerably more valuable when they do not come at the expense of the computational and memory efficiency that motivates the use of LoRA in the first place.

The empirical evaluation further supports the practical relevance of the proposed approach. Across multiple downstream tasks and model architectures, NA-LoRA demonstrates consistent improvements over conventional LoRA, suggesting that the benefits of nonlinear adaptation are not restricted to a particular experimental setting. The consistency of these improvements is encouraging and provides empirical support for the paper's central motivation. Overall, I appreciate that the work combines an interesting approximation-theoretic perspective with a relatively simple and practically efficient method, rather than relying on a substantially more complicated adaptation architecture to achieve better performance.

Critical weaknesses

My major concern is clarity, the main body should be self contained, please move the essential math part from the appendix to the main body. Another presentation issue is the excessive use of the "intuition" statements through the paper. In my view, most of these intuition statements do not need to be presented as standalone, formally labeled components. They would be more effective if integrated directly into the surrounding paragraphs, where they could motivate the relevant definitions, derivations, and algorithmic choices in a continuous narrative. Overall I believe extensive revising is need before the resubmission or paper acceptance.

Technically, I found the proposed framework and optimzaiton method is not well connected. The paper motivates NA-LoRA through a functional approximation perspective. In Eq. (3), the local functional discrepancy is characterized by

Q(E)=Ex[∣Jg(x)Ex∣22],Q(E)=\mathbb{E}_x\left[|J_g(x)Ex|_2^2\right],

where the Jacobian incorporates nonlinear gate derivatives, transmitted values, and downstream projection effects.

However, the micro-level controller in Eqs. (8)–(9) prioritizes channels according to

Eb,s[∣ψ′(zb,s,c)∣].\mathbb{E}_{b,s}\left[|\psi'(z{b,s,c})|\right].

Furthermore for the experimental results, While the experimental results are encouraging, several reported performance improvements are small relative to the standard deviations across runs, raising concerns about their statistical robustness. Could the authors provide formal statistical hypothesis tests to demonstrate whether the reported improvements over the strongest baselines and the individual ablation variants are statistically significant?

Questions for authors

See the weakness

Reviewer_YSseOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NA-LoRA uses the nonlinear state of an FFN to guide fixed-rank adaptation. The paper first separates gate and value mismatch and shows why matrix approximation error can rank updates differently from their effect on the FFN output. The method then combines a smoothed channel mask on the score-producing LoRA B gradient with coordinated gradient scaling across SwiGLU's three projections. Both controls operate during training. Experiments use two Llama models, T5, and CLIP, with mask-only control for T5 and CLIP.

Critical strengths

Equation (3) gives a concrete reason to look beyond matrix fit: the gate derivative, transmitted values, and output mixing all affect which residuals matter. The rank-one example makes this easy to see. The treatment of the different activations is careful, and Appendix F.5 usefully explains the effect of AdamW normalization on gradient scaling. These qualifications make the analysis more credible.

The gains over plain LoRA are substantial in several settings. On Llama-3.1-8B, rank-8 NA-LoRA improves GSM8K from 71.52 to 75.16 and HumanEval from 41.05 to 46.34. Table 5 also shows gains with matched FFN-only targets. Results on T5 and CLIP give some evidence that the mask transfers across activation families. Keeping the usual LoRA forward pass and mergeable factors is attractive in practice.

Critical weaknesses

I am less convinced by the evidence for the specific controller. Table 4 shows what happens when Micro or Macro is removed, but it does not show whether the gate statistics help more than a simpler change to gradient history. A fixed or shuffled mask and a state-independent scale schedule would help answer this. Appendix F.5 makes that comparison particularly relevant: AdamW can largely cancel a constant positive gradient multiplier. The component gains also need a closer look across seeds. Removing Micro lowers GSM8K by 0.63 points, against reported standard deviations of 1.34 and 1.82. Figures 3-4 show greater gate-state movement, but do not show whether it reduces the motivating mismatch. The authors acknowledge this limitation in Appendix K; the remaining issue is the missing link between that diagnosis and the benefit of the chosen controls.

The strongest-baseline comparisons leave some uncertainty. T5 is essentially tied with LoRA-Pro (88.43 versus 88.44), CLIP is 0.13 points above GoRA, and Llama-3.1-8B GSM8K is 0.23 points above LoRA-Pro. Small margins can still be useful, but they make protocol and tuning differences consequential. Appendix I.4 explains many inherited results, while the source or rerun status of most Llama-3.1-8B baseline rows remains unclear. It is also unclear how the extra controller settings were selected. I lean toward weak rejection because I would like stronger evidence that these controls give a reliable benefit over simpler alternatives and well-tuned baselines.

Questions for authors
  1. On one of the existing tasks, how does the method compare with a fixed mask, a shuffled assignment of the same mask values, and a state-independent projection-scale schedule? Paired results across seeds would also show how consistent the small component gains in Table 4 are.

  2. Which Llama-3.1-8B baseline rows were rerun? For the inherited rows, were the training recipe, evaluation or judge, and checkpoint selection matched? Please also explain how the controller settings were chosen. A comparison with one strong baseline under the same tuning budget would address this concern.

  3. Is there a small controlled probe showing reduced gate or FFN-output discrepancy toward an explicit target? For example, using common layer inputs and an available full-fine-tuning reference would help connect the response analysis to the learned adapters.

Comments on the paper

1 comment

Dear Reviewers,

The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.