- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the discrepancy between weight-space approximation and actual nonlinear feature transmission in low-rank adaptation. The authors identify that small weight residuals can cause disproportionate mismatches in feature selection because of nonlinear gating mechanisms. The mitigation is Nonlinearity-Aware LoRA, a method that applies training-time gradient modulation. This modulation uses a first (mirco) controller to mask channels based on historical responsiveness and a second (macro) controller to scale gradients based on the operative regimes of the activation gate. Experiments across natural language generation, understanding, and image classification demonstrate improvements over standard LoRA and other baselines without adding inference overhead.
Critical strengths
The paper provides a interesting perspective on low-rank adaptation by moving beyond purely weight-based or gradient-based criteria to explicitly account for the nonlinear activation dynamics. The formulation of the gate-selection mismatch offers a interesting theoretical justification for why standard matrix fitting can be suboptimal for feature transmission.
The method is implemented entirely during the backward pass via gradient modulation to ensure zero inference overhead over standard LoRA.
The empirical results across multiple architectures and domains validate the practical utility of this approach.
Critical weaknesses
A significant weakness of the paper is its conceptual overlap and incremental novelty relative to prior works interpreting neural activations through multi-criteria decision making. The theoretical grounding heavily relies on existing frameworks, notably FleS (Cai et al., "Toward Principled Flexible Scaling for Self-Gated Neural Activation", ICLR 2026) and IIEU (Cai, "IIEU: Rethinking Neural Feature Activation from Decision-Making", ICCV 2023), which already establish the score-weight-response paradigm for activations. While applying this to low-rank adaptation is new, the core interpretive lens is inherited rather than innovated.
The operational mechanics of the proposed method share significant overlap with existing parameter efficient techniques. To the authors' credit, the paper does cite related methods like AdaLoRA (Zhang et al., 2023), EVA (Paischer et al., 2025), and LoRA+ (Hayou et al., 2024), and it includes AdaLoRA and LoRA+ as empirical baselines. However, the conceptual distance to these approaches remains relatively small. Specifically, NA-LoRA’s micro controller generates a temporal mask based on an importance score (the activation derivative magnitude). This is mechanically very similar to AdaLoRA’s sensitivity-based scoring, but applied to gradient modulation instead of rank allocation. Similarly, the "macro controller" scales the gradients of the A and B factors based on layer activation statistics. While the authors argue their statistics uniquely capture FFN nonlinearity, the resulting operation effectively yields an adaptive, activation-informed version of the factor-specific gradient scaling introduced by LoRA+.
Given this overlap, the paper needs to more strongly justify why its specific FFN-derived statistics offer a fundamentally novel capability compared to these existing sensitivity and scaling mechanisms.
Questions for authors
How sensitive is NA-LoRA to the choice of hyperparameters for the gradient scale bounds, and could an adaptive or trained mechanism remove the need for this manual tuning?
Can you provide a more explicit comparison isolating the benefits of the proposed gradient masking versus a simpler activation-magnitude based or Fisher-information based gradient masking?
Given the reliance on previous multi-criteria decision making frameworks for neural networks (appendix A.1), what is the most significant new theoretical hurdle this paper overcomes that was not already addressed by FleS (Cai et al., "Toward Principled Flexible Scaling for Self-Gated Neural Activation", ICLR 2026) or IIEU (Cai, "IIEU: Rethinking Neural Feature Activation from Decision-Making", ICCV 2023)?
Dear Reviewers,
The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.
Thank you very much for your time and valuable contributions!