acceptodds
Under review as a conference paper at ICLR 2027

SLoRA: Learning Structured Low-Rank Updates for Cross-Domain Transfer

Abstract

Parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) have become standard tools for adapting large pre-trained models. However, conventional LoRA applies uniform low-rank updates across all adaptation directions, which can limit flexibility and lead to unstable optimization or degraded generalization under domain shift, particularly in multimodal settings. We propose SLoRA, a lightweight extension of LoRA that introduces structured basis-space reparameterization and adaptive gating into low-rank updates to improve stability and robustness during transfer. Specifically, SLoRA transforms the low-rank adaptation space into a fixed orthogonal basis representation, applies learnable basis scaling to redistribute adaptation capacity across structured directions, and employs local basis modulation together with a global input-adaptive gate to control update strength. These mechanisms are batch-stabilized, allowing SLoRA to operate reliably on mixed-modality data without requiring explicit modality labels. We further provide theoretical insights suggesting that the proposed design promotes smoother update dynamics and acts as a form of structured adaptation regularization. Empirically, we evaluate SLoRA across four diverse domains, radiology, pathology, remote sensing, and art, on six multimodal visual question answering (VQA) benchmarks and eleven image classification datasets using multiple general-purpose vision and vision-language backbones. SLoRA consistently outperforms standard LoRA, achieving average gains of over 2.2% on VQA tasks and 2.9% on classification tasks, while remaining fully plug-and-play and parameter-efficient. These results demonstrate that structured low-rank reparameterization with input-adaptive modulation provides an effective and practical solution for robust parameter-efficient transfer under domain shift.

Primary area:
optimization
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Stable Adaptation, Structured Reparameterization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PvkfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLoRA, a structured low-rank adaptation method for parameter-efficient transfer under domain shift. It transforms the LoRA adaptation space into a fixed orthogonal basis using FFT, applies learnable structured basis modulation to redistribute adaptation capacity across low-rank directions, and introduces an input-adaptive gate to control the strength of the LoRA update. The method is validated across six multimodal VQA benchmarks and eleven image-classification datasets spanning radiology, pathology, remote sensing, and art, with experiments on multiple vision and vision-language backbones and comparisons against standard LoRA and several recent PEFT methods.

Critical strengths
  1. The motivation is clear: standard LoRA treats low-rank adaptation directions uniformly, while different directions may contribute differently under domain shift.
  2. The experimental evaluation is thorough and demonstrates the plug-and-play nature of SLoRA across multiple pre-trained architectures. Across diverse VQA and image-classification datasets, replacing LoRA with SLoRA generally leads to consistent performance improvements without model-specific adaptation. The paper also reports computational overhead relative to LoRA, making the practical trade-off between improved performance and additional training cost relatively transparent.
Critical weaknesses
  1. The theoretical justification is incomplete. The paper claims to provide theoretical insights and refers to a detailed theoretical analysis in the Appendix, but the submitted manuscript does not contain a formal theorem, proposition, proof, or theoretical bound. As a result, I am not fully convinced by the claimed theoretical roles of the individual components, particularly why the basis transformation, structured modulation, and adaptive gating should lead to smoother optimization or structured regularization beyond the empirical observations. Since the theoretical justification is highlighted as part of the contribution, this is a significant gap.
  2. There are noticeable citation-formatting issues.
  3. The Appendix is missing from the submitted manuscript. Several parts refer to the Appendix for additional theoretical analysis, implementation details, experimental settings, ablation studies, robustness analyses, and supplementary results, but the Appendix is not included in the submitted manuscript. Details are needed to assess the completeness and reproducibility of the method.
Questions for authors
  1. Could the authors report inference latency and memory overhead relative to standard LoRA? Table 5 evaluates training efficiency, but the input-adaptive modulation and gating are performed dynamically and can introduce additional inference cost.
1 comment
AuthorsOpenReview ↗

Response to Reviewer Pvkf

1. Theoretical properties

Appendix B distinguishes the static rank-rr matrix class from input-conditioned pointwise rank. The fixed transform preserves LoRA's rank bound; each reconstructed update has rank at most rr. Under the specified sigmoid, tanh, and clamp ranges, the basis multipliers lie in (0,2)(0,2). FFT/inverse FFT and the modulation are differentiable almost everywhere, permitting gradient-based training.

The input differential includes both the global gate and the input-dependent reconstructed factor, as well as normalization. Shared batch descriptors can also couple samples. These are properties of the parameterization and its derivatives; the training curves supply the optimization evidence rather than a formal convergence result.

2. Appendix and citations

The complete appendix follows the references. Appendix B provides the mathematical properties; C gives implementation details and controlled experiments; D discusses the findings; and E contains Algorithm 1 (p. 32). The pure-transform reconstruction and gradient check appears in Table 32 (p. 29). Narrative and parenthetical citations for CLIP and LoRA have been corrected, and LoRA's bibliography entry now uses its ICLR conference information.

3. Inference latency and memory

Table 35 (p. 30) reports batch sizes 1 and 32. LoRA, FouRA, and SLoRA have mean latencies of 18.0/53.0, 20.0/60.0, and 20.3/60.0 ms, respectively; corresponding SDs are 0.8/2.7, 0.9/2.9, and 1.0/3.0 ms, with P95 values of 20.0/59.5, 22.4/67.2, and 22.7/67.5 ms. Peak memory is 1.05/1.38, 1.13/1.52, and 1.15/1.53 GB. SLoRA's mean/P95/memory overheads relative to LoRA are 12.8/13.2%, 13.50/13.45%, and 9.5/10.9% for the two batch sizes.

Panel B of Table 35 specifies the common remeasurement protocol: one NVIDIA V100 32GB GPU, FP32 precision, 224×224 images, 77-token text inputs, 50 warm-up iterations, and 500 synchronized CUDA-event-timed forward passes. Timing includes adapter reconstruction and gating but excludes model loading and data preparation. Table 5 (p. 8) separately reports full training-step overhead of 13.3%.

Reviewer_zf86OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SLoRA, a lightweight add-on to LoRA that reparameterizes the low-rank adaptation space with a fixed orthogonal (FFT) basis and adds input-adaptive gating, aiming to make cross-domain transfer more stable for vision and vision-language models. It's evaluated across six VQA benchmarks, eleven classification datasets, and several backbones, and consistently beats standard LoRA.

Critical strengths

Across six cross-domain VQA benchmarks, eleven classification datasets, and six vision/VL backbones, SLoRA consistently beats standard LoRA (about +2.2% on VQA, +2.9% on classification) and in many cases matches or exceeds full fine-tuning. That breadth is more convincing than the usual one- or two-dataset PEFT study. They test sensitivity to the partition ratio ρ, the clamp range, batch- vs per-sample gating, activation functions, and random-seed stability, plus an extension to multimodal LLMs. This makes the "what actually matters" claims fairly credible. The method is lightweight and practical: plug-and-play on top of LoRA, negligible extra parameters, no modality labels needed, and only ~13% extra step time. The pseudocode and full results in the appendix also help reproducibility.

Critical weaknesses

The FFT "structured basis modulation" is just a fixed circular rescaling of the LoRA factors. Since F is linear and parameter-free, ˜W_down = F⁻¹( diag(m) · F(W_down) ) is a fixed linear operator on the rank dimension — a structured rescaling, not a new adaptation mechanism. Choosing the FFT also implicitly assumes the rank directions have a meaningful low-/high-frequency ordering (so that two-way splitting is sensible); that prior is never justified, and the LoRA rank axis has no inherent frequency semantics.

"Local basis modulation" is a misnomer. In ˜W_f^(i) = W_f^(i) ⊙ (1 + β·tanh(g_i)·d(x)), both β and d(x) are scalars shared across the whole group, and only tanh(g_i) varies per coefficient — and it is a learned constant, not input-dependent. So the modulation is really "group-level scalar scaling multiplied by a fixed per-coefficient weight," not the per-coefficient, input-adaptive modulation the name suggests.

The "input-adaptive" gating likely degenerates to a near-constant scaling. a(x) = σ(mean_i |x_i|) and d(x) = clamp(σ(W_sᵀx), 0.3, 1.0) are both single scalars that are sigmoid-squashed, clamped, and batch-averaged. After layer normalization, mean_i |x_i| reflects the learned scale/shift rather than a sample's activation strength, so a(x) may vary little across inputs — which undercuts both the "input-adaptive" claim and the "works without modality labels" framing (the latter may simply mean the gate is too weak to discriminate). Please clarify whether a(x) uses pre- or post-norm activations.

Questions for authors

The whole premise is that LoRA's uniform adaptation is suboptimal, so we need structured + adaptive modulation. But the most direct test of that premise — plain LoRA plus a learnable per-rank scaling vector (a single r-dim vector weighting the rank directions) — is absent from the ablations (Table 17 covers "mixing" and "fixed orthogonal mixing," but not per-rank scaling). If per-rank scaling alone matches SLoRA, then the FFT basis, grouping, and gating are unnecessary complexity on top of a trivial idea. Could you add this baseline and report it?

Is the gain "structured adaptation," or just regularization?​ Under domain shift, LoRA can overfit; the fix might be ordinary regularization (weight decay, dropout, or a smaller rank). No "LoRA + strong regularization" baseline is reported. What evidence shows the improvement comes from the structured/adaptive mechanism specifically, rather than from generic regularization that any of these simpler knobs would provide?

The adaptation is sample-level, but the problem is domain-level. The setup is source-free, and the "input-adaptive" gate responds to per-sample activation statistics, not to the domain. Domain shift typically calls for domain-level adaptation (a fixed per-domain bias), whereas SLoRA modulates per sample. How does a sample-level gate address a domain-level problem? Do you have evidence that the gate actually separates domains — e.g., that a(x)/d(x) differ systematically across domains rather than just across samples?

1 comment
AuthorsOpenReview ↗

Response to Reviewer zf86

1. Expressivity and basis choice

A fixed FFT with fixed scaling can be absorbed into LoRA's trainable factors. Section 2.1 and Appendix B distinguish this static matrix class from input-conditioned modulation: each reconstructed update has rank at most rr, but its coefficients can vary with the input. FFT provides coordinates along the latent rank axis, not semantic frequencies.

Table 33 (p. 29) matches r=16r=16, α=32\alpha=32, training budget, preprocessing, gates, and nine independent scaling coefficients. The 16 real coordinates share coefficients via p(j)=⌊9j/16⌋p(j)=\lfloor 9j/16\rfloor, whereas rFFT uses nine nonredundant complex positions. FFT reaches 69.1/60.8/70.4% on VQA-RAD/PathVQA/OVQA, compared with 68.9/60.6/70.4% for DCT; they tie on OVQA. The original basis comparison is separate because its budgets differ. Table 32 (p. 29) checks reconstruction and gradients under a pure transform round trip.

2. Direct per-rank scaling

On CLIP, LoRA achieves 67.1/59.4/68.1%, learnable per-rank scaling 67.6/59.7/68.6%, per-rank scaling with a gate 68.1/60.0/69.2%, and SLoRA 69.2/60.9/70.5%. Across the nine backbone–dataset pairs in Table 9 (p. 9), SLoRA exceeds direct scaling by 1.79 percentage points (pp) and scaling plus gating by 1.26 pp on average. The comparison isolates what these simpler changes recover before adding the full structured adapter.

3. Regularization and preprocessing

Table 25 (p. 25) evaluates nine LoRA tuning candidates rather than a full factorial grid. Starting from r=16r=16, α=32\alpha=32, dropout 0, and weight decay 10−610^{-6}, one-factor alternatives adjust rank (8/32), α\alpha (16/64), dropout (0.05/0.10), or weight decay (10−410^{-4}/10−210^{-2}). The reported protocol uses AdamW, learning rate 10−410^{-4}, 300 epochs, batch size 32, common insertion locations and splits, and validation-accuracy selection before test evaluation.

The selected candidate IDs on VQA-RAD/PathVQA/OVQA are 7/6/8 for CLIP, 6/8/6 for BLIP, and 8/6/7 for GIT. All retain r=16r=16 and α=32\alpha=32. Candidate 6 uses dropout 0.05 and decay 10−610^{-6}; candidate 7 uses dropout 0.10 and decay 10−610^{-6}; candidate 8 uses dropout 0 and decay 10−410^{-4}. Their corresponding tuned accuracies are 68.0/60.1/69.1% (CLIP), 68.6/60.6/68.8% (BLIP), and 72.3/60.3/72.1% (GIT). Across nine pairs, SLoRA leads tuned LoRA by 1.24 pp.

Table 23 (p. 24) matches LayerNorm and dropout individually and jointly. The CLIP controls obtain 67.5/59.7/68.5%, 67.3/59.5/68.4%, and 67.8/59.9/68.9%, respectively. SLoRA exceeds the joint control by 1.4/1.0/1.6 pp (mean 1.33 pp). Normalization settings, trainable affine flags, dropout placement, and evaluation behavior are matched across the reference branches.

Table 24 (p. 24) reports three common seeds. For per-rank scaling, scaling plus gate, LoRA-Tuned, and SLoRA, VQA-RAD mean±SD is 67.6±0.5, 68.1±0.5, 68.0±0.6, and 69.2±0.4%; PathVQA is 59.7±0.5, 60.0±0.5, 60.1±0.5, and 60.9±0.5%; OVQA is 68.6±0.5, 69.2±0.6, 69.1±0.6, and 70.5±0.4%. The paired configurations use common seeds and splits; these results describe variation without claiming a significance test.

4. Input conditioning

The vectors g1,g2g_1,g_2 are learned basis-wise constants; β\beta balances the two groups, and dBd_B is a shared input-conditioned scalar. The global gate is concentrated near 0.69 (within-layer sample SD 0.006–0.007), whereas the local signal varies more (SD 0.11–0.12). Retraining with fixed global or local coefficients reduces mean accuracy by 0.44 pp; inference-only replacement reduces it by 0.24/0.36 pp. A retrained learnable global scalar reduces it by 0.31 pp. These controls support the contribution of learned basis-wise structure together with a smaller dynamic increment (Tables 26–27; C.14).

5. Domain shift

The model is trained with labeled target-domain samples and no source-domain data. Input conditioning adjusts residual updates within the target-trained adapter rather than performing domain routing. For the same checkpoint and visual layers, Table 30 (p. 27) reports global/local mean ranges of 0.004/0.028 across domains, smaller than within-domain standard deviations. This measures input-distribution-associated variation, not explicit domain classification.

Reviewer_152zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SLoRA, a parameter-efficient fine-tuning method designed to improve the adaptation of pretrained vision and vision-language models to specialized domains. The method introduces two main mechanisms: (1) structured basis-space modulation, which applies a Fourier transform, group-wise partitioning, and learnable scaling to adjust the contributions of different low-rank basis directions; and (2) input-adaptive gating, which dynamically controls the magnitude of low-rank updates based on input activations.The authors evaluate SLoRA on six VQA benchmarks and eleven image classification datasets spanning medical imaging, remote sensing, and art, using multiple pretrained backbones. The reported results show average accuracy improvements of approximately 2.2 and 2.9 percentage points over standard LoRA on VQA and classification tasks, respectively. Comparisons with other PEFT methods, component ablations, and training dynamics analyses are also presented.

Critical strengths
  1. SLoRA incorporates structured modulation and input-dependent update control while largely preserving the parameter efficiency of LoRA. The reported efficiency experiments indicate a relatively moderate increase in trainable parameters and training cost. Component ablations further suggest that combining the proposed mechanisms is more effective than using partial configurations. These properties make the approach potentially useful for practical parameter-efficient adaptation.

  2. The paper provides relatively comprehensive experiments across multiple model architectures, task types, and specialized domains. SLoRA demonstrates reasonably consistent improvements over standard LoRA on six VQA benchmarks and achieves encouraging results on eleven image classification datasets. The comparisons also include relevant PEFT methods, such as DoRA, AdaLoRA, FouRA, FourierFT, and LoCA. This broad evaluation provides useful evidence of the method's practical effectiveness and applicability across architectures.

Critical weaknesses

One central motivation of the paper is that conventional LoRA does not explicitly distinguish the importance of different low-rank directions. However, standard LoRA can already learn direction-dependent contributions through its trainable low-rank factors. Therefore, characterizing conventional LoRA as uniformly adapting all directions is somewhat imprecise.

More importantly, LoRA's latent rank dimensions do not have an inherent frequency interpretation or canonical ordering. The paper does not sufficiently explain why applying a Fourier transform along these dimensions should provide a more suitable inductive bias than alternative basis transformations or direct rank-wise scaling.

Questions for authors
  1. Given that LoRA's latent rank dimensions have no inherent frequency interpretation, could the authors further justify the choice of FFT?

  2. Section 2.1 indicates that (d(x)) is averaged across the mini-batch during training. Is the same operation performed during inference? Does an individual sample produce consistent predictions under different batch sizes or batch compositions?

  3. Could the authors report the empirical distribution of (a(x)) across datasets and individual inputs, and compare SLoRA with a fixed gate, a learnable scalar gate, or standard LoRA with a retuned scaling factor? These comparisons would help distinguish genuine input-adaptive benefits from simple rescaling effects. Additionally, Section 2.2 states that the standard (\alpha/r) scaling is retained, but this factor is not explicitly included in the final output equation. Please clarify the implementation.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 152z

1. LoRA motivation and FFT

The original uniform-direction description was imprecise. The revised Abstract, Introduction, and Section 2 clarify that LoRA already learns unequal contributions across latent rank directions through its trainable factors. SLoRA instead makes their modulation explicit through a fixed rank-space transform, learned basis-wise coefficients, a shared input-conditioned signal, and a residual gate. FFT operates on latent rank coordinates, which have no intrinsic semantic frequency ordering; the static rank-rr matrix class is unchanged.

Table 33 (p. 29) compares transforms at r=16r=16, α=32\alpha=32, and nine independent scaling coefficients. For real transforms, 16 coordinates share these coefficients via p(j)=⌊9j/16⌋p(j)=\lfloor 9j/16\rfloor; rFFT uses nine nonredundant complex positions. FFT achieves 69.1/60.8/70.4% on VQA-RAD/PathVQA/OVQA, versus 68.9/60.6/70.4% for DCT: a 0.2-point gain on the first two and a tie on OVQA. The earlier basis comparison is reported separately because its scaling budgets differ. Table 32 (p. 29) checks reconstruction and gradients for a pure FFT–inverse FFT round trip.

2. Input adaptivity and scaling

Table 9 (p. 9) compares SLoRA with direct rank scaling and tuned LoRA. Across nine backbone–dataset pairs, SLoRA exceeds LoRA-Tuned by 1.24 percentage points (pp) and per-rank scaling with an input gate by 1.26 pp on average. Fixed-scalar, learnable-scalar, and retuned-α\alpha controls leave average gaps of 1.86, 1.56, and 1.64 pp, respectively. Matching LayerNorm and dropout leaves a 1.33 pp gap to SLoRA.

Table 26 (p. 25) distinguishes retraining from inference-only coefficient replacement. Retrained fixed global or local coefficients each lower mean accuracy by 0.44 pp; a learned global scalar lowers it by 0.31 pp. Replacing global or local coefficients only at inference lowers accuracy by 0.24 or 0.36 pp. Thus input dynamics contribute modestly beyond the learned basis-wise structure. In Table 27 (p. 26), the global gate averages 0.690/0.691/0.693 with within-layer sample SD 0.006–0.007, while local modulation averages 0.51/0.50/0.52 with SD 0.11–0.12. These are sample-level statistics before aggregation.

3. Batch composition and inference

The local coefficient is dB=clamp⁡(B−1∑iσ(Ws⊤zi),0.3,1)d_B=\operatorname{clamp}(B^{-1}\sum_i\sigma(W_s^\top z_i),0.3,1); the global gate is aB=σ(B−1∑imi)a_B=\sigma(B^{-1}\sum_i m_i), where mim_i is the mean absolute magnitude of normalized input ziz_i. The local sigmoid precedes averaging and clamping; the global sigmoid follows averaging. Primary task evaluation uses batch size B=1B=1. Table 28 (p. 27) tests shared gates with different accompanying samples at B=8B=8 and B=32B=32.

Accuracy on VQA-RAD/PathVQA/OVQA is 69.2/60.9/70.5% at B=1B=1, 69.0/60.8/70.3% at B=8B=8, and 68.9/60.7/70.1% at B=32B=32. Agreement with the corresponding B=1B=1 prediction is 98.8/99.0/98.7% at B=8B=8 and 98.2/98.5/98.0% at B=32B=32. Table 28 also reports mean and maximum logit differences; C.14 reports post-aggregation clamp rates. Per-sample coefficients remove companion dependence within the adapter, while whole-model consistency may also depend on padding and numerical computation.

4. Final output equation

Equation 1 (p. 4) explicitly retains the standard scaling:

yi=yorig,i+αraBWup(W~down(B)ui).y_i=y_{\mathrm{orig},i}+\frac{\alpha}{r}a_B W_{\mathrm{up}}\bigl(\widetilde W_{\mathrm{down}}(B)u_i\bigr).

Figure 3 and Algorithm 1 use the same residual update, with branch dropout enabled only during training. The reconstructed down-projection depends on dBd_B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.