- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SLoRA, a structured low-rank adaptation method for parameter-efficient transfer under domain shift. It transforms the LoRA adaptation space into a fixed orthogonal basis using FFT, applies learnable structured basis modulation to redistribute adaptation capacity across low-rank directions, and introduces an input-adaptive gate to control the strength of the LoRA update. The method is validated across six multimodal VQA benchmarks and eleven image-classification datasets spanning radiology, pathology, remote sensing, and art, with experiments on multiple vision and vision-language backbones and comparisons against standard LoRA and several recent PEFT methods.
Critical strengths
- The motivation is clear: standard LoRA treats low-rank adaptation directions uniformly, while different directions may contribute differently under domain shift.
- The experimental evaluation is thorough and demonstrates the plug-and-play nature of SLoRA across multiple pre-trained architectures. Across diverse VQA and image-classification datasets, replacing LoRA with SLoRA generally leads to consistent performance improvements without model-specific adaptation. The paper also reports computational overhead relative to LoRA, making the practical trade-off between improved performance and additional training cost relatively transparent.
Critical weaknesses
- The theoretical justification is incomplete. The paper claims to provide theoretical insights and refers to a detailed theoretical analysis in the Appendix, but the submitted manuscript does not contain a formal theorem, proposition, proof, or theoretical bound. As a result, I am not fully convinced by the claimed theoretical roles of the individual components, particularly why the basis transformation, structured modulation, and adaptive gating should lead to smoother optimization or structured regularization beyond the empirical observations. Since the theoretical justification is highlighted as part of the contribution, this is a significant gap.
- There are noticeable citation-formatting issues.
- The Appendix is missing from the submitted manuscript. Several parts refer to the Appendix for additional theoretical analysis, implementation details, experimental settings, ablation studies, robustness analyses, and supplementary results, but the Appendix is not included in the submitted manuscript. Details are needed to assess the completeness and reproducibility of the method.
Questions for authors
- Could the authors report inference latency and memory overhead relative to standard LoRA? Table 5 evaluates training efficiency, but the input-adaptive modulation and gating are performed dynamically and can introduce additional inference cost.
Response to Reviewer Pvkf
1. Theoretical properties
Appendix B distinguishes the static rank-r matrix class from input-conditioned pointwise rank. The fixed transform preserves LoRA's rank bound; each reconstructed update has rank at most r. Under the specified sigmoid, tanh, and clamp ranges, the basis multipliers lie in (0,2). FFT/inverse FFT and the modulation are differentiable almost everywhere, permitting gradient-based training.
The input differential includes both the global gate and the input-dependent reconstructed factor, as well as normalization. Shared batch descriptors can also couple samples. These are properties of the parameterization and its derivatives; the training curves supply the optimization evidence rather than a formal convergence result.
2. Appendix and citations
The complete appendix follows the references. Appendix B provides the mathematical properties; C gives implementation details and controlled experiments; D discusses the findings; and E contains Algorithm 1 (p. 32). The pure-transform reconstruction and gradient check appears in Table 32 (p. 29). Narrative and parenthetical citations for CLIP and LoRA have been corrected, and LoRA's bibliography entry now uses its ICLR conference information.
3. Inference latency and memory
Table 35 (p. 30) reports batch sizes 1 and 32. LoRA, FouRA, and SLoRA have mean latencies of 18.0/53.0, 20.0/60.0, and 20.3/60.0 ms, respectively; corresponding SDs are 0.8/2.7, 0.9/2.9, and 1.0/3.0 ms, with P95 values of 20.0/59.5, 22.4/67.2, and 22.7/67.5 ms. Peak memory is 1.05/1.38, 1.13/1.52, and 1.15/1.53 GB. SLoRA's mean/P95/memory overheads relative to LoRA are 12.8/13.2%, 13.50/13.45%, and 9.5/10.9% for the two batch sizes.
Panel B of Table 35 specifies the common remeasurement protocol: one NVIDIA V100 32GB GPU, FP32 precision, 224×224 images, 77-token text inputs, 50 warm-up iterations, and 500 synchronized CUDA-event-timed forward passes. Timing includes adapter reconstruction and gating but excludes model loading and data preparation. Table 5 (p. 8) separately reports full training-step overhead of 13.3%.