acceptodds
Under review as a conference paper at ICLR 2027

MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweighting

Abstract

Parameter-efficient fine-tuning (PEFT) of vision backbones is usually posed as a question of how many parameters to train. We argue that for dense prediction the more consequential question is where a fixed budget should be spent. We study two ways of spending a small budget on a frozen backbone: an additive low-rank residual over intermediate feature maps, and the same residual modulated by an input-dependent, spatially varying multiplicative gate. At budgets of the same order, the multiplicative form is consistently better for localization-heavy tasks and roughly neutral for global classification, which is the pattern one expects if the gate supplies spatial selectivity that a channel-wise low-rank update cannot express. We instantiate this finding as MSLoRA, an adapter that gates a grouped low-rank projection with a multi-scale depth-wise branch. Because MSLoRA acts on feature maps rather than on architecture-specific weight matrices, the same feature-space formulation applies across convolutional, hierarchical-transformer, hybrid, and plain ViT backbones; a separate adapter is still trained per backbone and task, as for other PEFT methods. On COCO with Cascade Mask R-CNN, MSLoRA reaches 42.9 box AP / 38.4 mask AP on ResNet-50 while training 0.7M parameters (2.7% of the backbone), and is competitive with backbone-specialized PEFT methods on Swin-B at roughly half their trainable-parameter count. We report the full cost profile, including the cases where MSLoRA is slower than full fine-tuning, and we audit the strength of the full fine-tuning baselines we compare against.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, vision adaptation, dense prediction, multi-scale feature modulation, object detection and segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ijgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MSLoRA (Multi-Scale Low-Rank Adaptation), a parameter-efficient fine-tuning (PEFT) method designed specifically for dense vision tasks like object detection and segmentation. The proposed idea is to introduce a spatially varying multiplicative gate alongside the standard additive low-rank residual used in LoRA. Specifically, a frozen feature map is updated by adding a low-rank projection that is element-wise multiplied by a multi-scale spatial attention map computed via parallel depth-wise convolutions. The authors hypothesize that while spatially uniform low-rank updates (like standard LoRA) are sufficient for global image classification, dense prediction tasks benefit heavily from spatial selectivity. Experiments across various architectures (ResNet, Swin, ConvNeXt) and datasets (COCO, ADE20K) demonstrate that MSLoRA achieves competitive or superior performance compared to full fine-tuning and baseline PEFT methods, all while utilizing a small parameter budget.

Critical strengths

The paper provides an insightful analysis of parameter budget allocation in parameter-efficient fine-tuning (PEFT). By identifying that standard LoRA suffers from spatial uniformity, the authors explicitly hypothesize and demonstrate that allocating part of a fixed parameter budget to a multi-scale spatial gate is more valuable for dense prediction than spending it entirely on channel recombination. This budget-allocation principle is well validated in their controlled experiments.

The empirical results include a variety of backbone architectures, including both CNNs and ViTs. The proposed method consistently achieves or exceeds full fine-tuning performance on dense prediction tasks with a very small parameter budget, proving it is an effective improvement for localized representation requirements.

Critical weaknesses

The primary weakness of this submission is its limited architectural novelty and substantial conceptual overlap with existing literature on spatial attention and parameter-efficient fine-tuning. The core operation in MSLoRA combines a low-rank linear projection with a multi-scale depth-wise convolutional attention gate. However, injecting spatial priors via depth-wise convolutions into adapters is already heavily explored in prior works such as ConvAdapter (Chen et al., CVPR 2024) and ViT-Adapter (Chen et al., ICLR 2023). Furthermore, the concept of element-wise spatial modulation of feature maps is a standard paradigm in computer vision, extensively explored in foundational attention mechanisms like the Convolutional Block Attention Module (CBAM) (Woo et al., 2018) and dynamic convolutions (Chen et al., 2020). Modifying the additive branch of a standard adapter with a spatially modulated gate represents a relatively straightforward fusion of two well-established techniques, which limits the theoretical and architectural novelty of the contribution.

Additionally, the efficiency claims require more nuanced qualification. Because MSLoRA relies on an input-dependent, spatially varying multiplicative gate, it fundamentally loses the re-parameterization property of standard LoRA. Meaning the adapter weights cannot be merged back into the frozen backbone during inference. This introduces a persistent computational and memory overhead during deployment. Although the authors acknowledge this limitation and show lower training throughput in Table 5, the lack of an inference latency profile makes it difficult to assess the true deployment cost compared to structurally re-parameterizable approaches like RepAdapter (Luo et al., 2023) for example.

As a result I am borderline leaning on a weak reject for the paper in the current state.

Chen, H., et al. Conv-Adapter: Exploring Parameter Efficient Transfer Learning for Convnets. CVPR 2024

Chen, Z., et al. Vision Transformer Adapter for Dense Predictions. In International Conference on Learning Representations, ICLR 2023.

Woo, S., et al. CBAM: Convolutional Block Attention Module, ECCV 2018.

Chen, Y. et al. Dynamic Convolution: Attention over Convolution Kernels, CVPR 2020.

Luo, G., et al. Towards Efficient Visual Adaption via Structural Re-parameterization. arXiv preprint arXiv:2302.08106.

Questions for authors

Could the authors provide a detailed profile of the inference latency and memory overhead? Since the gating mechanism precludes weight merging, this overhead could be a significant bottleneck for real-time dense prediction tasks compared to standard LoRA.

How does MSLoRA compare empirically against other spatially-aware or dynamic PEFT methods that modulate parameter updates based on input features, such as Dynamic Visual Prompt Tuning (Derakhshani et al., 2023) or mixture-of-expert adapter approaches like ParaX (Wang et al., 2024)? Including these baselines would help contextualize the performance gains of your input-dependent design.

Could you clarify if the hyperparameter configurations, such as the bottleneck dimension and group size, require extensive dataset-specific tuning when migrating across fundamentally different backbones, or is the default configuration robust across all settings?

Derakhshani, M. M., et al. Variational Prompt Tuning Improves Generalization of Vision-Language Foundation Models, ICLR 2023.

Wang, S., Song, et al. Parameters as Experts (ParaX): Adapting Vision Models with Dynamic Parameter Routing and Shared Expert Centers, ICML 2025.

Reviewer_kPY7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MSLoRA combines grouped low-rank feature projections with input-dependent multi-scale spatial gating to adapt frozen vision backbones. Experiments suggest stronger benefits for dense prediction than classification, with broad architectural compatibility and explicit reporting of memory and throughput trade-offs.

Critical strengths
  1. A common feature-space formulation supports convolutional and transformer backbones across several visual tasks.
  2. Dense-prediction experiments show competitive accuracy with relatively few trainable backbone parameters.
  3. The paper transparently discusses unmatched budgets, baseline strength, and slower training throughput.
Critical weaknesses
  1. The gate-only control recovers most of the gain, leaving the necessity of combining gating with low-rank projection only weakly established.
  2. Single runs without fixed seeds or uncertainty estimates cannot substantiate the stated precision thresholds or small performance gains.
  3. Lower throughput, nonmergeable adapters, and incomplete end-to-end inference profiling limit the practical efficiency case.
  4. The related-work survey insufficiently covers efficient visual adapters and convolutional low-rank adaptation [1-10].

[1] V-PETL Bench: A Unified Visual Parameter-Efficient Transfer Learning Benchmark. NeurIPS. 2024.

[2] Parameter-efficient is not Sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions. ACM Multimedia. 2024.

[3] Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model. ICLR. 2024.

[4] Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach. CVPR. 2024.

[5] TEA: A Training-Efficient Adapting Framework for Tuning Foundation Models in Remote Sensing. IEEE TGRS. 2024.

[6] Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation. CVPR. 2025.

[7] Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation. CVPR. 2024.

[8] AiRs: Adapter in Remote Sensing for Parameter-Efficient Transfer Learning. IEEE TGRS. 2024.

[9] Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition. CVPR. 2025.

[10] Efficient Adaptation of Pre-trained Vision Transformer underpinned by Approximately Orthogonal Fine-Tuning Strategy. ICCV. 2025.

Questions for authors
2 comments

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

Dear Reviewer,

Thank you again for your valuable feedback. We have provided a detailed response to your concerns and would greatly appreciate it if you could take a look when you have a chance.

We would be grateful to know whether our clarifications have adequately addressed your main concerns, or if there are any remaining questions that we could further clarify during the discussion period.

Thank you for your time and consideration.

Reviewer_zeq4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates feature allocation for a frozen visual backbone under a limited PEFT budget, comparing additive low-rank feature residuals with input-dependent spatially gated residuals. The paper proposed MSLoRA, which integrates grouped low-rank projections and multi-scale deep gating. The method is evaluated on both CNN and Transformer backbones for classification and dense prediction tasks.

Critical strengths

1.Cost and applicability boundaries are well reported, allowing users to clearly evaluate suitable use cases. 2. MSLoRA achieves 42.9 box AP and 38.4 mask AP with 0.7M backbone-adaptation parameters, outperforming the corresponding full fine-tuning baseline by 1.7 AP and 2.5 AP, respectively.

Critical weaknesses

1.The mathematical definition of the proposed module is dimensionally inconsistent as presented in Equations 1–4 and Fig 1. 2. The paper presents its main contribution as a budget-allocation, but claim is not sufficiently supported by experiments. The additive variant already achieves 41.6/37.6 AP, while the multiplicative variant reaches 41.9/37.8 AP. Given that these results come from single runs, the stability of this marginal improvement remains unclear.

Questions for authors
  1. Most gains are already attained by the nonlinear additive branch, while the multiplicative interaction only brings an additional 0.3 box AP and 0.2 mask AP. The paper explicitly states that all reported results come from single runs without fixed random seeds, so it remains unclear whether these marginal differences are stable.
  2. Does the actual multiplication take place before or after dimension lifting? Please provide the forward formulation or pseudocode with channel dimensions, and reconcile the order in Figure 1, Equations (1)–(4), as well as LayerNorm and GELU. 3.Please report a parameter-matched additive baseline for the COCO experiment in Table 1.
  3. Can the observed performance gap between classification and dense prediction be reproduced under the same backbone and identical adaptation module?
3 comments

We thank the reviewer for the time and effort devoted to our paper and for the constructive questions. We have updated the related-work references in the manuscript (updated paper: https://openreview.net/pdf?id=jmMCBwohWI).

1. Dimensional consistency and operation order.

The multiplication is performed in the D-dimensional bottleneck, before lifting to C channels. The implementation uses the following forward pass for the base module, omitting the batch dimension:

F: C × H × W
(x, q) = split(GConv_down(F))         # C → 2D; each branch: D × H × W
c = PWConv(sum_k GELU(DWConv_k(q)))   # D × H × W; k in {3, 5, 7}
z = x + x * c                         # element-wise multiplication at width D
z = GELU(LayerNorm_channels(z))       # D × H × W
F_out = F + GConv_up(z)               # D → C

The down-projection produces separate content and conditioning features; a single up-projection lifts their combined result. This resolves the ambiguity in Eqs. 1–4 and Fig. 1: the product belongs inside the up-projection, and LayerNorm/GELU follows the combination. Input pre-normalization is additionally used for pre-norm backbones. The original 41.9/37.8 ablation omits post-combination LayerNorm/GELU; the base configuration below includes it.

2. Budget allocation, the nonlinear additive control, and stability.

The important distinction is between adding more linear capacity and allocating capacity to spatial nonlinear adaptation. We added a parameter-matched linear residual to test whether MSLoRA's gain can be explained simply by its larger trainable budget.

UpdateBox APMask AP
Linear residual, D=12840.536.6
Parameter-matched linear residual, D=25240.836.9
MSLoRA base, D=12842.538.2

Nearly doubling the linear budget yields only +0.3/+0.3 AP; MSLoRA improves on that widened baseline by +1.7/+1.3 AP at the same budget. Thus, extra parameter count alone does not explain the improvement. Widening the linear branch increases channel capacity, whereas the spatial branch introduces local nonlinear interactions that a wider pointwise linear map cannot express.

Your observation about the original 41.6/37.6 versus 41.9/37.8 comparison is also important. We do not base the contribution on the small additive-versus-product difference: the supported finding is that how the adaptation budget is used matters substantially, and MSLoRA provides an effective realization of that allocation.

Regarding stability, our final configurations were run three times and agreed at one decimal place; we ran the required ablation again and got the same AP. These repeat checks establish observed reproducibility, rather than a universal significance threshold.

3. Same backbone and adapter for classification and dense prediction.

Yes. For classification, the consistent improvements across all three datasets are unlikely to result from random fluctuations. To verify this, we repeated the training of MSLoRA on all three datasets and obtained an average standard deviation of only 0.04 percentage points, substantially smaller than the performance gaps between MSLoRA and competing methods.

Furthermore, full fine-tuning of large backbones such as ResNet-50 and Swin on relatively small classification datasets involves optimizing tens of millions of parameters, potentially increasing the risk of overfitting and sensitivity to training conditions. In contrast, MSLoRA updates fewer than 2.5M parameters, providing a more constrained adaptation space that may help mitigate overfitting. This is consistent with the low variance observed in our repeated experiments and further supports the reliability of our classification results.

Responses fail to resolve run‑to‑run variance, implementation ambiguity and baseline‑matching concerns. The provided parameter‑matched linear‑residual baseline represents only one single comparison point, which is overly simplistic. To convincingly validate the benefits brought by spatial nonlinear adaptation, richer ablation variants with comparable parameter budgets should be included. In addition, I still consider the novelty is increamental.

Thank you for the time and effort you have put into this discussion. You raise a fair question about how much one matched-budget comparison can establish. Two observations bear directly on it. A spatial adapter with fewer parameters also outperforms the widened linear control, and the advantage of the final architecture extends across several frozen CNN backbones.

Budget matching and run-to-run variation. The existing ResNet-50 results below compare the same update forms on classification and dense prediction. All use D=128 except the widened linear control.

UpdateOxford Pets top-1 (%)COCO box APCOCO mask AP
Linear residual93.4040.536.6
Widened linear residual, matched to MSLoRA base—40.836.9
Nonlinear spatial additive93.7342.538.1
MSLoRA base93.5442.538.2

Widening the linear residual improves box and mask AP by 0.3 each. The spatial additive variant uses fewer parameters than the widened linear residual, yet gains 1.7 box and 1.2 mask AP over it. At the matched budget, MSLoRA gains 1.7 and 1.3 AP. This gives a second test of the capacity explanation. Even when the linear adapter has the larger budget, both spatial variants perform better. Widening a pointwise projection increases its capacity to recombine channels at one location. The spatial branch can also use neighboring features to compute the update at that location.

The submitted kernel ablation provides a further check within the spatial branch. Moving from a single 3×3 kernel to {3,5,7} raises AP from 41.4/37.4 to 41.9/37.8. The more expensive {9,11,13} set also reaches 41.9/37.8. Increasing capacity again yields no further gain here.

We treat the spatial additive and multiplicative variants as tied. The final configurations agreed to one decimal place in the repeat runs reported in our first reply. The architectural conclusion rests on the larger gaps to the linear controls. The table also answers your same-backbone question directly. At D=128, changing from linear to MSLoRA on ResNet-50 gives +0.14 percentage points on Pets and +2.0 box / +1.6 mask AP on COCO.

Implementation. The base module projects C channels to 2D and splits the result into content x and conditioning features q, each of width D. The multiscale depthwise convolutions act on q. Their GELU-transformed outputs are summed and mixed by a pointwise convolution to produce c. The module computes x + x ⊙ c at width D, applies channel LayerNorm followed by GELU, projects once back to C, and adds the result to the frozen feature map. This is the order that Eqs. 1–4 and Fig. 1 should depict. Table 1's 41.9/37.8 variant omits the post-combination LayerNorm/GELU. The 42.5/38.2 base above includes it.

One adapter design across frozen CNN detectors and vision Transformers. The architectural contribution of MSLoRA is that a single adaptation design can outperform full fine-tuning on classical CNN detectors and remain competitive with specialized adapters on Transformers. Improving LoRA on a Transformer establishes what can be achieved by adapting its attention and feed-forward layers. It leaves open a different design question: can the same adapter also recover the accuracy of full fine-tuning on a ResNet detector, where there are no attention projections to tune and the backbone’s spatial filters remain frozen?

MSLoRA addresses this by learning the correction on the intermediate feature grid that these backbones share. The adapter uses multiscale context to determine how strongly to update the feature at each location. A ResNet supplies that grid directly, and a plain ViT supplies it through its patch tokens. Changing the backbone therefore preserves the core adaptation rule.

The ResNet-50 COCO comparison shows why that distinction matters. Under the same Cascade Mask R-CNN protocol, both LoRA and ConvAdapter fall below full fine-tuning on box and mask AP. MSLoRA keeps the backbone frozen and exceeds full fine-tuning by 1.7 box AP and 2.5 mask AP. The same design also exceeds full fine-tuning on ResNeXt-101, approaches Mona on Swin-B, and applies to plain ViT-B/16. We believe the ability to obtain this level of dense-prediction accuracy on a frozen convolutional backbone while retaining one adaptation rule across backbone families is the architectural advance that deserves weight in the novelty assessment.

We ask you to reconsider the rating in light of this result. A single spatial adaptation design recovers accuracy that both linear and convolutional adapters leave behind on the frozen ResNet detector, and the original submission demonstrates how that design carries across backbone families.

Comments on the paper

4 comments

Thank you for the detailed rebuttal. I appreciate the transparency of the inference latency profiling and the new matched-budget ablation.

However, while the new experiments clarify the empirical behavior of the method, they do not resolve my fundamental concerns regarding the conceptual novelty and the practical deployment costs expected at a venue like ICLR.

Significance of the Contribution

The matched-budget ablation (showing a 1.7/1.3 AP gap against a widened linear LoRA) is good empirical hygiene. It effectively proves that spatial gating is a better use of parameters than capacity alone. However, reframing the paper around a "budget allocation principle" does not change the underlying mechanics of the method. At its core, MSLoRA is an architectural combination of standard low-rank adaptation and well-established spatial multiplicative gating (highly reminiscent of CBAM, dynamic convolutions, and recent multi-scale adapters). While the empirical observation that this specific combination works well for dense prediction is useful, it remains a relatively straightforward architectural fusion.

Inference Latency and Lack of Re-parameterization

I appreciate the latency profiling provided in the rebuttal. However, a ~40% latency overhead on ResNet-50 (49.3 ms -> 68.3 ms), compared to the negligible overhead of ConvAdapter (49.3 ms -> 50.9 ms), confirms my concern that the lack of structural re-parameterization is a significant bottleneck. A major appeal of additive PEFT methods in vision is the ability to fold weights during deployment, incurring zero inference cost. By relying on an input-dependent dynamic gate, MSLoRA sacrifices this property. While the method remains competitive against Mona on heavier architectures like Swin-B, the substantial latency penalty on standard CNN backbones is a fundamental limitation for a method claiming to be an efficient adapter.

The authors have executed a solid empirical study and demonstrated a useful engineering trade-off for dense prediction adapters. However, due to the heavy reliance on existing architectural primitives, the lack of fundamental conceptual novelty, and the introduced inference latency penalties on standard backbones, I believe the paper falls short of the high bar for this conference.

I will maintain my current score for now but I am curious to see if the other reviewers weigh the empirical gains higher than the conceptual and deployment limitations.

We thank the reviewer for the thoughtful follow-up and the time and effort devoted to our paper. We appreciate your recognition of the matched-budget experiment and inference profiling. We would like to clarify why these findings, together with the comparison against an existing spatial adapter, support a meaningful contribution.

1. Contribution beyond established components.

We agree that combining established components does not automatically establish a significant contribution. Equally, identifying familiar components does not by itself establish that the resulting contribution is merely incremental. For MSLoRA, the relevant evidence is the improvement over an existing spatial adapter, together with controlled experiments showing why simply increasing linear adaptation capacity does not recover that improvement. We ask that the design be assessed through these distinctions and results, alongside its acknowledged deployment costs.

ConvAdapter is an informative comparison precisely because it already incorporates spatial processing. Under our ResNet-50 COCO protocol, it reaches 38.8/35.1 box/mask AP, below full fine-tuning at 41.2/35.9. MSLoRA reaches 42.9/38.4, exceeding both. The +4.1/+3.3 AP gain therefore demonstrates an improvement over an existing spatial adapter. Prior demonstrations that spatial priors are useful do not make this improvement an already-achieved result.

The controlled ablation addresses a complementary question: could the gain simply be recovered by enlarging the linear adapter? Increasing its width improves AP from 40.5/36.6 to 40.8/36.9, while MSLoRA reaches 42.5/38.2 at matched budget. The remaining 1.7/1.3 AP gap identifies a benefit that additional linear capacity does not recover. The spatial nonlinear additive control reaches 42.5/38.1, further showing that this finding extends beyond the particular multiplicative implementation.

Together, these experiments establish both a practical advance over an existing spatial adapter and evidence for a specific design choice: allocating limited adaptation capacity to spatial nonlinear processing rather than solely widening channel transformations. We do not claim that multiplication itself is new or uniquely necessary.

The same MSLoRA formulation is evaluated across CNN and Transformer backbones with shared structural defaults. We believe this combination of a reusable adaptation design, substantial improvements over established baselines, and controlled evidence explaining the capacity choice deserves weight in the assessment of significance.

2. Inference latency together with the accuracy it delivers.

We acknowledge the approximately 40% inference latency overhead on ResNet-50. The main COCO results and measured latency make the corresponding accuracy–cost trade-off explicit:

MethodBox APMask APΔ AP vs. FT (Box/Mask)Latency
Full fine-tuning41.235.9—49.3 ms
ConvAdapter38.835.1−2.4 / −0.850.9 ms
MSLoRA42.938.4+1.7 / +2.568.3 ms

Although ConvAdapter introduces considerably less inference overhead, it falls below full fine-tuning by 2.4 box AP and 0.8 mask AP. MSLoRA instead exceeds full fine-tuning by 1.7 box AP and 2.5 mask AP, while outperforming ConvAdapter by 4.1 box AP and 3.3 mask AP.

These substantial accuracy differences matter to the deployment comparison. ConvAdapter represents a faster operating point, but it does not provide an equivalent-accuracy replacement for MSLoRA.

There is also a concrete adaptation-time benefit: ResNet-50 peak training memory decreases from 5512 MB to 3999 MB, a 27.5% reduction relative to full fine-tuning. Peak inference memory increases by less than 3% over the plain detector. These measurements distinguish the training-memory savings from the persistent inference cost.

Weight merging is valuable, and MSLoRA does not offer zero-overhead deployment. However, the inability to merge an input-dependent adapter does not by itself determine whether its accuracy and resource trade-off is useful. Here, the additional computation accompanies substantially better dense-prediction accuracy and lower training memory than full fine-tuning.

We respectfully ask whether the substantial gains over an existing spatial adapter, the controlled evidence for the design choice, and the quantified accuracy–cost trade-off warrant reconsidering the overall rating. We would be happy to address any remaining questions.

Thank you for your answer and time spent addressing my concerns

After reviewing the rebuttal, my assessment of the paper remains unchanged for the following reasons.

Conceptual insight vs. confounded thesis

While I agree with the ICLR 2027 Reviewer Guidelines [6] that novelty does not require an entirely new algorithmic primitive, combining existing components still requires clear, well-isolated insights rather than primarily empirical performance gains. Whether those insights justify acceptance is ultimately a matter of reviewer judgment.

More importantly, the rebuttal introduces a direct tension with the core thesis of the submitted manuscript. The title ("via Attention Reweighting"), abstract, and hypothesis (Sec. 1, Sec. 3.2, Table 1) attribute the method's advantage to input-dependent multiplicative spatial gating over an additive residual. Even in Table 1, the gate-only additive variant already recovered most of the gain (41.6/37.6 vs. 41.9/37.8 AP). In the rebuttal's new matched-budget experiment, the nonlinear spatial additive control reaches 42.5/38.1 AP compared to 42.5/38.2 AP for MSLoRA base. This is no difference of box AP and 0.1 mask AP, which is negligible relative to the 0.1 AP reporting resolution defined in Section 4.1. As the rebuttal itself notes, this identifies "spatial nonlinear adaptation, rather than multiplication alone, as the primary source of improvement." Table 1 therefore confounds spatial nonlinearity with multiplicative gating, which makes the comparison to foundational paradigms like LoRA or SENet overstated and places the core mechanism even closer to existing nonlinear spatial convolutional adapters.

Deployment latency vs. efficient PEFT alternatives

Framing the ResNet-50 latency table as an unavoidable accuracy-latency trade-off overlooks two issues.

First, although MSLoRA reduces peak training memory by 27.5% (1.5 GB: 3999 vs. 5512 MB on an A5000, Table 5), it still uses more memory than LoRA (3565 MB), is 25% slower in training throughput than full fine-tuning (1.2 vs. 1.6 iter/s, Table 5), and incurs an acknowledged ~40% inference latency overhead on ResNet-50 (68.3 vs. 49.3 ms for MSLoRA base).

Second, existing peer-reviewed PEFT methods (most already cited in the manuscript) show that competitive performance does not inherently require both training slowdowns and persistent inference latency penalties. Re-parameterizable approaches such as RepAdapter [1] (46.71 vs. 45.63 mIoU-MS for VPT+Bias on ADE20K SETR-L), RLRR [2], and GLoRA [3] merge their learned updates into the frozen weights for zero inference overhead. On dense prediction benchmarks specifically, E3VA [4] achieves comparable accuracy to full fine-tuning on COCO, Pascal VOC, and ADE20K while reducing both training memory (up to 62.2%) and training time (14.8% faster on COCO), in contrast to MSLoRA's 25% training slowdown. Similarly, V-PETL Bench [5] reports on Pascal VOC (Swin-L) that zero-inference-overhead methods such as LoRA (87.5%), BitFit (85.7%), and LN TUNE (85.8%) can outperform full fine-tuning (83.5%) without introducing non-mergeable inference branches.

Rebuttal scope

The volume of new post-submission experiments, particularly controls that shift the paper's primary causal claim away from "attention reweighting", goes beyond validating existing results and would require substantially rewriting the title, abstract, and framing of the manuscript. Under the ICLR 2027 Reviewer Guidelines [6], "New experiments should not significantly change the content of the submission. Rather, they should be limited in scope and serve to more thoroughly validate existing results from the submission." In its current state, the paper is not ready for publication and would benefit from a full revision cycle incorporating the reviewer feedback for a future venue.

[1] Luo, G., et al. Towards Efficient Visual Adaption via Structural Re-parameterization. arXiv:2302.08106, 2023.

[2] Dong, W., et al. Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach. CVPR 2024.

[3] Chavan, A., et al. One-for-All: Generalized LoRA for Parameter-Efficient Fine-Tuning. ICLR 2024.

[4] Yin, D., et al. Parameter-efficient is not sufficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions. ACM MM 2024.

[5] Xin, Y., et al. V-PETL Bench: A Unified Visual Parameter-Efficient Transfer Learning Benchmark. NeurIPS 2024.

[6] ICLR 2027 Reviewer Guidelines. https://iclr.cc/Conferences/2027/ReviewerGuidelines

Thank you for the time and effort you have devoted to this discussion. We agree that the two spatial variants are effectively tied. We believe one architectural contribution deserves more weight in the assessment: MSLoRA defines the adaptation on the spatial feature grid shared by CNNs and Transformers. Its value is particularly clear on the frozen CNN detectors, where the success of low-rank adaptation on Transformers leaves an empirical question open.

One adapter design across frozen CNN detectors and vision Transformers. The architectural contribution of MSLoRA is that a single adaptation design can outperform full fine-tuning on classical CNN detectors and remain competitive with specialized adapters on Transformers. Improving LoRA on a Transformer establishes what can be achieved by adapting its attention and feed-forward layers. It leaves open a different design question: can the same adapter also recover the accuracy of full fine-tuning on a ResNet detector, where there are no attention projections to tune and the backbone’s spatial filters remain frozen?

MSLoRA addresses this by learning the correction on the intermediate feature grid that these backbones share. The adapter uses multiscale context to determine how strongly to update the feature at each location. A ResNet supplies that grid directly, and a plain ViT supplies it through its patch tokens. Changing the backbone therefore preserves the core adaptation rule.

The ResNet-50 COCO comparison shows why that distinction matters. Under the same Cascade Mask R-CNN protocol, both LoRA and ConvAdapter fall below full fine-tuning on box and mask AP. MSLoRA keeps the backbone frozen and exceeds full fine-tuning by 1.7 box AP and 2.5 mask AP. The same design also exceeds full fine-tuning on ResNeXt-101, approaches Mona on Swin-B, and applies to plain ViT-B/16. We believe the ability to obtain this level of dense-prediction accuracy on a frozen convolutional backbone while retaining one adaptation rule across backbone families is the architectural advance that deserves weight in the novelty assessment.

Frozen backboneMSLoRA box / mask APΔ vs. full fine-tuning (box / mask)
ResNet-5042.9 / 38.4+1.7 / +2.5
ResNet-10144.0 / 39.2+1.1 / +1.9
ResNeXt-10144.8 / 39.9+0.5 / +1.6
RegNetX-4.044.3 / 39.7−0.1 / +1.1
ConvNeXt-S48.5 / 41.9+0.6 / +0.3
Swin-B52.7 / 45.9+0.3 / +0.8

Mechanism and rebuttal scope. Section 1 hypothesizes an advantage over spending the entire budget on a linear low-rank residual. Section 4.2 already reports that the spatial additive control recovers most of this gain and explicitly identifies unequal budgets as a limitation. The widened linear control addresses that exact limitation. At matched budget it reaches 40.8/36.9, compared with 42.5/38.2 for MSLoRA. The spatial additive control reaches 42.5/38.1. We therefore give the two spatial variants equal standing in this ablation. “Attention reweighting” describes how the proposed model computes its correction. The empirical advantage we can attribute to that design is over the linear low-rank baseline.

The reviewer guidelines allow additional experiments that validate existing submission results. Here the original manuscript identifies the budget confound, includes the additive spatial control, and reports all six backbone comparisons above. The rebuttal resolves the stated budget question and measures the inference cost requested in the review. We respectfully disagree that these checks change the scope of the submission. ICLR 2027 Reviewer Guidelines

Deployment cost. We agree that mergeable methods offer a deployment advantage and that E³VA demonstrates real training savings. On our ResNet-50 detector, MSLoRA trains at 1.2 versus 1.6 iter/s for full fine-tuning, while peak training memory falls from 5512 to 3999 MB. At inference, the profiled MSLoRA base reaches 42.5/38.2 AP at 68.3 ms, compared with 38.8/35.1 at 50.9 ms for ConvAdapter. The additional 17.4 ms accompanies gains of 3.7 box and 3.1 mask AP in that comparison. On Swin-B, the profiled MSLoRA base runs at 154.6 ms versus 158.9 ms for Mona. These measurements let practitioners judge the cost on their target backbone. E³VA

We respectfully ask you to reconsider the significance of a single spatial adaptation design that improves on full fine-tuning across classical frozen CNN detectors and remains competitive on Transformers. This contribution is present in the submitted architecture and results, and it holds even when the two spatial variants are treated as tied.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.