acceptodds
Under review as a conference paper at ICLR 2027

AeroLoRA: You Need Few of Them

Abstract

Low-rank adaptation (LoRA) reduces the number of trainable parameters for fine-tuning large language models (LLMs), but two limitations remain: (i) maintaining separate adapters for multiple tasks incurs cumulative storage costs; and (ii) all low-rank channels remain active for every input, leaving input-dependent sparsity unexploited. We propose AeroLoRA (Adaptive Expert Routing with Orthogonal LoRA), a two-stage sparse adaptation framework that reduces both trainable parameter counts and the number of active adapter parameters per token. In the first stage, we derive a task-specific binary mask from the weight magnitudes of the trained up-projection matrix , restricting subsequent adaptation to the selected parameter subset. In the second stage, we update only the selected parameters and perform TopK routing based on activations from a frozen sparse projection, dynamically selecting a small subset of low-rank channels for each input without an additional trainable router. Experiments on Mistral and Llama across four benchmarks show that AeroLoRA achieves the best performance among the evaluated methods in single-task adaptation, multi-task adapter merging, and continual learning. During the second stage, trainable parameters account for only 0.24% and 0.22% of the total model parameters for Mistral and Llama, respectively, while AeroLoRA also activates the fewest adapter parameters per token among the evaluated methods. Spectral analysis shows that its adapter update matrices exhibit the highest effective rank among the compared methods. Subspace analysis further reveals that the update subspaces associated with TopK-selected channels align with the dominant directions of task gradients. These findings provide empirical evidence that diverse, task-relevant update directions and strong adaptation performance can coexist with very few trainable parameters and sparse, input-dependent channel activation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Sparse Adaptation, Dynamic Channel Routing, Multi-Task Adapter Merging, Continual Learning, Effective Rank

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mTe3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AeroLoRA combines task-specific weight masking with input-dependent rank-channel selection for parameter-efficient adaptation. The first stage trains the up-projection matrix and identifies a sparse support through magnitude-based selection. The second stage resets the up-projection and retrains the selected weights with TopK routing, reusing the same frozen sparse projection. The paper evaluates single-task adaptation, adapter merging, and sequential adaptation, and reports improved performance with fewer trainable and active adapter parameters.

Critical strengths
  1. The method has a clear structure that distinguishes task-level weight selection from token-level channel selection. Combining these two forms of sparsity is reasonable, and the frozen projection supports routing without introducing an additional trainable router.

  2. The reported single-task results show mean improvements across two backbones and four benchmarks, including comparisons with the directly relevant LoRI-S and FlyLoRA baselines. At the same Stage 2 trainable-parameter budget as LoRI-S, AeroLoRA improves accuracy while reducing nominal per-token active support.

Critical weaknesses
  1. Limited incremental contribution and insufficient validation of the central design choices. The method primarily combines LoRI’s calibration-and-masking procedure with FlyLoRA’s activation-based channel selection. While this combination is reasonable, the paper does not sufficiently establish what new insight or necessary methodological advance it provides beyond these components. In particular, the current ablations do not establish the importance of all-channel calibration or resetting the calibrated weights. Comparisons against random masks at matched sparsity, calibration with routing enabled, and continued training without resetting would help determine whether the proposed procedure is necessary for the gains.

  2. The experiments do not sufficiently establish the practical significance and generality of the improvements. The efficiency evidence mainly consists of Stage 2 trainable-parameter counts and nominal active B support. These quantities do not establish reductions in total training cost, peak memory, or inference latency. The calibration stage and routing overhead must also be considered. Moreover, input-dependent routing prevents exact merging into a fixed backbone weight matrix, making comparison with standard LoRA’s merged deployment particularly relevant. The paper therefore does not demonstrate when the targeted sparsity corresponds to an important practical bottleneck. What's more, evaluation is limited to Llama-3.1-8B, Mistral-7B-v0.3, and four established benchmarks, with no total-rank sweep for AeroLoRA. Evidence on newer backbones, more challenging tasks, and a wider range of rank budgets would strengthen the conclusions.

Questions for authors
  1. What are the total training time, peak GPU memory, complete adapter storage, and inference latency relative to LoRA, LoRI-S, and FlyLoRA, including both training stages and all inference-time operations?

  2. How does AeroLoRA compare with matched random masks, calibration under routing, and retraining without resetting? What explains the ScienceQA difference between the “without element mask” variant in Table 5 (74.5%) and FlyLoRA in Table 2 (79.1%)?

  3. How are the gates, routing biases, and channel budgets handled in the Linear and Concat merging experiments? What is the GSM8K performance after sequential adaptation to the new tasks?

Reviewer_TeK4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AeroLoRA, a two-stage sparse adaptation framework that combines parameter masking with input-dependent Top-K routing to reduce both trainable and active adapter parameters. Experiments demonstrate strong performance across multiple models and tasks, including single-task adaptation, adapter merging, and continual learning. However, the proposed method appears to mainly combine existing ideas from LoRI and FlyLoRA, making its methodological novelty somewhat incremental.

Critical strengths
  1. The paper proposes an input-dependent sparse LoRA framework that effectively reduces both the number of trainable parameters and the number of active adapter parameters per token.

  2. The proposed method achieves competitive results across different models and tasks, demonstrating its effectiveness in single-task adaptation, multi-task adapter merging, and continual learning.

Critical weaknesses
  1. The proposed method appears to combine existing techniques from LoRI and FlyLoRA, making the overall contribution somewhat incremental. The paper does not sufficiently clarify the fundamental differences or novel insights beyond these existing approaches.
Questions for authors

none

Reviewer_ABLgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AeroLoRA combines task-specific weight sparsity with input-dependent channel selection to reduce the parameters trained and activated by a LoRA adapter. In the first stage, the up-projection B is trained with all channels active, and its weight magnitudes determine a binary mask. In the second stage, B is reset and only the retained entries are retrained. Both stages share a frozen sparse projection A; its activations drive token-dependent TopK routing without an additional trainable router. Experiments on Mistral and Llama cover single-task adaptation, adapter merging, and sequential adaptation, with reported gains over the compared methods. The paper also reports Stage 2 trainable-parameter counts and per-token active support, and uses spectral and subspace analyses to study the learned updates and their alignment with task gradients.

Critical strengths

The single-task results are consistently positive, and the component ablation supports combining masking and routing. The paper is also careful to distinguish trainable parameters, active weights per token, and actual computation. Keeping A fixed makes the two training stages easy to understand.

Critical weaknesses

The calibration step is not directly tested. The mask is chosen without routing, although it will be used with routing afterward. Table 5 removes masking or routing, but does not show whether the calibrated mask is better than a random mask at the same sparsity. A matched random-mask control would distinguish the contribution of calibration from the benefit of sparsity itself.

The headline trainable fraction refers to Stage 2, although Stage 1 trains all of B. The cost description should include both stages and distinguish parameter savings from measured runtime savings. Fewer active entries do not necessarily make an unstructured sparse update faster at inference.

The main uncertainty is whether mask calibration adds value beyond sparsity itself. The efficiency claim also needs to account for the cost of both training stages.

Questions for authors
  1. Does the calibrated mask outperform a random mask at the same sparsity and total training budget?
  2. Which reported efficiency claims cover both training stages?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.