- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AeroLoRA combines task-specific weight masking with input-dependent rank-channel selection for parameter-efficient adaptation. The first stage trains the up-projection matrix and identifies a sparse support through magnitude-based selection. The second stage resets the up-projection and retrains the selected weights with TopK routing, reusing the same frozen sparse projection. The paper evaluates single-task adaptation, adapter merging, and sequential adaptation, and reports improved performance with fewer trainable and active adapter parameters.
Critical strengths
-
The method has a clear structure that distinguishes task-level weight selection from token-level channel selection. Combining these two forms of sparsity is reasonable, and the frozen projection supports routing without introducing an additional trainable router.
-
The reported single-task results show mean improvements across two backbones and four benchmarks, including comparisons with the directly relevant LoRI-S and FlyLoRA baselines. At the same Stage 2 trainable-parameter budget as LoRI-S, AeroLoRA improves accuracy while reducing nominal per-token active support.
Critical weaknesses
-
Limited incremental contribution and insufficient validation of the central design choices. The method primarily combines LoRI’s calibration-and-masking procedure with FlyLoRA’s activation-based channel selection. While this combination is reasonable, the paper does not sufficiently establish what new insight or necessary methodological advance it provides beyond these components. In particular, the current ablations do not establish the importance of all-channel calibration or resetting the calibrated weights. Comparisons against random masks at matched sparsity, calibration with routing enabled, and continued training without resetting would help determine whether the proposed procedure is necessary for the gains.
-
The experiments do not sufficiently establish the practical significance and generality of the improvements. The efficiency evidence mainly consists of Stage 2 trainable-parameter counts and nominal active B support. These quantities do not establish reductions in total training cost, peak memory, or inference latency. The calibration stage and routing overhead must also be considered. Moreover, input-dependent routing prevents exact merging into a fixed backbone weight matrix, making comparison with standard LoRA’s merged deployment particularly relevant. The paper therefore does not demonstrate when the targeted sparsity corresponds to an important practical bottleneck. What's more, evaluation is limited to Llama-3.1-8B, Mistral-7B-v0.3, and four established benchmarks, with no total-rank sweep for AeroLoRA. Evidence on newer backbones, more challenging tasks, and a wider range of rank budgets would strengthen the conclusions.
Questions for authors
-
What are the total training time, peak GPU memory, complete adapter storage, and inference latency relative to LoRA, LoRI-S, and FlyLoRA, including both training stages and all inference-time operations?
-
How does AeroLoRA compare with matched random masks, calibration under routing, and retraining without resetting? What explains the ScienceQA difference between the “without element mask” variant in Table 5 (74.5%) and FlyLoRA in Table 2 (79.1%)?
-
How are the gates, routing biases, and channel budgets handled in the Linear and Concat merging experiments? What is the GSM8K performance after sequential adaptation to the new tasks?