- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BLAde, a framework that uses forward activation statistics to allocate LoRA-MoE capacity before fine-tuning. Output variance guides expert placement and counts, while a normalized joint effective-rank score guides rank allocation and active expert counts. Experiments evaluate accuracy and computational costs across several backbones and reasoning and generation tasks.
Critical strengths
-
Jointly determining expert placement, expert configuration, and ordinary LoRA ranks under a factor budget addresses a practically relevant limitation of uniform adapter configurations.
-
The evaluation covers multiple backbones and task types, includes initialization and training costs, and compares against a gradient-based allocation variant. The consistently competitive results support the practical potential of forward-only calibration.
Critical weaknesses
-
The experiments do not sufficiently isolate the contribution of the proposed allocation signals. Comparisons with existing methods and BLAde-G do not establish whether the gains require the specific variance and effective-rank criteria, rather than heterogeneous allocation itself. Budget-matched random or shuffled allocations, placement-only and rank-only variants, and simpler scoring rules are needed to substantiate the central contribution.
-
The theoretical motivation remains disconnected from parts of the practical allocation procedure. The variance argument compares constant correction vectors, whereas shared LoRA already produces input-dependent corrections. Replacing backward errors with normalized forward outputs introduces another approximation, and the effective-rank proxy has only modest average correlation with its gradient counterpart (0.229). The mappings from these scores to expert counts and active expert counts remain heuristic. Consequently, the analysis does not yet establish why these particular allocation rules should identify useful adaptation capacity.
Questions for authors
-
Under matched factor budgets and comparable active computation, how does BLAde compare with random or shuffled module allocations? How much improvement comes separately from expert placement, expert-count allocation, top-k allocation, and ordinary LoRA rank redistribution?
-
Do the proposed module scores predict the measured benefit of adding capacity to individual modules, beyond correlating with gradient statistics? Would simpler output-variance-only allocation achieve comparable results?
-
Does BLAde retain its advantage under matched end-to-end training time or active-computation budgets? Figure 4 reports approximately 29% higher per-step training time than LoRA, so a compute-matched comparison would clarify its practical efficiency.
3 comments
Thank you for the detailed response and the additional experiments. The random and fixed-configuration baselines strengthen the evidence for BLAde’s joint allocation strategy, and the approximately time-matched comparison addresses my concern about whether its advantage over LoRA persists under a similar training-time budget.
My main remaining concern is the contribution of the proposed forward-only scores. The new ablations compare different allocation configurations, but they do not show how much the effective-rank score contributes beyond a simpler variance-only rule under the same budget. It also remains unclear whether modules with higher scores benefit more from additional capacity. Since the choice of allocation signals is central to the paper, I am maintaining my original rating.
Thank you for your thoughtful question. To further strengthen the empirical evidence for our method, we have conducted additional ablation studies and incorporated the results into Appendix 4.6, Table 24 of the revised manuscript. The new content is highlighted in red for ease of reference.
We compare two allocation signals: Variance, which uses output variance alone for MoE allocation, expert count, LoRA rank, and top-k; and BLAde, which uses variance for MoE allocation and expert count, and effective rank for LoRA rank and top-k. For each, Top assigns higher ranks and prioritizes MoE replacement for modules whose scores are closer to the maximum. Median favors modules near the median score percentile, with capacity priority decreasing toward either tail. Bottom favors modules whose scores are closer to the minimum. All three settings use the same underlying score estimates, target parameter budget, and rank bounds. However, they recompute the allocation under their respective priorities.
| Mode | BoolQ | PIQA | SIQA | HellaS | WinoG | ARC-E | ARC-C | OBQA | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Variance-Bottom | 75.1 | 89.3 | 82.0 | 95.3 | 89.8 | 92.6 | 80.5 | 87.6 | 86.5 |
| Variance-Median | 75.8 | 89.6 | 82.4 | 95.8 | 89.2 | 92.3 | 80.7 | 89.0 | 86.9 |
| Variance-Top | 76.2 | 90.1 | 83.2 | 95.6 | 90.1 | 92.8 | 80.6 | 88.6 | 87.1 |
| BLAde-Bottom | 74.5 | 88.6 | 81.7 | 95.3 | 88.5 | 91.9 | 80.0 | 87.0 | 85.9 |
| BLAde-Median | 75.6 | 89.6 | 82.8 | 95.9 | 89.7 | 93.4 | 82.4 | 88.4 | 87.2 |
| BLAde-Top | 76.7 | 90.3 | 83.2 | 96.3 | 90.1 | 93.2 | 82.3 | 89.2 | 87.7 |
The results suggest that BLAde’s combined signals guide capacity allocation more effectively than variance alone. Prioritizing high-scoring modules achieves the best overall performance, while favoring low-scoring modules substantially degrades performance. The ordering of BLAde-Top, BLAde-Median, and BLAde-Bottom further supports the practical value of its allocation signals.
We thank the reviewer for these valuable suggestions. We have revised the manuscript, with all changes highlighted in red.
1. Response to Weakness 1 and Question 1
We have added six alternatives in Sec. 4.6 (Table 4): FFN-only MoE, Attention-only MoE, and Random-MoE-allocation use 2-in-8 routing with MoE placed in FFN modules, attention modules, or 20% of randomly selected modules, respectively. Fixed 2in8 and Fixed 8 experts retain BLAde's placement but use fixed 2-in-8 routing or 8 experts with adaptive top-k, respectively. All-random randomizes all allocation decisions.
BLAde achieves the highest average accuracy, supporting the benefits of joint allocation over simpler alternatives, as discussed in Sec. 4.6.
2. Response to Weakness 2
We have revised Sec. 3.2.2 to clarify that the analysis concerns weight-update matrices rather than correction vectors. The shared matrix approximation error is
\overset{\mathbf{M}=\mathbb{E}[\mathbf{U}_t]}{=} \operatorname{Tr}(\mathbf{C}_U),$$ where $\mathbf{U}_t$ is the token-specific update matrix and $\mathbf{C}_U$ its vectorized covariance, yielding the bound in Eq. (7). Thus, this is a terminological correction, not a mathematical change. LoRA shares one update matrix across tokens, while LoRA-MoE enables token-dependent updates. We have clarified the limitations of forward-only statistics in Secs. 3.2.3 and 3.3 and Appendix D.3. Gradient directions depend on both inputs and backward errors, making effective-rank estimation from forward activations challenging. We therefore use normalized outputs to approximate gradient geometry rather than reconstruct it exactly. As shown in Appendix H.1 (Table 16), BLAde achieves the highest correlation (0.229) and rank overlap (0.352), while all alternatives show negative correlations, supporting our proxies as more informative allocation priors. Its performance also closely matches gradient-based BLAde-G (Sec. 4.1, Table 2), suggesting exact gradient reconstruction is unnecessary. We have also clarified the expert-count and top-$k$ mappings in Sec. 3.4 and Appendix B.2. Uniform configurations overlook heterogeneous capacity demands, while directly estimating module-wise marginal benefits requires costly training or search. We instead use forward statistics to guide budget-constrained allocation. Although the mappings are heuristics without global optimality guarantees, Sec. 4.6 (Table 4) demonstrates their benefits over fixed and random configurations. We have also added Appendix H.2 (Stability Under Calibration Sampling), which evaluates the consistency of module rankings and allocation decisions across repeated calibration samples, different sample sizes, and sketch widths. ### 3. Response to Question 2 We have clarified the motivation and limitations of our allocation proxies in Secs. 3.1–3.4 and Appendices C and D. Directly measuring module-level marginal gains is challenging: increasing capacity changes the total budget, while maintaining a fixed budget requires reducing capacity elsewhere. Module interactions also make individual gains non-additive, and small differences may be obscured by evaluation variability. As formulated in Sec. 3.1 (Eq. 2), directly optimizing global allocation would require training many candidate architectures. Instead, we use forward-only proxies motivated by the geometry of task-dependent weight updates, whose dimensionality and variation inform adaptation demands. We do not claim these scores precisely predict isolated performance gains. Their effectiveness is supported by proxy–gradient agreement (Table 1 and Appendix H.1), downstream improvements, and performance close to gradient-based BLAde-G (Table 2). ### 4. Response to Question 3 We have added compute-matched comparisons in Appendix H.7 (Table 22). To approximately match end-to-end training time, we reduce BLAde's training steps to 1/1.293 of LoRA's best-checkpoint steps, compensating for its 1.293× per-step latency (Fig. 4) under the same experimental setup. |Method|BoolQ|PIQA|SIQA|HellaS.|WinoG.|ARC-E|ARC-C|OBQA|Avg.| |-|-|-|-|-|-|-|-|-|-| |LoRA|75.4|88.2|82.1|95.4|88.1|92.8|81.0|88.4|86.4| |BLAde (early stopped)|76.6|90.1|82.8|96.0|89.6|93.2|82.1|89.2|87.5| |BLAde|76.7|90.3|83.2|96.3|90.1|93.2|82.3|89.2|87.7| As shown in Appendix H.7 (Table 22), BLAde retains a better average accuracy advantage under matched training time.