acceptodds
Under review as a conference paper at ICLR 2027

Forward-Only LoRA-MoE Allocation via Module Activation Geometry

Abstract

Low-rank adaptation (LoRA) mixtures of experts (LoRA-MoE) increase adaptation capacity through conditional expert activation. However, many conventional LoRA-MoE methods adopted in large language models (LLMs) typically restrict MoE placement to predefined modules, such as the feed-forward networks (FFNs), and use uniform expert configurations across them. This design overlooks that different modules may need different computational capacity for a downstream task. For a more adaptive and more efficient solution, we introduce BLAde (Budget-constrained LoRA Mixture-of-Experts Adaptive Allocation), a task-adaptive framework that jointly determines where MoE capacity should be introduced and how the available factor budget should be allocated across LoRA modules. Using lightweight statistics collected from forward activations, BLAde identifies promising modules for conditional expert specialization, adaptively determines their number of experts and top- routing configurations, and reallocates LoRA ranks across the remaining modules under a factor budget. In this way, BLAde reformulates LoRA-MoE design from a manually specified, architecture-wide configuration into a task-adaptive and budget-aware allocation problem, allocating conditional capacity according to module activation geometry.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Parameter Efficient Fine-tuning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KoVkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BLAde, a framework that uses forward activation statistics to allocate LoRA-MoE capacity before fine-tuning. Output variance guides expert placement and counts, while a normalized joint effective-rank score guides rank allocation and active expert counts. Experiments evaluate accuracy and computational costs across several backbones and reasoning and generation tasks.

Critical strengths
  1. Jointly determining expert placement, expert configuration, and ordinary LoRA ranks under a factor budget addresses a practically relevant limitation of uniform adapter configurations.

  2. The evaluation covers multiple backbones and task types, includes initialization and training costs, and compares against a gradient-based allocation variant. The consistently competitive results support the practical potential of forward-only calibration.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of the proposed allocation signals. Comparisons with existing methods and BLAde-G do not establish whether the gains require the specific variance and effective-rank criteria, rather than heterogeneous allocation itself. Budget-matched random or shuffled allocations, placement-only and rank-only variants, and simpler scoring rules are needed to substantiate the central contribution.

  2. The theoretical motivation remains disconnected from parts of the practical allocation procedure. The variance argument compares constant correction vectors, whereas shared LoRA already produces input-dependent corrections. Replacing backward errors with normalized forward outputs introduces another approximation, and the effective-rank proxy has only modest average correlation with its gradient counterpart (0.229). The mappings from these scores to expert counts and active expert counts remain heuristic. Consequently, the analysis does not yet establish why these particular allocation rules should identify useful adaptation capacity.

Questions for authors
  1. Under matched factor budgets and comparable active computation, how does BLAde compare with random or shuffled module allocations? How much improvement comes separately from expert placement, expert-count allocation, top-k allocation, and ordinary LoRA rank redistribution?

  2. Do the proposed module scores predict the measured benefit of adding capacity to individual modules, beyond correlating with gradient statistics? Would simpler output-variance-only allocation achieve comparable results?

  3. Does BLAde retain its advantage under matched end-to-end training time or active-computation budgets? Figure 4 reports approximately 29% higher per-step training time than LoRA, so a compute-matched comparison would clarify its practical efficiency.

3 comments

We thank the reviewer for these valuable suggestions. We have revised the manuscript, with all changes highlighted in red.

1. Response to Weakness 1 and Question 1

We have added six alternatives in Sec. 4.6 (Table 4): FFN-only MoE, Attention-only MoE, and Random-MoE-allocation use 2-in-8 routing with MoE placed in FFN modules, attention modules, or 20% of randomly selected modules, respectively. Fixed 2in8 and Fixed 8 experts retain BLAde's placement but use fixed 2-in-8 routing or 8 experts with adaptive top-kk, respectively. All-random randomizes all allocation decisions.

MethodBoolQPIQASIQAHellaS.WinoG.ARC-EARC-COBQAAvg.
FFN-only MoE75.789.681.795.888.792.081.987.886.6
Attention-only MoE75.189.482.695.788.292.482.387.886.7
Random-MoE-allocation74.889.582.296.089.292.682.188.086.8
All-random75.489.181.295.588.992.480.988.486.5
Fixed 2in875.289.483.095.889.792.981.988.287.0
Fixed 8 experts75.789.683.396.189.493.682.588.887.4
BLAde76.790.383.296.390.193.282.389.287.7

BLAde achieves the highest average accuracy, supporting the benefits of joint allocation over simpler alternatives, as discussed in Sec. 4.6.

2. Response to Weakness 2

We have revised Sec. 3.2.2 to clarify that the analysis concerns weight-update matrices rather than correction vectors. The shared matrix approximation error is

\overset{\mathbf{M}=\mathbb{E}[\mathbf{U}_t]}{=} \operatorname{Tr}(\mathbf{C}_U),$$ where $\mathbf{U}_t$ is the token-specific update matrix and $\mathbf{C}_U$ its vectorized covariance, yielding the bound in Eq. (7). Thus, this is a terminological correction, not a mathematical change. LoRA shares one update matrix across tokens, while LoRA-MoE enables token-dependent updates. We have clarified the limitations of forward-only statistics in Secs. 3.2.3 and 3.3 and Appendix D.3. Gradient directions depend on both inputs and backward errors, making effective-rank estimation from forward activations challenging. We therefore use normalized outputs to approximate gradient geometry rather than reconstruct it exactly. As shown in Appendix H.1 (Table 16), BLAde achieves the highest correlation (0.229) and rank overlap (0.352), while all alternatives show negative correlations, supporting our proxies as more informative allocation priors. Its performance also closely matches gradient-based BLAde-G (Sec. 4.1, Table 2), suggesting exact gradient reconstruction is unnecessary. We have also clarified the expert-count and top-$k$ mappings in Sec. 3.4 and Appendix B.2. Uniform configurations overlook heterogeneous capacity demands, while directly estimating module-wise marginal benefits requires costly training or search. We instead use forward statistics to guide budget-constrained allocation. Although the mappings are heuristics without global optimality guarantees, Sec. 4.6 (Table 4) demonstrates their benefits over fixed and random configurations. We have also added Appendix H.2 (Stability Under Calibration Sampling), which evaluates the consistency of module rankings and allocation decisions across repeated calibration samples, different sample sizes, and sketch widths. ### 3. Response to Question 2 We have clarified the motivation and limitations of our allocation proxies in Secs. 3.1–3.4 and Appendices C and D. Directly measuring module-level marginal gains is challenging: increasing capacity changes the total budget, while maintaining a fixed budget requires reducing capacity elsewhere. Module interactions also make individual gains non-additive, and small differences may be obscured by evaluation variability. As formulated in Sec. 3.1 (Eq. 2), directly optimizing global allocation would require training many candidate architectures. Instead, we use forward-only proxies motivated by the geometry of task-dependent weight updates, whose dimensionality and variation inform adaptation demands. We do not claim these scores precisely predict isolated performance gains. Their effectiveness is supported by proxy–gradient agreement (Table 1 and Appendix H.1), downstream improvements, and performance close to gradient-based BLAde-G (Table 2). ### 4. Response to Question 3 We have added compute-matched comparisons in Appendix H.7 (Table 22). To approximately match end-to-end training time, we reduce BLAde's training steps to 1/1.293 of LoRA's best-checkpoint steps, compensating for its 1.293× per-step latency (Fig. 4) under the same experimental setup. |Method|BoolQ|PIQA|SIQA|HellaS.|WinoG.|ARC-E|ARC-C|OBQA|Avg.| |-|-|-|-|-|-|-|-|-|-| |LoRA|75.4|88.2|82.1|95.4|88.1|92.8|81.0|88.4|86.4| |BLAde (early stopped)|76.6|90.1|82.8|96.0|89.6|93.2|82.1|89.2|87.5| |BLAde|76.7|90.3|83.2|96.3|90.1|93.2|82.3|89.2|87.7| As shown in Appendix H.7 (Table 22), BLAde retains a better average accuracy advantage under matched training time.

Thank you for the detailed response and the additional experiments. The random and fixed-configuration baselines strengthen the evidence for BLAde’s joint allocation strategy, and the approximately time-matched comparison addresses my concern about whether its advantage over LoRA persists under a similar training-time budget.

My main remaining concern is the contribution of the proposed forward-only scores. The new ablations compare different allocation configurations, but they do not show how much the effective-rank score contributes beyond a simpler variance-only rule under the same budget. It also remains unclear whether modules with higher scores benefit more from additional capacity. Since the choice of allocation signals is central to the paper, I am maintaining my original rating.

Thank you for your thoughtful question. To further strengthen the empirical evidence for our method, we have conducted additional ablation studies and incorporated the results into Appendix 4.6, Table 24 of the revised manuscript. The new content is highlighted in red for ease of reference.

We compare two allocation signals: Variance, which uses output variance alone for MoE allocation, expert count, LoRA rank, and top-k; and BLAde, which uses variance for MoE allocation and expert count, and effective rank for LoRA rank and top-k. For each, Top assigns higher ranks and prioritizes MoE replacement for modules whose scores are closer to the maximum. Median favors modules near the median score percentile, with capacity priority decreasing toward either tail. Bottom favors modules whose scores are closer to the minimum. All three settings use the same underlying score estimates, target parameter budget, and rank bounds. However, they recompute the allocation under their respective priorities.

ModeBoolQPIQASIQAHellaSWinoGARC-EARC-COBQAAvg
Variance-Bottom75.189.382.095.389.892.680.587.686.5
Variance-Median75.889.682.495.889.292.380.789.086.9
Variance-Top76.290.183.295.690.192.880.688.687.1
BLAde-Bottom74.588.681.795.388.591.980.087.085.9
BLAde-Median75.689.682.895.989.793.482.488.487.2
BLAde-Top76.790.383.296.390.193.282.389.287.7

The results suggest that BLAde’s combined signals guide capacity allocation more effectively than variance alone. Prioritizing high-scoring modules achieves the best overall performance, while favoring low-scoring modules substantially degrades performance. The ordering of BLAde-Top, BLAde-Median, and BLAde-Bottom further supports the practical value of its allocation signals.

Reviewer_WLrwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BLAde, a framework for allocating ordinary LoRA adapters and routed LoRA experts under limited parameter budget. In this method, a forward-only calibration stage computes normalized output variance and a marginal-normalized effective-rank score. These statistics guide MoE placement, expert allocations, and ordinary LoRA ranks. Experiments on reasoning and generation tasks with 3B–8B backbones report improvements over standard LoRA and competitive performance against adaptive-rank and LoRA-MoE baselines.

Critical strengths

Practical allocation pipeline with inexpensive cost. BLAde provides a concrete mechanism for deciding which modules receive routed experts while preserving capacity for ordinary adapters. BLAde chooses the adapter configuration without computing gradients, using obviously lower GPU memory than baseline methods. This reduces the memory needed to set up the adapters before fine-tuning.

Substantial improvements under the evaluated settings. BLAde improves the displayed average reasoning accuracy over LoRA by approximately 1.2–1.4 percentage points across three backbones and also improves generation results. These results support the practical usefulness of the overall framework.

Critical weaknesses

The contribution of the proposed method is insufficiently explained. GuiLoMo already studies task-dependent joint allocation of expert counts and ranks, while EVA uses forward activations for rank redistribution. BLAde’s distinctive contribution therefore depends substantially on selective MoE placement and its particular proxy-driven allocation rules. The reported comparisons do not cleanly separate these contributions.

The accuracy gains come with additional computational cost. On LLaMA3-8B, BLAde increases training-step time by approximately 29% and reported forward-pass memory by 61% relative to LoRA. The paper does not compare accuracy under the same training-time or memory, leaving unclear whether the improvement justifies the additional cost.

The benefit of adaptive expert counts is not justified by the generation experiments. Equation (20) leaves only four experts admissible from the candidate set {2,4,8,16}, so every routed module uses 4 rank-5 experts. These experiments therefore do not test adaptive expert-count allocation. A comparison against fixed expert counts in settings with multiple admissible choices would help.

Questions for authors

The method uses p=8 for allocation, but the proxy evaluation uses p=16. Why are these settings different?

Does BLAde perform better than selecting modules randomly or placing experts only in FFN modules under the same parameter budget?

2 comments

We thank the reviewer for these valuable suggestions. We have revised the manuscript, with all changes highlighted in red. Our responses are provided below.

1. Response to Weakness 1 and Question 2

Unlike GuiLoMo's expert-capacity allocation and EVA's activation-based rank redistribution, BLAde uses forward-only proxies of gradient-related adaptation geometry to jointly allocate MoE placement, expert configurations, and LoRA ranks under a fixed budget. We have added six alternatives in Sec. 4.6 (Table 4): FFN-only MoE, Attention-only MoE, and Random-MoE-allocation use 2-in-8 routing with MoE placed in FFN modules, attention modules, or 20% of randomly selected modules, respectively. Fixed 2in8 and Fixed 8 experts retain BLAde's placement but use fixed 2-in-8 routing or 8 experts with adaptive top-kk, respectively. All-random randomizes all allocation decisions.

MethodBoolQPIQASIQAHellaS.WinoG.ARC-EARC-COBQAAvg.
FFN-only MoE75.789.681.795.888.792.081.987.886.6
Attention-only MoE75.189.482.695.788.292.482.387.886.7
Random-MoE-allocation74.889.582.296.089.292.682.188.086.8
All-random75.489.181.295.588.992.480.988.486.5
Fixed 2in875.289.483.095.889.792.981.988.287.0
Fixed 8 experts75.789.683.396.189.493.682.588.887.4
BLAde76.790.383.296.390.193.282.389.287.7

BLAde achieves the highest average accuracy, supporting joint allocation over simpler alternatives, as discussed in Sec. 4.6.

2. Response to Weakness 2

We have added training-time and memory-matched comparisons in Appendix H.7 (Table 22). Since BLAde requires 1.293× the per-step training time of LoRA (Fig. 4), we reduce its training steps by 22.7% to approximately match training time. For memory matching, we compare LoRA r=32 with BLAde r=8 under comparable GPU memory usage (approximately 3,595 MiB for BLAde r=8), accounting for MoE routing and expert computation overhead.

MethodBoolQPIQASIQAHellaS.WinoG.ARC-EARC-COBQAAvg.
LoRA r=3275.488.282.195.488.192.881.088.486.4
BLAde r=32 (early stopped)76.690.182.896.089.693.282.189.287.5
BLAde r=3276.790.383.296.390.193.282.389.287.7
BLAde r=874.489.282.395.888.694.080.487.886.6

BLAde outperforms LoRA by 1.1 points under approximately matched training time and achieves comparable accuracy under matched memory, demonstrating its favorable performance–efficiency trade-off.

3. Response to Weakness 3

To address this concern, we additionally compare BLAde's adaptive expert allocation against fixed expert-count configurations. As detailed in our response to Weakness 1 and Question 2, we have added comparisons on eight reasoning tasks where multiple expert counts are admissible (Sec. 4.6, Table 4). We further compare adaptive BLAde with fixed 4- and 8-expert baselines on NLG tasks, keeping other allocation settings unchanged (Appendix H.8, Table 23).

MethodGSM8KMBPPHumanEvalNQ
Fixed 4 experts63.049.443.941.8
Fixed 8 experts63.949.044.541.1
BLAde63.550.244.541.8

BLAde achieves the highest accuracy on MBPP, ties the best results on HumanEval and NQ, and remains competitive on GSM8K. Together with the reasoning-task results, these comparisons support adaptive allocation over fixed expert configurations.

4. Response to Question 1

BLAde uses p=8p=8 for efficient allocation with a 64×6464\times64 covariance matrix, while p=16p=16 provides richer features for proxy–oracle evaluation. We have clarified these settings and added the p=8p=8 results in Table 1.

SignalρS(v)\rho_S(v)ρS(q)\rho_S(q)MoE overlapRank overlap
Input only-0.243-0.0950.0470.152
Output only0.672-0.1500.4140.176
Input × output0.190-0.1260.2150.160
Ours0.6720.1510.3670.277

Despite lower effective-rank agreement than p=16p=16, our proxy retains the highest effective-rank correlation and rank overlap among the evaluated forward-only alternatives, supporting p=8p=8 as an efficient allocation setting.

Thank you for the response. However, most of my major concerns remain unresolved in the current version, and addressing them is essential to the work's completeness. I also found that the proposed method's improvement is not that significant. Therefore, I will maintain my original rating.

Reviewer_Sr9hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BLAde (Budget-constrained LoRA Mixture-of-Experts Adaptive Allocation), a forward-only framework that jointly determines where to introduce LoRA-MoE capacity and how to distribute the available factor budget across LoRA modules. The key insight is that two complementary statistics computed from forward activations—normalized output variance and marginal-normalized paired effective rank—are sufficient to guide these decisions without requiring backward calibration. Specifically, response variance identifies modules suitable for token-routed expert specialization and determines their expert counts, while effective rank guides MoE eligibility, active expert counts per token, and ordinary LoRA rank redistribution. BLAde is evaluated across reasoning and generation tasks on LLaMA2-7B, LLaMA3-8B, LLaMA3.2-3B, Gemma-7B, and Mistral-7B, demonstrating consistent improvements over standard LoRA and other adaptive baselines while maintaining low initialization and training overhead.

Critical strengths
  • Comprehensive and Fair Experimental Comparison: The paper provides a thorough evaluation against a wide range of baselines, including standard LoRA, rank-adaptive methods (AdaLoRA, EVA), and adaptive LoRA-MoE methods (MoRE, AdaMoLE, AlphaLoRA, GuiLoMo). The comparisons are conducted across multiple backbones and tasks, with careful parameter-count reporting. Additionally, the paper includes detailed overhead comparisons (training time per step, forward-pass latency, forward-pass memory, initialization time, and GPU memory), which are often neglected in similar works. This gives a very complete picture of the practical trade-offs.
  • Well-Motivated Forward-Only Design: The decision to use only forward activations for allocation decisions is both principled and practical. The theoretical motivation connecting output variance to the upper bound of conditional specialization gain (Theorem C.2) and effective rank to necessary LoRA rank (Theorem C.1) provides a clear rationale. The proxy-oracle agreement experiment (Table 1) empirically validates that the forward statistics are informative proxies for gradient-based signals, and the comparison with BLAde-G shows minimal accuracy loss (within 0.2 points) at substantially lower cost.
  • Thorough Hyperparameter Sensitivity Analysis: The paper systematically evaluates sensitivity to target rank, MoE fraction ρ, initial-rank multiplier µ, and calibration-set size. The results demonstrate robustness across a range of settings, and the excessive-scaling control (Table 11) provides useful boundary information. The calibration-size experiment shows that BLAde can derive informative signals from as few as 256 examples.
  • Intuitive and Practical Framework: The overall design philosophy—deciding whether a module needs conditional specialization (variance) versus how much shared capacity it needs (effective rank)—is intuitive and well-motivated. The budget-constrained formulation ensures that the total parameter count remains comparable to a uniform-rank baseline, making the comparison fair and the method deployable.
Critical weaknesses
  • Lack of Post-Training Adaptive Behavior Analysis: While the paper provides extensive allocation maps (Figures 5–12) showing which modules are selected and how ranks are distributed before training, there is limited analysis of what happens after training. Specifically, the paper does not examine: (a) whether the selected routed modules actually develop distinct expert specializations post-training; (b) whether the router distributions become more or less concentrated after fine-tuning; (c) whether modules assigned higher ranks by BLAde indeed capture more task-relevant gradient signal in the final model; or (d) how the learned routing patterns differ across tasks. Such analysis would significantly strengthen the mechanistic understanding of why the allocation works.
  • Scalability to larger models: All experiments use 3B–8B models. The paper does not discuss whether the forward proxy statistics remain informative at the scale of 70B+ models, where module interactions may be qualitatively different.
Questions for authors
  • Post-training routing behavior: After fine-tuning with BLAde's allocated architecture, how do the learned routing distributions evolve? Specifically, (a) do the selected routed modules develop task-specific expert specializations, and can this be measured via expert utilization entropy or pairwise expert output similarity? (b) How does the effective number of experts used per token change from initialization to convergence? This would help explain why the allocation works beyond just reporting accuracy.
  • Cross-task allocation stability: Figures 5–12 show that allocations vary across tasks (e.g., BoolQ selects q_proj and k_proj while others do not). If a model needs to serve multiple tasks simultaneously, how would BLAde handle this? Have you considered a multi-task calibration setting, and if so, does the resulting allocation remain effective for individual tasks?
  • Comparison with search-based methods: The paper compares against GuiLoMo, which uses bilevel optimization. However, how does BLAde compare against simpler search baselines, such as a grid search over a few discrete MoE placement configurations (e.g., "all FFN layers with 4 experts" vs. "attention layers with 8 experts") trained to convergence? This would help quantify how much of BLAde's gain comes from the principled allocation versus simply having any heterogeneous configuration.
1 comment

We thank the reviewer for these valuable suggestions. We have revised the manuscript, with all changes highlighted in red. Our responses are provided below.

1. Weakness 1

BLAde allocates capacity before fine-tuning without modifying routing or training; expert behavior thus emerges rather than being explicitly optimized. Nevertheless, we have added further analyses in Appendices H.3–H.6.

(a) Expert specialization. The mean pairwise cosine similarity between expert outputs after fine-tuning is 0.354 across eight reasoning tasks, indicating diverse expert outputs (Table 18).

(b) Router distributions. We measure normalized selection entropy from top-kk expert frequencies and token router entropy from pre-top-kk softmax probabilities. Both range from 0 to 1, with higher values indicating more balanced routing. Their averages across eight tasks are 0.977 and 0.956, respectively, indicating balanced routing (Table 21).

(c) Ranks and gradient signals. Higher rank does not necessarily imply greater gradient capture, as gradient concentration varies across modules. BLAde instead estimates the effective rank of gradient-driven weight updates. Across the same ordinary-LoRA modules and calibration data, the Spearman correlation between initial ranks and post-training gradient capture is 0.4208 for BLAde vs. 0.2129 for EVA, averaged over eight tasks, supporting better alignment with gradient information (Table 19).

(d) Cross-task routing. Direct comparisons are difficult because tasks use independently trained MoE architectures with different expert identities. Our entropy results nevertheless suggest balanced routing. Since BLAde targets task-dependent capacity allocation, comparing allocated configurations and performance is more relevant. Shared-architecture cross-task routing analysis remains future work.

2. Weakness 2

We agree that 3B–8B experiments cannot establish 70B+ scalability. Nevertheless, BLAde uses dimension-normalized statistics (Eqs. 8–11) and relative allocation criteria (Eq. 22), supporting potential scale transfer. Its forward-only calibration uses 256 examples, adding just 165 MiB and a few hundred seconds at 8B (Fig. 3), far less costly than gradient-based alternatives. We further evaluate proxy–oracle agreement on LLaMA3.2-3B:

SignalρS(v)\rho_S(v)ρS(q)\rho_S(q)MoE overlapRank overlap
Input only-0.133-0.0290.1140.193
Output only0.658-0.0910.4500.157
Input × output0.215-0.0590.2430.164
Ours0.6580.1790.4140.257

Our proxy achieves the highest effective-rank correlation and rank overlap, supporting reliability at the tested scale.

3. Question 1

As detailed in Appendix H.5 (Table 20), we measure effective experts via exponentiated routing entropy: Selected uses normalized top-kk probabilities, while Full uses pre-top-kk softmax probabilities. Results are averaged over probe tokens, MoE modules, and eight tasks.

StepSelected effective expertsSelected/kFull-router effective expertsFull/E
04.24650.96147.27600.9190
10004.24170.96097.27820.9193
20004.24210.96107.27810.9193

The results indicate balanced routing and support BLAde's initial MoE allocation.

4. Question 2

BLAde can support multi-task calibration by jointly computing variance and effective-rank statistics from multiple tasks, with adjustable sampling proportions to reflect task importance. However, BLAde focuses on budget-aware LoRA-MoE allocation rather than multi-task learning, and conditional adaptation does not require task-specific experts. Thus, Figures 5–12 illustrate task-dependent adaptation requirements, not the necessity of separate architectures. BLAde can derive a unified allocation from combined calibration data, although multi-task validation remains future work.

5. Question 3

We have added six alternatives in Sec. 4.6 (Table 4). FFN-only MoE, Attention-only MoE, and Random-MoE-allocation convert all targeted FFN modules, all targeted attention modules, or 20% of randomly selected modules into MoE, respectively, all using 2-in-8 routing. Fixed 2in8 and Fixed 8 experts retain BLAde's MoE placement but use fixed 2-in-8 routing or 8 experts with adaptive top-kk, respectively. All-random randomizes MoE placement, LoRA ranks, expert counts, and top-kk.

MethodBoolQPIQASIQAHellaS.WinoG.ARC-EARC-COBQAAvg.
FFN-only MoE75.789.681.795.888.792.081.987.886.6
Attention-only MoE75.189.482.695.788.292.482.387.886.7
Random-MoE-allocation74.889.582.296.089.292.682.188.086.8
All-random75.489.181.295.588.992.480.988.486.5
Fixed 2in875.289.483.095.889.792.981.988.287.0
Fixed 8 experts75.789.683.396.189.493.682.588.887.4
BLAde76.790.383.296.390.193.282.389.287.7

The results support joint allocation over simpler alternatives, as discussed in Sec. 4.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.