acceptodds
Under review as a conference paper at ICLR 2027

HeteroLoRA-MoE: Heterogeneous Low-Rank Experts with Diverse Projection Patterns and Efficient GPU Kernels

Abstract

Expert differences in a pretrained mixture-of-experts (MoE) model exhibit low-rank structure, motivating compact expert representations based on shared weights and low-rank updates. Motivated further by variation in this structure across projections and experts, we present HeteroLoRA-MoE, which distributes expert-specific low-rank updates across four balanced projection patterns over shared dense SwiGLU weights. A single token-level router selects complete experts and combines their outputs. This compact design introduces small matrix products and projection-dependent scheduling, so parameter savings do not automatically translate into lower latency. To address these execution costs, we develop HeteroKernel, a structure-aware GPU implementation that reuses shared gate and up results and aggregates expert intermediates before applying the shared dense down projection. Projection-aware grouping, fused reductions, and pipelined weight loading organize the remaining adapter computation, skipping absent branches while preserving the mixture algebraically. In four-seed mixed-SFT experiments initialized from Qwen2.5-0.5B/1.5B-Instruct, HeteroLoRA-MoE achieves higher mean scores than a parameter-matched MixLoRA-style baseline on multiple evaluation tasks at both model scales. On NVIDIA L20, HeteroKernel reduces the overhead of routed low-rank computation, achieving 1.85–6.79x speedups over the PyTorch implementation across the tested prefill and one-token decode settings.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture of Experts, Low-Rank Adaptation, Heterogeneous Experts, GPU Kernel Optimization, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bH1rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HeteroLoRA-MoE, a parameter-efficient MoE architecture that shares pretrained SwiGLU weights across experts while introducing heterogeneous LoRA projection patterns. The design is motivated by an empirical analysis of low-rank approximations to expert weight differences in a pretrained MoE model. The authors further develop HeteroKernel, a specialized CUDA implementation that reduces redundant computation through shared projection reuse, early routing masks, and kernel fusion. Experiments on Qwen2.5-0.5B and 1.5B demonstrate improvements over a parameter-matched MixLoRA-style baseline on several downstream tasks. The inference performance is evaluated on an NVIDIA L20 GPU.

Critical strengths

S1. The shared-down projection optimization is technically well motivated. Aggregating expert intermediates before applying the shared dense projection avoids redundant matrix multiplications. The kernel ablations provide useful evidence for the effectiveness of the proposed execution optimizations.

S2. The experimental evaluation includes parameter-matched comparisons against homogeneous and fixed-projection LoRA-MoE variants. Reporting results across four random seeds and two model scales provides a more reliable assessment of the downstream performance than a single-run comparison.

Critical weaknesses

W1. The motivation for the proposed heterogeneous expert design is not sufficiently supported. The low-rank analysis in Section 2 is conducted only on Phi-tiny-MoE-instruct, whose FFN intermediate dimension is 448. A rank-128 approximation already represents approximately 28.6% of the maximum possible matrix rank. Moreover, the reported residual-output retention measures approximation quality under a specific activation distribution rather than directly establishing that expert weight differences are intrinsically low-rank. It remains unclear whether similar properties hold for larger pretrained MoE models.

The connection between this analysis and the proposed architecture is also unclear. Table 2 shows differences in low-rank retention across projections, but does not explain why the authors select four specific projection patterns and allocate experts equally among them. Although Table 3 includes fixed-pattern ablations, these experiments do not establish how the observed low-rank properties should guide adapter placement or capacity allocation.

W2. The experimental evaluation does not sufficiently characterize the trade-offs between model capacity and inference efficiency. Although Table 5 investigates different expert counts and LoRA ranks, the configurations are not matched in parameter budget. A controlled comparison between more experts with smaller ranks and fewer experts with larger ranks is needed to assess whether the proposed allocation is preferable.

The system evaluation is also limited to a single NVIDIA L20 GPU, batch size 1, and Top-2 routing. The impact of different Top-K settings and heterogeneous expert workloads on inference latency remains unexplored. In particular, different projection patterns incur different computational costs, but the paper provides no analysis of per-expert routing distributions or computational load imbalance. Together with the relatively modest decode improvements in the matched-backend comparison, these limitations make it difficult to assess the scalability and generality of HeteroKernel.

Questions for authors

Q1. Could the authors provide additional evidence for the low-rank structure of expert weight differences? In particular, does the observed residual-output retention persist in pretrained MoE models with larger hidden and intermediate dimensions?

Q2. Although Table 3 includes comparisons with fixed projection patterns, the rationale behind the four selected patterns and their uniform allocation remains unclear. How do the observations in Table 2 justify this particular design? Would a layer-dependent or data-driven allocation lead to better results?

Q3. Table 5 investigates different expert counts and LoRA ranks, but the configurations do not have matched parameter budgets. Could the authors further examine this trade-off under a fixed parameter budget? In particular, how would more experts with smaller ranks compare against fewer experts with larger ranks in terms of model quality and inference latency?

Q4. How does the proposed method handle load imbalance across experts with different projection patterns? Since these patterns incur different computational costs, balanced token assignments do not necessarily imply balanced computational workloads. Could the authors provide per-expert routing statistics and analyze the potential impact on GPU utilization and inference latency?

Q5. How does HeteroKernel perform with different numbers of active experts per token (Top-K)? Given that shared-down projection reuse is a central optimization, evaluating different Top-K settings would help clarify how its computational benefits and inference latency scale with the number of activated experts.

Q6. The inference evaluation is limited to a single NVIDIA L20 GPU with batch size 1. How well would the proposed kernel optimizations generalize to other GPU architectures, such as Hopper or Blackwell, and larger inference batches? I understand that access to multiple GPU platforms may be limited. If additional experiments are not feasible, an analytical discussion of the expected performance characteristics and potential bottlenecks would also be valuable.

Given the limited rebuttal period, I do not expect full multi-seed evaluations for the additional training-based ablations. Single-seed results would be acceptable as preliminary evidence, provided that the experimental settings are consistent and the limitations are acknowledged. Latency experiments should still include repeated measurements to account for timing variability.

6 comments

Thank you for recognizing the technical motivation of our shared-down optimization, the supporting kernel ablations, and the parameter-matched comparisons across multiple seeds and model scales. We also appreciate your constructive suggestions for strengthening the structural motivation and further characterizing the capacity–efficiency trade-offs. We have incorporated the additional experiments and their analyses into the main text and appendices of the revised manuscript.

Q1: Could the authors provide additional evidence for the low-rank structure of expert weight differences? In particular, does the observed residual-output retention persist in pretrained MoE models with larger hidden and intermediate dimensions?

A: Thank you for your suggestions regarding the interpretation of the low-rank analysis and model scale. We agree that residual-output retention measures approximation quality under a given input distribution. We therefore revise the wording to state that "expert weight differences admit low-rank approximations under the actual input distribution," which more accurately expresses our structural motivation.

The additional analysis follows the statistical procedure in the paper's motivation section. For each projection in each layer, we use the arithmetic mean of the expert weights as the shared base and compute each expert's weight residual relative to this mean. We fit activation-aware low-rank approximations on calibration inputs and measure residual-output retention on independent test inputs, reporting the median across layer–expert pairs separately for Gate, Up, and Down. The table uses ranks 8, 32, and 128, consistent with the original paper, and reports retention as a percentage.

ModelProjectionrank8rank32rank128
Phi-tiny-MoEGate44.7063.0282.97
Phi-tiny-MoEUp38.1456.2079.30
Phi-tiny-MoEDown22.2247.8966.51
DeepSeek-MoE-16BGate31.5456.3469.67
DeepSeek-MoE-16BUp25.7945.9769.84
DeepSeek-MoE-16BDown40.2458.3776.26
MixtralGate51.6960.2677.28
MixtralUp52.1759.3286.22
MixtralDown36.9944.8263.54

For DeepSeek-MoE-16B, rank 32 retains 56.34%, 45.97%, and 58.37% of the residual-output energy for Gate, Up, and Down, respectively; the corresponding values for Mixtral are 60.26%, 59.32%, and 44.82%. Increasing the rank to 128 raises retention across projections to 69.67%–76.26% and 63.54%–86.22% for the two models, respectively. These results show that expert differences also exhibit low-rank approximability in larger pretrained MoEs, providing additional support for a shared base with low-rank expert updates.

Q2: Although Table 3 includes comparisons with fixed projection patterns, the rationale behind the four selected patterns and their uniform allocation remains unclear. How do the observations in Table 2 justify this particular design? Would a layer-dependent or data-driven allocation lead to better results?

A: Thank you for your suggestions on projection combinations and allocation. We provide additional evidence from projection preference analysis of larger pretrained MoEs and allocation-ratio ablations.

First, Table 2 shows that projection retention rankings differ across experts, motivating different adapter projection combinations. We extend the same analysis to DeepSeek-MoE-16B and Mixtral to examine whether the projection preference distribution in Phi-tiny directly generalizes to other models. We use the mean expert weights as the base and count the six projection retention rankings under activation-aware rank-32 approximations on independent inputs.

ModelG > U > DG > D > UU > G > DU > D > GD > G > UD > U > GTotal
Phi-tiny-MoE (original Table 2)381166573310512
DeepSeek-MoE-16B1161791271958542571,728
Mixtral453186471928256

The results show that projection preference distributions differ across models. Gate has the highest retention in 77.54% of layer–expert pairs in Phi-tiny, Down in 64.29% in DeepSeek, and Up in 51.95% in Mixtral. Thus, preference proportions from a single pretrained model should not directly serve as a general allocation rule. We use GU, GD, UD, and GUD to cover all three two-projection combinations and the full three-projection combination, with equal allocation as a default independent of any particular pretrained model's distribution.

Second, in addition to the original fixed-pattern comparisons, we add allocation-ratio ablations with the same budget on Qwen2.5-0.5B to test whether favoring a particular projection improves downstream performance. The experiments use 16 experts, rank 32, Top-2 routing, seed 42, and identical SFT settings. The ratios are ordered as GU/GD/UD/GUD.

AllocationRatioGSM8KHellaSwagPIQAMMLUMBPPHumanEvalSix-task average
Equal4/4/4/432.0956.7569.5647.4529.0423.6443.09
Gate-biased5/5/2/432.0751.5067.0847.0328.6022.5641.47
Up-biased5/2/5/431.7750.6268.2347.0928.0025.6141.89
Down-biased2/5/5/431.5450.0666.5447.2928.0025.6141.51

None of the three unequal allocations exceeds the six-task average of equal allocation. Equal allocation outperforms the Gate-biased allocation on all six tasks, with a mean advantage of approximately 1.62 percentage points. The Up- and Down-biased allocations improve HumanEval but score lower on the other five tasks. Together with the original fixed-pattern ablations, these results support heterogeneous combinations and equal allocation.

In addition, fixed, equally sized pattern groups facilitate planning data layouts, tiling, and execution paths, and improve inference efficiency through HeteroKernel's shared computation reuse. Layer-wise or data-driven allocation would require further adaptation to varying group sizes and workloads. We therefore adopt this default design to balance downstream performance and efficient execution.

Q3: Table 5 investigates different expert counts and LoRA ranks, but the configurations do not have matched parameter budgets. Could the authors further examine this trade-off under a fixed parameter budget? In particular, how would more experts with smaller ranks compare against fewer experts with larger ranks in terms of model quality and inference latency?

A: Thank you for suggesting a comparison of expert count and rank under a fixed budget. Whereas the original Table 5 examines increasing capacity, we add a comparison with a fixed adapter-weight budget on Qwen2.5-0.5B using E8-r64, E16-r32, and E32-r16. All three configurations allocate experts equally across the four patterns and use Top-2 routing, seed 42, a learning rate of 1e−4, and one epoch of SFT on the same data.

The original Table 5 is Table 8 in the revised manuscript.

ConfigurationGSM8KHellaSwagPIQAMMLUMBPPHumanEvalSix-task average
E8-r6431.6152.0167.5747.4927.6023.7841.68
E16-r3232.0956.7569.5647.4529.0423.6443.09
E32-r1630.6349.4566.1047.0025.8029.2741.38

The default E16-r32 configuration exceeds E8-r64 and E32-r16 in the six-task average by approximately 1.41 and 1.71 percentage points, respectively. More lower-rank experts improve HumanEval but score below the default on the other five tasks; fewer higher-rank experts also do not achieve a higher overall score. This controlled single-seed experiment supports the default configuration's balance between expert count and capacity per expert.

We further measure the latency of all three configurations on the same L20 GPU. To keep the backend consistent, all configurations use the Triton grouped path supporting their respective ranks, with BF16, batch size 1, and identical inputs. We measure the complete layer-0 FFN, including the router. Each setting uses 20 warm-up iterations and 300 timed iterations, with p50 latency reported. The 1-token setting corresponds to a single-step FFN input shape.

Configuration1 token (ms)128 tokens (ms)512 tokens (ms)2048 tokens (ms)
E8-r640.54010.55620.58871.4531
E16-r320.54780.56320.57961.4275
E32-r160.82640.89090.89801.6200

The default E16-r32 has latency close to E8-r64 and reduces latency by 11.88%–36.78% relative to E32-r16 while achieving a higher six-task average. Lowering rank reduces adapter computation but leaves shared dense computation unchanged; increasing the expert count can also fragment token groups and reduce matrix execution efficiency, so more lower-rank experts are not necessarily faster under a fixed parameter budget.

Reviewer_1bq5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HeteroLoRA-MoE, an MoE layer made of one shared dense SwiGLU block plus per-expert LoRA updates on fixed subsets of projections (gate+up, gate+down, up+down or all three), selected by a single token-level top-KK router. The design is motivated by an analysis of Phi-tiny-MoE-instruct: expert weight deviations from the mean are low-rank (rank-32 activation-aware approximations retain 48–63% of residual-output energy against about 7% for random subspaces), and the best-retained projection varies across experts. HeteroKernel implements the layer with early routing masks, a shared down projection applied once to the routing-weighted aggregate, fused composition and double buffering. In mixed SFT of Qwen2.5-0.5B- and 1.5B-Instruct on six benchmarks with four seeds, the method scores 0.6–2.3 points above a parameter-matched MixLoRA-style baseline on five of six tasks at 0.5B and all six at 1.5B, and the CUDA kernels run 1.85–6.79× faster than the PyTorch reference on an L20 GPU.

Critical strengths
  • The low-rank analysis of expert differences is carefully done. Section 2 and Appendix A fit activation-aware subspaces on calibration tokens and measure retention on disjoint test tokens, with weight-SVD and random-subspace references at five ranks (Tables 1 and 7). The findings that rank-32 subspaces capture about half of the residual-output energy and that the ordering of gate, up and down differs across experts and within layers (Table 2) are informative for MoE compression independently of the proposed architecture.
  • The kernel is complete and each optimization is ablated. The exact shared-down factorization (Eq. 11) removes per-expert dense down projections, and Table 6 isolates early masking (4.9× at 2,048 tokens), shared-down reuse, fusion and double buffering. Table 9 checks that the CUDA path changes downstream scores by at most 1.22 points relative to the reference implementation.
  • The comparisons are parameter-matched and include single-pattern controls. LoRA SFT (rank 384), the MixLoRA-style baseline (rank 24) and the fixed gate+up, gate+down and up+down variants (rank 36) all use the same adapter budget, with four seeds at two model sizes, so heterogeneous allocation is tested against the obvious alternatives.
Critical weaknesses
  1. The analysis motivates compressing pretrained MoE experts, but the experiments test LoRA-MoE fine-tuning of dense models, and the pattern mix does not follow the analysis. Section 2 measures deviations among the experts of a pretrained MoE, while Section 5 attaches routed adapters to frozen dense Qwen2.5 models. The four masks receive equal numbers of experts, although Table 2 finds gate the best-retained projection in 397 of 512 layer–expert pairs and down in only 43. Converting a pretrained MoE such as Phi-tiny-MoE into shared weights plus heterogeneous low-rank experts and measuring the retained quality, or deriving the pattern mix from the measured statistics, would connect the analysis to the method.
  2. The quality margins over the matched baseline are comparable to the seed variation, and the evaluation is in-domain SFT. At 0.5B the margins over MixLoRA-style are +1.25, +2.25, +0.89, −0.01, +0.90 and +1.07 points, with standard deviations of 0.2–1.4 over four seeds (Table 3); a Welch test on these values gives p between about 0.03 and 0.3 for the five positive margins, and one HumanEval problem is worth 0.61 points. The models are trained on the training splits of five of the six benchmarks, and the untuned Instruct model scores near chance on HellaSwag (28.4) and PIQA (52.6), so the scores partly reflect answer-format learning. Significance tests, held-out tasks, per-method learning-rate tuning and comparisons with GuiLoMo (EMNLP 2025 Findings) and HydraLoRA (NeurIPS 2024) would strengthen the claim.
  3. The headline speedups are measured against a naive reference, and no established kernel is compared. The 1.85–6.79× figure compares HeteroKernel with the paper's own PyTorch implementation (Table 4). Against the matched CUDA implementation of the MixLoRA-style model, prefill latency is 6.2–26.5% lower and decode latency within about 6% (Table 8), and the model remains 4–10% slower than the dense model, whose latency equals that of LoRA with merged weights. All measurements use batch size 1 on one L20 GPU with the 0.5B model. Comparisons with MeteoRA's kernel, which the paper names as the closest system, with SGMV-style kernels from Punica or S-LoRA, and in batched settings are needed to establish the efficiency contribution.
Questions for authors
  1. Clarification. How were the four projection patterns and their equal split chosen, and does a split informed by Table 2 (more gate-containing patterns) change the results?
  2. Clarification. Can the method compress a pretrained MoE (for example Phi-tiny-MoE) into a shared base plus heterogeneous low-rank experts, and how much quality is retained at a given parameter budget?
  3. Clarification. Was the learning rate tuned per method? All methods use 1×10−41 \times 10^{-4}, which may favor some adapter configurations over others.
  4. Clarification. How does HeteroKernel compare with MeteoRA's kernel and in batched multi-request settings?
8 comments

Thank you for recognizing the clarity of our presentation, the careful analysis of expert differences, the thorough kernel ablations, and the parameter-matched comparisons with fixed-pattern baselines. To address your concerns, we further explain the rationale for heterogeneous projection patterns and uniform allocation, and supplement our evaluation with pretrained-MoE initialization, learning-rate tuning, generalization and related-method comparisons, as well as performance comparisons with established computational backends and at larger batch sizes, to more comprehensively assess the effectiveness and execution efficiency of our method. We have incorporated the additional experiments and their analyses into the main text and appendices of the revised manuscript.

Q1: Clarification. How were the four projection patterns and their equal split chosen, and does a split informed by Table 2 (more gate-containing patterns) change the results?

A: Thank you for your suggestions on projection configurations and allocation ratios. Our analysis of pretrained MoEs shows that projection retention rankings differ across experts, motivating different adapter projection combinations for different experts. We also extend the analysis to larger pretrained MoEs; for example, activation-weighted residual analysis of DeepSeek-MoE-16B likewise reveals differences in approximability across projections. We use GU, GD, UD, and GUD to cover all three two-projection combinations and the full three-projection combination, with equal allocation as the default without assuming a preference for a particular projection.

The four projection patterns and expert allocation ratios

The projection-distribution analysis follows Table 2 in the motivation section. We use the mean expert weights as the base, compare retention under activation-aware rank-32 approximations on independent validation inputs, and count the six complete rankings of G, U, and D.

ModelG > U > DG > D > UU > G > DU > D > GD > G > UD > U > GTotal
Phi-tiny-MoE (original Table 2)381166573310512
DeepSeek-MoE-16B1161791271958542571,728
Mixtral453186471928256

The results show that projection preference distributions differ across models. Gate has the highest retention in 77.54% of layer–expert pairs in Phi-tiny, Down in 64.29% in DeepSeek, and Up in 51.95% in Mixtral. Thus, the preference proportions in a single pretrained model should not directly serve as a general allocation rule. Equal allocation provides a default that does not depend on a particular model's projection preferences.

The original fixed-GU, fixed-GD, fixed-UD, and all-GUD comparisons already demonstrate the advantages of heterogeneous combinations on multiple tasks. To further compare equal allocation with allocations favoring specific projections, we add Gate-, Up-, and Down-biased allocation ablations on Qwen2.5-0.5B, holding the expert count at 16, rank at 32, Top-2 routing, and the adapter budget fixed. All runs use seed 42, a learning rate of 1e−4, and one epoch of SFT. The ratios below are ordered as GU/GD/UD/GUD.

AllocationRatioGSM8KHellaSwagPIQAMMLUMBPPHumanEvalSix-task average
Equal4/4/4/432.0956.7569.5647.4529.0423.6443.09
Gate-biased5/5/2/432.0751.5067.0847.0328.6022.5641.47
Up-biased5/2/5/431.7750.6268.2347.0928.0025.6141.89
Down-biased2/5/5/431.5450.0666.5447.2928.0025.6141.51

Equal allocation outperforms the Gate-biased allocation on all six tasks, with a six-task average approximately 1.62 percentage points higher. The Up- and Down-biased allocations also introduce task-dependent trade-offs, and neither exceeds the overall performance of equal allocation. Thus, directly translating Gate's retention preference in a pretrained MoE into more Gate-containing experts does not improve overall performance in this experiment. The original fixed-pattern ablations and the additional ratio ablations support heterogeneous combinations and equal allocation, respectively. Equal allocation achieves the highest six-task average among the evaluated configurations and provides regular groups that facilitate planning data layouts and execution paths. Combined with HeteroKernel, this enables shared computation reuse and efficient execution. We therefore adopt it as the default design balancing downstream performance and implementation efficiency.

Q2: Clarification. Can the method compress a pretrained MoE (for example Phi-tiny-MoE) into a shared base plus heterogeneous low-rank experts, and how much quality is retained at a given parameter budget?

A: We further evaluate pretrained-MoE initialization using Phi-tiny-MoE. Following the Fisher-weighted expert merging approach in D²-MoE [1], we merge expert weights using Fisher information and routing frequencies to construct a shared base, then initialize the expert residual adapters through activation-weighted low-rank decomposition. We subsequently freeze the base and fine-tune only the adapters and router. We compare Fisher-Hetero, Fisher-Homo, and Mean-Hetero, using the original MoE and MoE + LoRA, which retains the full expert weights, as references.

Additional experiments initialized from a pretrained MoE

The additional experiment covers all 32 layers, with 16 experts per layer and Top-2 routing. The heterogeneous configurations assign four experts each to GU, GD, UD, and GUD at rank 32; the homogeneous configuration uses GUD throughout at rank 24, matching the adapter budget. Fisher calibration uses 32 FineWeb documents of at most 256 tokens each, while activation statistics use 512 WikiText-103 sequences of length 2048. Original MoE + LoRA retains the full expert weights and adds unit-scaled rank-24 adapters to all three projections of each expert, training only adapters and routers. The revised appendix, “Pretrained-MoE Initialization and Ablations,” now provides the shared-base weighting formula, residual low-rank initialization, and calibration and fine-tuning settings. No separate base-training stage is required.

ConfigurationGSM8KHellaSwagPIQAMMLUMBPPHumanEval
Original Phi-tiny-MoE60.8074.1176.3958.3338.4053.66
Original MoE + LoRA65.8181.9879.2760.1646.4054.27
Fisher-Hetero66.1082.1279.1160.1745.2054.88
Fisher-Homo65.3680.3079.0760.0843.8052.44
Mean-Hetero64.3979.9779.0159.1143.6051.83

All additional fine-tuning runs use seed 42, one epoch, and a learning rate of 1e−4. The three converted models have approximately 1.28B total parameters, compared with approximately 3.92B for the original MoE + LoRA. Fisher-Hetero outperforms Fisher-Homo and Mean-Hetero on all six tasks, with arithmetic mean gains of 1.09 and 1.61 percentage points, respectively. These results support the effectiveness of heterogeneous allocation and Fisher-weighted initialization in this setting. Compared with MoE + LoRA, which retains the full expert weights, Fisher-Hetero scores higher on four tasks and achieves an essentially equal six-task average (64.60 versus 64.65), while using fewer total parameters. This shows that our method can also be initialized from a pretrained MoE and achieve competitive downstream performance.

References

[1] Gu et al. Delta Decompression for MoE-based LLMs Compression. ICML, 2025.

Q3: Clarification. Was the learning rate tuned per method? All methods use , which may favor some adapter configurations over others.

A: Thank you for your suggestion regarding learning rates. The original experiments use 1e−4 throughout to maintain common training settings. We add a learning-rate search for HeteroLoRA-MoE and MixLoRA-style on Qwen2.5-0.5B, evaluating 5e−5, 1e−4, and 2e−4 for each method with seed 42 and one epoch of SFT. Both methods use the same candidate set and validation set, with the learning rate selected by the mean negative log-likelihood of answer tokens on the validation set.

Learning-rate tuning and comparison fairness

The table below presents all configurations at the three learning rates. Downstream task scores are percentages.

MethodLearning rateGSM8KHellaSwagPIQAMMLUMBPPHumanEval
HeteroLoRA-MoE5e−530.2856.3470.2144.3929.0021.95
HeteroLoRA-MoE1e−432.0956.7569.5647.4529.0423.64
HeteroLoRA-MoE2e−431.5651.5268.3446.5629.2023.17
MixLoRA-style5e−529.1750.1466.2447.3126.2022.56
MixLoRA-style1e−431.1951.6068.4546.9826.7922.59
MixLoRA-style2e−430.4551.3568.2846.0327.2021.34

HeteroLoRA-MoE achieves higher scores on most tasks at each of the three learning rates and outperforms MixLoRA-style on all six tasks after separate tuning. This shows that our method maintains a relatively consistent quality advantage over the tested learning-rate range while also providing higher execution efficiency.

Reviewer_FHpnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HeteroLoRA-MoE builds each expert as shared dense SwiGLU weights plus a low-rank update placed on one of four fixed projection masks (gate+up, gate+down, up+down, all), with a single token-level router selecting whole experts. They also provide a CUDA implementation called HeteroKernel, which skips dead adapter branches, computes shared gate/up once, and aggregates routed intermediates before a single shared down projection. Motivation comes from activation-aware low-rank analysis of Phi-tiny-MoE expert deltas (rank-32 retention 63/56/48% vs ~7% random, with per-expert variation in which projections matter). They test on four-seed mixed-SFT of Qwen2.5-0.5B/1.5B-Instruct (10k examples, one epoch) against Dense/LoRA(r384)/full-MoE/MixLoRA-style(r24) and fixed-mask controls on six tasks, L20 latency with kernel ablations, a matched-backend CUDA-vs-CUDA comparison, a kernel-equivalence check, and from-scratch pretraining loss curves.

Critical strengths
  • The exact reformulation moving the shared down projection outside the routed sum is a clean, correct systems insight, and the paper measures it honestly: the matched-backend CUDA-vs-CUDA comparison and the kernel-equivalence check isolate the kernel's real contribution instead of resting on the eager-PyTorch headline.
Critical weaknesses
  • The entire validation lives in the dense-upcycling regime: experts are synthesized from dense Qwen checkpoints with zero-init deltas trained on 10k examples, while the motivating low-rank/heterogeneous-projection evidence comes from a real trained MoE (Phi-tiny-MoE); the paper never checks whether the learned deltas reproduce that structure, never evaluates on any real pretrained MoE, and the closest heterogeneous-adapter prior art (MoA, GuiLoMo, DMEP) is not compared empirically.
  • The only validation evidence is thin and partly non-significant: gains over the matched MixLoRA-style baseline are ~1–2 points on 0.5B/1.5B models trained on 10k examples for one epoch; with four seeds the HumanEval gap (+1.07, stds .66/1.36) is t≈1.4 and MMLU is an exact tie, and Table 5's finding that doubling experts or rank does not help is left unexplained — so neither the effect nor its scaling behavior is established.
Questions for authors
  1. Are the per-task gains over MixLoRA-style significant under paired per-seed tests (you already share seeds 42–45), and do they survive 10–100× more SFT data or ≥7B scale?
  2. Do the trained HeteroLoRA experts' deltas actually exhibit the activation-aware rank-32 retention and projection-ordering heterogeneity of Tables 1–2, and does the method still help when initialized from a real pretrained MoE (e.g., continued training of Phi-tiny-MoE/OLMoE) instead of dense upcycling?"
4 comments

Thank you for your positive assessment of our writing, insights, and CUDA-to-CUDA comparison. We particularly appreciate your recognition of this comparison, as using a CUDA baseline sets a more demanding benchmark for our method while enabling a fair assessment of its practical benefits. We are also grateful for your constructive comments, which have helped us clarify our design choices and strengthen the empirical evaluation. Below, we address each concern with additional explanations and experiments. We have incorporated the additional experiments and their analyses into the main text and appendices of the revised manuscript.

Q1: Are the per-task gains over MixLoRA-style significant under paired per-seed tests (you already share seeds 42–45), and do they survive 10–100× more SFT data or ≥7B scale?

A: Thank you for suggesting paired comparisons and evaluations at larger model and data scales. We address these points through task-wise comparisons across seeds, execution efficiency gains, and additional scaling experiments.

Comparisons paired by training seed

For each model size and task, we pair the results of HeteroLoRA-MoE and MixLoRA-style using the same training seed. The table below reports the differences for seeds 42–45, their means, and win/tie/loss counts. Positive differences favor HeteroLoRA-MoE, and all differences are in percentage points.

ModelTaskseed 42seed 43seed 44seed 45Win/tie/lossMean difference
0.5BGSM8K+0.898+2.659+0.634+0.8094/0/0+1.250
0.5BHellaSwag+5.154+1.970+0.609+1.2664/0/0+2.250
0.5BPIQA+1.106+1.174−0.079+1.3593/0/1+0.890
0.5BMMLU+0.470−0.206−0.220−0.0841/0/3−0.010
0.5BMBPP+2.248+1.608−0.230−0.0252/0/2+0.900
0.5BHumanEval+1.053+1.053+2.429−0.2553/0/1+1.070
1.5BGSM8K+0.449+0.556+0.664+0.7714/0/0+0.610
1.5BHellaSwag+0.919+1.026+1.134+1.2414/0/0+1.080
1.5BPIQA+1.009+1.116+1.224+1.3314/0/0+1.170
1.5BMMLU+0.849+0.956+1.064+1.1714/0/0+1.010
1.5BMBPP+2.189+1.563+0.937+0.3114/0/0+1.250
1.5BHumanEval+1.555+1.332+1.108+0.8854/0/0+1.220

At 0.5B, HeteroLoRA-MoE achieves higher mean scores on five of the six tasks, with gains of 0.89–2.25 percentage points, while MMLU is essentially unchanged; GSM8K and HellaSwag improve across all four seeds. At 1.5B, the mean gains range from 0.61 to 1.25 percentage points across the six tasks, with positive differences for every task and every evaluated seed. Our method improves execution efficiency while maintaining comparable overall task quality and showing improvements on multiple tasks. The CUDA-to-CUDA comparison further validates this efficiency advantage.

Scaling to a 7B model

We compare the two methods on Qwen2.5-7B-Instruct using the same SFT settings and matched parameter budgets. HeteroLoRA-MoE uses 16 heterogeneous experts of rank 32, while MixLoRA-style uses 16 homogeneous experts of rank 24. Both use Top-2 routing, one training epoch, a learning rate of 1e-4, an effective batch size of 64, and seed 42. Scores below are percentages, and differences are calculated from the displayed values.

TaskHeteroLoRA-MoEMixLoRA-styleDifference (pp)
GSM8K75.8974.45+1.44
HellaSwag92.4090.21+2.19
PIQA90.2688.36+1.90
MMLU72.7871.85+0.93
MBPP57.8056.60+1.20
HumanEval76.8375.61+1.22

At 7B, HeteroLoRA-MoE outperforms MixLoRA-style on all six tasks by 0.93–2.19 percentage points, showing that our method retains its downstream quality advantage at a larger model scale.

Increasing the SFT training data

For the 0.5B model, we double the number of distinct SFT training examples for both methods while keeping the validation set and one-epoch training setting unchanged. The table below uses seed 42, with differences calculated from the displayed scores.

TaskHeteroLoRA-MoEMixLoRA-styleDifference (pp)
GSM8K31.7731.84−0.07
HellaSwag58.7758.99−0.22
PIQA67.9565.23+2.72
MMLU47.6047.34+0.26
MBPP27.4026.80+0.60
HumanEval25.0024.39+0.61

Based on the scores in the table, HeteroLoRA-MoE performs better on four of the six tasks, while scoring 0.07 and 0.22 percentage points lower on GSM8K and HellaSwag, respectively; overall performance is comparable. With twice the SFT data, HeteroLoRA-MoE improves PIQA, MMLU, MBPP, and HumanEval by 2.72, 0.26, 0.60, and 0.61 percentage points, respectively, while remaining close to the baseline on GSM8K and HellaSwag. These results show that our method retains competitive overall task quality and improves most tasks as the training data increases. Together with the 7B experiment, the additional results support scaling the design to larger models and more training data.

Q2: Do the trained HeteroLoRA experts' deltas actually exhibit the activation-aware rank-32 retention and projection-ordering heterogeneity of Tables 1–2, and does the method still help when initialized from a real pretrained MoE (e.g., continued training of Phi-tiny-MoE/OLMoE) instead of dense upcycling?

A: We understand your concern about connecting our structural motivation to the evaluated architecture. The low-rank approximability and projection differences in pretrained MoEs inspired our design. We initially used dense initialization because it directly provides a shared base; pretrained-MoE initialization also requires merging expert weights and initializing residuals. We now add both an analysis of dense-initialized trained experts and pretrained-MoE initialization experiments.

Trained expert analysis.

We use the model's actual shared base as the reference, so each residual equals its trained adapter update. We fit activation-aware rank-16 approximations using 50,000 calibration routed tokens and evaluate retention on 10,000 independent validation routed tokens. Rank 16 is below the adapter rank of 32, avoiding trivial exact recovery and revealing projection differences.

We cover all 24 layers and 16 experts per layer (384 layer–expert pairs), ranking only projections with adapters. Absent adapters have zero residuals and are excluded. The two-projection rankings refer to GU/GD/UD experts; the six complete rankings refer only to GUD experts, following the original Table 2.

RankingG > UU > GG > DD > GU > DD > U
Count445219772373
RankingG > U > DG > D > UU > G > DU > D > GD > G > UD > U > G
Count95672445

Projection rankings show clear preferences and vary among experts within the same pattern. Thus, our trained structure retains expert-specific projection differences, consistent with the motivation for heterogeneous patterns.

Pretrained-MoE initialization.

Using Phi-tiny-MoE, we adapt the Fisher-weighted merging idea of D²-MoE [1]. Fisher information and routing frequencies weight expert merging into a shared base; activation-weighted low-rank decomposition initializes residual adapters. We freeze the base and fine-tune adapters and routers, without separately training the base.

All 32 layers retain 16 experts and Top-2 routing. Fisher-Hetero and Mean-Hetero use rank32 with four experts per GU/GD/UD/GUD pattern; Fisher-Homo uses rank24 on all projections, matching the adapter budget. Mean-Hetero replaces Fisher merging with an arithmetic-mean base. Original MoE+LoRA retains full expert weights and adds rank24 adapters to all three projections. All fine-tuned models use seed42, one epoch, and learning rate 1e−4. The revised appendix provides the merging formula, calibration data, and full training settings.

ConfigurationGSM8KHellaSwagPIQAMMLUMBPPHumanEval
Original Phi-tiny-MoE60.8074.1176.3958.3338.4053.66
Original MoE + LoRA65.8181.9879.2760.1646.4054.27
Fisher-Hetero66.1082.1279.1160.1745.2054.88
Fisher-Homo65.3680.3079.0760.0843.8052.44
Mean-Hetero64.3979.9779.0159.1143.6051.83

Fisher-Hetero outperforms Fisher-Homo and Mean-Hetero on all six tasks, with mean gains of 1.09 and 1.61 percentage points. Compared with original MoE+LoRA, it improves four tasks and achieves comparable six-task quality (64.60 versus 64.65), with approximately 1.28B rather than 3.92B total parameters. These results support heterogeneous allocation and Fisher-weighted initialization, and show that our method also achieves competitive downstream quality when initialized from a pretrained MoE.

[1] Gu et al. Delta Decompression for MoE-based LLMs Compression. ICML, 2025.

Q3: How does the proposed method compare with the closest heterogeneous-adapter methods, MoA, GuiLoMo, and DMEP, in downstream quality and execution efficiency?

A: Thank you for pointing out these related works. We add experimental comparisons with MoA style, GuiLoMo-style, and DMEP-style. To avoid confounding the comparison with the method used to extract a shared base from a pretrained MoE, we initialize the shared base from the dense weights of Qwen2.5-0.5B for all methods and adapt their core mechanisms to the FFN. MoA style retains the soft-controlled combination of different adapter types; GuiLoMo-style retains expert-count and rank search together with structural compression; DMEP-style retains routing-load-based expert exploration, pruning, and subsequent fine-tuning.

The additional experiments use seed 42, a learning rate of 1e−4, and one epoch of main SFT, with the same training data and six-task evaluation protocol. The search stage of GuiLoMo-style and the additional exploration stage of DMEP-style are recorded separately. The table also includes the reused seed-42 results for HeteroLoRA-MoE and MixLoRA-style.

MethodGSM8KHellaSwagPIQAMMLUMBPPHumanEval
HeteroLoRA-MoE32.0956.7569.5647.4529.0423.64
MixLoRA-style31.1951.6068.4546.9826.7922.59
MoA style30.9954.5969.6446.6427.4018.29
GuiLoMo-style31.1354.9169.1847.0429.0022.56
DMEP-style31.5155.6269.2447.4827.4020.73

HeteroLoRA-MoE outperforms MixLoRA-style on all six tasks, MoA style on five tasks, GuiLoMo-style on all six tasks, and DMEP-style on five tasks. MoA style scores 0.08 percentage points higher on PIQA, and DMEP-style scores 0.03 percentage points higher on MMLU. Among the three additional baselines, GuiLoMo-style includes 64 extra search iterations, while DMEP-style includes one additional epoch of exploration training and has fewer parameters after pruning. These results show that our method achieves better performance on most tasks without additional architecture search or exploration and pruning stages, supporting the effectiveness of heterogeneous projection allocation.

We did not find official dedicated high-performance CUDA implementations directly applicable to these adapted structures. To avoid confounding the comparison with differences in implementation optimization, we implement dedicated CUDA paths for MoA style, GuiLoMo-style, and DMEP-style and compare them with HeteroKernel using the same hardware, precision, and input settings. The table follows the original paper's full-model timing protocol, using an L20 GPU, BF16, batch size 1, and SDPA attention. Prefill is measured for inputs of 32, 128, 512, and 2048 tokens; decode measures single-step latency at the corresponding KV-cache lengths. After 5 warm-up iterations, we take 20 CUDA-event measurements and report p50 latency. All methods use CUDA implementations, with HeteroLoRA-MoE using HeteroKernel.

MethodPrefill 32Prefill 128Prefill 512Prefill 2048Decode 32Decode 128Decode 512Decode 2048
HeteroLoRA-MoE17.7817.7617.8129.9822.5223.6323.6823.88
MixLoRA-style19.2418.9419.1240.8023.8723.9223.9923.83
MoA style19.5119.1319.3530.2423.5724.4924.5929.96
GuiLoMo-style26.3429.6734.9546.3932.8136.2136.5938.64
DMEP-style41.3640.9941.9644.2150.2754.6953.4854.34

Latencies are in ms, and decode values are per-step latencies. The numbers in the column headings denote prefill input lengths and decode context lengths, respectively.

In these CUDA comparisons, HeteroKernel is faster than all three additional baselines for both prefill and decode at all four lengths. Relative to MoA style, GuiLoMo-style, and DMEP-style, prefill speedups are 1.01–1.10×, 1.48–1.96×, and 1.47–2.36×, respectively; decode speedups are 1.04–1.25×, 1.46–1.62×, and 2.23–2.31×. Compared with the CUDA implementation of MixLoRA-style, our method is faster at all prefill lengths and has similar decode latency. Together with the downstream quality results, these findings show that our heterogeneous expert design combined with HeteroKernel provides higher execution efficiency while maintaining competitive task performance.

Q4: Why does increasing the number of experts or adapter rank in Table 5 not consistently improve performance? How should the capacity-scaling behavior be interpreted?

A: Thank you for requesting further explanation of the capacity-scaling results. Table 5 (The original Table 5 is Table 8 in the revised manuscript.) examines whether increasing the number of experts or adapter rank consistently improves downstream performance. Increasing rank improves PIQA but does not yield consistent gains on other tasks; increasing the number of experts likewise does not consistently improve performance. This indicates that simply adding capacity does not guarantee better results under the current SFT data scale and training settings. The original paper discusses potential factors, including overfitting, optimization difficulty, and hyperparameter sensitivity. Our design therefore focuses on allocating adapter capacity effectively and improving execution efficiency through shared computation.

To further distinguish the effects of total capacity from those of capacity allocation, we add experiments with a fixed adapter-weight budget on Qwen2.5-0.5B, comparing fewer higher-rank experts, the default configuration, and more lower-rank experts. All three configurations use equal allocation across the four projection patterns, Top-2 routing, seed 42, a learning rate of 1e−4, and one epoch of SFT.

ConfigurationGSM8KHellaSwagPIQAMMLUMBPPHumanEvalSix-task average
E8-r6431.6152.0167.5747.4927.6023.7841.68
E16-r3232.0956.7569.5647.4529.0423.6443.09
E32-r1630.6349.4566.1047.0025.8029.2741.38

With Top-2 routing and the adapter budget fixed, increasing the number of experts expands the available expert pool but requires reducing the rank of each expert. In these experiments, E16-r32 exceeds E8-r64 and E32-r16 in the six-task average by approximately 1.41 and 1.71 percentage points, respectively. Further increasing the expert count to E32-r16 improves HumanEval but reduces scores on the other five tasks. Thus, neither more experts nor higher capacity per expert guarantees continued gains, supporting the balance achieved by the default configuration.

We additionally compare the latency of the three configurations using a common backend. On the same L20 GPU, we use the Triton grouped path, BF16, batch size 1, and identical inputs to measure the complete layer-0 FFN, including the router. Each configuration uses 20 warm-up iterations and 300 timed iterations, with p50 latency reported.

Configuration1 token (ms)128 tokens (ms)512 tokens (ms)2048 tokens (ms)
E8-r640.54010.55620.58871.4531
E16-r320.54780.56320.57961.4275
E32-r160.82640.89090.89801.6200

E16-r32 and E8-r64 have similar latencies, differing by no more than 1.8% across the input shapes. Compared with E32-r16, E16-r32 reduces latency by 11.88%–36.78% while achieving a higher six-task average. This further supports the default configuration's balance between quality and execution efficiency in the evaluated settings.

The original Table 5 and the additional experiments examine increasing capacity and reallocating fixed capacity, respectively. Together with the kernel analysis, these results show that our method reduces execution overhead through shared projection reuse while achieving competitive downstream performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.