- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HeteroLoRA-MoE, a parameter-efficient MoE architecture that shares pretrained SwiGLU weights across experts while introducing heterogeneous LoRA projection patterns. The design is motivated by an empirical analysis of low-rank approximations to expert weight differences in a pretrained MoE model. The authors further develop HeteroKernel, a specialized CUDA implementation that reduces redundant computation through shared projection reuse, early routing masks, and kernel fusion. Experiments on Qwen2.5-0.5B and 1.5B demonstrate improvements over a parameter-matched MixLoRA-style baseline on several downstream tasks. The inference performance is evaluated on an NVIDIA L20 GPU.
Critical strengths
S1. The shared-down projection optimization is technically well motivated. Aggregating expert intermediates before applying the shared dense projection avoids redundant matrix multiplications. The kernel ablations provide useful evidence for the effectiveness of the proposed execution optimizations.
S2. The experimental evaluation includes parameter-matched comparisons against homogeneous and fixed-projection LoRA-MoE variants. Reporting results across four random seeds and two model scales provides a more reliable assessment of the downstream performance than a single-run comparison.
Critical weaknesses
W1. The motivation for the proposed heterogeneous expert design is not sufficiently supported. The low-rank analysis in Section 2 is conducted only on Phi-tiny-MoE-instruct, whose FFN intermediate dimension is 448. A rank-128 approximation already represents approximately 28.6% of the maximum possible matrix rank. Moreover, the reported residual-output retention measures approximation quality under a specific activation distribution rather than directly establishing that expert weight differences are intrinsically low-rank. It remains unclear whether similar properties hold for larger pretrained MoE models.
The connection between this analysis and the proposed architecture is also unclear. Table 2 shows differences in low-rank retention across projections, but does not explain why the authors select four specific projection patterns and allocate experts equally among them. Although Table 3 includes fixed-pattern ablations, these experiments do not establish how the observed low-rank properties should guide adapter placement or capacity allocation.
W2. The experimental evaluation does not sufficiently characterize the trade-offs between model capacity and inference efficiency. Although Table 5 investigates different expert counts and LoRA ranks, the configurations are not matched in parameter budget. A controlled comparison between more experts with smaller ranks and fewer experts with larger ranks is needed to assess whether the proposed allocation is preferable.
The system evaluation is also limited to a single NVIDIA L20 GPU, batch size 1, and Top-2 routing. The impact of different Top-K settings and heterogeneous expert workloads on inference latency remains unexplored. In particular, different projection patterns incur different computational costs, but the paper provides no analysis of per-expert routing distributions or computational load imbalance. Together with the relatively modest decode improvements in the matched-backend comparison, these limitations make it difficult to assess the scalability and generality of HeteroKernel.
Questions for authors
Q1. Could the authors provide additional evidence for the low-rank structure of expert weight differences? In particular, does the observed residual-output retention persist in pretrained MoE models with larger hidden and intermediate dimensions?
Q2. Although Table 3 includes comparisons with fixed projection patterns, the rationale behind the four selected patterns and their uniform allocation remains unclear. How do the observations in Table 2 justify this particular design? Would a layer-dependent or data-driven allocation lead to better results?
Q3. Table 5 investigates different expert counts and LoRA ranks, but the configurations do not have matched parameter budgets. Could the authors further examine this trade-off under a fixed parameter budget? In particular, how would more experts with smaller ranks compare against fewer experts with larger ranks in terms of model quality and inference latency?
Q4. How does the proposed method handle load imbalance across experts with different projection patterns? Since these patterns incur different computational costs, balanced token assignments do not necessarily imply balanced computational workloads. Could the authors provide per-expert routing statistics and analyze the potential impact on GPU utilization and inference latency?
Q5. How does HeteroKernel perform with different numbers of active experts per token (Top-K)? Given that shared-down projection reuse is a central optimization, evaluating different Top-K settings would help clarify how its computational benefits and inference latency scale with the number of activated experts.
Q6. The inference evaluation is limited to a single NVIDIA L20 GPU with batch size 1. How well would the proposed kernel optimizations generalize to other GPU architectures, such as Hopper or Blackwell, and larger inference batches? I understand that access to multiple GPU platforms may be limited. If additional experiments are not feasible, an analytical discussion of the expected performance characteristics and potential bottlenecks would also be valuable.
Given the limited rebuttal period, I do not expect full multi-seed evaluations for the additional training-based ablations. Single-seed results would be acceptable as preliminary evidence, provided that the experimental settings are consistent and the limitations are acknowledged. Latency experiments should still include repeated measurements to account for timing variability.
6 comments
Q2: Although Table 3 includes comparisons with fixed projection patterns, the rationale behind the four selected patterns and their uniform allocation remains unclear. How do the observations in Table 2 justify this particular design? Would a layer-dependent or data-driven allocation lead to better results?
A: Thank you for your suggestions on projection combinations and allocation. We provide additional evidence from projection preference analysis of larger pretrained MoEs and allocation-ratio ablations.
First, Table 2 shows that projection retention rankings differ across experts, motivating different adapter projection combinations. We extend the same analysis to DeepSeek-MoE-16B and Mixtral to examine whether the projection preference distribution in Phi-tiny directly generalizes to other models. We use the mean expert weights as the base and count the six projection retention rankings under activation-aware rank-32 approximations on independent inputs.
| Model | G > U > D | G > D > U | U > G > D | U > D > G | D > G > U | D > U > G | Total |
|---|---|---|---|---|---|---|---|
| Phi-tiny-MoE (original Table 2) | 381 | 16 | 65 | 7 | 33 | 10 | 512 |
| DeepSeek-MoE-16B | 116 | 179 | 127 | 195 | 854 | 257 | 1,728 |
| Mixtral | 45 | 31 | 86 | 47 | 19 | 28 | 256 |
The results show that projection preference distributions differ across models. Gate has the highest retention in 77.54% of layer–expert pairs in Phi-tiny, Down in 64.29% in DeepSeek, and Up in 51.95% in Mixtral. Thus, preference proportions from a single pretrained model should not directly serve as a general allocation rule. We use GU, GD, UD, and GUD to cover all three two-projection combinations and the full three-projection combination, with equal allocation as a default independent of any particular pretrained model's distribution.
Second, in addition to the original fixed-pattern comparisons, we add allocation-ratio ablations with the same budget on Qwen2.5-0.5B to test whether favoring a particular projection improves downstream performance. The experiments use 16 experts, rank 32, Top-2 routing, seed 42, and identical SFT settings. The ratios are ordered as GU/GD/UD/GUD.
| Allocation | Ratio | GSM8K | HellaSwag | PIQA | MMLU | MBPP | HumanEval | Six-task average |
|---|---|---|---|---|---|---|---|---|
| Equal | 4/4/4/4 | 32.09 | 56.75 | 69.56 | 47.45 | 29.04 | 23.64 | 43.09 |
| Gate-biased | 5/5/2/4 | 32.07 | 51.50 | 67.08 | 47.03 | 28.60 | 22.56 | 41.47 |
| Up-biased | 5/2/5/4 | 31.77 | 50.62 | 68.23 | 47.09 | 28.00 | 25.61 | 41.89 |
| Down-biased | 2/5/5/4 | 31.54 | 50.06 | 66.54 | 47.29 | 28.00 | 25.61 | 41.51 |
None of the three unequal allocations exceeds the six-task average of equal allocation. Equal allocation outperforms the Gate-biased allocation on all six tasks, with a mean advantage of approximately 1.62 percentage points. The Up- and Down-biased allocations improve HumanEval but score lower on the other five tasks. Together with the original fixed-pattern ablations, these results support heterogeneous combinations and equal allocation.
In addition, fixed, equally sized pattern groups facilitate planning data layouts, tiling, and execution paths, and improve inference efficiency through HeteroKernel's shared computation reuse. Layer-wise or data-driven allocation would require further adaptation to varying group sizes and workloads. We therefore adopt this default design to balance downstream performance and efficient execution.
Q3: Table 5 investigates different expert counts and LoRA ranks, but the configurations do not have matched parameter budgets. Could the authors further examine this trade-off under a fixed parameter budget? In particular, how would more experts with smaller ranks compare against fewer experts with larger ranks in terms of model quality and inference latency?
A: Thank you for suggesting a comparison of expert count and rank under a fixed budget. Whereas the original Table 5 examines increasing capacity, we add a comparison with a fixed adapter-weight budget on Qwen2.5-0.5B using E8-r64, E16-r32, and E32-r16. All three configurations allocate experts equally across the four patterns and use Top-2 routing, seed 42, a learning rate of 1e−4, and one epoch of SFT on the same data.
The original Table 5 is Table 8 in the revised manuscript.
| Configuration | GSM8K | HellaSwag | PIQA | MMLU | MBPP | HumanEval | Six-task average |
|---|---|---|---|---|---|---|---|
| E8-r64 | 31.61 | 52.01 | 67.57 | 47.49 | 27.60 | 23.78 | 41.68 |
| E16-r32 | 32.09 | 56.75 | 69.56 | 47.45 | 29.04 | 23.64 | 43.09 |
| E32-r16 | 30.63 | 49.45 | 66.10 | 47.00 | 25.80 | 29.27 | 41.38 |
The default E16-r32 configuration exceeds E8-r64 and E32-r16 in the six-task average by approximately 1.41 and 1.71 percentage points, respectively. More lower-rank experts improve HumanEval but score below the default on the other five tasks; fewer higher-rank experts also do not achieve a higher overall score. This controlled single-seed experiment supports the default configuration's balance between expert count and capacity per expert.
We further measure the latency of all three configurations on the same L20 GPU. To keep the backend consistent, all configurations use the Triton grouped path supporting their respective ranks, with BF16, batch size 1, and identical inputs. We measure the complete layer-0 FFN, including the router. Each setting uses 20 warm-up iterations and 300 timed iterations, with p50 latency reported. The 1-token setting corresponds to a single-step FFN input shape.
| Configuration | 1 token (ms) | 128 tokens (ms) | 512 tokens (ms) | 2048 tokens (ms) |
|---|---|---|---|---|
| E8-r64 | 0.5401 | 0.5562 | 0.5887 | 1.4531 |
| E16-r32 | 0.5478 | 0.5632 | 0.5796 | 1.4275 |
| E32-r16 | 0.8264 | 0.8909 | 0.8980 | 1.6200 |
The default E16-r32 has latency close to E8-r64 and reduces latency by 11.88%–36.78% relative to E32-r16 while achieving a higher six-task average. Lowering rank reduces adapter computation but leaves shared dense computation unchanged; increasing the expert count can also fragment token groups and reduce matrix execution efficiency, so more lower-rank experts are not necessarily faster under a fixed parameter budget.
Thank you for recognizing the technical motivation of our shared-down optimization, the supporting kernel ablations, and the parameter-matched comparisons across multiple seeds and model scales. We also appreciate your constructive suggestions for strengthening the structural motivation and further characterizing the capacity–efficiency trade-offs. We have incorporated the additional experiments and their analyses into the main text and appendices of the revised manuscript.
Q1: Could the authors provide additional evidence for the low-rank structure of expert weight differences? In particular, does the observed residual-output retention persist in pretrained MoE models with larger hidden and intermediate dimensions?
A: Thank you for your suggestions regarding the interpretation of the low-rank analysis and model scale. We agree that residual-output retention measures approximation quality under a given input distribution. We therefore revise the wording to state that "expert weight differences admit low-rank approximations under the actual input distribution," which more accurately expresses our structural motivation.
The additional analysis follows the statistical procedure in the paper's motivation section. For each projection in each layer, we use the arithmetic mean of the expert weights as the shared base and compute each expert's weight residual relative to this mean. We fit activation-aware low-rank approximations on calibration inputs and measure residual-output retention on independent test inputs, reporting the median across layer–expert pairs separately for Gate, Up, and Down. The table uses ranks 8, 32, and 128, consistent with the original paper, and reports retention as a percentage.
For DeepSeek-MoE-16B, rank 32 retains 56.34%, 45.97%, and 58.37% of the residual-output energy for Gate, Up, and Down, respectively; the corresponding values for Mixtral are 60.26%, 59.32%, and 44.82%. Increasing the rank to 128 raises retention across projections to 69.67%–76.26% and 63.54%–86.22% for the two models, respectively. These results show that expert differences also exhibit low-rank approximability in larger pretrained MoEs, providing additional support for a shared base with low-rank expert updates.