- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper empirically studies four design choices for PEFT of pretrained MoE models: adaptation position, adaptation granularity, PEFT method, and load-balancing strength. Experiments cover three backbones and two downstream tasks, with MMLU-Pro and IFEval measuring capability retention. The authors derive practical recommendations and use them to redesign MoE²-LoRA.
Critical strengths
-
The study covers several practically relevant design choices across different MoE backbones, with substantial experimental effort.
-
Evaluating downstream performance together with general-capability retention provides a useful perspective on adaptation trade-offs.
Critical weaknesses
-
The experimental comparisons do not sufficiently isolate the factors underlying the recommendations. The position study uses a module-level bypass rather than native expert-weight updates, while subsequent studies enable router adaptation throughout. The granularity comparison also uses unequal parameter budgets, particularly on GPT-OSS (36M, 9M, and 12M for module-level, expert-level, and rank-1 weight-level adaptation). Consequently, the observed differences cannot be cleanly attributed to the individual design choices.
-
The practical guidance provided by the insights remains insufficiently established. Since the contribution is primarily empirical, its value depends on yielding reliable and consequential recommendations. However, rank-1 weight-level adaptation achieves nearly identical average downstream accuracy to module-level adaptation (68.5 versus 68.4), and the insight-guided MoE²-LoRA redesign improves retention while reducing Math and Intent accuracy by 4.2 and 2.2 points. These are potentially useful trade-offs, but do not establish a clearly preferable adaptation recipe. The absence of accuracy uncertainty estimates further weakens recommendations based on small differences.
Questions for authors
-
Do the granularity conclusions hold under matched parameter budgets, and do the placement conclusions hold with native expert-weight LoRA and controlled router treatment?
-
Can the original MoE²-LoRA achieve a comparable downstream–retention trade-off through capacity reduction or regularization?
-
To what extent is the higher spectral consistency of weight-level adaptation a structural consequence of passing updates through pretrained projections, rather than an explanation of better adaptation? Do randomly initialized, scale-matched adapters exhibit similar consistency, and does this metric predict downstream performance or retention beyond the choice of adaptation granularity?
6 comments
Response to Reviewer cau4 (Part 2 / 4)
W2: The practical guidance provided by the insights remains insufficiently established. Since the contribution is primarily empirical, its value depends on yielding reliable and consequential recommendations. However, rank-1 weight-level adaptation achieves nearly identical average downstream accuracy to module-level adaptation (68.5 versus 68.4), and the insight-guided MoE²-LoRA redesign improves retention while reducing Math and Intent accuracy by 4.2 and 2.2 points. These are potentially useful trade-offs, but do not establish a clearly preferable adaptation recipe. The absence of accuracy uncertainty estimates further weakens recommendations based on small differences.
Response to W2:
Thank you for your review. We add significance tests, clarify retention benefits, and report further insight-guided designs.
1. Reliability.
The original controlled studies used three learning rates × three random seeds (nine runs) per model–task–configuration. For each insight, we retain candidates not significantly outperformed on mean downstream performance, then apply the same rule to retention among survivors. Retention averages MMLU-Pro and IFEval per run; both stages equally weight three backbones and two tasks. Tests combine paired differences from shared seeds within backbone–task settings using a Welch–Satterthwaite t approximation.
- Position: Expert-only and experts-plus-router survive both stages. Expert-only leads downstream (68.60%; attention-only: 61.50%; attention-plus-router: 55.90%) and significantly outperforms attention-involving alternatives (p<0.05); its Intent gain over attention-plus-router is 25.02 points (p=0.000422).
- Granularity: At closely matched budgets, weight-level alone survives with router LoRA, outperforming module-level and expert-level bypass by 2.14 (p=0.000836) and 4.17 points (p=3.63×10⁻⁹). Without router LoRA, weight-level and module-level survive; weight-level outperforms expert-level downstream (p=0.000490) and has the highest means on all three metrics.
- Method: LoRA and DoRA survive both stages, averaging 68.49% and 68.53% downstream versus at most 65.62% for six alternatives (all p<0.05). On Math, LoRA improves retention over DAS-LoRA and FourierFT by 1.54 (p=0.000970) and 1.50 points (p=0.001965).
- Auxiliary loss: 0 and 10⁻⁴ survive both stages; larger coefficients significantly underperform downstream (p<0.05). On Intent, zero exceeds 10⁻³ and 10⁻¹ by 2.79 (p=0.0054) and 11.48 points (p=0.000148).
We added three-seed means and sample standard deviations to Figures 3–6, per-seed results to the appendix, and statistical assessments to the Section 4.
2. Practical value.
We prioritize downstream learning while considering retention. Math and Intent probe established and less-developed capabilities; MMLU-Pro and IFEval assess knowledge and instruction following.
Originally, rank-1 weight-level adaptation has similar downstream performance to module-level bypass and better retention on both benchmarks:
| Granularity (original) | Average downstream↑ | MMLU-Pro change↑ | IFEval change↑ |
|---|---|---|---|
| Module | 68.4% | −0.5 pp | −3.7 pp |
| Weight | 68.5% | +0.8 pp | −1.4 pp |
At closely matched budgets (revised Figure 4), weight-level significantly improves downstream performance and has higher mean IFEval than module-level with router LoRA; without it, weight-level leads all three means. Both controls support Insight 2.
| Router LoRA | Adaptation granularity | Average downstream↑ | MMLU-Pro↑ | IFEval↑ |
|---|---|---|---|---|
| Enabled | Module | 67.37 | 41.81 | 24.18 |
| Weight | 69.51 | 42.01 | 26.33 | |
| Disabled | Module | 68.34 | 40.47 | 23.38 |
| Weight | 68.80 | 42.65 | 26.29 |
3. Original redesign (MoE²-LoRA-WT).
The Table 1 weight-level redesign improves MMLU-Pro and IFEval on both tasks with fewer trainable parameters but lower downstream accuracy. It is useful when retention or parameter budget takes priority and some accuracy loss is acceptable.
4. Further design (MoE²-LoRA-SF).
The original redesign adapts fused gate/up and down projections, with gate/up sharing a low-rank input factor. MoE²-LoRA-SF gives gate, up, and down separate input and output factors, allowing distinct input subspaces while preserving pool selection conditioned on native MoE routing context.
On Qwen3-30B-A3B-Base, MoE²-LoRA-SF outperforms original MoE²-LoRA on all three metrics on both tasks and on average, with slightly fewer parameters:
| Task | Method | Params (M) | Downstream↑ | MMLU-Pro↑ | IFEval↑ |
|---|---|---|---|---|---|
| Intent | Original MoE²-LoRA | 47.71 | 84.00 | 45.26 | 18.11 |
| MoE²-LoRA-WT | 39.58 | 81.80 | 51.20 | 36.04 | |
| MoE²-LoRA-SF | 47.24 | 84.40 | 47.47 | 28.10 | |
| Math | Original MoE²-LoRA | 47.71 | 60.00 | 48.19 | 35.12 |
| MoE²-LoRA-WT(original Table 1) | 39.58 | 55.80 | 50.63 | 40.30 | |
| MoE²-LoRA-SF | 47.24 | 60.20 | 52.53 | 37.34 |
These granularity-guided improvements are now reported in Section 5.
Response to Reviewer cau4 (Part 3 / 4)
Q2: Can the original MoE²-LoRA achieve a comparable downstream–retention trade-off through capacity reduction or regularization?
Response to Q2:
Thank you for this question. We now complement the regularization experiments in Appendix D (Tables 7 and 8) with direct capacity-reduction experiments on original MoE²-LoRA. These controls produce different downstream–retention trade-offs, while our weight-level redesigns achieve higher average scores on both retention benchmarks.
1. Direct capacity reduction of original MoE²-LoRA.
We separately halve the adapter rank from 32 to 16 and the total pool entries from 256 to 128, retaining the original module-level design and task-specific learning rates. These changes reduce trainable parameters from 47.71M to 30.93M and 30.15M, respectively.
The original Table 1 redesign achieves higher average downstream performance and both retention scores than either capacity-reduced configuration. Halving rank can improve a particular retention metric: on Math, IFEval increases from 35.12% to 39.93%, while downstream accuracy decreases from 60.00% to 56.00% and MMLU-Pro from 48.19% to 46.12%. On Intent, both capacity reductions lower downstream accuracy and both retention scores relative to the original.
2. Trade-offs achievable through regularization.
We evaluate auxiliary load-balancing coefficients , keeping the original architecture and 47.71M trainable parameters fixed.
Regularization can improve retention at a downstream cost. For example, increases average MMLU-Pro from 46.73% to 47.37% and IFEval from 26.62% to 29.12%, while average downstream accuracy decreases from 72.00% to 70.70%.
The task-specific results illustrate this balance. On Intent, improves downstream accuracy and MMLU-Pro, while IFEval changes from 18.11% to 17.56%. On Math, all tested nonzero coefficients improve IFEval while reducing downstream accuracy and MMLU-Pro.
3. Comparison with our insight-guided redesigns.
The redesign in the original Table 1 achieves 68.80% average downstream accuracy, compared with 68.40% for halving the original adapter rank, alongside substantially higher retention scores: 50.92% versus 40.09% on MMLU-Pro, and 38.17% versus 27.27% on IFEval.
The further variants described in our response to W2 refine adaptation into separate gate, up, and down low-rank factors. Without tuning the auxiliary-loss coefficient, they achieve averages of 72.30% downstream, 50.00% MMLU-Pro, and 32.72% IFEval—higher than every tested capacity-reduction and fixed-coefficient regularization setting on all three metrics.
The following table reports equal averages over Intent and Math. Each capacity-reduction or regularization row uses the same rank, pool-size, or coefficient setting on both tasks. IFEval uses strict prompt-level accuracy.
| Method | Params (M) | Average downstream ↑ | Average MMLU-Pro ↑ | Average IFEval ↑ |
|---|---|---|---|---|
| Original MoE²-LoRA | 47.71 | 72.00 | 46.73 | 26.62 |
| Original, rank halved: | 30.93 | 68.40 | 40.09 | 27.27 |
| Original, pool size halved: | 30.15 | 67.60 | 42.23 | 23.11 |
| Original + | 47.71 | 70.70 | 47.37 | 29.12 |
| Original + | 47.71 | 69.00 | 45.94 | 24.86 |
| Original + | 47.71 | 68.20 | 33.60 | 26.62 |
| Original + | 47.71 | 60.60 | 29.70 | 29.95 |
| MoE²-LoRA-WT (original Table 1) | 39.58 | 68.80 | 50.92 | 38.17 |
| MoE²-LoRA-SF | 47.24 | 72.30 | 50.00 | 32.72 |
We have added these capacity-reduction results and the comparison with our redesigns to Appendix D.3.
Response to Reviewer cau4 (Part 1 / 4)
Response to W1 and Q1:
Thank you for this comment. In the original position study, we use a shared module-level bypass to keep the expert-adaptation form fixed and focus on where adaptation capacity is allocated. In the subsequent studies, router adaptation is enabled for all configurations within each comparison, keeping this setting consistent when examining each design choice.
In the original granularity study, we use the same rank setting for each granularity across backbones, including GPT-OSS. Since parameter counts depend on the expert count and backbone dimensions, these uniform rank settings yield different parameter budgets across granularities, particularly on GPT-OSS.
We provide three additional experiments to test these recommendations. Scores are averaged first across seeds, then equally across the three backbones and two tasks (Intent and Math).
1. Position comparison using native expert-weight updates.
We repeat the position comparison using weight-level LoRA, with and without router adaptation. Experts-only adaptation achieves the highest average score on all three metrics, supporting Insight 1: prioritize expert adaptation and showing that this recommendation also holds with native expert-weight updates.
2. Granularity comparison with a frozen router.
We compare expert-adaptation granularities with attention and the native router frozen, using both the original rank settings and closely matched parameter budgets.
Under the original rank settings, rank-1 weight-level adaptation combines strong downstream performance with the highest MMLU-Pro and IFEval scores.
We further repeat the comparison without router LoRA under closely matched parameter budgets. Weight-level adaptation achieves the highest overall mean on all three metrics.
3. Granularity comparison under closely matched parameter budgets.
To control for parameter budget, we adjust the ranks for each backbone to closely match trainable parameter counts across the three granularities.
With router adaptation enabled, the trainable budgets are approximately 23M, 80M, and 90M for GPT-OSS, Qwen3, and Gemma4, respectively, with a maximum within-backbone parameter-count gap of 4.3%. Router adaptation is enabled consistently across all configurations.
At these closely matched budgets, weight-level adaptation achieves the highest mean downstream score on each backbone and the highest average IFEval score, further supporting native expert-weight adaptation.
The corresponding three-backbone averages are:
We have updated Figure 4 with the budget-matched comparisons with and without router LoRA. Detailed configurations and results are reported in Appendix J.3, with individual seed results in Appendix K.2.