acceptodds
Under review as a conference paper at ICLR 2027

An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs

Abstract

Parameter-efficient fine-tuning (PEFT) methods are critical for adapting large models under resource-constrained settings. However, as mixture-of-experts (MoE) architectures, which offer efficient inference at massive parameter scales, gain increasing prevalence in the community, a critical gap arising from the lack of systematic recipes for applying PEFT to MoE models. To narrow this gap, we conduct a large-scale systematic study that investigates a set of essential design recipes, including which modules should be adapted, which granularity should be used to adapt MoE modules, which PEFT method to use, and to what extent should the load-balancing objective be used, across three MoE backbones on downstream tasks and general knowledge tasks. Our experiments yield several key insights. First, adapting MoE experts alone achieves the highest downstream performance with competitive general capability retention under a comparable parameter budget. Second, native expert-weight updates offer a better downstream–retention trade-off than adapters that bypass experts or entire MoE modules. Third, LoRA and DoRA together define the strongest operating region for high downstream accuracy among the evaluated PEFT methods. Fourth, no nonzero load-balancing coefficient Pareto-dominates the zero baseline. We finally evaluate a weight-level redesign of MoE-LoRA on Qwen3. Moving its module-level bypass pool into native expert weights improves all four upstream endpoints by 2.4 to 17.9 points with 17% fewer trainable parameters and a 3.2-point average downstream cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Peft, moe

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cau4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper empirically studies four design choices for PEFT of pretrained MoE models: adaptation position, adaptation granularity, PEFT method, and load-balancing strength. Experiments cover three backbones and two downstream tasks, with MMLU-Pro and IFEval measuring capability retention. The authors derive practical recommendations and use them to redesign MoE²-LoRA.

Critical strengths
  1. The study covers several practically relevant design choices across different MoE backbones, with substantial experimental effort.

  2. Evaluating downstream performance together with general-capability retention provides a useful perspective on adaptation trade-offs.

Critical weaknesses
  1. The experimental comparisons do not sufficiently isolate the factors underlying the recommendations. The position study uses a module-level bypass rather than native expert-weight updates, while subsequent studies enable router adaptation throughout. The granularity comparison also uses unequal parameter budgets, particularly on GPT-OSS (36M, 9M, and 12M for module-level, expert-level, and rank-1 weight-level adaptation). Consequently, the observed differences cannot be cleanly attributed to the individual design choices.

  2. The practical guidance provided by the insights remains insufficiently established. Since the contribution is primarily empirical, its value depends on yielding reliable and consequential recommendations. However, rank-1 weight-level adaptation achieves nearly identical average downstream accuracy to module-level adaptation (68.5 versus 68.4), and the insight-guided MoE²-LoRA redesign improves retention while reducing Math and Intent accuracy by 4.2 and 2.2 points. These are potentially useful trade-offs, but do not establish a clearly preferable adaptation recipe. The absence of accuracy uncertainty estimates further weakens recommendations based on small differences.

Questions for authors
  1. Do the granularity conclusions hold under matched parameter budgets, and do the placement conclusions hold with native expert-weight LoRA and controlled router treatment?

  2. Can the original MoE²-LoRA achieve a comparable downstream–retention trade-off through capacity reduction or regularization?

  3. To what extent is the higher spectral consistency of weight-level adaptation a structural consequence of passing updates through pretrained projections, rather than an explanation of better adaptation? Do randomly initialized, scale-matched adapters exhibit similar consistency, and does this metric predict downstream performance or retention beyond the choice of adaptation granularity?

6 comments
AuthorsOpenReview ↗

Response to Reviewer cau4 (Part 1 / 4)

W1: The experimental comparisons do not sufficiently isolate the factors underlying the recommendations. The position study uses a module-level bypass rather than native expert-weight updates, while subsequent studies enable router adaptation throughout. The granularity comparison also uses unequal parameter budgets, particularly on GPT-OSS (36M, 9M, and 12M for module-level, expert-level, and rank-1 weight-level adaptation). Consequently, the observed differences cannot be cleanly attributed to the individual design choices.
Q1: Do the granularity conclusions hold under matched parameter budgets, and do the placement conclusions hold with native expert-weight LoRA and controlled router treatment?

Response to W1 and Q1:

Thank you for this comment. In the original position study, we use a shared module-level bypass to keep the expert-adaptation form fixed and focus on where adaptation capacity is allocated. In the subsequent studies, router adaptation is enabled for all configurations within each comparison, keeping this setting consistent when examining each design choice.

In the original granularity study, we use the same rank setting for each granularity across backbones, including GPT-OSS. Since parameter counts depend on the expert count and backbone dimensions, these uniform rank settings yield different parameter budgets across granularities, particularly on GPT-OSS.

We provide three additional experiments to test these recommendations. Scores are averaged first across seeds, then equally across the three backbones and two tasks (Intent and Math).

1. Position comparison using native expert-weight updates.

We repeat the position comparison using weight-level LoRA, with and without router adaptation. Experts-only adaptation achieves the highest average score on all three metrics, supporting Insight 1: prioritize expert adaptation and showing that this recommendation also holds with native expert-weight updates.

Adaptation positionDownstream↑MMLU-Pro↑IFEval↑
Attention only61.5031.5821.24
Attention + router55.9040.0626.22
Experts only68.5443.4628.55
Experts + router68.4943.3227.87
Attention + experts67.4235.4722.05
Attention + experts + router67.1833.8423.07

2. Granularity comparison with a frozen router.

We compare expert-adaptation granularities with attention and the native router frozen, using both the original rank settings and closely matched parameter budgets.

Under the original rank settings, rank-1 weight-level adaptation combines strong downstream performance with the highest MMLU-Pro and IFEval scores.

Adaptation granularityDownstream↑MMLU-Pro↑IFEval↑
Module-level bypass, r=25668.2440.6724.35
Expert-level bypass, r=265.1036.7623.06
Weight-level, r=168.5443.4628.55
Weight-level, r=268.7942.3625.39

We further repeat the comparison without router LoRA under closely matched parameter budgets. Weight-level adaptation achieves the highest overall mean on all three metrics.

Adaptation granularityDownstream↑MMLU-Pro↑IFEval↑
Module-level bypass68.3440.4723.38
Expert-level bypass66.0733.5322.81
Weight-level68.8042.6526.29

3. Granularity comparison under closely matched parameter budgets.

To control for parameter budget, we adjust the ranks for each backbone to closely match trainable parameter counts across the three granularities.

With router adaptation enabled, the trainable budgets are approximately 23M, 80M, and 90M for GPT-OSS, Qwen3, and Gemma4, respectively, with a maximum within-backbone parameter-count gap of 4.3%. Router adaptation is enabled consistently across all configurations.

At these closely matched budgets, weight-level adaptation achieves the highest mean downstream score on each backbone and the highest average IFEval score, further supporting native expert-weight adaptation.

The corresponding three-backbone averages are:

Adaptation granularityDownstream ↑MMLU-Pro↑IFEval↑
Module-level bypass67.3741.8124.18
Expert-level bypass65.3435.1923.65
Weight-level69.5142.0126.33

We have updated Figure 4 with the budget-matched comparisons with and without router LoRA. Detailed configurations and results are reported in Appendix J.3, with individual seed results in Appendix K.2.

AuthorsOpenReview ↗

Response to Reviewer cau4 (Part 2 / 4)

W2: The practical guidance provided by the insights remains insufficiently established. Since the contribution is primarily empirical, its value depends on yielding reliable and consequential recommendations. However, rank-1 weight-level adaptation achieves nearly identical average downstream accuracy to module-level adaptation (68.5 versus 68.4), and the insight-guided MoE²-LoRA redesign improves retention while reducing Math and Intent accuracy by 4.2 and 2.2 points. These are potentially useful trade-offs, but do not establish a clearly preferable adaptation recipe. The absence of accuracy uncertainty estimates further weakens recommendations based on small differences.

Response to W2:

Thank you for your review. We add significance tests, clarify retention benefits, and report further insight-guided designs.

1. Reliability.

The original controlled studies used three learning rates × three random seeds (nine runs) per model–task–configuration. For each insight, we retain candidates not significantly outperformed on mean downstream performance, then apply the same rule to retention among survivors. Retention averages MMLU-Pro and IFEval per run; both stages equally weight three backbones and two tasks. Tests combine paired differences from shared seeds within backbone–task settings using a Welch–Satterthwaite t approximation.

  • Position: Expert-only and experts-plus-router survive both stages. Expert-only leads downstream (68.60%; attention-only: 61.50%; attention-plus-router: 55.90%) and significantly outperforms attention-involving alternatives (p<0.05); its Intent gain over attention-plus-router is 25.02 points (p=0.000422).
  • Granularity: At closely matched budgets, weight-level alone survives with router LoRA, outperforming module-level and expert-level bypass by 2.14 (p=0.000836) and 4.17 points (p=3.63×10⁻⁹). Without router LoRA, weight-level and module-level survive; weight-level outperforms expert-level downstream (p=0.000490) and has the highest means on all three metrics.
  • Method: LoRA and DoRA survive both stages, averaging 68.49% and 68.53% downstream versus at most 65.62% for six alternatives (all p<0.05). On Math, LoRA improves retention over DAS-LoRA and FourierFT by 1.54 (p=0.000970) and 1.50 points (p=0.001965).
  • Auxiliary loss: 0 and 10⁻⁴ survive both stages; larger coefficients significantly underperform downstream (p<0.05). On Intent, zero exceeds 10⁻³ and 10⁻¹ by 2.79 (p=0.0054) and 11.48 points (p=0.000148).

We added three-seed means and sample standard deviations to Figures 3–6, per-seed results to the appendix, and statistical assessments to the Section 4.

2. Practical value.

We prioritize downstream learning while considering retention. Math and Intent probe established and less-developed capabilities; MMLU-Pro and IFEval assess knowledge and instruction following.

Originally, rank-1 weight-level adaptation has similar downstream performance to module-level bypass and better retention on both benchmarks:

Granularity (original)Average downstream↑MMLU-Pro change↑IFEval change↑
Module68.4%−0.5 pp−3.7 pp
Weight68.5%+0.8 pp−1.4 pp

At closely matched budgets (revised Figure 4), weight-level significantly improves downstream performance and has higher mean IFEval than module-level with router LoRA; without it, weight-level leads all three means. Both controls support Insight 2.

Router LoRAAdaptation granularityAverage downstream↑MMLU-Pro↑IFEval↑
EnabledModule67.3741.8124.18
Weight69.5142.0126.33
DisabledModule68.3440.4723.38
Weight68.8042.6526.29

3. Original redesign (MoE²-LoRA-WT).

The Table 1 weight-level redesign improves MMLU-Pro and IFEval on both tasks with fewer trainable parameters but lower downstream accuracy. It is useful when retention or parameter budget takes priority and some accuracy loss is acceptable.

4. Further design (MoE²-LoRA-SF).

The original redesign adapts fused gate/up and down projections, with gate/up sharing a low-rank input factor. MoE²-LoRA-SF gives gate, up, and down separate input and output factors, allowing distinct input subspaces while preserving pool selection conditioned on native MoE routing context.

On Qwen3-30B-A3B-Base, MoE²-LoRA-SF outperforms original MoE²-LoRA on all three metrics on both tasks and on average, with slightly fewer parameters:

TaskMethodParams (M)Downstream↑MMLU-Pro↑IFEval↑
IntentOriginal MoE²-LoRA47.7184.0045.2618.11
MoE²-LoRA-WT39.5881.8051.2036.04
MoE²-LoRA-SF47.2484.4047.4728.10
MathOriginal MoE²-LoRA47.7160.0048.1935.12
MoE²-LoRA-WT(original Table 1)39.5855.8050.6340.30
MoE²-LoRA-SF47.2460.2052.5337.34

These granularity-guided improvements are now reported in Section 5.

AuthorsOpenReview ↗

Response to Reviewer cau4 (Part 3 / 4)

Q2: Can the original MoE²-LoRA achieve a comparable downstream–retention trade-off through capacity reduction or regularization?

Response to Q2:

Thank you for this question. We now complement the regularization experiments in Appendix D (Tables 7 and 8) with direct capacity-reduction experiments on original MoE²-LoRA. These controls produce different downstream–retention trade-offs, while our weight-level redesigns achieve higher average scores on both retention benchmarks.

1. Direct capacity reduction of original MoE²-LoRA.

We separately halve the adapter rank from 32 to 16 and the total pool entries from 256 to 128, retaining the original module-level design and task-specific learning rates. These changes reduce trainable parameters from 47.71M to 30.93M and 30.15M, respectively.

The original Table 1 redesign achieves higher average downstream performance and both retention scores than either capacity-reduced configuration. Halving rank can improve a particular retention metric: on Math, IFEval increases from 35.12% to 39.93%, while downstream accuracy decreases from 60.00% to 56.00% and MMLU-Pro from 48.19% to 46.12%. On Intent, both capacity reductions lower downstream accuracy and both retention scores relative to the original.

2. Trade-offs achievable through regularization.

We evaluate auxiliary load-balancing coefficients λ∈{10−4,10−3,10−2,10−1}\lambda\in\{10^{-4},10^{-3},10^{-2},10^{-1}\}, keeping the original architecture and 47.71M trainable parameters fixed.

Regularization can improve retention at a downstream cost. For example, λ=10−4\lambda=10^{-4} increases average MMLU-Pro from 46.73% to 47.37% and IFEval from 26.62% to 29.12%, while average downstream accuracy decreases from 72.00% to 70.70%.

The task-specific results illustrate this balance. On Intent, λ=10−4\lambda=10^{-4} improves downstream accuracy and MMLU-Pro, while IFEval changes from 18.11% to 17.56%. On Math, all tested nonzero coefficients improve IFEval while reducing downstream accuracy and MMLU-Pro.

3. Comparison with our insight-guided redesigns.

The redesign in the original Table 1 achieves 68.80% average downstream accuracy, compared with 68.40% for halving the original adapter rank, alongside substantially higher retention scores: 50.92% versus 40.09% on MMLU-Pro, and 38.17% versus 27.27% on IFEval.

The further variants described in our response to W2 refine adaptation into separate gate, up, and down low-rank factors. Without tuning the auxiliary-loss coefficient, they achieve averages of 72.30% downstream, 50.00% MMLU-Pro, and 32.72% IFEval—higher than every tested capacity-reduction and fixed-coefficient regularization setting on all three metrics.

The following table reports equal averages over Intent and Math. Each capacity-reduction or regularization row uses the same rank, pool-size, or coefficient setting on both tasks. IFEval uses strict prompt-level accuracy.

MethodParams (M)Average downstream ↑Average MMLU-Pro ↑Average IFEval ↑
Original MoE²-LoRA47.7172.0046.7326.62
Original, rank halved: n256,r16n256,r1630.9368.4040.0927.27
Original, pool size halved: n128,r32n128,r3230.1567.6042.2323.11
Original + λ=10−4\lambda=10^{-4}47.7170.7047.3729.12
Original + λ=10−3\lambda=10^{-3}47.7169.0045.9424.86
Original + λ=10−2\lambda=10^{-2}47.7168.2033.6026.62
Original + λ=10−1\lambda=10^{-1}47.7160.6029.7029.95
MoE²-LoRA-WT (original Table 1)39.5868.8050.9238.17
MoE²-LoRA-SF47.2472.3050.0032.72

We have added these capacity-reduction results and the comparison with our redesigns to Appendix D.3.

Reviewer_n4UsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs proposes an analysis of multiple approaches to fine-tuning of MoE LLMs, including PEFT technique and placement. This work uses 3 models across 4 tasks, with a final section dedicated to the rank analysis of some of the insights. While the question is interesting and the work well done, the central weakness is the lack of power to strongly justify the claimed insights.

Critical strengths
  1. The problem is relevant, and this work is a good step towards improving the clarity of the field
  2. The paper is well written and clear
  3. The PEFT placement choices are thorough, with a good overview of the possibilities
  4. The spectral analysis is interesting (though not quite novel)
Critical weaknesses
  1. The effect sizes for most insights are tiny; this would normally not be an issue if the effect were consistent, but given the point of this work is to pinpoint each insight individually, it is.
  2. Anything that is not LoRA is underperforming, perhaps again because of hyperaparameter optimization, or maybe some other phenomenon that it is necessary to investigate
  3. There are other works that investigate the placement of LoRAs in MoE, for example MixLoRA, which is cited but not investigated
Questions for authors
  1. Multi seed runs must be done and the effect sizes considered more carefully, possibly through statistical methods
  2. While choice of r is presented as-is and ablated once (r=1 vs r=2 in expert weights), this is a critical hyperparameter and a much more thorough ablation is necessary
  3. The same problem applies to the choice of models and tasks, they are too few and too similar (e.g., the models are all 20 to 30B, what about other scales?)
  4. Averaging the 6 cells seems to mix confounders, as the behavior of intent-based tasks and math-based tasks is not the same
13 comments
AuthorsOpenReview ↗

Response to Reviewer n4Us (Part 1 / 6)

W1: The effect sizes for most insights are tiny; this would normally not be an issue if the effect were consistent, but given the point of this work is to pinpoint each insight individually, it is.
Q1: Multi seed runs must be done and the effect sizes considered more carefully, possibly through statistical methods

Response to W1 and Q1:

Thank you for highlighting the importance of effect sizes and consistency. We provide an additional systematic analysis of all four insights, quantifying effect sizes and assessing the recommendations under a common statistical procedure.

1. Repeated runs and a unified statistical assessment.

In the original experiments, we evaluated each model–task–configuration at three learning rates with three random seeds each, giving nine training runs. Our insights and recommendations are drawn from this extensive set of repeated experiments across models, tasks, and configurations.

For this rebuttal, we add a systematic statistical assessment, applying the same downstream-first, retention-second procedure to all four insights:

  • Downstream stage: compare all candidates on average downstream performance and retain those not significantly outperformed by another candidate.
  • Retention stage: among the remaining candidates, apply the same rule to average general-capability retention, calculated by equally averaging MMLU-Pro and IFEval within each run.

Both averages equally weight the three backbones and two tasks. Tests combine paired differences from actual shared seeds within each backbone–task setting using a Welch–Satterthwaite t approximation with a nominal two-sided threshold of p<0.05p<0.05. Descriptive means use all three seeds.

2. Effect sizes and statistical evidence for each insight.

  • Adaptation position. Expert-only and experts-plus-router adaptation remain after both stages; the four configurations involving attention adaptation are excluded by downstream comparisons (p<0.05p<0.05). Expert-only achieves the highest downstream mean, 68.60%, exceeding attention-only (61.50%) and attention-plus-router (55.90%) by 7.10 and 12.70 points, respectively. These results support prioritizing expert adaptation, consistent with Insight 1.
  • Adaptation granularity. In the revised budget-matched comparison with router LoRA, weight-level adaptation achieves 69.51% average downstream accuracy, compared with 67.37% for module-level bypass and 65.34% for expert-level bypass. The gains of 2.14 and 4.17 points are significant (p=0.000836p=0.000836 and p=3.63×10−9p=3.63\times10^{-9}, respectively), and weight-level adaptation also achieves the highest average IFEval score (25.83%). Without router LoRA, it achieves the highest overall mean on all three metrics, improving downstream accuracy, MMLU-Pro, and IFEval over module-level bypass by 0.46, 1.34, and 2.24 points, respectively. These results support Insight 2 through significant downstream gains with router adaptation and a favorable mean downstream–retention profile with the router frozen.
  • PEFT formulation. LoRA and DoRA remain after both stages, with downstream means of 68.49% and 68.53%. For comparison, the highest downstream mean among the six alternatives is 65.62%, achieved by MoE-LoRA, giving LoRA and DoRA mean advantages of 2.87 and 2.91 points, respectively. All six alternatives are excluded by downstream comparisons (p<0.05p<0.05). These results support LoRA and DoRA as practical starting points, consistent with Insight 3.
  • Auxiliary loss. Coefficients 0 and 10−410^{-4} remain after both stages, with downstream means of 68.49% and 68.14% and retention means of 35.60% and 35.45%. The three larger coefficients are excluded by downstream comparisons (p<0.05p<0.05). At 10−110^{-1}, mean downstream performance is 58.58%, compared with 68.49% at zero—a 9.91-point difference. These results support zero or a small auxiliary-loss coefficient, consistent with Insight 4.

3. Presenting effect sizes and uncertainty explicitly.

We have updated Figures 3–6 to report mean ± sample standard deviation across three seeds for each model–task–configuration at its selected learning rate, and provide individual seed scores, seed identifiers, and task-specific results in the appendix. We have incorporated this additional statistical analysis into the main-text discussions of the four insights in Section 4.

AuthorsOpenReview ↗

Response to Reviewer n4Us (Part 2 / 6)

W2: Anything that is not LoRA is underperforming, perhaps again because of hyperaparameter optimization, or maybe some other phenomenon that it is necessary to investigate.

Response to W2:

Thank you for this important comment. We clarify the tuning protocol and provide supporting evidence below.

1. Consistent tuning and comparable parameter budgets.

For each model–task–method configuration, we evaluate learning rates in {1e-5, 1e-4, 1e-3}, running three independent seeds at every candidate rate, and select the rate with the highest mean downstream score. All methods receive the same LR search budget. And trainable parameter counts are approximately matched across methods, except for IA3 and last-layer partial fine-tuning, whose counts are determined by the scaling-vector dimensions and the selected layer’s full weights, respectively.

2. Consistency with prior empirical studies.

Method-specific LR tuning follows prior PEFT evaluation practice [1, 2]. Both studies find that properly tuned vanilla LoRA remains competitive with its variants, and further identify LR as particularly influential relative to other hyperparameters. These findings support both our tuning protocol and the strength of LoRA as a baseline.

3. Additional hyperparameter sensitivity analysis.

Our additional analysis on Qwen3-30B-A3B-Base with Intent shows that LR produces the largest score variation for all five methods across the tested ranges. LoRA and DoRA also retain their leading Intent performance across the tested batch-size and weight-decay settings, supporting prioritizing LR tuning and reinforcing the observed advantage of LoRA/DoRA.

We sweep effective batch sizes {72, 144, 216, 288} at LR=1e-3 and WD=0, and weight decays {0, 0.01, 0.05, 0.1} at LR=1e-3 and BS=144. The LR sweep fixes BS=144 and WD=0. The table reports the maximum minus minimum downstream score within each sweep, in percentage points.

MethodLR sweep (pp)BS sweep (pp)WD sweep (pp)
LoRA58.86.81.4
DoRA57.06.43.0
IA35.83.00.8
OFT35.05.44.0
FourierFT6.44.00.8

We have included this analysis in Appendix I.1.

[1] He et al. A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation. 2026.

[2] Lee et al. Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning. 2026.

AuthorsOpenReview ↗

Response to Reviewer n4Us (Part 3 / 6)

W3: There are other works that investigate the placement of LoRAs in MoE, for example MixLoRA, which is cited but not investigated

Response to W3:

Thank you for highlighting this related work. We clarify the distinction in setting and provide additional MixLoRA comparisons below.

1. Different starting points: constructing MoE adapters versus adapting pretrained MoEs.

MixLoRA introduces multiple LoRA-based experts and an additional top-k router into a frozen dense backbone, sharing the pretrained FFN weights among its experts [3]. Our study examines pretrained native MoE backbones, which already contain specialized experts and a native router. We investigate how PEFT should interact with this existing expert structure to achieve effective downstream adaptation while preserving general capabilities.

2. The need for our systematic study of PEFT on pretrained MoEs.

Existing tools such as Hugging Face PEFT and Tinker support parameter-efficient adaptation of pretrained MoEs, with different implementation and configuration choices [4, 5]. This diversity motivates systematic evaluation of several connected design questions: where to allocate adaptation capacity, at what granularity to adapt experts, which PEFT formulation to use, and how to configure load balancing. Our study provides empirical insights into these choices, jointly considering downstream performance and general-capability retention to guide practical MoE adaptation.

3. Additional MixLoRA comparisons on Qwen3-30B-A3B-Base.

To extend the method comparison, we adapt MixLoRA to Qwen3-30B-A3B-Base at two granularities: layer-level MixLoRA, which adds parallel low-rank adaptation branches at the MoE layer output, and weight-level MixLoRA, which applies LoRA to the FFN’s gate, up, and down projections and aggregates the resulting nonlinear expert outputs.

Native weight-level LoRA achieves higher downstream accuracy and IFEval scores than both MixLoRA configurations on both tasks, using fewer trainable parameters (40.16M versus 51.95–53.53M). This reinforces our method-selection recommendation in Insight 3. Within MixLoRA, weight-level adaptation achieves higher downstream accuracy and MMLU-Pro scores than layer-level adaptation on both tasks, with a trade-off in IFEval, further supporting our granularity recommendation in Insight 2.

For each method and task, the learning rate is selected based on downstream performance, and general capabilities are evaluated on the corresponding checkpoint. Scores are percentages, and IFEval reports strict prompt-level accuracy.

TaskMethodParams (M)Downstream ↑MMLU-Pro ↑IFEval ↑
IntentNative weight-level LoRA40.1681.8749.4532.90
IntentLayer-level MixLoRA, r3251.9561.4024.4716.64
IntentWeight-level MixLoRA, r1653.5378.2032.459.06
MathNative weight-level LoRA40.1659.0051.4236.35
MathLayer-level MixLoRA, r3251.9556.8048.4734.01
MathWeight-level MixLoRA, r1653.5357.0051.7033.64

We have incorporated MixLoRA into the main-text method comparison and provide detailed configurations in Appendix I.2.

[3] Li et al. MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts.

[4] Hugging Face PEFT. MoE LoRA parameterization and per-expert low-rank factors.

[5] Tinker. MoE shared-outer LoRA implementation.

Reviewer_vBEYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies parameter-efficient fine-tuning of MoE LLMs, focusing on the trade-offs among downstream performance, trainable-parameter cost, and retention of general capabilities. Across three MoE backbones and two downstream tasks—mathematical reasoning and intent classification—the authors systematically compare adaptation positions, adaptation granularities, fine-tuning methods, and auxiliary load-balancing coefficients, using MMLU-Pro and IFEval to assess general-capability retention. Within the evaluated settings, the results favor low-rank adaptation of native expert weights, identify LoRA and DoRA as effective choices, and show that stronger load-balancing regularization generally reduces downstream performance. Functional-response and spectral analyses further investigate how weight-level adaptation reuses pretrained expert structure. Guided by these findings, the authors redesign MoE²-LoRA by moving its module-level bypass adaptation into native expert weights. On Qwen3, this redesign reduces trainable parameters by approximately 17% and improves general-capability retention, at the cost of a 3.2-percentage-point decrease in average downstream performance.

Critical strengths
  1. Joint evaluation of adaptation and capability retention provides practical value.The paper examines four fine-tuning design choices across three MoE backbones and two downstream tasks, evaluating MMLU-Pro and IFEval on the same downstream-selected checkpoints. This evaluation exposes task gains alongside their forgetting costs, providing useful evidence for selecting MoE fine-tuning configurations.
  2. The granularity findings are supported by analysis and a concrete application.Functional-response and spectral analyses provide a plausible explanation of how weight-level adaptation reuses pretrained expert structure. The resulting MoE²-LoRA redesign reduces trainable parameters by approximately 17% and improves all four general-capability endpoints on Qwen3, with a 3.2-point decrease in average downstream performance. This application demonstrates the findings’ value for adjusting the trade-off between parameter cost and capability retention.
Critical weaknesses

Uncertainty in the key performance results is insufficiently reported. Although Section 3.2 and Appendix B.1 specify three-seed experiments, Figures 3–6 provide no standard deviations, confidence intervals, or per-seed performance results. This makes the stability of some fine-grained recommendations difficult to assess, particularly the preference for DoRA based on an approximately 0.3-percentage-point average IFEval advantage. This limits my confidence in these specific recommendations.

Questions for authors

Section 3.2 and Appendix B.1 state that learning rates are selected using the mean downstream score across three seeds. Are the data used for learning-rate selection independent of the MATH-500 and 500-example Intent evaluation sets used for the reported results? If a separate validation set is used, please clarify its construction and size. This clarification would help me assess whether model-selection bias could affect the reported downstream results. Beyond the evaluation protocol, I would also appreciate clarification of the MoE²-LoRA redesign. Appendix A.1 introduces the mapping πℓ(j)=(e,t)\pi_\ell(j)=(e,t), but its construction is unclear. How are pool entries assigned to native experts and their gate/up or down projections, and how does pool selection interact with native expert routing? Please also clarify which pool-routing and rank settings are shared between the original and redesigned variants in Table 1. These details would help assess the reproducibility of the redesign and the attribution of its reported gains.

2 comments
AuthorsOpenReview ↗

Response to Reviewer vBEY (Part 1 / 2)

W1: Uncertainty in the key performance results is insufficiently reported. Although Section 3.2 and Appendix B.1 specify three-seed experiments, Figures 3–6 provide no standard deviations, confidence intervals, or per-seed performance results. This makes the stability of some fine-grained recommendations difficult to assess, particularly the preference for DoRA based on an approximately 0.3-percentage-point average IFEval advantage. This limits my confidence in these specific recommendations.

Response to W1:

Thank you for highlighting the importance of uncertainty reporting. We have expanded the presentation of our three-seed results and clarify the settings supporting our method recommendations.

1. Three-seed evaluation at every candidate learning rate.

For each model–task–configuration, we evaluate learning rates in {1e-5, 1e-4, 1e-3}, with three random seeds per learning rate, giving nine training runs. We select the learning rate using the mean downstream score across seeds and evaluate general capabilities on the corresponding checkpoints.

2. Explicit uncertainty reporting in the figures and appendix.

We have updated Figures 3–6 to report the mean and sample standard deviation across three seeds for each model–task–configuration at its selected learning rate. Individual seed scores and seed identifiers have been provided in appendix K, making both effect sizes and variation across runs directly accessible.

3. DoRA’s IFEval gains across tasks and backbones.

DoRA achieves higher average IFEval performance than LoRA, with larger gains on Intent, consistent with the average instruction-following retention advantage reported in Insight 3. The following table reports IFEval scores (%) as mean ± standard deviation over three seeds. Differences are computed as DoRA minus LoRA.

TaskBackboneLoRADoRADifference (pp)
IntentQwen332.90 ± 1.7632.53 ± 0.81−0.37
IntentGemma48.75 ± 2.229.67 ± 2.81+0.92
IntentGPT-OSS32.72 ± 3.2136.72 ± 1.57+4.00
MathQwen336.35 ± 0.1135.49 ± 0.67−0.86
MathGemma422.55 ± 1.3322.80 ± 1.30+0.25
MathGPT-OSS33.95 ± 0.8333.40 ± 1.85−0.55

On Intent, DoRA improves IFEval by 1.52 points on average across the three backbones. The largest gain occurs on GPT-OSS (+4.00 points), where DoRA also exhibits a smaller observed standard deviation (1.57 versus 3.21).

4. Statistical assessment of the recommendations.

We additionally assess all four insights using the same downstream-first, retention-second procedure. Within each insight, we first retain configurations not significantly outperformed on average downstream performance, then apply the same rule to average general-capability retention, defined as the mean of MMLU-Pro and IFEval. Both averages equally weight the three backbones and two tasks. Tests combine paired differences from actual shared seeds within each backbone–task setting using a Welch–Satterthwaite t approximation with a nominal two-sided threshold of p<0.05p<0.05.

  • Adaptation position: Expert-only and experts-plus-router adaptation remain after both stages; the four attention-involving configurations are excluded by downstream comparisons (p<0.05p<0.05). Expert-only achieves the highest downstream mean, 68.60%, compared with 61.50% for attention-only and 55.90% for attention-plus-router, supporting prioritizing expert adaptation.
  • Adaptation granularity: Under closely matched parameter budgets with router LoRA, weight-level adaptation significantly improves average downstream accuracy over module-level and expert-level bypass by 2.14 points (p=0.000836p=0.000836) and 4.17 points (p=3.63×10−9p=3.63\times10^{-9}), respectively, making it the sole configuration retained by the assessment. Without router LoRA, weight-level and module-level adaptation remain after both stages, with weight-level adaptation significantly exceeding expert-level bypass downstream (p=0.000490p=0.000490). These results provide statistical support for native expert-weight adaptation under both parameter-budget and router controls.
  • PEFT formulation: LoRA and DoRA remain after both stages, with downstream means of 68.49% and 68.53%. For comparison, the highest downstream mean among the six alternatives is 65.62%; all six are excluded by downstream comparisons (p<0.05p<0.05). This supports LoRA and DoRA as practical starting points.
  • Auxiliary loss: Coefficients **0 and 10−410^{-4} remain after both stages, with downstream means of 68.49% and 68.14% and retention means of 35.60% and 35.45%. The three larger coefficients are excluded by downstream comparisons (p<0.05p<0.05), supporting zero or a small auxiliary-loss coefficient.

We have incorporated this additional statistical analysis in Section 4.

AuthorsOpenReview ↗

Response to Reviewer vBEY (Part 2 / 2)

Q1: Section 3.2 and Appendix B.1 state that learning rates are selected using the mean downstream score across three seeds. Are the data used for learning-rate selection independent of the MATH-500 and 500-example Intent evaluation sets used for the reported results? If a separate validation set is used, please clarify its construction and size. This clarification would help me assess whether model-selection bias could affect the reported downstream results. Beyond the evaluation protocol, I would also appreciate clarification of the MoE²-LoRA redesign. Appendix A.1 introduces the mapping (\pi_\ell(j)=(e,t)), but its construction is unclear. How are pool entries assigned to native experts and their gate/up or down projections, and how does pool selection interact with native expert routing? Please also clarify which pool-routing and rank settings are shared between the original and redesigned variants in Table 1. These details would help assess the reproducibility of the redesign and the attribution of its reported gains.

Response to Q1:

Thank you for your thoughtful comments. We clarify the evaluation protocol and the MoE²-LoRA redesign below.

1. Learning-rate selection and evaluation.

We use benchmark-based learning-rate selection, an evaluation practice also used in prior PEFT studies[1], without constructing an additional validation split for each task. For each model–task–configuration, we select from {10−5,10−4,10−3}\{10^{-5},10^{-4},10^{-3}\} using the mean downstream score across three seeds on the same 500-example MATH-500 or 500-example held-out Intent set used for downstream reporting. We do not use a separate task-specific validation split.All compared methods follow this protocol.

MMLU-Pro (12,032 examples) and IFEval (541 prompts) are evaluated on the corresponding checkpoints at the selected learning rate. Neither benchmark is used for selection or reselection. We have specified the selection data and procedure explicitly in Section 3.2 and Appendix B.1.

2. Pool selection and weight-level adaptation.

The πℓ\pi_\ell notation was introduced to describe how pool-based low-rank components are applied within native expert projections. In MoE²-LoRA-WT, each selected pool entry pairs a fused gate/up component with a down component. MoE²-LoRA-SF instead uses separate gate, up, and down pools with independent factors. In both variants, the selected updates are shared across the native experts dispatched for the token. Original MoE²-LoRA instead adds the adapter outputs after the MoE block. We have clarified these computations in Appendix A.1 and added Algorithm 1 in Section 5.

3. Shared settings in Table 1.

Original MoE²-LoRA and MoE²-LoRA-WT share two layer groups, pool top-k = 8, routing bottleneck db=128d_b=128, and adapter rank r=32r=32. The original uses 128 pool entries per group ; WT uses 64 paired entries per group. Attention and native router weights remain frozen, and the auxiliary-loss coefficient is zero. These configurations are summarized in Appendix A.1, Table 2.

With these shared routing and rank settings, the redesign improves MMLU-Pro and IFEval on both tasks with fewer trainable parameters, supporting weight-level adaptation for general-capability retention. We have added the explicit computation and configuration summary to Appendix A.1 and D.1.

[1] Biderman et al. LoRA Learns Less and Forgets Less, 2024.


If you have any further questions or concerns, please feel free to raise them. We would be happy to discuss and exchange ideas further. Once again, we sincerely thank you for your efforts in helping us improve the quality of our manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chairs, Senior Area Chairs, and Program Chairs,

We thank the reviewers and chairs for their constructive feedback. Below, we summarize our core contributions and the key outcomes of the rebuttal.

1. Core Contributions

Our paper systematically studies four essential design choices for PEFT of pretrained MoE LLMs: adaptation position, adaptation granularity, PEFT formulation, and load-balancing strength. By jointly evaluating downstream learning, parameter cost, and general-capability retention, we establish practical defaults within the evaluated settings: prioritize native expert-weight adaptation, start with LoRA/DoRA, and use zero or very small load-balancing coefficients. We further translate these findings into MoE²-LoRA-SF, which uses separate gate, up, and down factors within native expert weights.

2. Summary of Strengths

Reviewers recognized the following strengths:

  • Practical relevance and breadth: “The problem is relevant” and “The PEFT placement choices are thorough” (n4Us); “several practically relevant design choices across different MoE backbones, with substantial experimental effort” (cau4).
  • Joint assessment of learning and retention: “provides practical value” (vBEY) and “a useful perspective on adaptation trade-offs” (cau4).
  • Analysis and actionable application: “The granularity findings are supported by analysis and a concrete application” (vBEY).

3. Key Concerns Resolved

  • Generality, including the follow-up on SF. We extended the granularity study to DeepSeek-V2-Lite, GPT-OSS-120B, code generation, and legal judgment generation. Crucially, in response to cau4's follow-up about SF's single-backbone evaluation, we added GPT-OSS-20B Math/Intent and Law on both Qwen3 and GPT-OSS. SF now improves downstream performance and both retention metrics over original MoE²-LoRA in all six evaluated backbone–task combinations, with equal or slightly fewer parameters. On GPT-OSS Math, gains are +1.20/+9.15/+13.31 points on downstream/MMLU-Pro/IFEval; its Law score rises from 9.5 to 22.0. (n4Us Q3; cau4 follow-up.)
  • Isolating granularity and placement. We repeated placement comparisons using native expert-weight LoRA and added granularity comparisons at closely matched parameter budgets, both with and without router adaptation. With router LoRA, weight-level adaptation improves average downstream accuracy over module/expert bypass by 2.14/4.17 points (p<0.001 for both). With the router frozen, it has the highest mean on all three metrics. Revised Figure 4 uses these controlled comparisons. (cau4 W1/Q1; n4Us follow-up Concern 2.)
  • Statistical reliability and task-specific guidance. Figures 3–6 now report three-seed means and standard deviations, with individual seed results and paired statistical assessments in the appendix. Following n4Us's questions, we removed universal rank recommendations from the main text, extended every Math rank setting to three seeds, and clarified that 62.00 was a single-seed result whereas 59.00 was the three-seed mean. Separate task analyses also show why Math matters: Intent alone retains zero auxiliary loss, while the combined assessment retains both zero and 10⁻⁴. (vBEY W1; n4Us follow-up Concerns 2–4; cau4 W2.)
  • Tuning fairness and evaluation transparency. In response to the learning-rate follow-up, we clarified that every method receives the same search range and selects its own rate. Additional batch-size and weight-decay sweeps retain LoRA/DoRA as the top two methods on Qwen3–Intent; recommendations are explicitly scoped to the tested ranges. We also disclosed that downstream reporting sets are used for learning-rate selection, while MMLU-Pro and IFEval are never used for selection, and added MixLoRA comparisons. (n4Us W2/W3 and follow-up Concern 1; vBEY Q1.)
  • Practical gains beyond capacity reduction. We tested halving the original MoE²-LoRA rank or pool size and adding load-balancing regularization. On Qwen3, SF achieves higher average downstream, MMLU-Pro, and IFEval scores than every tested capacity-reduction or fixed-coefficient regularization setting. Relative to original MoE²-LoRA, these averages improve from 72.00/46.73/26.62 to 72.30/50.00/32.72, with slightly fewer parameters. (cau4 W2/Q2.)
  • Mechanistic interpretation and reproducibility. Ten scale-matched random controls per configuration distinguish structural spectral bias from learned alignment. We accordingly present spectral consistency as a diagnostic, without claiming independent predictive or causal validity. We also clarified pool routing, factorization, and shared settings, and added explicit algorithmic details. (cau4 Q3; vBEY Q1.)

These additions strengthen the evidence and clarify the scope of our recommendations. We hope this summary assists your assessment of the revised work. Thank you again for your time and consideration.

Best regards,

Paper7319 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.