acceptodds
Withdrawn from ICLR 2027

Exploring Expert Concentration for Parameter-efficient Fine-tuning of Mixture-of-Expert LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling large language models (LLMs) with the Mixture-of-Experts (MoE) architecture has emerged as a powerful alternative to dense models. However, fine-tuning MoE models for domain- or task-specific adaptation remains challenging: full-model tuning is prohibitively expensive, while existing parameter-efficient fine-tuning (PEFT) methods, mostly adapted from dense models, suffer from unstable optimization due to MoE’s sparse expert activation. In this work, we conduct an empirical study on the fine-tuning dynamics of MoE models. We first introduce the Domain Advantage Score (DAS), a simple yet effective metric for identifying domain-relevant experts. Our findings uncover an expert concentration phenomenon: during domain-specific fine-tuning, the overall DAS of the top experts consistently increases, indicating a progressive enhancement of domain concentration. Building on this, we propose a lightweight two-stage PEFT framework: (1) fine-tuning only the attention and router layers to sharpen expert specialization, and (2) selectively fine-tuning parameters on the identified experts. This approach updates only a small fraction of parameters while achieving performance on par with full fine-tuning, and it effectively preserves the model's general capabilities. Experiments on nine benchmarks show the effectiveness and efficiency of our method. Our code and data will be publicly released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MoE, PEFT

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mjc9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a two-stage adaptation strategy for pretrained MoE models. It first tunes attention and router parameters with experts frozen, then selectively tunes experts identified by the Domain Advantage Score (DAS), which contrasts routing scores on domain-specific and general data. Experiments cover three MoE backbones, mathematics and coding tasks, and capability-retention evaluations.

Critical strengths
  1. Contrasting domain and general routing statistics provides an intuitive way to distinguish domain preference from overall expert popularity, addressing a relevant question in selective expert adaptation.

  2. The evaluation includes three backbones, alternative expert-selection criteria, component ablations, and retention experiments. DAS-Tune consistently improves over ESFT on the individual downstream results reported in Table 1.

Critical weaknesses
  1. Reporting inconsistencies and missing details limit confidence in the results. For example, the four DeepSeek-V2-Lite DAS-Tune scores in Table 1 average to 36.52, rather than the reported 42.82. The aggregation rule requires clarification or correction. Despite reporting three-seed results, accuracy variability is absent. The general reference dataset, precise routing-score definition, and global versus per-layer expert-selection procedure are also insufficiently specified.

  2. The experiments do not adequately isolate the benefit of routing-induced expert selection. The component ablations compare different training pathways, but do not clearly establish whether post-stage-1 selection is better than alternative selections starting from the same first-stage checkpoint. Moreover, increasing C-DAS@k demonstrates greater relative concentration of positive DAS, rather than necessarily stronger absolute domain affinity or expert specialization. The proposed mechanism is therefore less firmly supported than the downstream improvements.

Questions for authors
  1. How are the reported averages computed? Please explain or correct the discrepancies and provide variability across training seeds.

  2. Starting from the same first-stage checkpoint and matching second-stage budgets, how does post-stage-1 DAS selection compare with pre-stage-1 DAS selection, random selection, and absolute routing-score selection? Does Table 5 follow this protocol?

  3. What data and routing-score convention are used to compute DAS, and how is the expert budget distributed across layers? Can separate numerator and denominator trajectories for C-DAS@k confirm that its increase reflects stronger head-expert domain advantage?

Reviewer_zmNEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a two-stage parameter-efficient fine-tuning framework for Mixture-of-Experts (MoE) LLMs, built on the observation of an “expert concentration” phenomenon quantified by a Domain Advantage Score (DAS). It addresses the high cost of full-model MoE tuning and the instability of existing PEFT methods under sparse expert activation by first tuning only attention and router layers, then selectively fine-tuning DAS-identified experts. The method is validated on multiple math and coding benchmarks across three MoE backbones, together with general-ability and continual-learning evaluations.

Critical strengths

1. Clear and practical idea. The paper identifies an expert concentration phenomenon during MoE fine-tuning and turns it into a simple, usable expert-selection metric (DAS). The two-stage design—router/attention tuning followed by selective expert tuning—is intuitive and easy to follow.

2. Reasonably thorough experiments. The method is evaluated on three MoE backbones, multiple domain benchmarks, general-ability benchmarks, continual-learning settings, and several ablation studies (component tuning, expert-identification variants, expert count, layer distribution). This gives a reasonably complete picture of the method’s behavior.

Critical weaknesses

1. Limited insight and novelty. Although the empirical study is well executed, the main conclusion is quite intuitive and already widely recognized in the community. The methodological contribution is also limited—it is closer to a training pipeline than a new algorithmic idea.

2. Unclear performance–parameter trade-off and source of DAS-LoRA degradation. From the comparison with DAS-LoRA and the ablation results, Attention Tuning Only already brings substantial gains. It would be interesting to know whether using LoRA only in stage 2 (without LoRA in stage 1) could yield a better performance–parameter trade-off. In addition, is the performance drop of DAS-LoRA relative to DAS-Tune caused by applying LoRA to the router during training?

3. Lack of strong baselines. ESFT is a 2024 baseline. The authors are encouraged to compare against more recent MoE-PEFT baselines, especially 2025–2026 methods that are already cited in the related work.

Questions for authors
  1. Could the authors report results for the setting where LoRA is used only in stage 2 (no LoRA in stage 1)? Would this improve the performance–parameter trade-off?

  2. In DAS-LoRA, is the performance gap to DAS-Tune mainly due to applying LoRA to the router? An ablation isolating router-LoRA vs. expert-LoRA would help.

  3. Since the paper already cites 2025–2026 MoE-PEFT work, could the authors add at least one recent baseline (e.g., Li et al., 2025 or MoE-Sieve) to the main comparison, or clearly explain why such methods are not comparable or not publicly available?

Reviewer_QKQuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DAS-Tune is an empirical study of MoE fine-tuning. The Domain Advantage Score is the gap between an expert's routing frequency on a target domain and on a general corpus, and the paper observes that the cumulative DAS of the top experts rises during domain tuning The method tunes attention and router layers with all experts frozen, recomputes DAS, then tunes only the top experts' FFNs.

Critical strengths
  1. The paper is well-written. The methods are well presented with comprehensive experiments. The figures and tables are clear enough to show their method’s advantages.
  2. The retention evidence is stronger than the accuracy. Table 2 shows it achieves 60.27 on the general task against a 60.36 backbone, while full fine-tuning drops to 57.96, ESFT to 60.08 and LoRA to 59.26. Losing 0.09 general points while gaining 7.37 reasoning points, and beating FFT's retention by 2.31, is a convincing practical case for selective expert updates.
  3. The finding replicates across three model designs. DeepSeek-V2-Lite, DeepSeek-MoE-Base and Qwen1.5-MoE differ in expert count, shared-expert scheme and gating, and DAS-Tune leads ESFT and LoRA on all three. The performance improvement is consistent even where the margin is small.
Critical weaknesses
  1. The paper's attribution does not match its own ablation. In Table 4, DAS-selected expert only improves 1.46 over attention-only. In addition, DAS ranking only improves 0.60 over the simplest heuristic, while plain attention-plus-router tuning contributes 5.91 of the 7.37 gain in Table 5. Therefore, most of the improvements are not actually comes from the DAS-tune.
  2. ‘On par with full fine-tuning’ seems to be overclaimed. The gaps to FFT are 0.49 on DeepSeek-V2-Lite, 1.02 on Qwen and 3.02 on DeepSeek-MoE-Base. I do not think the gaps that more than 1 point can be considered ‘on par’ in PEFT.
  3. The motivation is only described but not quantified. Figure 1 shows C-DAS rising during domain tuning, but the paper does not report how much concentration each backbone achieves or relate it to the size of the gain. For example, on DeepSeek-MoE-Base, where the shortfall to FFT is largest, the analysis is not connected to the result. If concentration is the mechanism, it should predict where the method works, as presented it motivates the selection rule without constraining it.
Questions for authors
  1. Please consider adding the missing ablation study: two-stage training where stage 2 selects the same number of experts according to ESFT, or at random, at the same budget.
  2. On DeepSeek-MoE-Base the gap to FFT is 3.02 points, the largest of the three backbones, and every method scores lowest there. Does the concentration phenomenon appear at all on that backbone?
  3. The baseline only FFT, LoRA and ESFT. ESFT is the only expert-selection baseline, and there is no attention-plus-router LoRA baseline. Since the paper is about the ordering of the stages and the placement of parameters, why not compare against LoRA applied to attention and router, and against a method that also uses a routing-derived module?
  4. How many experts are selected per layer, and is that number fixed across backbones? “8% of parameters” corresponds to very different expert counts in DeepSeek-V2-Lite and Qwen1.5-MoE. If the count is chosen per backbone, and how?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.