- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a two-stage adaptation strategy for pretrained MoE models. It first tunes attention and router parameters with experts frozen, then selectively tunes experts identified by the Domain Advantage Score (DAS), which contrasts routing scores on domain-specific and general data. Experiments cover three MoE backbones, mathematics and coding tasks, and capability-retention evaluations.
Critical strengths
-
Contrasting domain and general routing statistics provides an intuitive way to distinguish domain preference from overall expert popularity, addressing a relevant question in selective expert adaptation.
-
The evaluation includes three backbones, alternative expert-selection criteria, component ablations, and retention experiments. DAS-Tune consistently improves over ESFT on the individual downstream results reported in Table 1.
Critical weaknesses
-
Reporting inconsistencies and missing details limit confidence in the results. For example, the four DeepSeek-V2-Lite DAS-Tune scores in Table 1 average to 36.52, rather than the reported 42.82. The aggregation rule requires clarification or correction. Despite reporting three-seed results, accuracy variability is absent. The general reference dataset, precise routing-score definition, and global versus per-layer expert-selection procedure are also insufficiently specified.
-
The experiments do not adequately isolate the benefit of routing-induced expert selection. The component ablations compare different training pathways, but do not clearly establish whether post-stage-1 selection is better than alternative selections starting from the same first-stage checkpoint. Moreover, increasing C-DAS@k demonstrates greater relative concentration of positive DAS, rather than necessarily stronger absolute domain affinity or expert specialization. The proposed mechanism is therefore less firmly supported than the downstream improvements.
Questions for authors
-
How are the reported averages computed? Please explain or correct the discrepancies and provide variability across training seeds.
-
Starting from the same first-stage checkpoint and matching second-stage budgets, how does post-stage-1 DAS selection compare with pre-stage-1 DAS selection, random selection, and absolute routing-score selection? Does Table 5 follow this protocol?
-
What data and routing-score convention are used to compute DAS, and how is the expert budget distributed across layers? Can separate numerator and denominator trajectories for C-DAS@k confirm that its increase reflects stronger head-expert domain advantage?