- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MaskCoFT, a masked co-adaptive fine-tuning method for memory-efficient Mixture-of-Experts (MoE) inference. MaskCoFT jointly fine-tunes routers and experts while learning binary routing masks that concentrate token activations on subsets of experts. Once the masks converge, they are fixed, and the routers and experts continue to be fine-tuned under the restricted routing. During inference, the learned mask distribution serves as a soft prior to favor frequently selected experts while keeping all experts accessible. Experiments on Mixtral-8×7B and DeepSeek-V2-Lite demonstrate reductions in expert fetches of up to 23.7% and 10.1%, respectively, and decoding latency improvements of up to 16.4% and 5.5%, while maintaining average accuracy above the pretrained baselines.
Critical strengths
The paper demonstrates practical improvements in MoE inference efficiency, reducing expert fetches and decoding latency in a real offloading system.
Critical weaknesses
1. Lack of validation of the main research questions. The paper identifies two potential limitations of ReMoE regarding temporal locality and frozen experts, and explains how MaskCoFT addresses them. However, the empirical or theoretical evidence supporting these claims is limited. In particular, Table 3 only reports aggregate accuracy and cache hit rates, making it difficult to determine whether the proposed method actually overcomes the two limitations that motivate the work.
2. Limited novelty and insufficient comparison with existing approaches. The proposed method builds on existing techniques, including learnable masking and joint fine-tuning, but its additional technical contribution is not clearly established. The paper lacks ablations demonstrating the individual contributions of these components and the two-stage training procedure. Although the authors acknowledge the limitations of their comparison with ReMoE, the results do not establish whether MaskCoFT offers additional benefits over the closest prior work.
Questions for authors
Q1. What concrete empirical or theoretical evidence supports the two limitations of ReMoE identified in Section 1?
Q2. What are the individual contributions of expert co-adaptation and the two-stage training procedure? How do the results compare with router-only fine-tuning under the same mask?
Q3. What evidence demonstrates the benefit of learned masking over simpler alternatives, such as fixed expert subsets or directly learned routing biases?
Q4. While Table 3 provides a comparison with ReMoE, it does not establish a clear advantage of MaskCoFT. What concrete experimental or theoretical results can demonstrate a meaningful improvement over ReMoE, given that addressing its limitations is a central motivation of this paper?
10 comments
For limitation regarding "frozen experts": we finetuned a DeepSeek-V2-Lite model using a modified ReMoE to jointly finetune routers and experts with LoRA. Then we compare the model capacity and simulation results with the original router-only ReMoE.
\begin{array}{lcc} \hline & \text{Generative} & \text{Multiple choice} \\\\ \begin{array}{l} \strut \text{Method} \\\\ \hline \strut \boldsymbol{\textit{DeepSeek-V2-Lite}} \\\\ \strut \text{ReMoE (router-only)} \\\\ \strut \text{ReMoE + expert LoRA} \\\\ \strut \text{MaskCoFT} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{GSM8K} & \text{HumanEval} \\\\ \hline \strut {} & {} \\\\ \strut 33.89 & 24.39 \\\\ \strut 36.24 & 26.83 \\\\ \strut \mathbf{43.82} & \mathbf{28.05} \\\\ \hline \end{array} & \begin{array}{ccccccc} \hline \strut \text{MMLU} & \text{ARC-E} & \text{ARC-C} & \text{HellaSwag} & \text{WinoGrande} & \text{PIQA} & \text{BoolQ} \\\\ \hline \strut {} & {} & {} & {} & {} & {} & {} \\\\ \strut 51.95 & 73.91 & 44.62 & 76.37 & 68.90 & 79.71 & 75.05 \\\\ \strut \mathbf{54.64} & 74.96 & 48.72 & \mathbf{79.06} & \mathbf{71.82} & \mathbf{80.58} & 79.57 \\\\ \strut 53.36 & \mathbf{75.72} & \mathbf{49.57} & 76.06 & \mathbf{71.82} & 80.36 & \mathbf{82.45} \\\\ \hline \end{array} \\\\ \end{array}$$ $$\small \begin{array}{lccc} \hline & \text{LRU} & \text{LFU} & \text{FIFO} \\\\ \begin{array}{l} \strut \text{Method} \\\\ \hline \strut \boldsymbol{\textit{DeepSeek-V2-Lite}}\quad B=12 \\\\ \strut \text{ReMoE (router-only)} \\\\ \strut \text{ReMoE + expert LoRA} \\\\ \strut \text{MaskCoFT} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{HR (\\%)}\\;\uparrow & \text{Fetches/tok}\\;\downarrow \\\\ \hline \strut {} & {} \\\\ \strut 48.61 & 80.16 \\\\ \strut \mathbf{49.17} & \mathbf{79.29} \\\\ \strut 47.85 & 81.36 \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{HR (\\%)}\\;\uparrow & \text{Fetches/tok}\\;\downarrow \\\\ \hline \strut {} & {} \\\\ \strut 48.12 & 80.94 \\\\ \strut 47.93 & 81.23 \\\\ \strut \mathbf{49.18} & \mathbf{79.28} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{HR (\\%)}\\;\uparrow & \text{Fetches/tok}\\;\downarrow \\\\ \hline \strut {} & {} \\\\ \strut 47.07 & 82.57 \\\\ \strut \mathbf{47.46} & \mathbf{81.96} \\\\ \strut 45.75 & 84.63 \\\\ \hline \end{array} \\\\ \end{array}$$ With the joint fine-tuning of routers and experts, ReMoE can preserve more model capacity and achieve a comparable hit rate compared with router-only fine-tuning.Q2. Individual contributions of expert co-adaptation and the two-stage training procedure.
To show the individual contribution of expert co-adaptation, we evaluate the model capacity of MaskCoFT with frozen experts and compare with original MaskCoFT with jointly fine-tuning of experts and routers on DeepSeek-V2-Lite. With expert co-adaptation, MaskCoFT has obviouly better preservation on model capacity.
\begin{array}{lcc} \hline & \text{Generative} & \text{Multiple choice} \\\\ \begin{array}{l} \strut \text{Method} \\\\ \hline \strut \boldsymbol{\textit{DeepSeek-V2-Lite}} \\\\ \strut \text{MaskCoFT(frozen experts)} \\\\ \strut \text{MaskCoFT(expert co-adaptation)} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{GSM8K} & \text{HumanEval} \\\\ \hline \strut {} & {} \\\\ \strut 27.75 & 20.12 \\\\ \strut \mathbf{43.82} & \mathbf{28.05} \\\\ \hline \end{array} & \begin{array}{ccccccc} \hline \strut \text{MMLU} & \text{ARC-E} & \text{ARC-C} & \text{HellaSwag} & \text{WinoGrande} & \text{PIQA} & \text{BoolQ} \\\\ \hline \strut {} & {} & {} & {} & {} & {} & {} \\\\ \strut 48.42 & 68.98 & 41.47 & 70.73 & 68.19 & 79.11 & 77.49 \\\\ \strut \mathbf{53.36} & \mathbf{75.72} & \mathbf{49.57} & \mathbf{76.06} & \mathbf{71.82} & \mathbf{80.36} & \mathbf{82.45} \\\\ \hline \end{array} \\\\ \end{array}$$ To show the individual contribution of the two-stage fine-tuning, we evaluate the model capacity of MaskCoFT with or without Stage 2. Without Stage 2, most of the benchmark scores have a drop in model capacity, especially a 7 point drop on GSM8K. $$\small \begin{array}{lcc} \hline & \text{Generative} & \text{Multiple choice} \\\\ \begin{array}{l} \strut \text{Method} \\\\ \hline \strut \boldsymbol{\textit{DeepSeek-V2-Lite}} \\\\ \strut \text{MaskCoFT(stage 1 only)} \\\\ \strut \text{MaskCoFT(stage 1\\&2)} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{GSM8K} & \text{HumanEval} \\\\ \hline \strut {} & {} \\\\ \strut 36.39 & 26.22 \\\\ \strut \mathbf{43.82} & \mathbf{28.05} \\\\ \hline \end{array} & \begin{array}{ccccccc} \hline \strut \text{MMLU} & \text{ARC-E} & \text{ARC-C} & \text{HellaSwag} & \text{WinoGrande} & \text{PIQA} & \text{BoolQ} \\\\ \hline \strut {} & {} & {} & {} & {} & {} & {} \\\\ \strut 52.43 & 74.66 & 48.63 & \mathbf{77.09} & \mathbf{72.14} & 80.47 & 82.09 \\\\ \strut \mathbf{53.36} & \mathbf{75.72} & \mathbf{49.57} & 76.06 & 71.82 & \mathbf{80.36} & \mathbf{82.45} \\\\ \hline \end{array} \\\\ \end{array}$$
We thank the reviewer for constructive feedback. Please see our responses to each weakness and question below.
Q1. Validation to address the two limitations of ReMoE.
For limitation regarding "temporal locality": All of ReMoE's locality terms act within one sequence. They compare each token with earlier tokens of the same sequence, over a lag or a window. MaskCoFT's mask has no window. It is shared by all tokens in a batch, so it skews expert use the same way for every token and every request.
To isolate this difference, we compare the two designs with adapted experts in both. We add expert LoRA to ReMoE and compare it with MaskCoFT on DeepSeek-V2-Lite.
Shared mask versus temporal locality terms (DeepSeek-V2-Lite, B=12)
The two designs reward different caches. MaskCoFT leads under LFU (+1.25 HR), which keeps the most-used experts and rewards a stable skew across all tokens. ReMoE + expert LoRA leads under LRU (+1.32) and FIFO (+1.71), which keep recent experts and reward reuse between nearby tokens. MaskCoFT also keeps 1.0 point more average accuracy. The two forms of concentration are therefore complementary, and the better choice depends on the eviction policy.
On the long-context benchmarks, MaskCoFT keeps more capacity than our ReMoE runs (table below). It leads by 3.3 to 6.7 points on LongBench v2 on all three models, and by 5.5 to 6.4 points on LongGenBench on DeepSeek-V2-Lite and Mixtral-8$\times$7B. This is consistent with the advantage of a global set. ReMoE's locality terms pull each token toward the experts of earlier tokens, which over a long context can limit the experts available to later tokens. MaskCoFT applies no such constraint, and at inference no expert is removed.
Long-context capacity (%)
(response continues)