acceptodds
Under review as a conference paper at ICLR 2027

MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference

Abstract

Mixture-of-experts (MoE) language models often exceed the memory of a single GPU. Expert offloading keeps most experts in host memory and loads them on demand, so decoding speed depends on how many experts each token must fetch. Caching and prefetching reduce this cost only as far as the routing allows. Router-only fine-tuning can reshape the routing to reuse experts, but it keeps the experts frozen, so they cannot adapt to the tokens the new routing sends them. We propose MaskCoFT, a masked co-adaptive fine-tuning method that trains routers and experts together with the cross-entropy loss alone. During fine-tuning, a learnable binary mask restricts the Top- routing of each layer to a subset of experts, and the experts adapt to the tokens redirected to them. At inference, the learned mask becomes a soft prior that re-ranks experts, so every expert remains selectable. We simulate a GPU cache of 4 experts per layer for Mixtral-8×7B and 12 for DeepSeek-V2-Lite. MaskCoFT cuts expert fetches per token by 23.7% and 10.1% relative to the base model. In real offloading system serving, it lowers the time per output token by up to 16.4% and 5.5%, respectively. Its average accuracy over nine benchmarks stays above the base model by 0.92 and 0.53 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-experts, Hardware-aware finetuning, Efficient expert offloading, Memory-constrained LLM inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SpSpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MaskCoFT, a masked co-adaptive fine-tuning method for memory-efficient Mixture-of-Experts (MoE) inference. MaskCoFT jointly fine-tunes routers and experts while learning binary routing masks that concentrate token activations on subsets of experts. Once the masks converge, they are fixed, and the routers and experts continue to be fine-tuned under the restricted routing. During inference, the learned mask distribution serves as a soft prior to favor frequently selected experts while keeping all experts accessible. Experiments on Mixtral-8×7B and DeepSeek-V2-Lite demonstrate reductions in expert fetches of up to 23.7% and 10.1%, respectively, and decoding latency improvements of up to 16.4% and 5.5%, while maintaining average accuracy above the pretrained baselines.

Critical strengths

The paper demonstrates practical improvements in MoE inference efficiency, reducing expert fetches and decoding latency in a real offloading system.

Critical weaknesses

1. Lack of validation of the main research questions. The paper identifies two potential limitations of ReMoE regarding temporal locality and frozen experts, and explains how MaskCoFT addresses them. However, the empirical or theoretical evidence supporting these claims is limited. In particular, Table 3 only reports aggregate accuracy and cache hit rates, making it difficult to determine whether the proposed method actually overcomes the two limitations that motivate the work.

2. Limited novelty and insufficient comparison with existing approaches. The proposed method builds on existing techniques, including learnable masking and joint fine-tuning, but its additional technical contribution is not clearly established. The paper lacks ablations demonstrating the individual contributions of these components and the two-stage training procedure. Although the authors acknowledge the limitations of their comparison with ReMoE, the results do not establish whether MaskCoFT offers additional benefits over the closest prior work.

Questions for authors

Q1. What concrete empirical or theoretical evidence supports the two limitations of ReMoE identified in Section 1?

Q2. What are the individual contributions of expert co-adaptation and the two-stage training procedure? How do the results compare with router-only fine-tuning under the same mask?

Q3. What evidence demonstrates the benefit of learned masking over simpler alternatives, such as fixed expert subsets or directly learned routing biases?

Q4. While Table 3 provides a comparison with ReMoE, it does not establish a clear advantage of MaskCoFT. What concrete experimental or theoretical results can demonstrate a meaningful improvement over ReMoE, given that addressing its limitations is a central motivation of this paper?

10 comments

We thank the reviewer for constructive feedback. Please see our responses to each weakness and question below.

Q1. Validation to address the two limitations of ReMoE.

For limitation regarding "temporal locality": All of ReMoE's locality terms act within one sequence. They compare each token with earlier tokens of the same sequence, over a lag or a window. MaskCoFT's mask has no window. It is shared by all tokens in a batch, so it skews expert use the same way for every token and every request.

To isolate this difference, we compare the two designs with adapted experts in both. We add expert LoRA to ReMoE and compare it with MaskCoFT on DeepSeek-V2-Lite.

Shared mask versus temporal locality terms (DeepSeek-V2-Lite, B=12B=12)

MethodExpertsAvg. acc. (%)HR LRUHR LFUHR FIFO
Base modelnot trained61.1944.4543.4543.34
ReMoEfrozen58.7548.6148.1247.07
ReMoE + expert LoRALoRA61.3849.1747.9347.46
MaskCoFTLoRA62.3647.8549.1845.75

The two designs reward different caches. MaskCoFT leads under LFU (+1.25+1.25 HR), which keeps the most-used experts and rewards a stable skew across all tokens. ReMoE + expert LoRA leads under LRU (+1.32+1.32) and FIFO (+1.71+1.71), which keep recent experts and reward reuse between nearby tokens. MaskCoFT also keeps 1.0 point more average accuracy. The two forms of concentration are therefore complementary, and the better choice depends on the eviction policy.

On the long-context benchmarks, MaskCoFT keeps more capacity than our ReMoE runs (table below). It leads by 3.3 to 6.7 points on LongBench v2 on all three models, and by 5.5 to 6.4 points on LongGenBench on DeepSeek-V2-Lite and Mixtral-8$\times$7B. This is consistent with the advantage of a global set. ReMoE's locality terms pull each token toward the experts of earlier tokens, which over a long context can limit the experts available to later tokens. MaskCoFT applies no such constraint, and at inference no expert is removed.

Long-context capacity (%)

MethodLongBench v2LongGenBench
DeepSeek-V2-Lite
Base model26.6711.00
CE-only23.894.20
ReMoE24.4419.10
MaskCoFT30.5624.60
Mixtral-8$\times$7B
Base model35.5613.50
CE-only38.3327.60
ReMoE34.444.10
MaskCoFT37.7810.50
Qwen3.5-35B-A3B
Base model46.6721.60
CE-only51.116.00
ReMoE42.7817.30
MaskCoFT49.4418.00

(response continues)

For limitation regarding "frozen experts": we finetuned a DeepSeek-V2-Lite model using a modified ReMoE to jointly finetune routers and experts with LoRA. Then we compare the model capacity and simulation results with the original router-only ReMoE.

\begin{array}{lcc} \hline & \text{Generative} & \text{Multiple choice} \\\\ \begin{array}{l} \strut \text{Method} \\\\ \hline \strut \boldsymbol{\textit{DeepSeek-V2-Lite}} \\\\ \strut \text{ReMoE (router-only)} \\\\ \strut \text{ReMoE + expert LoRA} \\\\ \strut \text{MaskCoFT} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{GSM8K} & \text{HumanEval} \\\\ \hline \strut {} & {} \\\\ \strut 33.89 & 24.39 \\\\ \strut 36.24 & 26.83 \\\\ \strut \mathbf{43.82} & \mathbf{28.05} \\\\ \hline \end{array} & \begin{array}{ccccccc} \hline \strut \text{MMLU} & \text{ARC-E} & \text{ARC-C} & \text{HellaSwag} & \text{WinoGrande} & \text{PIQA} & \text{BoolQ} \\\\ \hline \strut {} & {} & {} & {} & {} & {} & {} \\\\ \strut 51.95 & 73.91 & 44.62 & 76.37 & 68.90 & 79.71 & 75.05 \\\\ \strut \mathbf{54.64} & 74.96 & 48.72 & \mathbf{79.06} & \mathbf{71.82} & \mathbf{80.58} & 79.57 \\\\ \strut 53.36 & \mathbf{75.72} & \mathbf{49.57} & 76.06 & \mathbf{71.82} & 80.36 & \mathbf{82.45} \\\\ \hline \end{array} \\\\ \end{array}$$ $$\small \begin{array}{lccc} \hline & \text{LRU} & \text{LFU} & \text{FIFO} \\\\ \begin{array}{l} \strut \text{Method} \\\\ \hline \strut \boldsymbol{\textit{DeepSeek-V2-Lite}}\quad B=12 \\\\ \strut \text{ReMoE (router-only)} \\\\ \strut \text{ReMoE + expert LoRA} \\\\ \strut \text{MaskCoFT} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{HR (\\%)}\\;\uparrow & \text{Fetches/tok}\\;\downarrow \\\\ \hline \strut {} & {} \\\\ \strut 48.61 & 80.16 \\\\ \strut \mathbf{49.17} & \mathbf{79.29} \\\\ \strut 47.85 & 81.36 \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{HR (\\%)}\\;\uparrow & \text{Fetches/tok}\\;\downarrow \\\\ \hline \strut {} & {} \\\\ \strut 48.12 & 80.94 \\\\ \strut 47.93 & 81.23 \\\\ \strut \mathbf{49.18} & \mathbf{79.28} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{HR (\\%)}\\;\uparrow & \text{Fetches/tok}\\;\downarrow \\\\ \hline \strut {} & {} \\\\ \strut 47.07 & 82.57 \\\\ \strut \mathbf{47.46} & \mathbf{81.96} \\\\ \strut 45.75 & 84.63 \\\\ \hline \end{array} \\\\ \end{array}$$ With the joint fine-tuning of routers and experts, ReMoE can preserve more model capacity and achieve a comparable hit rate compared with router-only fine-tuning.

Q2. Individual contributions of expert co-adaptation and the two-stage training procedure.

To show the individual contribution of expert co-adaptation, we evaluate the model capacity of MaskCoFT with frozen experts and compare with original MaskCoFT with jointly fine-tuning of experts and routers on DeepSeek-V2-Lite. With expert co-adaptation, MaskCoFT has obviouly better preservation on model capacity.

\begin{array}{lcc} \hline & \text{Generative} & \text{Multiple choice} \\\\ \begin{array}{l} \strut \text{Method} \\\\ \hline \strut \boldsymbol{\textit{DeepSeek-V2-Lite}} \\\\ \strut \text{MaskCoFT(frozen experts)} \\\\ \strut \text{MaskCoFT(expert co-adaptation)} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{GSM8K} & \text{HumanEval} \\\\ \hline \strut {} & {} \\\\ \strut 27.75 & 20.12 \\\\ \strut \mathbf{43.82} & \mathbf{28.05} \\\\ \hline \end{array} & \begin{array}{ccccccc} \hline \strut \text{MMLU} & \text{ARC-E} & \text{ARC-C} & \text{HellaSwag} & \text{WinoGrande} & \text{PIQA} & \text{BoolQ} \\\\ \hline \strut {} & {} & {} & {} & {} & {} & {} \\\\ \strut 48.42 & 68.98 & 41.47 & 70.73 & 68.19 & 79.11 & 77.49 \\\\ \strut \mathbf{53.36} & \mathbf{75.72} & \mathbf{49.57} & \mathbf{76.06} & \mathbf{71.82} & \mathbf{80.36} & \mathbf{82.45} \\\\ \hline \end{array} \\\\ \end{array}$$ To show the individual contribution of the two-stage fine-tuning, we evaluate the model capacity of MaskCoFT with or without Stage 2. Without Stage 2, most of the benchmark scores have a drop in model capacity, especially a 7 point drop on GSM8K. $$\small \begin{array}{lcc} \hline & \text{Generative} & \text{Multiple choice} \\\\ \begin{array}{l} \strut \text{Method} \\\\ \hline \strut \boldsymbol{\textit{DeepSeek-V2-Lite}} \\\\ \strut \text{MaskCoFT(stage 1 only)} \\\\ \strut \text{MaskCoFT(stage 1\\&2)} \\\\ \hline \end{array} & \begin{array}{cc} \hline \strut \text{GSM8K} & \text{HumanEval} \\\\ \hline \strut {} & {} \\\\ \strut 36.39 & 26.22 \\\\ \strut \mathbf{43.82} & \mathbf{28.05} \\\\ \hline \end{array} & \begin{array}{ccccccc} \hline \strut \text{MMLU} & \text{ARC-E} & \text{ARC-C} & \text{HellaSwag} & \text{WinoGrande} & \text{PIQA} & \text{BoolQ} \\\\ \hline \strut {} & {} & {} & {} & {} & {} & {} \\\\ \strut 52.43 & 74.66 & 48.63 & \mathbf{77.09} & \mathbf{72.14} & 80.47 & 82.09 \\\\ \strut \mathbf{53.36} & \mathbf{75.72} & \mathbf{49.57} & 76.06 & 71.82 & \mathbf{80.36} & \mathbf{82.45} \\\\ \hline \end{array} \\\\ \end{array}$$
Reviewer_CVHVOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose MaskCoFT, a masked co-adaptive fine-tuning method that trains routers and experts together with the cross-entropy loss. The key design is a learnable binary mask that restricts the Top-K routing, and the experts adapt to the tokens redirected to them. By simulation and real offloading system serving, it cuts expert fetches per token and lowers the time per token relative to the base model.

Critical strengths
  • The authors provide relatively clear motivation and practical relevance in the Introduction part. The measured results with NVIDIA Nsight Systems in Figure 1 verify the point that experts dominate weights while loading dominates decoding.

  • MaskCoFT jointly fine-tunes routers and experts by a learnable binary routing mask. Particularly, it does not require tuning loss hyperparameters during training, as it does not need to auxiliary locality or load balancing loss.

Critical weaknesses
  • The authors mentioned that ReMoE is the closest prior work. However, the major evaluation results (Table 1, Table 2, Figure 3, Figure 4) do not have the specific performance of ReMoE. Particularly, although Table 3 shows the results reported by ReMoE, it seems that the proposed MaskCoFT has little improvement compared with ReMoE. Thus, I am doubtful that the authors have addressed the two open questions of ReMoE mentioned in Introduction, along with its corresponding effectiveness.

  • Regarding the point that ReMoE has the question “its reuse comes from nearby tokens selecting the same experts, so its benefit may shrink when adjacent tokens need different experts” (line 80-81),ReMoE also incorporates longer-horizon routing consistency and working-set compression to encourage expert concentration with temporal windows. Therefore, I think the advantage of global expert concentration in MaskCoFT should be discussed more carefully.

  • Compared with router-only fine-tuning, MaskCoFT introduces additional training operations, including binary-mask optimization and LoRA-based expert adaptation. However, the corresponding training and deployment costs are not sufficiently quantified.

Questions for authors
  • Could the authors provide more evaluation results with ReMoE, or other evidence to testify that MaskCoFT truly has better performance than ReMoE (not confined to baseline or CE-only) ?

  • Could the authors provide more careful discussion about the advantage of global expert concentration? Specifically, related ablation study or analysis are encouraged to identify them.

  • How much is the additional training/inference operations of MaskCoFT? Besides, how are the LoRA adapters deployed during inference, and do they introduce additional computation or expert-transfer overhead?

  • I also wonder how robust is the learned binary routing mask under domain shifts or changing inference workloads? And does the method require retraining when the model backbones or workload distribution changes significantly?

13 comments

We thank the reviewer for constructive feedback. Please see our responses to each weakness and question below.

Q1. More evaluation results to compare with ReMoE.

We added a 2026 MoE model, Qwen3.5-35B-A3B, two recent long-context benchmarks, and implemented a recent baseline.

Key observations (updated Tables 1 and 4 below):

  • MaskCoFT improves efficiency on every model, while CE-only fine-tuning decays it. Across all three models and all three cache policies, CE-only lowers the hit rate below the baseline in all settings, while MaskCoFT improves it (e.g., Mixtral, LRU: baseline 61.38, CE-only 60.17, MaskCoFT 67.79).

  • MaskCoFT preserves better model capacity than ReMoE with comparable efficiency. MaskCoFT outperforms ReMoE on 25 of 33 benchmark entries, with large margins on generative tasks (GSM8K: +9.9 on DeepSeek and +11.6 on Mixtral; HumanEval: +11.6 on Mixtral). The two methods reach comparable hit rates: MaskCoFT is higher on Mixtral under all policies and on DeepSeek under LFU, and ReMoE is slightly higher on DeepSeek under LRU and FIFO and on Qwen3.5 (within 2.5 points).

  • MaskCoFT outperforms ReMoE under long-context workload due to global expert concentration. MaskCoFT gains 3 to 7 points over ReMoE on the long-context understanding and long-form generation benchmarks, except for a comparable LongGenBench performance on Qwen model. A likely reason is that ReMoE's temporal-locality and routing-reuse regularizers push each token to activate the same experts as previous tokens within a temporal window. Over a long context, this local constraint limits the experts available to later tokens and therefore degrades model capacity. MaskCoFT instead concentrates expert activations globally across all tokens, and no experts get pruned during inference. so routing remains free to vary within the concentrated subset and more capacity is preserved.

(response continues)

Table 1. Scores (%) on model capacity benchmarks. Bold numbers are the best results among all methods. Underlined numbers are results of MaskCoFT better than ReMoE.

GenerativeMultiple choiceLong context/generation
MethodGSM8KHumanEvalMMLUARC-EARC-CHellaSwagWinoGrandePIQABoolQLongBenchV2LongGenBench
DeepSeek-V2-Lite
Baseline38.3626.8355.5374.3745.9977.7371.2780.2580.3426.6711.00
CE-only40.0328.0554.7377.2350.5179.0772.0680.6982.0823.894.20
ReMoE33.8924.3951.9573.9144.6276.3768.9079.7175.0524.4419.10
MaskCoFT (Ours)43.82‾\underline{\mathbf{43.82}}28.05‾\underline{\mathbf{28.05}}53.36‾\underline{53.36}75.72‾\underline{75.72}49.57‾\underline{49.57}76.0671.82‾\underline{71.82}80.36‾\underline{80.36}82.45‾\underline{\mathbf{82.45}}30.56‾\underline{\mathbf{30.56}}24.60‾\underline{\mathbf{24.60}}
Mixtral-8$\times$7B
Baseline58.6835.3768.1682.5359.4784.2477.1183.3085.9635.5613.50
CE-only66.8740.8568.2083.3360.8483.9076.7283.4687.9838.3327.60
ReMoE54.5932.3267.4581.4858.7083.5176.6483.3085.0534.444.10
MaskCoFT (Ours)66.19‾\underline{66.19}43.90‾\underline{\mathbf{43.90}}67.1882.91‾\underline{82.91}58.4583.56‾\underline{83.56}76.5683.73‾\underline{\mathbf{83.73}}88.44‾\underline{\mathbf{88.44}}37.78‾\underline{37.78}10.50‾\underline{10.50}
Qwen3.5-35B-A3B
Baseline86.6665.8582.5781.8661.9583.7075.8582.7587.4046.6721.60
CE-only85.6065.2480.2682.4963.1482.0574.0381.8890.3151.116.00
ReMoE83.8565.2480.4779.7158.3682.3170.4881.5681.9942.7817.30
MaskCoFT (Ours)83.7068.29‾\underline{\mathbf{68.29}}80.2982.74‾\underline{\mathbf{82.74}}61.95‾\underline{61.95}81.5875.45‾\underline{75.45}80.9690.12‾\underline{90.12}49.44‾\underline{49.44}18.00‾\underline{18.00}

Table 4. Trace-driven cache simulation results. B is the GPU budget indicating how many experts per MoE layer can be stored in the cache.

LRULFUFIFO
MethodHR (%) ↑\uparrowFetches/tok ↓\downarrowHR (%) ↑\uparrowFetches/tok ↓\downarrowHR (%) ↑\uparrowFetches/tok ↓\downarrow
DeepSeek-V2-Lite B=12B=12
Baseline44.4586.6643.4588.2143.3488.39
CE-only43.0888.8041.4891.2941.5991.12
ReMoE48.6180.1648.1280.9447.0782.57
MaskCoFT (Ours)47.8581.3649.1879.2845.7584.63
Mixtral-8$\times$7B B=4B=4
Baseline61.3824.7261.5824.5960.4925.29
CE-only60.1725.4960.6225.2159.2126.10
ReMoE65.5922.0264.7722.5564.5722.68
MaskCoFT (Ours)67.7920.6170.6818.7765.9521.79
Qwen3.5-35B-A3B B=16B=16
Baseline47.49168.0442.87182.8345.09175.70
CE-only46.13172.3741.38187.5843.76179.96
ReMoE51.10156.4746.62170.8348.43165.03
MaskCoFT (Ours)48.65164.3144.57177.3746.04172.66

Q2. The advantage of global expert concentration.

You are right that ReMoE also has longer-horizon terms. Besides its one-step reuse and smoothness terms, it compares routing with earlier tokens at several lags and penalizes the entropy of routing averaged over windows of tokens.

The difference that remains is scope. All of ReMoE's locality terms act within one sequence. They compare each token with earlier tokens of the same sequence, over a lag or a window. MaskCoFT's mask has no window. It is shared by all tokens in a batch, so it skews expert use the same way for every token and every request.

To isolate this difference, we compare the two designs with adapted experts in both. We add expert LoRA to ReMoE and compare it with MaskCoFT on DeepSeek-V2-Lite.

Shared mask versus temporal locality terms (DeepSeek-V2-Lite, B=12B=12)

MethodExpertsAvg. acc. (%)HR LRUHR LFUHR FIFO
Base modelnot trained61.1944.4543.4543.34
ReMoEfrozen58.7548.6148.1247.07
ReMoE + expert LoRALoRA61.3849.1747.9347.46
MaskCoFTLoRA62.3647.8549.1845.75

The two designs reward different caches. MaskCoFT leads under LFU (+1.25+1.25 HR), which keeps the most-used experts and rewards a stable skew across all tokens. ReMoE + expert LoRA leads under LRU (+1.32+1.32) and FIFO (+1.71+1.71), which keep recent experts and reward reuse between nearby tokens. MaskCoFT also keeps 1.0 point more average accuracy. The two forms of concentration are therefore complementary, and the better choice depends on the eviction policy.

On the long-context benchmarks, MaskCoFT keeps more capacity than our ReMoE runs (table below). It leads by 3.3 to 6.7 points on LongBench v2 on all three models, and by 5.5 to 6.4 points on LongGenBench on DeepSeek-V2-Lite and Mixtral-8$\times$7B. This is consistent with the advantage of a global set. ReMoE's locality terms pull each token toward the experts of earlier tokens, which over a long context can limit the experts available to later tokens. MaskCoFT applies no such constraint, and at inference no expert is removed.

Long-context capacity (%)

MethodLongBench v2LongGenBench
DeepSeek-V2-Lite
Base model26.6711.00
CE-only23.894.20
ReMoE24.4419.10
MaskCoFT30.5624.60
Mixtral-8$\times$7B
Base model35.5613.50
CE-only38.3327.60
ReMoE34.444.10
MaskCoFT37.7810.50
Qwen3.5-35B-A3B
Base model46.6721.60
CE-only51.116.00
ReMoE42.7817.30
MaskCoFT49.4418.00
Reviewer_8xNfOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an optimization method for accelerating MoE inference with expert offloading. It jointly fine-tunes the router and expert parameters and applies a soft mask to the router to steer tokens toward a shared, smaller subset of experts. This reduces frequent expert offloading and loading, thereby accelerating inference.

Critical strengths
  1. The two-stage training procedure is well motivated: the first stage learns a layer-wise mask to select an expert subset for each layer, and the second stage jointly fine-tunes the routers and experts based on the selected subsets.
  2. The experimental setup includes two large-scale MoE models, DeepSeek and Mixtral, lending credibility to the results.
  3. Inference experiments on samples drawn from ShareGPT show meaningful improvements in cache hit rate and reductions in expert fetches per token.
Critical weaknesses
  1. The evaluated models and benchmarks are somewhat outdated. DeepSeek-V2-Lite and Mixtral-8×7B date back to 2024. The selected benchmarks are relatively simple, and the evaluation does not adequately assess capability preservation under long-context inputs or long-form generation.
  2. MaskCoFT still degrades model performance relative to CE-only fine-tuning. This is consistent with practical experience in tuning MoE models: hard-masking a subset of experts can noticeably impair performance. Although MaskCoFT attempts to compensate through joint router–expert fine-tuning, the extent of adaptation achievable through fine-tuning remains limited compared with the extensive training performed during pretraining. MaskCoFT does not introduce a novel mechanism to address this limitation.
  3. MaskCoFT does not establish a clear empirical advantage over ReMoE. While the paper demonstrates efficiency gains over the original base models, it does not demonstrate a better quality–efficiency trade-off than its closest prior work, ReMoE. In particular, despite introducing expert adaptation and mask learning, the reported results show largely comparable or weaker improvements in cache efficiency, alongside more pronounced degradation in some capabilities. The practical value of these additional mechanisms therefore remains unclear.
Questions for authors
  1. Although omitting the load-balancing loss can encourage tokens to concentrate on a small expert subset, what happens if severe load imbalance also occurs within that subset? This may lead to GPU out-of-memory errors or reduced inference efficiency. Could the author show expert load balancing performance after MaskCoFT?
  2. Could the authors evaluate MaskCoFT on more recent models, such as MoE models in the Qwen3.5 family, and assess both capability degradation and inference speedups on benchmarks that better capture long-context capabilities?
  3. The reported results do not establish a clear quality–efficiency advantage over ReMoE, despite introducing expert adaptation and mask learning. What substantive advance does MaskCoFT offer beyond an alternative approach to the same problem? In the absence of demonstrated gains in effectiveness, simplicity, or cost, or a clearly established new insight, the significance of the contribution remains unclear.
12 comments

We thank the reviewer for constructive feedback. Please see our responses to each weakness and question below.

Q1. The evaluated models and benchmarks are somewhat outdated.

We added a 2026 MoE model, Qwen3.5-35B-A3B, two recent long-context benchmarks, and implemented a recent baseline.

Key observations (updated Tables 1 and 4 below):

  • MaskCoFT improves efficiency on every model, while CE-only fine-tuning decays it. Across all three models and all three cache policies, CE-only lowers the hit rate below the baseline in all settings, while MaskCoFT improves it (e.g., Mixtral, LRU: baseline 61.38, CE-only 60.17, MaskCoFT 67.79).

  • MaskCoFT preserves better model capacity than ReMoE with comparable efficiency. MaskCoFT outperforms ReMoE on 25 of 33 benchmark entries, with large margins on generative tasks (GSM8K: +9.9 on DeepSeek and +11.6 on Mixtral; HumanEval: +11.6 on Mixtral). The two methods reach comparable hit rates: MaskCoFT is higher on Mixtral under all policies and on DeepSeek under LFU, and ReMoE is slightly higher on DeepSeek under LRU and FIFO and on Qwen3.5 (within 2.5 points).

  • MaskCoFT outperforms ReMoE under long-context workload due to global expert concentration. MaskCoFT gains 3 to 7 points over ReMoE on the long-context understanding and long-form generation benchmarks, except for a comparable LongGenBench performance on Qwen model. A likely reason is that ReMoE's temporal-locality and routing-reuse regularizers push each token to activate the same experts as previous tokens within a temporal window. Over a long context, this local constraint limits the experts available to later tokens and therefore degrades model capacity. MaskCoFT instead concentrates expert activations globally across all tokens, and no experts get pruned during inference. so routing remains free to vary within the concentrated subset and more capacity is preserved.

(response continues)

Table 1. Scores (%) on model capacity benchmarks. Bold numbers are the best results among all methods. Underlined numbers are results of MaskCoFT better than ReMoE.

GenerativeMultiple choiceLong context/generation
MethodGSM8KHumanEvalMMLUARC-EARC-CHellaSwagWinoGrandePIQABoolQLongBenchV2LongGenBench
DeepSeek-V2-Lite
Baseline38.3626.8355.5374.3745.9977.7371.2780.2580.3426.6711.00
CE-only40.0328.0554.7377.2350.5179.0772.0680.6982.0823.894.20
ReMoE33.8924.3951.9573.9144.6276.3768.9079.7175.0524.4419.10
MaskCoFT (Ours)43.82‾\underline{\mathbf{43.82}}28.05‾\underline{\mathbf{28.05}}53.36‾\underline{53.36}75.72‾\underline{75.72}49.57‾\underline{49.57}76.0671.82‾\underline{71.82}80.36‾\underline{80.36}82.45‾\underline{\mathbf{82.45}}30.56‾\underline{\mathbf{30.56}}24.60‾\underline{\mathbf{24.60}}
Mixtral-8$\times$7B
Baseline58.6835.3768.1682.5359.4784.2477.1183.3085.9635.5613.50
CE-only66.8740.8568.2083.3360.8483.9076.7283.4687.9838.3327.60
ReMoE54.5932.3267.4581.4858.7083.5176.6483.3085.0534.444.10
MaskCoFT (Ours)66.19‾\underline{66.19}43.90‾\underline{\mathbf{43.90}}67.1882.91‾\underline{82.91}58.4583.56‾\underline{83.56}76.5683.73‾\underline{\mathbf{83.73}}88.44‾\underline{\mathbf{88.44}}37.78‾\underline{37.78}10.50‾\underline{10.50}
Qwen3.5-35B-A3B
Baseline86.6665.8582.5781.8661.9583.7075.8582.7587.4046.6721.60
CE-only85.6065.2480.2682.4963.1482.0574.0381.8890.3151.116.00
ReMoE83.8565.2480.4779.7158.3682.3170.4881.5681.9942.7817.30
MaskCoFT (Ours)83.7068.29‾\underline{\mathbf{68.29}}80.2982.74‾\underline{\mathbf{82.74}}61.95‾\underline{61.95}81.5875.45‾\underline{75.45}80.9690.12‾\underline{90.12}49.44‾\underline{49.44}18.00‾\underline{18.00}

Table 4. Trace-driven cache simulation results. B is the GPU budget indicating how many experts per MoE layer can be stored in the cache.

LRULFUFIFO
MethodHR (%) ↑\uparrowFetches/tok ↓\downarrowHR (%) ↑\uparrowFetches/tok ↓\downarrowHR (%) ↑\uparrowFetches/tok ↓\downarrow
DeepSeek-V2-Lite B=12B=12
Baseline44.4586.6643.4588.2143.3488.39
CE-only43.0888.8041.4891.2941.5991.12
ReMoE48.6180.1648.1280.9447.0782.57
MaskCoFT (Ours)47.8581.3649.1879.2845.7584.63
Mixtral-8$\times$7B B=4B=4
Baseline61.3824.7261.5824.5960.4925.29
CE-only60.1725.4960.6225.2159.2126.10
ReMoE65.5922.0264.7722.5564.5722.68
MaskCoFT (Ours)67.7920.6170.6818.7765.9521.79
Qwen3.5-35B-A3B B=16B=16
Baseline47.49168.0442.87182.8345.09175.70
CE-only46.13172.3741.38187.5843.76179.96
ReMoE51.10156.4746.62170.8348.43165.03
MaskCoFT (Ours)48.65164.3144.57177.3746.04172.66

Q2. MaskCoFT still degrades model performance relative to CE-only fine-tuning.

We agree that MaskCoFT does not match CE-only fine-tuning with respect to model performance on every benchmark. We view this gap as the price of the efficiency gain rather than a weakness of the method.

CE-only fine-tuning maximizes task performance and brings no efficiency benefit. However, our proposed MaskCoFT concentrates routing on a subset of experts, which restricts model capacity to trade for higher efficiency. Therefore, some capacity gap to CE-only is as expected. It is also worth noting that the compromise in model performance is small. MaskCoFT's average is within 0.4 points of CE-only on all three models (DeepSeek: −0.36-0.36, Mixtral: −0.14-0.14, Qwen: +0.01+0.01). CE-only also lowers the hit rate below the base in all nine settings of Table 4. At inference MaskCoFT removes no expert. The learned prior only re-ranks them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.