acceptodds
Withdrawn from ICLR 2027

SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-experts (MoE) models activate few experts per token, yet batched decoding can access nearly the entire expert pool, making expert-weight traffic a major bottleneck. Expert pruning reduces this traffic, but conventional approaches also prune compute-bound prefill, sacrificing model quality for little throughput benefit. We present SlimWise, a serving framework that tailors the expert pool to each inference phase. SlimWise performs prefill with the full model and decode with a pruned model that directly reuses the prefill-generated KV cache without conversion. Across two MoE backbones and three pruning criteria, this training-free KV cache handoff substantially narrows accuracy gaps relative to the full model in many settings. We also show that benchmark accuracy can conceal substantial pruning-induced changes in generation length. To address these distortions and residual accuracy loss, SlimWise introduces a low-cost distillation stage that trains the decoder to continue from full-model KV caches while updating only a small subset of parameters. Implemented in vLLM, SlimWise supports both prefill–decode (PD) disaggregation and PD-colocated serving. On Qwen3.6-35B-A3B, SlimWise improves decode throughput by up to 1.81 at 50% expert pruning with minimal accuracy loss.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM, Pruning, MoE serving

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_1bAAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SlimWise separates expert capacity across the prefill and decode phases of MoE inference. It performs prefill with the full model and lets a pruned decoder directly reuse the resulting KV cache. To recover remaining quality loss and generation-length distortions, it introduces distillation in which the student continues from teacher-generated prefix caches. The framework supports both disaggregated and colocated serving through separate pruned checkpoints or phase-aware router masking. Evaluation covers two MoE backbones, three pruning criteria, mathematical reasoning, coding, and tool use, together with vLLM-based decode-throughput measurements.

Critical strengths
  1. Useful quality recovery with supporting controls. Preserving full-model prefill can substantially improve a pruned decoder without changing its retained expert set. For example, at 75% REAP pruning on Qwen3.6, HumanEval+ accuracy increases from 0.331 to 0.856 through cache handoff alone. Table 2 also includes conventional-distillation controls that help distinguish the benefits of additional training from those of inheriting teacher-generated caches.

  2. Generation-length analysis adds useful information beyond accuracy. The paper shows that pruning can cause premature termination or excessively long generation while leaving benchmark accuracy relatively unchanged. The greedy-decoding and phase-swap experiments provide additional support for these observations. Evaluating accuracy alongside generation behavior is consequential for assessing compression in serving settings.

Critical weaknesses
  1. The incremental contribution over existing phase-aware expert-selection approaches is not sufficiently established. Retaining all experts during prefill while reducing the expert pool during decode has already been explored in Lynx. SlimWise’s static weight removal and cache-conditioned distillation are meaningful distinctions, but their additional value requires clearer competitive validation. The evaluation largely compares against the full model and conventional static pruning, without a matched comparison against published decode-optimization methods such as OEA. Consequently, it remains unclear whether SlimWise offers a compelling quality–throughput–resource tradeoff beyond existing approaches.

  2. The system-wide resource benefit remains insufficiently validated. In colocated serving, all expert weights remain resident, so the method does not reduce model-weight memory. In disaggregated serving, the decoder’s smaller weight footprint is a real benefit, but the system still requires full-model prefill and the handoff of KV caches and, where applicable, recurrent states. Figure 4 primarily reports decode throughput under fixed input/output lengths; it does not clearly establish whether actual cross-instance state-transfer costs are included or quantify the overall benefit after accounting for both phases. These limitations are acknowledged, but the evidence remains incomplete for assessing the practical resource advantage of the proposed serving framework.

Questions for authors
  1. Communication accounting: Does Figure 4 include actual KV-cache and recurrent-state transfer between prefill and decode instances? Please clarify the interconnect configuration, transferred bytes per request, transfer latency, and any overlap with computation. How do these costs affect first-token latency and completed-request throughput when both prefill and decode resources are counted?

  2. Targeted comparison: Can the authors provide a representative comparison against an established method such as OEA under matched hardware, quality requirements, and per-user generation-speed constraints? This would help identify the additional benefit attributable to static expert removal and cache-conditioned distillation.

  3. Distillation generalization: How are the distillation prompts and teacher-generated responses separated from the evaluation examples? Do the available results indicate whether recovery extends beyond the tasks emphasized by the distillation corpus, and whether other capabilities are preserved? This is a clarification request about the scope of the recovery evidence.

Reviewer_SaoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SlimWise uses the full MoE for prefill and a pruned MoE for decode, directly handing the full model's KV cache to the smaller decoder. It also introduces deployment-matched distillation to make the pruned decoder better use the inherited cache. The method is implemented in vLLM and improves decode throughput while recovering part of the quality lost by conventional pruning.

Critical strengths
  • The method is simple and easy to integrate into existing serving systems.
  • The analysis of abnormal output lengths is useful and shows that benchmark accuracy alone can hide pruning failures.
  • A smaller decode checkpoint is attractive for PD-disaggregated serving because it leaves more memory for KV cache.
Critical weaknesses
  • The main motivation appears specific to expert-pool pruning that keeps the per-token Top-K unchanged. Dynamic expert-skipping methods such as NAEE and MoDES reduce the number of executed experts and can accelerate both prefill and decode. Therefore, the paper does not establish that full-model prefill is generally preferable.
  • The observation that prefill and decode have different bottlenecks is relatively straightforward, so the technical novelty appears limited beyond the deployment-matched distillation.
  • Since SlimWise leaves prefill unchanged, its end-to-end benefit depends strongly on the input/output ratio. The system evaluation focuses on decode throughput for a fixed 1k-input/8k-output workload, so prompt-heavy or multimodal workloads are not covered.
  • The training-free handoff is not consistently reliable; some Gemma settings require approximately 100M tokens of distillation to recover quality and output length, which complicates deployment across multiple pruning configurations.
Questions for authors
  1. Please compare, on the same model and at a matched quality or compute budget: (a) conventional fixed-pool pruning in both phases, (b) SlimWise with full prefill and pruned decode, and (c) NAEE/MoDES-style dynamic active-K reduction in both phases.
  2. Can dynamic expert skipping be used during prefill together with SlimWise during decode? This appears to be a natural combination.
  3. Please report end-to-end request throughput across several input/output ratios, including prompt-heavy, short-output, and multimodal workloads, rather than decode throughput alone.
  4. How often does KV handoff hurt, and does each retained expert set or pruning ratio require a separate distilled checkpoint? A small ablation over pruning ratios and distillation components would help clarify this cost.
Reviewer_8PjWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Mixture-of-experts models can suffer from costly expert-weight traffic during decoding. SlimWise addresses this by using the full model for prefill and a pruned model for decoding while directly reusing the KV cache. This phase-specific approach improves efficiency while preserving much of the full model’s quality. A lightweight distillation stage further reduces accuracy loss and mitigates pruning-induced changes in generation behavior.

Critical strengths
  • This paper addresses an important and timely problem for the computation and memory efficiency of scaling sparse MoE.
  • The overall design is clear and easy to follow where prefill uses the full list of experts while decode use pruned experts.
  • Evaluation also demonstrate its performance advantages over those existing solutions in terms of the accuracy and runtime performance.
Critical weaknesses
  • Evaluation are on the relatively small models and platform settings.
  • Writing clarity can be improved at the evaluation section.
Questions for authors

Thanks for your submission to ICLR'2027, below are detailed comments.

  • Evaluation are on the relatively small models and platform settings. Please consider Qwen3 (235B total, 22B active) or GLM-4.5 (355B total, 32B active) to demonstrate the scalability of the proposed design.
  • What are the current GPU computation and memory resource utilization (in %) for those throughput evaluation? since I want to understand after applying pruned decode whether it can better leverage the GPUs.
  • The currently evaluation math or coding are on the simpler side, while longer reasoning task should be considered. such as LongBench v2 (https://longbench2.github.io/) and OlympiadBench (https://github.com/OpenBMB/OlympiadBench).
  • Writing clarity can be improved at the evaluation section. It is unclear to me what is the clear definition of "baseline" used in those Table-1 and 2 throughout the experimental sections and how does this "baseline" differ from those proposed solution.
  • I am also confused for experimental setup "6 hours on four B200" are used for distillation while on Figure-4 the paper mentioned that 2xA100 are used for throughput evaluation, why not all use the same set of GPUs for those two evaluation? I am wondering if there is any consistency issue when distillation is done on one type of GPU while perform regular serving on another type of GPUs.
Reviewer_UhjzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SlimWise, a serving framework for reducing expert-weight traffic in batched MoE decoding. The core idea is to assign different expert pool sizes to the prefill and decode phases. This is motivated by the observation that decode is bottlenecked by expert-weight movement, while compute-bound prefill gains nothing from a smaller pool. Since expert pruning preserves the KV cache format, the pruned decoder can reuse the full model's cache without conversion. SlimWise adds two further components: a low-cost distillation stage that trains the decoder on inherited caches, and phase-aware router masking for PD-colocated execution. Experiments cover math, coding, and tool-use benchmarks on Qwen3.6-35B-A3B and Gemma 4-26B-A4B. Accuracy lands within a few points of the full model after distillation, with 1.40–1.81× reported decode throughput at 50% pruning.

Critical strengths
  • The paper addresses an important problem: the expert-weight traffic that limits throughput in batched MoE decoding.
  • The measurements of expert-weight volume and phase-wise throughput are well executed and make the phase asymmetry concrete.
  • The method is practical in spirit, since the KV cache handoff requires no conversion and the router-masking formulation lets a single implementation cover both deployment modes.
  • The analysis of generation-length distortion is a useful observation for the compression literature.
Critical weaknesses
  • The core contribution of this paper is incremental. The expert-selection pipeline (calibration mixture and importance criteria) comes from prior work. The remaining idea, that a smaller pool helps only memory-bound decode, is intuitive and follows from a roofline argument once k is held fixed.
  • The training-free claim does not provide a consistent performance guarantee. On Gemma 4 with EAN, the handoff lowers GSM8K from 0.816 under conventional pruning to 0.173. The paper offers no mechanism or criterion for predicting when the handoff helps, so a deployer cannot know in advance which regime they are in. Distillation therefore appears indispensable rather than optional.
  • The paper's core novel component is supported by a single ablation. The comparison isolating handoff-aware distillation from conventional distillation appears only in Table 2, at one pruning ratio, on one backbone and one criterion.
  • The headline throughput numbers do not count useful work. All speedups are reported as tokens per second per GPU, yet pruned decode inflates generation length, and distillation reduces this without eliminating it: at m′=64, MATH-500 p90 falls from the 32.8k budget to 14.0k but remains 2.3× the baseline's 6.1k. Tokens from a prolonged reasoning trace are not useful work, so the reported gains need not translate to completed requests.
  • The evaluation setting is favorable to the reported gains in ways the paper does not discuss. Throughput is measured on a fixed 1k-input/8k-output workload, which maximizes the decode fraction, and no end-to-end number is given.
Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.