- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SlimWise separates expert capacity across the prefill and decode phases of MoE inference. It performs prefill with the full model and lets a pruned decoder directly reuse the resulting KV cache. To recover remaining quality loss and generation-length distortions, it introduces distillation in which the student continues from teacher-generated prefix caches. The framework supports both disaggregated and colocated serving through separate pruned checkpoints or phase-aware router masking. Evaluation covers two MoE backbones, three pruning criteria, mathematical reasoning, coding, and tool use, together with vLLM-based decode-throughput measurements.
Critical strengths
-
Useful quality recovery with supporting controls. Preserving full-model prefill can substantially improve a pruned decoder without changing its retained expert set. For example, at 75% REAP pruning on Qwen3.6, HumanEval+ accuracy increases from 0.331 to 0.856 through cache handoff alone. Table 2 also includes conventional-distillation controls that help distinguish the benefits of additional training from those of inheriting teacher-generated caches.
-
Generation-length analysis adds useful information beyond accuracy. The paper shows that pruning can cause premature termination or excessively long generation while leaving benchmark accuracy relatively unchanged. The greedy-decoding and phase-swap experiments provide additional support for these observations. Evaluating accuracy alongside generation behavior is consequential for assessing compression in serving settings.
Critical weaknesses
-
The incremental contribution over existing phase-aware expert-selection approaches is not sufficiently established. Retaining all experts during prefill while reducing the expert pool during decode has already been explored in Lynx. SlimWise’s static weight removal and cache-conditioned distillation are meaningful distinctions, but their additional value requires clearer competitive validation. The evaluation largely compares against the full model and conventional static pruning, without a matched comparison against published decode-optimization methods such as OEA. Consequently, it remains unclear whether SlimWise offers a compelling quality–throughput–resource tradeoff beyond existing approaches.
-
The system-wide resource benefit remains insufficiently validated. In colocated serving, all expert weights remain resident, so the method does not reduce model-weight memory. In disaggregated serving, the decoder’s smaller weight footprint is a real benefit, but the system still requires full-model prefill and the handoff of KV caches and, where applicable, recurrent states. Figure 4 primarily reports decode throughput under fixed input/output lengths; it does not clearly establish whether actual cross-instance state-transfer costs are included or quantify the overall benefit after accounting for both phases. These limitations are acknowledged, but the evidence remains incomplete for assessing the practical resource advantage of the proposed serving framework.
Questions for authors
-
Communication accounting: Does Figure 4 include actual KV-cache and recurrent-state transfer between prefill and decode instances? Please clarify the interconnect configuration, transferred bytes per request, transfer latency, and any overlap with computation. How do these costs affect first-token latency and completed-request throughput when both prefill and decode resources are counted?
-
Targeted comparison: Can the authors provide a representative comparison against an established method such as OEA under matched hardware, quality requirements, and per-user generation-speed constraints? This would help identify the additional benefit attributable to static expert removal and cache-conditioned distillation.
-
Distillation generalization: How are the distillation prompts and teacher-generated responses separated from the evaluation examples? Do the available results indicate whether recovery extends beyond the tasks emphasized by the distillation corpus, and whether other capabilities are preserved? This is a clarification request about the scope of the recovery evidence.