acceptodds
Withdrawn from ICLR 2027

CCQ: Compensation-Conditioned Quadratic Selection for Absorbing Skipped Experts in MoE LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Executing only half of the routed experts per token halves the routed-expert work of a fine-grained Mixture-of-Experts (MoE) model, but the absent experts leave an error. We split it into a common mode, the mean expert output scaled by the absent routing mass, and an expert-specific remainder. Restoring each part exactly on three MoEs shows that both matter, in model-dependent proportions, and that their costs nearly add, as a second-order analysis of the KL divergence predicts when their interaction is small. We therefore compensate them with different tools. A mass-gated low-rank readout of the always-executed shared expert, trained by KL distillation, recovers what the shared expert predicts of the absent output. Diagnostics show it is not merely a common-mode estimate, yet on the model where the common mode dominates, it removes as much KL as the exact common mode. The readout leaves most of the specific part, but each routed expert's output energy is predictable. We skip the half with the least predicted cost and add a sketch of the costliest absent expert at an eleventh of its width, trained on the residual the readout leaves. Implemented in vLLM with CUDA kernels, on Moonlight-16B-A3B, DeepSeek-V2-Lite and Qwen3.6-35B-A3B, the method, CCQ, lowers the full-vocabulary KL to the full model by 23–80% against skipping alone and by 13–18% against a LoRA with the same distillation budget, the strongest of seven baselines. Selection and sketch provide this gain; the readout alone only matches the LoRA. The method retains 98.0–99.2% of the full model's average score on 12 benchmarks, though mostly not significantly different from the readout or the LoRA. In vLLM's throughput benchmark on one GPU, it generates 1.19–1.23× as many tokens per second as the full model at batch size 1, up to 1.48× at batch size 4 and 1.06–1.15× at 256, mostly owing to skipping itself.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, expert skipping, efficient inference, large language models, KL distillation, second-order analysis, vLLM, CUDA kernels

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TaFoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed CCQ, which compensates for the skipped experts with three components --- absorber, energy-guided selection, and residual sketch. The method is implemented in vLLM with customized CUDA kernels and achieves lower KL to the full model than seven baselines across three MoE models.

Critical strengths
  1. The error decomposition analysis is interesting and well motivates the method. The three components are designed to address distinct parts of the error, and the ablations in Table 2 show that each component contributes. The authors also implement customized CUDA kernels.
  2. The authors comprehensively evaluate their method across various MoE-based models.
  3. The paper is clearly written and easy to follow.
Critical weaknesses
  1. The novelty of this paper is somewhat limited.
  • The speedup appears to come mostly from skipping itself rather than CCQ. CCQ is only 2-9% faster than plain top-K/2 skipping, owing to the custom kernels.
  • In the downstream benchmarks, CCQ is mostly not significantly different from the absorber alone or the LoRA baseline. The LoRA baseline is merged into the weights and adds no inference cost. Therefore, it is hard to argue that the KL improvements from selection and sketching translates into performance gain.
  1. The experimental setting is limited.
  • The method is evaluated only a fixed budget. Executing one more expert achieves much lower KL than CCQ (from Table 3), which raises the question of when a strict fixed budget is required in practice.
  • The models are relatively small, and all measurements are on single GPU. It is not verified whether the method remains effective for larger models or multi-GPU deployments.
Questions for authors

Are results available for multiple training seed? If so, what are the mean and standard deviation of the KL and benchmark scores?

Reviewer_8JB6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Paper describes an analysis they did on the errors observed by using only 50% of the recommended experts for a given MoE model as a method of reducing inference costs. They decomposed error into a "common" part (computed using the mean expert and scaling based on the dropped experts) and a "specific" part (the remainder). They were able to recover/predict the "common" portion of the error by utilizing the always-on shared expert.

Critical strengths

The paper addresses a common need, that of improving inference efficiency of MoE models by reducing the number of experts engaged at runtime. It carefully analyses errors from this and leverages findings from this analysis to propose a way to mitigate the approximation errors.

Critical weaknesses

The paper introduces terms like "common mode", "expert specific", "shared-expert readout", and "absorber" without clearly defining or explaining them. Figure 1 goes a long way to help understand at a high level, but the text is still confusing to follow for specifics.

Questions for authors

n/a

Reviewer_n9ZsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CCQ studies the error introduced by executing half of a fine-grained MoE model's routed experts. It decomposes the omitted output into a common component and expert-specific residual, and combines a trained low-rank absorber, predicted-energy-based expert selection, and a narrow sketch of an omitted expert. The analysis motivates the design through a conditional quadratic approximation. Experiments cover three MoE models, KL fidelity, component ablations, downstream benchmarks, and a single-GPU vLLM implementation with custom kernels.

Critical strengths
  1. The error decomposition is supported by informative diagnostics and ablations. Exact-component restoration in Figure 2 shows that the relative importance of common and expert-specific error varies across models. Tables 2 and 3 help identify the effects of compensation, selection, and sketching. The paper explicitly states that the quadratic analysis is conditional and that the deployed total-energy score is a proxy rather than a guaranteed optimum. These results provide useful understanding beyond an unexplained combination of components.

  2. The KL improvement is consistent across the three tested models and has meaningful controls. Table 1 reports CCQ/LoRA KL values of 0.0711/0.0848, 0.0397/0.0483, and 0.0690/0.0794, with paired document intervals supporting the differences. The compute-matched absorber-plus-routing frontier also favors CCQ by approximately 9.3%, 15%, and 11% at the reported routed-work budget. Real vLLM throughput measurements show that the complete method can retain an end-to-end speedup over the full model.

Critical weaknesses
  1. The practical value of the additional components relative to simpler trained compensation remains insufficiently established when their full cost is considered. The KL gains are valid evidence of improved fidelity and do not need to be dismissed merely because benchmark accuracy is similar. However, Table 4's CCQ/LoRA average scores are 69.3/69.1, 61.2/60.9, and 70.9/71.1, with all three average-difference intervals including zero in Table 14. Meanwhile, Table 5 reports 2.6, 2.6, and 6.0 GB of sketch weights, plus predictor/absorber parameters—roughly 8–9% of BF16 model-weight storage—and additional fitting work. The routed-compute-matched comparisons are useful, but are not a complete storage/latency comparison. The submission leaves the deployment conditions under which this fidelity improvement justifies the added resources unclear. This is a question about the demonstrated cost–benefit of the complete method, not a requirement to improve every downstream benchmark.

  2. The incremental serving benefit is confounded by implementation differences in the comparison to stock skipping. The 1.19–1.23× batch-one throughput relative to the full model largely reflects skipping experts. Appendix F states that part of the small-batch advantage over stock top-K/2 comes from custom MoE and small-GEMM kernels. Consequently, these timings do not isolate the marginal deployment value of selection and sketching relative to a simpler compensator on a comparable optimized execution path. The complete CCQ measurements are single-GPU; the reported stock-skipping multi-GPU experiments should not be interpreted as multi-GPU validation of CCQ.

Questions for authors
  1. At representative existing configurations, can the authors report a compact quality–throughput–memory comparison of CCQ, absorber-only, and LoRA using comparable optimized kernels where applicable? Please distinguish parameter storage from measured peak memory, and identify the time saved by skipping versus kernel implementation.

  2. Which measured deployment requirement or quality criterion favors adding selection and sketching once their full cost is included? The KL improvement is established, but a concrete operating-point comparison would help assess its practical significance. Please also clarify that the reported document-bootstrap intervals condition on one training seed and do not measure training variability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.