- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this manuscript, the authors explore channel selection for sparse MoE decoding without checkpoint-specific calibration. It separates channel ranking from the rule determining how many channels to retain, and proposes a reader combining ternary weight sketches, fixed-budget top-F selection, exact channel computation, and weight-derived low-rank recovery. Experiments compare fidelity, task performance, and decoding speed across four MoE models.
Critical strengths
-
The comparison between calibrated thresholds and fixed-budget selection is informative. Keeping Prox’s score while changing its selection rule helps distinguish ranking quality from budget allocation.
-
The method combines a compact screening representation with recovery of omitted outputs. The ablations examine both components, including settings where recovery prevents generation failures.
-
The evaluation covers multiple model families, generation tasks, and decoding budgets. It distinguishes analytical reads from measured speed and reports cases where Prox performs better or the proposed method slows decoding.
Critical weaknesses
-
The design and several constants were selected using OLMoE development data. The results support transfer without checkpoint-specific calibration, but do not establish that calibration is generally unhelpful. Calibrated recovery also improves fidelity in several settings.
-
Read counts are analytical rather than profiler measurements, and latency comparisons are against stock vLLM rather than competing sparse implementations. End-to-end speed is reported for only two models; Qwen3.5 is slower at batch one.
-
The post-trained models’ chat panels were used during method development. An independent fidelity evaluation would strengthen the cross-model conclusions, particularly where differences from Prox are small.
-
Prox achieves comparable or better task performance with fewer reads on several Qwen settings. The input-sparse extension is promising, but its advertised read savings are not realized by the current kernels. These distinctions should remain explicit in the headline conclusions.
Questions for authors
-
Can the post-trained models be evaluated on a fresh fidelity set after freezing all design choices and baseline allocations?
-
Can profiler measurements and comparisons with optimized sparse baselines establish whether analytical read savings translate into lower memory traffic and latency?
-
How sensitive are the results to sketch precision, recovery rank, and the Gaussian residual approximation? In particular, does the recovery-aware score consistently improve over a simpler point-estimate score?
3 comments
Response to Reviewer 294Z (2/2)
The benchmarks are already a held-out test.
No setting of our reader or of any baseline was chosen using benchmark scores; all were fixed on development data (Appendix B). To test fidelity the same way, we froze the recipe and evaluated each post-trained model on new text: 192 chat windows and 64 reasoning-trace windows at depths of 0, 500, and 1,000 tokens, disjoint from every fitting, tuning, and benchmark set. Prox's thresholds read different amounts on each panel, so we interpolate Prox's KL divergence, per document, between the two Prox settings that bracket our reads, and compare with paired standard errors over documents.
- At 17.8% of reads, our reader has lower KL than Prox in all 12 model-panel cells, by more than two standard errors. On chat: Gemma 4 −0.096 ± 0.005, Qwen3-Thinking −0.031 ± 0.003, Qwen3.5 −0.008 ± 0.001.
- At 12.8%, our reader's KL is lower on Qwen3.5 in all four panels. On Gemma 4, our reader's KL is lower in all four panels even than Prox's at its lowest setting, which reads 15% to 18% there. On Qwen3-Thinking, Prox's lowest setting reads 16% to 17%, so there is no matched comparison.
So on fresh held-out text, our weight-only reader keeps its fixed budget and reaches lower KL divergence than calibrated Prox in every matched comparison above. We will add these results to the appendix.
Prox matches our reader on the Qwen models only at reads it was not asked for.
On the two Qwen models, Prox reaches similar task accuracy to our reader when the two are compared at the reads Prox actually uses (Section 6.1, Appendix Figures 9 and 10). Yet those are not the reads Prox was calibrated for: calibrated for 13% of reads on Qwen3-Thinking, Prox reads about 6%, and calibrated for 18%, it reads 10% to 11%. At 6% of reads, Prox keeps under half of full-model accuracy on GPQA-Diamond and IFEval. A deployer sets the read budget in advance and cannot predict where a calibrated threshold will end up: calibrated thresholds read less than their budget in all 72 benchmark runs, by amounts that change with the model and task (Section 4.1). Top-k reads its budget for every token by construction, and at the budget that was set, the top-k version matches or beats the threshold version in 40 of 43 comparisons (Section 4.2).
The partial screen's savings are now measured.
The revised kernels implement the partial screen directly, so its read savings are realized on hardware and turn into decoding speedups. The Nsight Compute measurement in our first comment (18.3% measured against 17.8% counted) profiles this exact reader. With the partial screen in the kernels, the MoE block runs 1.60x to 1.90x faster than unmodified vLLM at batch one (Section 6.2), while keeping the fixed read budget on every token.
Sensitivity to sketch precision, recovery rank, and the Gaussian residual approximation.
At the heavy-sparsity budgets, no alternative we tested at equal reads beats the choices our reader makes.
- Recovery rank (192 held-out OLMoE documents). Rank 32 is the best tested rank at every budget, or within two standard errors of the best. A higher rank costs reads, so we compare ranks 64 and 128 with rank 32 that spends the same reads on more exact channels. Against ranks 64 and 128, rank 32 has lower KL divergence by 0.011 ± 0.003 and 0.030 ± 0.004 at 13.5% and 14.7% of reads, by 0.003 ± 0.002 and 0.011 ± 0.002 at 18.4% and 19.6%, and is level at 29.0% and 30.1%. Lower ranks free reads for more exact channels, yet ranks 8 and 16 are still worse by 0.003 to 0.009, except rank 16 at 17.8% (level).
- Sketch precision. At heavy sparsity, once its own reads are charged, a more precise sketch buys nothing. Replacing our ternary sketch with Prox's 4-bit weight copy matches our reader at the two lowest budgets and lowers KL divergence only from 28.5% of reads upward, at about twice the resident memory (Table 16). Before charging each screen's own reads (original version, Appendix E and Figure 4; OLMoE with 10% of channels computed exactly), the channels each screen selects carry 76.3% of the routed output energy with binary codes (5.2% of routed bytes), 81.2% with our ternary codes (7.8%), 83.2% with 2-bit codes (9.4%), and 86.1% with 4-bit codes (17.2%), against 86.4% when selecting by exact activations.
- Point estimate and the Gaussian residual. Without recovery, the point estimate is better on OLMoE development data: KL divergence 0.265 against 0.286 with 10% of channels computed exactly (original version, Table 6), and 0.547 against 0.603 with 5%. The revised reader therefore uses the point estimate at its two lowest budgets (Equation 1) and the recovery-aware score from 28.7% of reads. We will post the comparison with recovery at all four budgets in this thread.
We will add one sensitivity table (rank, precision, and score) to the appendix.
Follow-up: the promised results
We have uploaded a second revised version with the results promised in our reply. Table numbers below refer to it.
Measured reads on all three models (Appendix C, Table 7)
Gemma 4 and Qwen3.5 are now profiled the same way as Qwen3-Thinking. At batch one, Nsight Compute measures the routed experts' DRAM reads at 18.3% of the full model's on all three models, against the 17.8 to 17.9% we count. At batch four and eight, the measured share stays within 0.5 percentage points of our count.
Park et al.'s kernels on the same GPU (Appendix J, Table 23)
On Qwen3.5 at batch one, their MoE block runs at 0.67x the full model's speed and ours at 1.55x. Ours stays faster up to batch 32; theirs is faster at batch 64 and 128. As a whole model, ours is faster at every batch size up to 16. From batch 32 on, their kernels no longer fit in GPU memory, because their transposed copy of the down weights and their buffers take 24.2 GiB against our 5.1 GiB. On their protocol, our reader with the partial screen scores 69.6 at 12.8% of reads and 70.2 at 17.9%; the 69.3 and 69.9 in our reply came from the original version's reader.
Fresh held-out panels (Appendix I.2, Table 21)
The panel results from our reply are now in the appendix, cell by cell. At 17.8% of reads, our reader has lower KL divergence than Prox on all 12 model-panel pairs, beyond two standard errors.
Sensitivity (Appendix G, Table 18)
The table covers recovery rank, the score and sketch precision. On the Gaussian residual approximation: with recovery and at equal reads, the recovery-aware score that uses it is better only at 38.4% of reads. The two scores are level at 28.5%, and the point estimate has 16.5% and 10.0% lower KL divergence at 13.5% and 18.5%. Our reader uses the approximation only from 28.7% upward. One correction: the table ranks every sketch by the same score, so its output-energy values (binary 75.2%, ternary 81.0%, 2-bit 83.2%, 4-bit 86.1%) differ slightly from those in our reply, which mixed two scores.
This completes the results we promised earlier, thanks.
Response to Reviewer 294Z (1/2)
Thank you for the careful review. Section, table, and figure numbers refer to the revised version unless marked as the original version.
First, we want to establish that we consider a read budget as a contract. Top-k keeps it with no data. A calibrated threshold breaks it by an amount no one can predict, and buys no accuracy for it.
Constants chosen once are design, not calibration.
Calibration fits a cutoff or a map to a target model's activations, whereas our reader fits nothing to the target model. Its rotation block size, partial screen size, and recovery rank were chosen once on OLMoE development data and reused unchanged on Qwen3-Thinking, Gemma 4, and Qwen3.5 (Section 5). Those three settings are 128-coordinate rotation blocks, 80-coordinate partial screens, and rank-32 recovery. They transfer across models: on Gemma 4 development data, 80 coordinates again gives the lowest KL divergence at two of three budgets and is within 0.0003 of the lowest at the third (Appendix G). Prox's top-k version was set up the same way, with its input fractions chosen once on OLMoE, while every threshold was calibrated on each target model. This setup favors the threshold version, yet top-k still keeps its budget with no target-model data.
Lowering KL divergence does not translate to generation task accuracy.
Fitting a recovery map or layer budgets on calibration data lowers KL divergence on held-out text, yet neither improves task accuracy over a data-free rule.
First. Calibrated recovery lowers KL divergence on OLMoE (0.158 against 0.200 at 17.8% of reads). Yet in 18 benchmark comparisons on the three post-trained models, it never beats the weight-only map by more than two standard errors, and the weight-only map leads once (Qwen3-Thinking MMLU-Pro at 12.8% of reads, +2.1 ± 1.0).
Second. Calibrated per-layer budgets behave the same way (Appendix F.4). For channel selection itself, calibration delivers neither a known read budget nor an accuracy gain: calibrated thresholds read less than their budget in all 72 benchmark runs (Section 4.1), and top-k matches or beats them in 40 of 43 comparisons (Section 4.2).
Reads are measured, and speed is whole-model against the fastest stock path.
A hardware profiler confirms that, at batch one, our counted reads match the measured GPU memory traffic, and our kernels speed up both the MoE block and the whole model against unmodified vLLM.
Measured reads. The revised version measures GPU memory traffic with Nsight Compute on Qwen3-Thinking, over 12 of its 48 MoE layers, with the cache flushed before each kernel. At batch one, the MoE block without the router reads 13.9 MB per layer against 75.6 MB for the full model, or 18.3%, close to the 17.8% our sector count gives. The share rises to 21.8% at batch four and 25.7% at batch eight, because tokens in a batch select partly different channels (Appendix C, Table 7). We will add the same measurement for Gemma 4 and Qwen3.5.
Whole-model speed. Table 20 reports whole-model decoding speedups at batch sizes 1 to 128 on all three post-trained models, against unmodified vLLM in its fastest configuration at each batch size. Whole models decode 1.15x to 1.36x faster at batch one, up to 1.75x, 1.74x, and 1.88x at batch 8 or 16 (Qwen3-Thinking, Gemma 4, Qwen3.5), and 1.08x to 1.54x with 8k to 32k tokens of context (Section 6.2). Unmodified vLLM is a strong baseline: at batch one, its fused-MoE kernels stream weights at 83% to 88% of the read bandwidth we measured as achievable on this GPU (Appendix J).
Optimized sparse baselines. Park et al. are the only selector we compare with a public MoE implementation (Triton kernels in vLLM). Prox has not released code, and TEAL, CATS, and WINA were built for dense models. On Park et al.'s own zero-shot protocol on Qwen3.5 (original version, Table 16), their method needs 38.0% of reads to score 67.6 (68.4 with a per-layer threshold), while our reader scores 69.3 at 13.7% and 69.9 at 18.7%, against 71.2 for the full model. We will time Park et al.'s kernels against ours and unmodified vLLM on the same GPU and model, and post the result in this thread.