acceptodds
Under review as a conference paper at ICLR 2027

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

Abstract

Mixture-of-experts (MoE) language models decode slowly because every generated token must read its experts' weights from memory. Yet for a given token, a small share of each expert's channels carries most of its output. Methods that skip the other channels must choose which to compute before reading them, and the strongest tune thresholds on calibration data. We show that these data are unnecessary: replacing the calibrated thresholds of the strongest such method, Prox, by a fixed number of channels per token (top-k) changes its divergence from the full model by at most a quarter, whereas changing how channels are ranked changes it more than sixfold. Thresholds also make the cost depend on the input: on Gemma 4, Prox's thresholds read fewer bytes on reasoning than on their calibration data and collapse into repetition, which top-k selection with the same score avoids at the same reads. We therefore build a reader that ranks channels from the weights alone with a compact ternary sketch (7.8% of the expert bytes), computes the top-ranked channels exactly and recovers the rest with a low-rank map of the weights. Across four MoE models and nine baselines at equal memory reads, it is closer to the full model than Prox from 18% of expert reads up on OLMoE and Gemma 4 and level on the Qwen models, reads the same bytes on every input, and speeds up the MoE block by up to 1.9x over optimized vLLM. Channel selection can thus be applied to a new checkpoint from its weights alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, activation sparsity, channel selection, efficient inference, calibration-free, memory-bound decoding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_294ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this manuscript, the authors explore channel selection for sparse MoE decoding without checkpoint-specific calibration. It separates channel ranking from the rule determining how many channels to retain, and proposes a reader combining ternary weight sketches, fixed-budget top-F selection, exact channel computation, and weight-derived low-rank recovery. Experiments compare fidelity, task performance, and decoding speed across four MoE models.

Critical strengths
  1. The comparison between calibrated thresholds and fixed-budget selection is informative. Keeping Prox’s score while changing its selection rule helps distinguish ranking quality from budget allocation.

  2. The method combines a compact screening representation with recovery of omitted outputs. The ablations examine both components, including settings where recovery prevents generation failures.

  3. The evaluation covers multiple model families, generation tasks, and decoding budgets. It distinguishes analytical reads from measured speed and reports cases where Prox performs better or the proposed method slows decoding.

Critical weaknesses
  1. The design and several constants were selected using OLMoE development data. The results support transfer without checkpoint-specific calibration, but do not establish that calibration is generally unhelpful. Calibrated recovery also improves fidelity in several settings.

  2. Read counts are analytical rather than profiler measurements, and latency comparisons are against stock vLLM rather than competing sparse implementations. End-to-end speed is reported for only two models; Qwen3.5 is slower at batch one.

  3. The post-trained models’ chat panels were used during method development. An independent fidelity evaluation would strengthen the cross-model conclusions, particularly where differences from Prox are small.

  4. Prox achieves comparable or better task performance with fewer reads on several Qwen settings. The input-sparse extension is promising, but its advertised read savings are not realized by the current kernels. These distinctions should remain explicit in the headline conclusions.

Questions for authors
  1. Can the post-trained models be evaluated on a fresh fidelity set after freezing all design choices and baseline allocations?

  2. Can profiler measurements and comparisons with optimized sparse baselines establish whether analytical read savings translate into lower memory traffic and latency?

  3. How sensitive are the results to sketch precision, recovery rank, and the Gaussian residual approximation? In particular, does the recovery-aware score consistently improve over a simpler point-estimate score?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 294Z (1/2)

Thank you for the careful review. Section, table, and figure numbers refer to the revised version unless marked as the original version.

First, we want to establish that we consider a read budget as a contract. Top-k keeps it with no data. A calibrated threshold breaks it by an amount no one can predict, and buys no accuracy for it.

Constants chosen once are design, not calibration.

Calibration fits a cutoff or a map to a target model's activations, whereas our reader fits nothing to the target model. Its rotation block size, partial screen size, and recovery rank were chosen once on OLMoE development data and reused unchanged on Qwen3-Thinking, Gemma 4, and Qwen3.5 (Section 5). Those three settings are 128-coordinate rotation blocks, 80-coordinate partial screens, and rank-32 recovery. They transfer across models: on Gemma 4 development data, 80 coordinates again gives the lowest KL divergence at two of three budgets and is within 0.0003 of the lowest at the third (Appendix G). Prox's top-k version was set up the same way, with its input fractions chosen once on OLMoE, while every threshold was calibrated on each target model. This setup favors the threshold version, yet top-k still keeps its budget with no target-model data.

Lowering KL divergence does not translate to generation task accuracy.

Fitting a recovery map or layer budgets on calibration data lowers KL divergence on held-out text, yet neither improves task accuracy over a data-free rule.

First. Calibrated recovery lowers KL divergence on OLMoE (0.158 against 0.200 at 17.8% of reads). Yet in 18 benchmark comparisons on the three post-trained models, it never beats the weight-only map by more than two standard errors, and the weight-only map leads once (Qwen3-Thinking MMLU-Pro at 12.8% of reads, +2.1 ± 1.0).

Second. Calibrated per-layer budgets behave the same way (Appendix F.4). For channel selection itself, calibration delivers neither a known read budget nor an accuracy gain: calibrated thresholds read less than their budget in all 72 benchmark runs (Section 4.1), and top-k matches or beats them in 40 of 43 comparisons (Section 4.2).

Reads are measured, and speed is whole-model against the fastest stock path.

A hardware profiler confirms that, at batch one, our counted reads match the measured GPU memory traffic, and our kernels speed up both the MoE block and the whole model against unmodified vLLM.

Measured reads. The revised version measures GPU memory traffic with Nsight Compute on Qwen3-Thinking, over 12 of its 48 MoE layers, with the cache flushed before each kernel. At batch one, the MoE block without the router reads 13.9 MB per layer against 75.6 MB for the full model, or 18.3%, close to the 17.8% our sector count gives. The share rises to 21.8% at batch four and 25.7% at batch eight, because tokens in a batch select partly different channels (Appendix C, Table 7). We will add the same measurement for Gemma 4 and Qwen3.5.

Whole-model speed. Table 20 reports whole-model decoding speedups at batch sizes 1 to 128 on all three post-trained models, against unmodified vLLM in its fastest configuration at each batch size. Whole models decode 1.15x to 1.36x faster at batch one, up to 1.75x, 1.74x, and 1.88x at batch 8 or 16 (Qwen3-Thinking, Gemma 4, Qwen3.5), and 1.08x to 1.54x with 8k to 32k tokens of context (Section 6.2). Unmodified vLLM is a strong baseline: at batch one, its fused-MoE kernels stream weights at 83% to 88% of the read bandwidth we measured as achievable on this GPU (Appendix J).

Optimized sparse baselines. Park et al. are the only selector we compare with a public MoE implementation (Triton kernels in vLLM). Prox has not released code, and TEAL, CATS, and WINA were built for dense models. On Park et al.'s own zero-shot protocol on Qwen3.5 (original version, Table 16), their method needs 38.0% of reads to score 67.6 (68.4 with a per-layer threshold), while our reader scores 69.3 at 13.7% and 69.9 at 18.7%, against 71.2 for the full model. We will time Park et al.'s kernels against ours and unmodified vLLM on the same GPU and model, and post the result in this thread.

AuthorsOpenReview ↗

Response to Reviewer 294Z (2/2)

The benchmarks are already a held-out test.

No setting of our reader or of any baseline was chosen using benchmark scores; all were fixed on development data (Appendix B). To test fidelity the same way, we froze the recipe and evaluated each post-trained model on new text: 192 chat windows and 64 reasoning-trace windows at depths of 0, 500, and 1,000 tokens, disjoint from every fitting, tuning, and benchmark set. Prox's thresholds read different amounts on each panel, so we interpolate Prox's KL divergence, per document, between the two Prox settings that bracket our reads, and compare with paired standard errors over documents.

  • At 17.8% of reads, our reader has lower KL than Prox in all 12 model-panel cells, by more than two standard errors. On chat: Gemma 4 −0.096 ± 0.005, Qwen3-Thinking −0.031 ± 0.003, Qwen3.5 −0.008 ± 0.001.
  • At 12.8%, our reader's KL is lower on Qwen3.5 in all four panels. On Gemma 4, our reader's KL is lower in all four panels even than Prox's at its lowest setting, which reads 15% to 18% there. On Qwen3-Thinking, Prox's lowest setting reads 16% to 17%, so there is no matched comparison.

So on fresh held-out text, our weight-only reader keeps its fixed budget and reaches lower KL divergence than calibrated Prox in every matched comparison above. We will add these results to the appendix.

Prox matches our reader on the Qwen models only at reads it was not asked for.

On the two Qwen models, Prox reaches similar task accuracy to our reader when the two are compared at the reads Prox actually uses (Section 6.1, Appendix Figures 9 and 10). Yet those are not the reads Prox was calibrated for: calibrated for 13% of reads on Qwen3-Thinking, Prox reads about 6%, and calibrated for 18%, it reads 10% to 11%. At 6% of reads, Prox keeps under half of full-model accuracy on GPQA-Diamond and IFEval. A deployer sets the read budget in advance and cannot predict where a calibrated threshold will end up: calibrated thresholds read less than their budget in all 72 benchmark runs, by amounts that change with the model and task (Section 4.1). Top-k reads its budget for every token by construction, and at the budget that was set, the top-k version matches or beats the threshold version in 40 of 43 comparisons (Section 4.2).

The partial screen's savings are now measured.

The revised kernels implement the partial screen directly, so its read savings are realized on hardware and turn into decoding speedups. The Nsight Compute measurement in our first comment (18.3% measured against 17.8% counted) profiles this exact reader. With the partial screen in the kernels, the MoE block runs 1.60x to 1.90x faster than unmodified vLLM at batch one (Section 6.2), while keeping the fixed read budget on every token.

Sensitivity to sketch precision, recovery rank, and the Gaussian residual approximation.

At the heavy-sparsity budgets, no alternative we tested at equal reads beats the choices our reader makes.

  • Recovery rank (192 held-out OLMoE documents). Rank 32 is the best tested rank at every budget, or within two standard errors of the best. A higher rank costs reads, so we compare ranks 64 and 128 with rank 32 that spends the same reads on more exact channels. Against ranks 64 and 128, rank 32 has lower KL divergence by 0.011 ± 0.003 and 0.030 ± 0.004 at 13.5% and 14.7% of reads, by 0.003 ± 0.002 and 0.011 ± 0.002 at 18.4% and 19.6%, and is level at 29.0% and 30.1%. Lower ranks free reads for more exact channels, yet ranks 8 and 16 are still worse by 0.003 to 0.009, except rank 16 at 17.8% (level).
  • Sketch precision. At heavy sparsity, once its own reads are charged, a more precise sketch buys nothing. Replacing our ternary sketch with Prox's 4-bit weight copy matches our reader at the two lowest budgets and lowers KL divergence only from 28.5% of reads upward, at about twice the resident memory (Table 16). Before charging each screen's own reads (original version, Appendix E and Figure 4; OLMoE with 10% of channels computed exactly), the channels each screen selects carry 76.3% of the routed output energy with binary codes (5.2% of routed bytes), 81.2% with our ternary codes (7.8%), 83.2% with 2-bit codes (9.4%), and 86.1% with 4-bit codes (17.2%), against 86.4% when selecting by exact activations.
  • Point estimate and the Gaussian residual. Without recovery, the point estimate is better on OLMoE development data: KL divergence 0.265 against 0.286 with 10% of channels computed exactly (original version, Table 6), and 0.547 against 0.603 with 5%. The revised reader therefore uses the point estimate at its two lowest budgets (Equation 1) and the recovery-aware score from 28.7% of reads. We will post the comparison with recovery at all four budgets in this thread.

We will add one sensitivity table (rank, precision, and score) to the appendix.

AuthorsOpenReview ↗

Follow-up: the promised results

We have uploaded a second revised version with the results promised in our reply. Table numbers below refer to it.

Measured reads on all three models (Appendix C, Table 7)

Gemma 4 and Qwen3.5 are now profiled the same way as Qwen3-Thinking. At batch one, Nsight Compute measures the routed experts' DRAM reads at 18.3% of the full model's on all three models, against the 17.8 to 17.9% we count. At batch four and eight, the measured share stays within 0.5 percentage points of our count.

Park et al.'s kernels on the same GPU (Appendix J, Table 23)

On Qwen3.5 at batch one, their MoE block runs at 0.67x the full model's speed and ours at 1.55x. Ours stays faster up to batch 32; theirs is faster at batch 64 and 128. As a whole model, ours is faster at every batch size up to 16. From batch 32 on, their kernels no longer fit in GPU memory, because their transposed copy of the down weights and their buffers take 24.2 GiB against our 5.1 GiB. On their protocol, our reader with the partial screen scores 69.6 at 12.8% of reads and 70.2 at 17.9%; the 69.3 and 69.9 in our reply came from the original version's reader.

Fresh held-out panels (Appendix I.2, Table 21)

The panel results from our reply are now in the appendix, cell by cell. At 17.8% of reads, our reader has lower KL divergence than Prox on all 12 model-panel pairs, beyond two standard errors.

Sensitivity (Appendix G, Table 18)

The table covers recovery rank, the score and sketch precision. On the Gaussian residual approximation: with recovery and at equal reads, the recovery-aware score that uses it is better only at 38.4% of reads. The two scores are level at 28.5%, and the point estimate has 16.5% and 10.0% lower KL divergence at 13.5% and 18.5%. Our reader uses the approximation only from 28.7% upward. One correction: the table ranks every sketch by the same score, so its output-energy values (binary 75.2%, ternary 81.0%, 2-bit 83.2%, 4-bit 86.1%) differ slightly from those in our reply, which mixed two scores.

This completes the results we promised earlier, thanks.

Reviewer_2EGBOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates token-wise intra-expert channel selection to reduce memory bandwidth traffic during Mixture-of-Experts (MoE) decoding. Whereas prior methods rely on calibration data to tune per-layer activation thresholds or budgets, the authors decouple channel selection into a ranking score and a keep rule (calibrated threshold vs. fixed per-token top-kk). Holding the score of the strongest baseline, Prox (Liu et al., 2026), fixed and replacing its two calibrated thresholds with fixed top-kk selections ("Prox w/o calibration") changes KL divergence from the stock model by at most −17%-17\% to +24%+24\% across four MoEs, whereas changing the ranking score changes KL by up to 6.4×6.4\times (correlating at ρ=−0.96\rho = -0.96 with captured layer output energy). Furthermore, calibrated thresholds cause input-dependent read drift: on Gemma 4 26B-A4B-it, Prox thresholds calibrated to read 17.3% of routed bytes on chat drift down to 13.3% on GPQA-Diamond reasoning traces, causing 84.8% of outputs to degenerate into repetition loops (scoring 11.6% vs. 80.3% stock), whereas uncalibrated top-kk with Prox's own score at 13.4% reads achieves 62.1%.

Building on this insight, the authors design a weight-only channel reader ("Ours"):

  • Rotated Ternary Sketch Screen: Applies a block-128 randomized Hadamard transform RR and stores packed ternary codes (tb,c∈{−1,0,+1}t_{b,c} \in \{-1,0,+1\}), FP16 scale αb\alpha_b, and mean squared residual ρb\rho_b for each gate and up row (30 bytes per 128 coordinates, or 7.8% of routed BF16 bytes).
  • Global Top-FF Selection: Scores all N=KIN=KI candidate channels across a token's KK routed experts in output units by their expected squared omission error (Eq. 4, using 5-node Gauss-Hermite quadrature) and selects a single global top-FF per token.
  • Exact Completion & Weight-Only Low-Rank Recovery: Computes the FF selected channels exactly from native weights (with DeD_e stored channel-major) and approximates the N−FN-F omitted channels from their already-computed sketch activations pj=ϕ(g^j)v^jp_j = \phi(\hat{g}_j)\hat{v}_j via a rank-32 map UU formed by the top left singular vectors of the layer's stacked down projections DeD_e (costing 0.5–0.8% reads).

Evaluated in vLLM across OLMoE-1B-7B, Qwen3.5-35B-A3B, Qwen3-30B-A3B-Thinking-2507, and Gemma 4 26B-A4B-it against nine baselines under a 32-byte DRAM sector read model, the reader beats or matches Prox from 18% reads (P10) upward with half the resident state (8.4% vs. 15.6%) and speeds up the MoE block by up to 1.92×1.92\times (1.67×1.67\times at batch 1 on OLMoE; 1.38×1.38\times on Qwen3-Thinking) over tuned stock vLLM with CUDA graphs on an RTX PRO 6000.

Critical strengths
  • Compelling isolation of ranking vs. thresholding and discovery of threshold-drift repetition collapse: Factoring channel sparsity into a ranking score and a keep rule (Section 3, Section 6.1) yields a sharp empirical result: per-layer calibrated thresholds carry slightly less output energy than a simple per-token top-FF across every tested score (Table 7), while exposing decoding to severe read drift on out-of-distribution reasoning traces (Table 8) that triggers catastrophic repetition loops on Gemma 4 GPQA-Diamond and IFEval (Tables 1, 13, 14).
  • Hardware-aware cost modeling and effective weight-only low-rank recovery: Pricing memory accesses in 32-byte DRAM sectors on each method's best single layout explains why ungrouped coordinate sub-sampling fails on real GPUs (touching 88% of DRAM sectors for 19% logical reads, Appendix E). Moreover, recycling the already-computed sketch activations pjp_j of omitted channels through an SVD low-rank map of the down projections is highly effective: it requires zero calibration data, adds only 0.5–0.8% reads, reduces KL divergence by 10–44%, and lifts Gemma 4 GPQA-Diamond at P05 from 3.5% to 51.0% and IFEval at P10 from 81.3% to 93.0%. The vLLM integration and CUDA-graph benchmarks against tuned fused-MoE kernels are also well executed.
Critical weaknesses
  • At the sparsest budget (~13% reads, P05), "Prox w/o calibration" outperforms the implemented reader, while the "input-sparse screen" extension is only analytical: Because the full-row ternary sketch pays a fixed 7.8% of routed bytes up front, at P05 (13.4–13.7% total reads) only 5% of native channels can be completed. As a result, at ~13.4% reads on Gemma 4 GPQA-Diamond, "Prox w/o calibration" (which uses an input-sparse INT4 proxy) achieves 62.1% accuracy compared to 51.0% for "Ours" (Table 1), and on Qwen3-Thinking and Qwen3.5 below 16% reads, Prox reaches lower KL and matches task accuracy with 2.4–3.7 fewer points of reads. Although the authors plot "Ours + input-sparse screen" in Figures 2 and 3 to show strong quality at 12.8% reads (69.7% on Gemma 4 GPQA-D), Appendix D and Table 19 note that the coordinate-major sketch layout and gather kernel have not been implemented—the actual kernels still read the entire sketch (15.9% reads at P05) and run slower than "Ours" (1.57×1.57\times vs. 1.67×1.67\times in Table 19).
  • Batch-1 slowdown on fine-grained small-expert models and high relative sketch overhead under quantization: On Qwen3.5-35B-A3B (which routes 8 of 256 fine-grained experts of width I=512I=512, half of OLMoE's routed bytes per layer), fixed kernel and screening overhead outweigh the read savings at batch size 1, yielding a 0.85×0.85\times slowdown on the MoE block (and only 1.01×1.01\times on Gemma 4 at B=1B=1). Additionally, end-to-end TPOT speedups for Qwen3.5 and Gemma 4 are omitted from Table 2 and Table 19. Finally, because the 30-byte-per-128-coordinate sketch does not shrink when weights are quantized, its read cost doubles to ~15.6% on FP8 checkpoints and ~30% on INT4 checkpoints (where P10 reads ~43% of INT4 routed bytes, Appendix C).
Questions for authors
  • Clarification on combining "Prox w/o calibration" (or rotated 2-bit input-sparse screens) with weight-only recovery: Since "Prox w/o calibration" avoids threshold collapse and reaches 62.1% on Gemma 4 GPQA-Diamond at 13.4% reads even without a low-rank recovery path (beating "Ours" at 51.0%, Table 1), what happens if you attach your weight-only SVD recovery map (Section 4.4) and output-unit score directly to "Prox w/o calibration" or to the rotated 2-bit input-sparse proxy in Figure 4?
  • Clarification on missing end-to-end TPOT numbers in Table 19: Table 2 and Table 19 report end-to-end decode TPOT speedups for OLMoE and Qwen3-Thinking, but only report MoE-block speedups for Qwen3.5-35B-A3B and Gemma 4 26B-A4B-it. What are the end-to-end decode TPOT speedups across batch sizes B∈{1,…,128}B \in \{1, \dots, 128\} for Qwen3.5 and Gemma 4?
  • Suggestion for future work: Since Table 6 shows that the simple point estimate ϕ(g^j)2v^j2\phi(\hat{g}_j)^2 \hat{v}_j^2 matches or slightly beats the 5-node Gauss-Hermite quadrature score on OLMoE when no recovery bank is used (KL 0.265 vs. 0.286), replacing the quadrature in Equation (4) with the deterministic point-estimate residual wj2(nj2−∥bj∥2)pj2w_j^2 (n_j^2 - \Vert{}b_j\Vert{}^2) p_j^2{} could eliminate the variance parameter ρb\rho_b from the sketch (saving 2 bytes per block) and reduce screening compute at B=1B=1.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2EGB (1/2)

Thank you for the careful review and the concrete suggestions. The revised version adopts two of your suggestions and answers both questions. Section, table, figure, and equation numbers refer to the revised version unless marked as the original version. In the revised version, "Prox w/o calibration" is called "Prox's top-k version", the "input-sparse screen" is called the "partial screen", and "Ours" is called "our reader".

At the sparsest budget, our reader now leads, with the partial screen in the kernels.

The revised version implements the partial screen in the kernels, and our reader uses it at its two lowest budgets, 12.8% and 17.8% of reads. Its savings are measured on hardware: on Qwen3-Thinking at batch one, Nsight Compute measures the MoE block's GPU memory reads (without the router) at 18.3% of the full model's, close to the 17.8% we count (Table 7). On Gemma 4, our reader at 12.8% of reads scores 69.4 on GPQA-Diamond (four samples per question), 88.0 on IFEval, and 80.4 on MMLU-Pro. Prox's top-k version at 13.5% scores 56.3, 57.1, and 67.3 (Table 18). The 62.1 for Prox's top-k version on GPQA-Diamond in the original version came from a single sample, and our earlier comment on the submission lists all four per-sample scores from the rerun. Across the nine model-benchmark pairs, our reader at 12.8% of reads keeps 94.5% of full-model accuracy on average, against 87.0% for Prox's top-k version at 13.5% (Table 1).

Prox matches our reader on the Qwen models only at reads it was not asked for.

On the two Qwen models, Prox reaches task accuracy similar to our reader's when the two are compared at the reads Prox actually uses (Section 6.1, Appendix Figures 9 and 10). Yet those are not the reads Prox was calibrated for: calibrated for 13% of reads on Qwen3-Thinking, Prox reads about 6% and keeps under half of full-model accuracy on GPQA-Diamond and IFEval, and calibrated for 18%, it reads 10% to 11%. A read budget is a contract that a deployer sets in advance: top-k keeps it by construction with no data, whereas a calibrated threshold breaks it by an unpredictable amount and buys no accuracy for it. At the budget that was set, the top-k version matches or beats the threshold version in 40 of 43 comparisons (Section 4.2). On KL divergence, we froze the recipe and evaluated fresh held-out text, with Prox interpolated to our reads. At 17.8% of reads, our reader has lower KL divergence than Prox on both Qwen models in every panel, by more than two standard errors. At 12.8%, our reader has lower KL divergence than Prox on Qwen3.5 in all four panels, while on Qwen3-Thinking Prox's lowest setting reads 16% to 17%, so there is no matched comparison. More details are in our reply to Reviewer 294Z.

Our reader is now faster than optimized vLLM at batch one on every model, and Table 20 gives the whole-model speedups.

With the new kernels and the partial screen, the MoE block at batch one runs 1.60x faster than optimized vLLM on Qwen3.5 and 1.71x faster on Gemma 4, up from 0.85x and 1.01x in the original version. Table 20 reports per-token whole-model decoding speedups (time per output token) across batch sizes for all three post-trained models. At batch 1 / 2 / 4 / 8 / 16 / 32 / 64 / 128:

  • Qwen3.5: 1.15 / 1.22 / 1.47 / 1.65 / 1.88 / 1.74 / 1.32 / 1.24x
  • Gemma 4: 1.31 / 1.45 / 1.70 / 1.74 / 1.72 / 1.42 / 1.22 / 1.07x
AuthorsOpenReview ↗

Response to Reviewer 2EGB (2/2)

On quantized checkpoints, selecting channels beats dropping to fewer bits.

The sketch is sized for BF16 checkpoints, the setting this paper targets. It does not shrink with the weights, so next to 4-bit weights it takes a larger share of reads and of resident memory. The practical question is how to read less than a 4-bit checkpoint: quantize further or select channels. On OLMoE, our reader on GPTQ 4-bit weights reaches a KL divergence of 0.045 at 18.9% of BF16-equivalent reads, while AWQ 3-bit weights alone reach a KL divergence of 0.075 at 19.7% (paired difference −0.030 ± 0.003, Appendix G). So at about the same reads, selecting channels from 4-bit weights gives lower KL divergence than dropping to 3 bits. Matching the sketch's precision to quantized weights is a direct extension.

Both proposed combinations are now tested, and the second is our default reader.

Your second proposed combination is now our default reader at its two lowest budgets (12.8% and 17.8% of reads): a rotated partial screen paired with output-scaled top-k and weight-only recovery. Our reader keeps ternary codes rather than 2-bit codes for two reasons: a single sketch can serve every budget, and 2-bit codes need more resident memory, 9.4% against 7.8% of routed bytes (original version, Appendix E). The first combination is evaluated on held-out OLMoE data in Table 16. Pairing Prox's 4-bit weight copy with our output-scaled top-k and weight-only recovery matches our reader in KL divergence at 13.5% and 18.5% of reads (KL divergence 0.305 against 0.310, and 0.187 against 0.187), and has lower KL divergence from 28.5% upward (0.070 against 0.096, and 0.035 against 0.047), though it uses about twice the resident memory (2.16 GB against 1.0 GB next to the OLMoE checkpoint).

The point estimate is now used at the two lowest budgets.

Following your suggestion, our reader now ranks channels by the point estimate (Equation 1) at its two lowest budgets, 12.8% and 17.8% of reads. At these budgets it neither reads the per-block error ρ nor runs the quadrature, which gives the savings in reads and compute that you describe. From 28.7% of reads upward, the reader reads the full sketch and still uses ρ: it models the sketch's error with ρ and ranks channels by the expected squared output error each would leave after recovery if skipped, computed with five-node Gauss-Hermite quadrature (Section 5.2, Appendix G). So ρ stays in the sketch, but only these higher budgets read it. We will post the comparison of the two scores, with recovery, at all four budgets in this thread.

AuthorsOpenReview ↗

Follow-up: point estimate versus recovery-aware score, with recovery

As promised, here is the comparison of the two scores with recovery. It is Table 18(b) in Appendix G of the second revised version, where the point estimate is called the point score and the recovery-aware score the expected-error score.

The point estimate wins where our reader uses it

Both scores ran at equal reads on 192 held-out OLMoE documents, with the full sketch and the rank-32 recovery map. The point estimate gives 16.5% lower KL divergence at 13.5% of reads and 10.0% lower at 18.5%. At 28.5% the two are level. At 38.4% the recovery-aware score is better, and the point estimate has 14.2% higher KL divergence.

This is how the reader is already built: the point estimate at its two lowest budgets, and the recovery-aware score only from 28.7% of reads upward, where it is level or better. Your suggestion made the low-budget reader both cheaper and more accurate. Thank you for this suggestion.

Reviewer_wcF9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper analyzes the practice of improving inference efficiency for MoE models by approximating output by reading only a portion of each expert based on a pre-determined scoring methodology and calibrated threshold so that they don't have to load the entire expert weight into memory. They found that the scoring methodology dominated results and calibration was an unnecessary complication that could be replaced with simple top-k ranking for a pre-determined fixed k.

Critical strengths

Paper simplifies an existing method of inference optimization and justifies the simplification with empirical results.

Critical weaknesses

Improvement is relatively minor.

Questions for authors

n/a

2 comments
AuthorsOpenReview ↗

Response to Reviewer wcF9

Thank you for the review. Section, table, and figure numbers refer to the revised version.

Your summary states our main result: the standard calibration step can be replaced by top-k with a fixed k. That finding, not a small gain from a new method, is the claim the paper stands on. A read budget is a contract that a deployer sets in advance. Top-k keeps it with no data. A calibrated threshold breaks it by an amount no one can predict, and buys no accuracy for it.

A calibrated threshold provides neither a known read budget nor an accuracy gain over top-k.

Selectors calibrate a threshold on offline data to hit a target read budget and to gain accuracy at that budget, yet a calibrated threshold delivers neither. First, calibrated thresholds read less than their budget in all 72 benchmark runs (Section 4.1). Second, at the calibrated budget, each selector's top-k version matches or beats its threshold version on the same score in 40 of 43 comparisons (Section 4.2).

Where this claim is tested directly, by comparing a calibrated selector against its own top-k version, the effect of dropping calibration is large. On the weakest model-benchmark pair, Prox as published (calibrated for 13% of reads) keeps 0.0% of full-model accuracy, while Prox's top-k version keeps 61.1%. At 18%, the two keep 12.1% and 90.4% (Table 1, Figure 1). Reading too little does not explain this. On Gemma 4 GPQA-Diamond, where the full model scores 79.9, Prox with its threshold calibrated for 18% reads 13.3% and scores 11.7, while Prox's top-k version at about the same reads, 13.5%, scores 56.3 (Section 4.3).

Most of the accuracy gain comes from dropping calibration, and our reader builds the whole selector from the weights alone.

Comparing our reader with Prox's top-k version shows a smaller gap because removing calibration already recovers most of the lost accuracy, which is what our claim predicts. Across nine model-benchmark pairs on three post-trained models, Prox as published (calibrated for 13% and 18% of reads) keeps a mean of 51.5% and 68.5% of full-model accuracy. Replacing its calibrated threshold with top-k at the same budgets raises Prox's top-k version to 87.0% and 96.1%. Our reader reaches 94.5% and 98.1% at 12.8% and 17.8% of reads (Table 1).

Our reader shows that the entire selector can be built from the model's weights alone, with no calibration data. At heavy sparsity it keeps the highest accuracy, reaching 86.9% and 95.0% of full-model accuracy on the weakest pair, against 61.1% and 90.4% for Prox's top-k version (Table 1). Its new kernels also run the MoE block 1.60x to 1.90x faster than optimized vLLM at batch one (0.85x to 1.38x in the original version) and speed up whole-model decoding by up to 1.88x (Section 6.2).

The revised version is organized directly around these findings.

To make the paper easier to read, the revised version is structured around these results and adds one figure for each main finding (Figures 1 to 7). Our general comment above summarizes the changes, and we would appreciate a second look at the revised version.

Thank you again for your review. We uploaded a second revised version after our reply, and we would love to add one short note on your point that the improvement is minor.

Our claim is not a small gain from a better selector. Many earlier works on activation sparsity calibrate thresholds on the belief that a threshold adapts to each token and gives it the computation it needs, and we show that this belief is wrong, by showing that a calibrated threshold reads less than it was set to and gains nothing in accuracy.

So the calibration data that nearly every method requires turns out to be unnecessary, as your summary also puts it. That is the contribution we would be grateful to have the paper judged on. The reader is just an example of what a data-free reader can look like.

Thank you for your time. We would be glad to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.