- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A prompt-conditioned RL controller allocates layer sparsity budgets during prefill, followed by deterministic module allocation and cached magnitude thresholds. Fused kernels apply token-dependent masks during decoding. Evaluation across five backbones reports improved sparse-model perplexity and average task accuracy, with up to 1.71× decoding speedup.
Critical strengths
- Moving allocation decisions into prefill addresses a practical source of decoding overhead.
- An independently trained prompt-free control isolates prompt conditioning more convincingly than baseline comparisons alone.
Critical weaknesses
-
The incremental value of RL is insufficiently established. Much of the pipeline overlaps with WAS and related approaches, while the prompt-conditioning control covers one backbone/budget setting. The evidence does not yet establish why RL is preferable to simpler allocation or optimized static scheduling.
-
Average gains conceal a substantial reasoning tradeoff. Severe GSM8K degradation at high sparsity limits the practical claim of preserving useful model quality. The distribution-matching objective's relationship to reasoning preservation remains unexplained.
Questions for authors
- Which capability is new relative to WAS-style budget optimization, module routing, and fused sparse decoding?
- Why is RL necessary for prompt-conditioned allocation rather than a simpler conditional predictor or an optimized static schedule?
- Does the prompt-conditioning benefit persist across backbones, sparsity levels, and training seeds under matched optimization budgets?
- Why does GSM8K degrade so sharply, and what does this reveal about the KL objective's ability to preserve reasoning?
- How do realized sparsity and threshold calibration change during long continuations when prefill thresholds remain fixed?
- How does the method compare with WAS under a matched decoder and total prefill-plus-decode cost?
7 comments
Response to W2 and Q4: Reasoning degradation and the role of KL guidance.
The GSM8K results show that reasoning degradation at high sparsity is shared by the evaluated methods. On LLaMA-2-7B at 75% sparsity, GSM8K accuracy falls from 14.03% for the dense model to 0.99% for TEAL, 1.36% for WAS, and 1.36% for ours. This pattern points to a broader challenge of aggressive activation sparsification and does not identify our prompt-conditioned allocation as the cause.
KL guidance helps the sparse model retain the dense model’s token predictions. Its benefit is directly supported by our ablation: removing KL increases WikiText-2 PPL from 11.80 to 13.52 and reduces nine-task average accuracy from 42.63% to 41.54%. Mathematical reasoning requires a sequence of correct steps, however, and an error at a critical step can change the entire continuation. Matching token distributions therefore improves overall prediction quality without ensuring that every reasoning chain reaches the correct answer.
The results establish two distinct findings: KL guidance improves sparse-model quality, while preserving mathematical reasoning at aggressive sparsity remains a shared challenge. Our gains concern overall quality at a given sparsity budget; reasoning-intensive applications require choosing that budget using task-specific accuracy.
Response to Q3: Evidence across models, sparsity levels, and training seeds.
The matched prompt-free experiment directly measures the contribution of prompt conditioning. On LLaMA-2-7B at 75% sparsity, both controllers are independently trained with the same data, objective, optimization settings, Micro Router, and decoder:
| Macro Controller | WikiText-2 PPL ↓ | Nine-task average ↑ |
|---|---|---|
| Prompt-free, learned fixed schedule | 11.68 | 41.04% |
| Prompt-conditioned | 11.80 | 42.63% |
Prompt conditioning improves average downstream accuracy by 1.59 percentage points, with a 0.12 increase in PPL. Appendix E adds evidence on LLaMA-2-13B: replacing per-prompt budgets with a fixed WikiText-derived configuration reduces BBH and PIQA accuracy by 0.23 and 1.36 points, respectively.
The broader evaluations establish the applicability of the complete method: it achieves the highest or tied-highest average accuracy among the compared sparse methods across five backbones and all three sparsity levels (60%, 65%, 75%). On the additional Qwen3-8B backbone, average accuracy exceeds TEAL by 0.74 and 1.54 points at 60% and 75%, respectively.
Results on Qwen3-8B.
The table below reports results across nine downstream tasks, with dense performance included as a baseline. All scores are percentages; AVG gives equal weight to each benchmark.
Benchmark Shots Dense (0%) TEAL (60%) Ours (60%) TEAL (75%) Ours (75%)
--------------------------------------------------------------------------------
MMLU 5 76.76 68.96 69.67 54.76 55.92
C-Eval 5 82.24 74.00 74.22 58.92 59.44
ARC-E 25 87.79 86.15 86.70 77.57 79.38
GSM8K 5 82.49 74.98 74.75 36.01 34.72
HellaSwag 10 79.83 72.73 74.55 56.23 59.90
BBH 3 54.38 47.61 48.03 34.93 36.55
WinoGrande 5 76.01 70.09 71.27 58.48 60.14
PIQA 0 79.05 78.35 78.40 75.14 73.56
BoolQ 32 87.71 81.93 83.88 65.50 71.83
--------------------------------------------------------------------------------
AVG 78.47 72.75 73.50 57.50 59.05
--------------------------------------------------------------------------------
These experiments provide direct evidence for conditioning in the matched setting and complementary evidence across models and budgets. The matched prompt-free comparison uses one training seed; variation across training seeds remains to be measured.
Response to W1 and Q1–Q2: Added capability and the role of RL.
The added capability is learning how to distribute the sparsity budget for each prompt while keeping budget allocation outside the decoding loop. WAS optimizes a shared block-wise schedule; our controller uses the current prompt to choose layer budgets, which the Micro Router converts into module budgets during prefill. The resulting thresholds are then reused during decoding. This gives input-dependent allocation without repeated policy evaluation or budget search for each generated token.
The matched static control directly demonstrates the value of this capability. With the same training data, objective, optimization settings, Micro Router, and decoder, adding prompt information improves nine-task average accuracy from 41.04% to 42.63% on LLaMA-2-7B at 75% sparsity. Appendix E provides complementary evidence on LLaMA-2-13B: reusing budgets from a fixed WikiText prompt reduces BBH and PIQA accuracy by 0.23 and 1.36 percentage points, respectively.
RL provides a direct way to train this conditional predictor from the consequences of its decisions. Layer budgets interact and must satisfy a shared budget constraint, while training data provide no “correct budget” labels. RL evaluates the resulting sparse model through dense-model KL feedback and learns the allocation policy without requiring labeled budget targets or gradients through the discrete sparsification procedure. At inference, the trained policy simply predicts budgets during prefill.