- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Super Apriel, a weight-sharing supernet offering four mixer choices per decoder layer. Stochastic distillation, surrogate-guided placement search, and targeted SFT enable multiple speed–quality operating points from one checkpoint. Reported presets achieve 2–10.7× decoding throughput with 97–77% average quality retention under long-output benchmarks.
Critical strengths
- A single checkpoint supports multiple operating points, with competitive speed–quality trade-offs demonstrated at 15B scale.
- The locally decomposable surrogate enables exact dynamic programming over the surrogate objective, avoiding exhaustive architecture evaluation.
Critical weaknesses
- Quality is measured in eager mode, while throughput uses CUDA graphs despite acknowledged numerical instability. The reported trade-offs therefore lack validation under one consistent execution configuration.
- Main speedups use single-preset, long-output throughput tests. Realistic mixed traffic, request-level switching, and latency overhead remain insufficiently evaluated.
- Headline averages exclude NIAH, while removing FA substantially harms precise long-context retrieval. Math-based placement scoring may not generalize across workloads.
- Training-regime ablations mainly use the 0.5B model. Compute-matched fixed-architecture baselines and alternative-search comparisons would better isolate gains from supernet training and placement optimization.
Questions for authors
n/a
6 comments
2. On the main results
W2. "Main speedups use single-preset, long-output throughput tests. Realistic mixed traffic, request-level switching, and latency overhead remain insufficiently evaluated."
Again, we refer to a detailed discussion in our response to Reviewer ZpYn. We distinguish two serving modes:
- Single-preset mode (used for all main benchmarks): Only the selected placement is on the GPU, so memory use and throughput match a dedicated 15B model. Changing presets takes 5-15 s: the mixer weights that change are copied from CPU to GPU and CUDA graphs are recaptured, with no reload from disk. This mode targets changes made by an operator at deployment granularity (per tier, per SLA, or as traffic shifts), which happen over minutes to hours.
- Supernet mode (Table 7): Several placements are kept on the GPU with pre-captured CUDA graphs, so a switch needs no weight transfer. The scheduler mechanism (same-placement batching) is implemented and was used for Table 7. The routing policy, i.e. deciding which placement should serve a given request, is left to future work.
Supernet mode is not required for our main contribution, which is a single training run that yields the full, navigable frontier, and this is fully achieved with single-preset mode.
Long generation is the realistic workload for the reasoning models we target: responses include thinking traces of tens of thousands of tokens. We report three generation lengths (8k, 16k, and 32k in Fig. 6), which show how the speedups depend on length. For prefill-dominated or short workloads, operators can select a more conservative preset, including all-FA, from the same checkpoint.
3. NIAH and math-based scoring
W3. "Headline averages exclude NIAH, while removing FA substantially harms precise long-context retrieval. Math-based placement scoring may not generalize across workloads."
All per-task results are reported in Tables 3-4. Below we recompute the all-tasks average with MMLU and NIAH (from Table 3) added back:
| Preset | Speedup | All-tasks avg (16, incl. MMLU & NIAH) | Retention | Published retention |
|---|---|---|---|---|
| all-FA | 1.0× | 74.3 | 100% | 100% |
| Idealized|All-18 | 2.0× | 71.4 | 96% | 97% |
| Reg|Lklhd-26 | 2.9× | 72.2 | 97% | 96% |
| Reg|Lklhd-18 | 4.8× | 68.2 | 92% | 94% |
| Idealized|All-6 | 6.1× | 62.5 | 84% | 88% |
| Idealized|Lklhd-6 | 6.2× | 63.7 | 86% | 90% |
| Reg|Lklhd-13 | 6.9× | 58.4 | 79% | 81% |
| Reg|Lklhd-10 | 10.7× | 55.2 | 74% | 77% |
Retention changes minimally, by at most about 4 points, and the reported trade-off is essentially unchanged. We will add this table to the revision. On the original exclusions: NIAH scores are mostly bimodal across presets (either very good or very bad). RULER, the graded long-context retrieval benchmark, is included in the headline average. The dev MMLU score is a 10-question-per-subject subsample scored by log-likelihood. After SFT it became unreliable: all-FA drops from 64.7 to 49.8, while held-out generative MMLU-Pro stays at 76.8 (teacher: 78.8). We agree that long-context retrieval degrades as FA layers are removed; we state this in the Limitations, and conservative presets keep most of it (e.g. Reg|Lklhd-26: NIAH 98.8, RULER 74.4 at 2.9x).
We agree that the choice of proxy tasks should be specific to target domain. Adapting to a new workload requires only re-scoring placements under that workload's objective (inference) and rerunning DP. No re-distillation/training, no new checkpoints. The math-centric proxy used in this work serves to demonstrate the method.
1. Eager vs. CUDA-graph mode
We thank the reviewer for their time and comments. We address each weakness below.
W1. "Quality is measured in eager mode, while throughput uses CUDA graphs despite acknowledged numerical instability. The reported trade-offs therefore lack validation under one consistent execution configuration."
We kindly refer the reviewer to our full answer, in our response to Reviewer ZpYn (1. Eager vs. CUDA graphs). In brief:
load_format=dummy, input length 1, 16k/32k output tokens), so they reflect kernel cost only and cannot be inflated by numerical errors in the outputs.Each quantity is therefore measured in the regime where it is meaningful and comparable across all models.