acceptodds
Under review as a conference paper at ICLR 2027

Super Apriel: One Checkpoint, Many Speeds

Abstract

We introduce Super Apriel, a 15B-parameter supernet in which every decoder layer provides four trained mixer choices—Full Attention (FA), Sliding Window Attention (SWA), Kimi Delta Attention (KDA), and Gated DeltaNet (GDN). A placement selects one mixer per layer; placements can be switched between requests at serving time without reloading weights, enabling multiple speed presets from a single checkpoint. The shared checkpoint also enables speculative decoding without a separate draft model. The all-FA preset matches the Apriel 1.6 teacher performance; recommended hybrid presets span 2× to 10.7× decode throughput at 97% to 77% quality retention, with throughput advantages that compound at longer context lengths. With four mixer types across 48 layers, the configuration space is vast. A surrogate that predicts placement quality from the per-layer mixer assignment makes the speed–quality landscape tractable and identifies the best tradeoffs at each speed level. We investigate the convergence dynamics of the layer placement rankings. Rankings stabilize early at both scales, and while the ordering within the frontier placements stays somewhat volatile, the frontier set itself converges over distillation. Super Apriel is trained by stochastic distillation from a frozen Apriel 1.6 teacher, followed by supervised fine-tuning. We open-source the supernet weights, training code, vLLM serving code, and a placement optimization toolkit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
state space models, model efficiency, supernet, architecture search, inference efficiency, model distillation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_A3gTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Super Apriel, a weight-sharing supernet offering four mixer choices per decoder layer. Stochastic distillation, surrogate-guided placement search, and targeted SFT enable multiple speed–quality operating points from one checkpoint. Reported presets achieve 2–10.7× decoding throughput with 97–77% average quality retention under long-output benchmarks.

Critical strengths
  1. A single checkpoint supports multiple operating points, with competitive speed–quality trade-offs demonstrated at 15B scale.
  2. The locally decomposable surrogate enables exact dynamic programming over the surrogate objective, avoiding exhaustive architecture evaluation.
Critical weaknesses
  1. Quality is measured in eager mode, while throughput uses CUDA graphs despite acknowledged numerical instability. The reported trade-offs therefore lack validation under one consistent execution configuration.
  2. Main speedups use single-preset, long-output throughput tests. Realistic mixed traffic, request-level switching, and latency overhead remain insufficiently evaluated.
  3. Headline averages exclude NIAH, while removing FA substantially harms precise long-context retrieval. Math-based placement scoring may not generalize across workloads.
  4. Training-regime ablations mainly use the 0.5B model. Compute-matched fixed-architecture baselines and alternative-search comparisons would better isolate gains from supernet training and placement optimization.
Questions for authors

n/a

6 comments
AuthorsOpenReview ↗

1. Eager vs. CUDA-graph mode

We thank the reviewer for their time and comments. We address each weakness below.

W1. "Quality is measured in eager mode, while throughput uses CUDA graphs despite acknowledged numerical instability. The reported trade-offs therefore lack validation under one consistent execution configuration."

We kindly refer the reviewer to our full answer, in our response to Reviewer ZpYn (1. Eager vs. CUDA graphs). In brief:

  • The reported tradeoffs are internally consistent: All accuracy numbers are measured in eager mode, which is numerically stable, while throughput numbers use CUDA-graph mode, so all comparisons are apples-to-apples. External hybrids are evaluated under the same protocol (Appendix H).
  • Throughput does not depend on the instability: All throughput numbers, including every external baseline’s, are measured with CUDA graphs, vLLM’s default serving path. The measurements use dummy weights and a fixed generation length (Table 8: load_format=dummy, input length 1, 16k/32k output tokens), so they reflect kernel cost only and cannot be inflated by numerical errors in the outputs.
  • Eager-mode throughput would describe no real deployment: Linear mixers launch many small kernels per step, so eager execution is dominated by launch overhead and would understate our speedups. Furthermore CUDA-graphs are vLLM's default serving path for all models.

Each quantity is therefore measured in the regime where it is meaningful and comparable across all models.

AuthorsOpenReview ↗

2. On the main results

W2. "Main speedups use single-preset, long-output throughput tests. Realistic mixed traffic, request-level switching, and latency overhead remain insufficiently evaluated."

Again, we refer to a detailed discussion in our response to Reviewer ZpYn. We distinguish two serving modes:

  • Single-preset mode (used for all main benchmarks): Only the selected placement is on the GPU, so memory use and throughput match a dedicated 15B model. Changing presets takes 5-15 s: the mixer weights that change are copied from CPU to GPU and CUDA graphs are recaptured, with no reload from disk. This mode targets changes made by an operator at deployment granularity (per tier, per SLA, or as traffic shifts), which happen over minutes to hours.
  • Supernet mode (Table 7): Several placements are kept on the GPU with pre-captured CUDA graphs, so a switch needs no weight transfer. The scheduler mechanism (same-placement batching) is implemented and was used for Table 7. The routing policy, i.e. deciding which placement should serve a given request, is left to future work.

Supernet mode is not required for our main contribution, which is a single training run that yields the full, navigable frontier, and this is fully achieved with single-preset mode.

Long generation is the realistic workload for the reasoning models we target: responses include thinking traces of tens of thousands of tokens. We report three generation lengths (8k, 16k, and 32k in Fig. 6), which show how the speedups depend on length. For prefill-dominated or short workloads, operators can select a more conservative preset, including all-FA, from the same checkpoint.

AuthorsOpenReview ↗

3. NIAH and math-based scoring

W3. "Headline averages exclude NIAH, while removing FA substantially harms precise long-context retrieval. Math-based placement scoring may not generalize across workloads."

All per-task results are reported in Tables 3-4. Below we recompute the all-tasks average with MMLU and NIAH (from Table 3) added back:

PresetSpeedupAll-tasks avg (16, incl. MMLU & NIAH)RetentionPublished retention
all-FA1.0×74.3100%100%
Idealized|All-182.0×71.496%97%
Reg|Lklhd-262.9×72.297%96%
Reg|Lklhd-184.8×68.292%94%
Idealized|All-66.1×62.584%88%
Idealized|Lklhd-66.2×63.786%90%
Reg|Lklhd-136.9×58.479%81%
Reg|Lklhd-1010.7×55.274%77%

Retention changes minimally, by at most about 4 points, and the reported trade-off is essentially unchanged. We will add this table to the revision. On the original exclusions: NIAH scores are mostly bimodal across presets (either very good or very bad). RULER, the graded long-context retrieval benchmark, is included in the headline average. The dev MMLU score is a 10-question-per-subject subsample scored by log-likelihood. After SFT it became unreliable: all-FA drops from 64.7 to 49.8, while held-out generative MMLU-Pro stays at 76.8 (teacher: 78.8). We agree that long-context retrieval degrades as FA layers are removed; we state this in the Limitations, and conservative presets keep most of it (e.g. Reg|Lklhd-26: NIAH 98.8, RULER 74.4 at 2.9x).

We agree that the choice of proxy tasks should be specific to target domain. Adapting to a new workload requires only re-scoring placements under that workload's objective (inference) and rerunning DP. No re-distillation/training, no new checkpoints. The math-centric proxy used in this work serves to demonstrate the method.

Reviewer_YKUWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SUPERAPRIEL, a 15B-parameter supernet where the decoder layer contains four trained token-mixer choices: full attention, sliding-window attention, Kimi Delta Attention, and Gated DeltaNet. Training combines stochastic distillation, activation matching, forward and reverse KL losses, and targeted supervised fine-tuning on selected placements.

Critical strengths
  1. The problem is meaningful. The four trained mixer choices can enhance some related research.
  2. The method is easy to follow and the illustration is clear.
  3. The paper acknowledges meaningful limitations. RULER and other long-context retrieval tasks degrade as full attention is removed, and the authors discuss the approximately 67% parameter overhead of retaining four mixer options per layer.
Critical weaknesses
  1. The surrogate is interpreted too strongly. This is an engineering assumption, not a derived property of the network. No decay estimate, truncation-error bound, or held-out placement study is provided. The dynamic program is exact only for the fitted surrogate and additive cost model, not for the original network quality.

  2. The additive cost model is insufficiently validated. Per-mixer costs are fitted on one H100 setup and treated as functions of mixer counts rather than layer position, batch size, context length, kernel implementation, memory pressure, CUDA graphs, or concurrent placements.

  3. External baseline comparisons are not fully fair or causal. Falcon-H1, Apriel-H1, Qwen-3.5, and Nemotron-3-Nano differ in parameter count, training data, etc.

  4. The work is more like a eng

Questions for authors
  1. The paper should report held-out other results to analysis the assumption, and should call the result a surrogate-defined frontier unless stronger evidence is added.

  2. The authors should report more details and evaluate related costs.

  3. Stronger evidence would include matched fixed-hybrid and independently fine-tuned placements, equal active parameters and inference stack, and tuned heuristic, beam, evolutionary, or predictor-based NAS baselines under the same evaluation budget.

  4. This thesis is more of an engineering expolring.

5 comments
AuthorsOpenReview ↗

1. Surrogate interpretation, held-out evaluation, decay estimate, DP exactness

We thank the reviewer for their time and comments. Below, we address their concerns.

W1: "The surrogate is interpreted too strongly. This is an engineering assumption, not a derived property of the network."

If we understand correctly, the reviewer says that our justification for the feature locality assumption is motivated by engineering concerns (exact DP and tractable search) rather than a property of the network that derives it (as a cluster expansion in physics). We agree that locality is an empirical modeling assumption and cannot be derived from the network analytically. We do not impose it, however, but select it from the data. The expansion order and range are hyperparameters of the Bayesian ridge surrogate, chosen by maximizing the marginal likelihood. Higher-order and longer-range terms are admitted whenever the evidence supports them (Sec. 2.3, Appendix C). Additionally, the cross-validated analysis in Fig. 7 shows directly that higher-order terms add little predictive power across all cost levels.

“No decay estimate, truncation-error bound, or held-out placement study is provided.”

We provide held-out evaluation at three levels:

  • Cross-validation: Fig. 7 reports 5-fold cross-validated surrogate fit (Spearman ρ\rho and R2R^2), stratified by placement cost (n = 500 per bin, sampled from a 10k-placement exploration run).
  • Out-of-distribution placements: Placements proposed by the DP are unseen when the surrogate is fit. Fig. 3 compares their predicted frontier (blue) with the validated frontier (red), which measures out-of-distribution generalization. The two agree closely most of the time.
  • Canaries: At 0.5B scale, we also track 1,000 random held-out (OoD) canary placements throughout training (Fig. 9).

Decay estimate: The expansion-order sweep in Fig. 7 is an empirical decay estimate. Moving from allocation features to first-order features gives the largest gain at every cost level, and second- and third-order terms add progressively less. This directly measures how little longer-range interactions contribute.

"The dynamic program is exact only for the fitted surrogate and additive cost model, not for the original network quality." / "...should call the result a surrogate-defined frontier unless stronger evidence is added."

If we understand correctly, the point the reviewer is making is that there is a gap between the fitted surrogate and the real score of the proposed placement on the proxy tasks (and ultimately the downstream benchmarks). We agree and will adopt this terminology. We already distinguish “surrogate-predicted optima” from “validated optima” (Fig. 3), and the revision will use “surrogate-predicted frontier”.

AuthorsOpenReview ↗

2. Additive cost model

W2: “The additive cost model is insufficiently validated. Per-mixer costs are fitted on one H100 setup and treated as functions of mixer counts rather than layer position, batch size, context length, kernel implementation, memory pressure, CUDA graphs, or concurrent placements.”

We address this at multiple levels:

  • Mixer counts vs. layer position: All decoder layers have identical dimensions, so a given mixer performs the same computation at any depth. The measurements are consistent with this: a model based on counts alone fits measured throughput across 84 heterogeneous layouts with R2R^2 = 0.977 and a mean error of 7.6% (Appendix E, Fig. 10). On all 123 benchmarked placements, R2R^2 falls to 0.576. The deviations come from minority-mixer configurations, in which some mixer type occupies only one or two layers. We attribute this to how current serving stacks implement unified paging for hybrid models: mixing mixer types forces a shared page geometry through block-size inflation and padding [1, 2]. This engine-level behavior is outside the scope of the paper, and we expect it to improve as hybrid serving support matures. We therefore exclude such placements from the cost-model fit and from the search, which only admits placements with zero or at least three layers of each mixer type (Appendix E).
  • Dependence on serving conditions: We agree that per-mixer costs depend on batch size, context length, kernels, CUDA graphs, and memory pressure. The cost model’s only role is to guide the search. Every reported speedup is measured directly in vLLM (Fig. 6, Table 4, Appendix H) and none of the reported numbers is a cost-model prediction.
  • Robustness to the choice of cost model: Our paper contains an implicit ablation. The Idealized|* and Reg|* preset families were selected under two different cost models: a preliminary idealized model and the calibrated regression model (Appendix E). Both families produce competitive presets near the measured frontier (Fig. 5).
  • Adapting to new serving setups: The surrogate is a property of the trained weights and is fit once. The cost model, and every factor the reviewer lists, is a property of the serving setup. Recalibrating it for new hardware or a new serving stack takes about 100 benchmarked placements, after which the DP re-derives the full frontier from the same checkpoint in seconds on CPU. A fixed hybrid architecture, by contrast, commits to one cost regime before training. The dependence the reviewer identifies is therefore what this design is meant to accommodate.

[1] Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference, arXiv:2605.22416.
[2] Hybrid Models as First-Class Citizens in vLLM, PyTorch blog, Nov. 2025.

AuthorsOpenReview ↗

3. Baselines

W3: "External baseline comparisons are not fully fair or causal. Falcon-H1, Apriel-H1, Qwen-3.5, and Nemotron-3-Nano differ in parameter count, training data, etc."

We agree with the reviewer, but we clarify that we do not draw causal conclusions from them. All Pareto-frontier and retention claims compare presets of the same supernet, which share active parameters (~15B), the inference stack, and the evaluation protocol. External hybrids give context for absolute positioning only, and this is a common practice. We will state this explicitly and note that they differ in size, training data, and vLLM version (Table 8).

Q3: “Stronger evidence would include matched fixed-hybrid and independently fine-tuned placements, equal active parameters and inference stack, and tuned heuristic, beam, evolutionary, or predictor-based NAS baselines under the same evaluation budget.”

We would like to clarify the following:

  • Equal active parameters and inference stack: Every comparison between presets of our model meets both conditions.
  • Search baselines: We kindly refer to our response to Reviewer ZpYn on random, beam and heuristic search.
  • Matched fixed-hybrid baselines: We read this as a question about the cost of weight sharing compared with dedicated training. Our controlled 0.5B ablation (Sec. 3.3, Fig. 9) uses identical data. Fully stochastic training matches and eventually surpasses training targeted at the selected preset set, on those same presets. It also improves 1,000 held-out canary placements, which improve much more slowly under targeted training. At this scale, concentrating training on a set of presets did not beat stochastic weight sharing. This result motivated fully stochastic distillation at 15B. Extending it to a single dedicated architecture is an extrapolation, but we see no evidence that concentrating training further would reverse it. A 15B dedicated-hybrid control was beyond our compute budget, and we will list it as a limitation.
Reviewer_ZpYnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Super Apriel, a 15B model that supports different speed-quality trade-offs through a shared checkpoint with 4 mixer options that are available at each decoder layer. The model is trained though stochastic distillation and SFT while a Bayesian surrogate and dynamic programming are used in order to find suitable layer configurations. Also, the authors study the stability of placement rankings during training and explore the use of faster configurations as draft models for speculative decoding. SUPER APRIEL demonstrates that a single supernet checkpoint can serve multiple speed–quality operating points by training all mixer options at every layer. Surrogate search enables efficient navigation of the placement space, and targeted SFT preserves quality across the Pareto frontier, thus shifting the throughput–quality tradeoff from a release-time commitment to runtime control.

Critical strengths
  1. Practical Deployment Focus: The paper addresses a highly practical deployment challenge by enabling diverse speed-quality trade-offs within a single, shared model checkpoint.

  2. Scale and Promising Margins: The approach is demonstrated at a meaningful 15B parameter scale using extensive distillation and fine-tuning, yielding strong operating points such as 96% quality retention at a 2.9× speedup.

  3. Tractable Optimization Space: The introduction of a surrogate-based placement search paired with dynamic programming effectively makes a massive configuration space manageable, backed by a helpful analysis of layer placement stability during training. The evaluation across several benchmarks, including held-out tasks, also strengthens the experimental study.

Critical weaknesses
  1. Mismatched Execution and Evaluation Settings: The reported accuracy and speedup metrics are measured under conflicting execution modes. Accuracy is evaluated using eager mode due to numerical instability, whereas throughput numbers rely on CUDA graphs. Consequently, it remains unverified whether the model can simultaneously achieve the reported throughput speedups while preserving its claimed accuracy profile.

  2. Unproven Deployment Flexibility under Realistic Workloads: The practical utility of the flexible serving mechanism is not fully demonstrated. The 5–15 second overhead required to switch presets limits real-world adaptability, and per-request routing is still under development. Furthermore, the throughput benchmarks focus heavily on long generations, leaving performance under more realistic or mixed workloads unclear.

  3. Under-Justified Optimization Advantages: The explicit benefits of the proposed surrogate-based search method over simpler heuristics have not been rigorously established. The evaluation lacks baseline comparisons against random or standard heuristic search strategies under an identical evaluation budget. Additionally, the underlying cost model struggles on certain configurations, calling into question its reliability across the full search space.

  4. Averaging Effects and Indirect Performance Measurements: The reliance on macro-averaged results appears to obscure significant localized performance degradation, particularly on long-context retrieval and agent benchmarks. A standalone, held-out evaluation would clarify these architectural quality trade-offs. Additionally, the speculative decoding gains require end-to-end wall-clock runtime measurements rather than isolated simulation metrics to confirm the claimed acceleration.

Questions for authors

Please provide empirical or conceptual clarifications for the following:

  1. Simultaneous Accuracy and Speedup Validation: Can you provide accuracy numbers when running the model with CUDA graphs, or conversely, throughput measurements when running in eager mode? Given the noted numerical instability in eager mode, it is important to confirm that the reported throughput gains do not compromise model accuracy.

  2. Baseline Search Comparisons: How does your surrogate-based placement search compare against simpler alternatives, such as random search or basic configuration heuristics, when restricted to the same total evaluation budget?

  3. Serving Overheads and Workloads: How do you envision mitigating the 5–15 second preset switching latency in a production serving environment? Additionally, could you provide performance numbers under more varied workloads beyond long-generation tasks (e.g., short-context or balanced prompt/generation budgets)?

  4. End-to-End Speculative Decoding: Could you provide end-to-end wall-clock speedup measurements for the speculative decoding configurations to verify that the claimed acceleration holds in a realistic generation pipeline?

6 comments
AuthorsOpenReview ↗

1. Eager vs. CUDA graphs

We thank the reviewer for their time and effort spent reviewing our manuscript. We address each of their concerns in the following one by one.

1. Eager vs. CUDA graphs

“... it remains unverified whether the model can simultaneously achieve the reported throughput speedups while preserving its claimed accuracy profile.”

Why accuracy is measured in eager mode: Current CUDA-graph implementations of linear-mixer kernels have numerical issues in generation, so reliable accuracy numbers can only be obtained in eager mode, see e.g. OLMo-Hybrid (arXiv:2604.03444) who also adopt --enforce-eager for accuracy evaluation for the same reasons. The weights and the function they compute are identical, the issue is implementation maturity of specific kernels that we hope will go away with broader adoption.

“... Given the noted numerical instability in eager mode, it is important to confirm that the reported throughput gains do not compromise model accuracy.”

Numerical instabilities were noted in CUDA-graph mode (as in Olmo hybrid), not eager; eager is the numerically stable mode in which all the accuracies were measured, measuring them in cuda-graph mode would be unreliable. Hence note that all accuracy numbers are measured in eager mode and all throughput numbers use CUDA graphs — both internally consistent. Hence all headline claims are are apples to apples.

"... or conversely, throughput measurements when running in eager mode?"

Why eager-mode throughput would be meaningless: linear mixers execute many small kernels per step, so in eager mode CPU launch overhead is substantial. The full attention with fused kernels is less affected by this. These mixers are designed to exploit modern CUDA-graph capture-and-replay. Eager-mode throughput would therefore systematically understate our speedups while describing no real deployment scenario — hence meaningless.

These are seemingly well known issues at this point. Beyond Olmo-Hybrid, we kindly refer to the following sources for more details on this topic:

[1] https://github.com/triton-lang/triton/issues/2637

[2] PyTorch blog: Accelerating Mamba2 with Kernel Fusion (2026)

Overall, we report each quantity in the regime where it is meaningful: accuracy in the numerically reliable mode, throughput in the deployed mode. We don’t see how any other way to report it would make it more meaningful.

AuthorsOpenReview ↗

2. Unproven Deployment Flexibility under Realistic Workloads

We thank the reviewer for raising this point. We believe this concern is partially a result of a misalignment between the statements made in the abstract, abstracts says "can be switched between requests at serving time," which is a claim about per-request mode (supernet serving mode), while the headline evidence supports deployment-time selection (single-preset mode). As discussed in the answer to the reviewer FXTj (W1), we will correct this formulation in the revised pdf.

The goal of this work is deployment-time flexibility, not necessarily serving time: serving multiple speed–quality operating points from a single checkpoint. We emphasize that this goal under any serving mode, including the two we consider (single-preset and supernet mode), first requires the ability to obtain multiple presets with distinct throughput–accuracy characteristics — which is the main contribution of this work.

“...The practical utility of the flexible serving mechanism is not fully demonstrated.”

We want to clarify again our two serving modes detailed in Appendix G*:

(1) Single-preset mode — the primary deployment mode, used for all throughput benchmarks in the paper unless stated otherwise. Only the selected placement is GPU-resident. This is where the 5-15 seconds overhead applies.

(2) Supernet mode — the per-request-switchable extension, with all “registered” placements preloaded on GPU. Importantly, supernet-mode is not essential for the core contributions. Single-preset mode already delivers the central value proposition -- flexible navigation of the cost-quality frontier from one checkpoint for the realistic deployment scenarios: choosing an operating point per deployment tier, per customer SLA, or as traffic shape shifts over the course of a day. These all change on a timescale of minutes-to-hours, well within what the single-preset "switch" mode handles (order of tens of seconds reload + graph recapture).

* - we realized that out mode description in Appendix G is somewhat suboptimal, e.g. it describes the single-preset switch mechanism under a heading that names the other mode, we will correct it in the paper.

Furthermore, we kindly point the reviewer to an extensive elaboration of the practical utility of the proposed approach and serving methods in our response to Reviewer FXTj (Q1). These include "one training run instead of N conversions", "optimal placements for any cost target, chosen for a particular deployment", "a way to choose presets for new domains, with or without labeled data for speed of performance maximization" and "cheaper co-residency when needed".

“...The 5–15 second overhead required to switch presets limits real-world adaptability, and per-request routing is still under development.” “... How do you envision mitigating the 5–15 second preset switching latency in a production serving environment?”

As mentioned above, deployment scenarios we envision here are choosing an operating point per deployment tier, per customer SLA, or as traffic shape shifts over the course of a day. These all change on a timescale of minutes-to-hours, well within what the single-preset "switch" mode handles.

Now, for deployment scenario where faster switching is required e.g. per request switching (routing itself is an open research question), this is where we envision supernet serving mode with all registered presets pre-loaded on the GPU.

To summarize the scope: (a) for preset decisions at deployment granularity (the scenarios above), selection is fully supported today — an operator sets a cost/quality target, and surrogate search returns the best placement; (b) for per-request serving, the mechanism is implemented, while the routing policy — deciding, from request content, which placement should serve it — is an open research question in its own, which we explicitly scope as future work. We will make this scoping explicit in the revised manuscript.

“... Furthermore, the throughput benchmarks focus heavily on long generations, leaving performance under more realistic or mixed workloads unclear.” “... Additionally, could you provide performance numbers under more varied workloads beyond long-generation tasks”

Long generation is the realistic workload for this model class: Super Apriel is a reasoning model whose responses include thinking traces of tens of thousands of tokens — this is what we benchmark. This matches the motivation for hybrid architectures generally — long-sequence patterns as the driving workloads. For short-context, short-generation requests there is comparatively little inference cost to reduce; the supernet addresses mixed conditions by construction, since we can select presets suited to each workload from the same checkpoint including all-FA, which is likely better suited for prefill dominated workloads. The length-dependence of the speedups is also shown explicitly in the experiments.

AuthorsOpenReview ↗

3. Under-Justified Optimization Advantages

“... The evaluation lacks baseline comparisons against random or standard heuristic search strategies under an identical evaluation budget. “

“... How does your surrogate-based placement search compare against simpler alternatives, such as random search or basic configuration heuristics”

Random search. We considered evaluating explicit random-search or beam search, but judged it as not very informative. Fig. 3 at every cost level, the vast majority of evaluated placements (grey points) — and of the full surrogate-scored space (grayscale density) — lie far below the frontier: frontier-quality placements form a thin tail of the score distribution. With any realistic budget (in a space of 4^48), the expected best of a random sample falls in the bulk, not in the tail. That's also why prior work on mixer placement does not use random search either — Jet-Nemotron (arXiv 2508.15884) employs beam search. In our procedure, the frontier is reached by surrogate proposal, not by sampling: initial sampling locates the bulk, and the surrogate-guided rounds plus DP climb from the bulk to the frontier.

Comparison to guided search (beam). Relative to Jet-Nemotron's beam search, our surrogate + cost-constrained DP delivers the optimum for all cost constraints in a single execution, yielding a full frontier rather than a single operating point for a fixed cost; it is unclear whether beam search extends efficiently to multiple cost targets.

Standard heuristics (uniform FA interleaving, attention at first/last layers, greedy layer swapping). Our results indicate optimal placements are not heuristic-shaped: attention concentrates in mid-network layers rather than at uniform intervals or at the ends (Fig. 3, bottom panels), and the eviction order FA→SWA→KDA→GDN emerges from the search — not fixed prior rule encoded in advance. We also note that heuristic placements are represented in our evaluation: the external hybrid baselines instantiate a fixed, hand-designed mixer layout given the full benefit of dedicated end-to-end training. Our searched presets dominate the most directly comparable case — Apriel-H1, a fixed-layout hybrid distilled from a teacher in the same model family. These comparisons are not fully controlled of course (the baselines differ in size and training data), which is why we point to the within-model evidence that optimal placements are not heuristic-shaped (Fig. 3, bottom panel).

Reviewer_FXTjOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Super Apriel, a 15B supernet distilled from Apriel 1.6 in which every layer offers four trained token mixers (FA, SWA, GDN, KDA), so one checkpoint can serve multiple speed–quality presets switchable at deployment time. Training combines stochastic per-layer mixer sampling during distillation with targeted SFT on selected presets, and placements are chosen with a cluster-expansion Bayesian surrogate optimized exactly via cost-constrained Viterbi DP. The all-FA preset matches the teacher, and hybrid presets reach 2.0×–10.7× decode speedup at 32k context with 97%–77% quality retention. Super Apriel outperforms the comparable Apriel-H1 but is dominated by Nemotron-3-Nano 30B, and the paper adds ranking-stability ablations and a self-speculative decoding analysis.

Critical strengths

1. A working system at a scale that matters. The authors train a 15B reasoning model in which every layer carries four mixers, and the all-FA path still matches the teacher (74.2 vs. 73.9 average). Adding the extra mixers therefore does not seem to hurt the original model. The hybrid presets give a fairly smooth trade-off, e.g. 96% of the quality at 2.9× and 94% at 4.8× decode speedup at 32k. They also clearly beat Apriel-H1, the closest comparable distilled hybrid (71.1 at 2.9× vs. 58.4 at 2.0×). Most prior conversion work ships a single fixed hybrid, so showing that one set of weights can cover this whole range is the main reason I lean toward acceptance. The planned release of the weights, vLLM plugin and search code would make the work directly usable by others.

2. The placement search is principled and cheap. Modeling placement quality as a cluster expansion fitted with Bayesian linear regression yields a chain-structured model. As a result, the best placement for every mixer allocation can be found exactly in a single Viterbi pass that takes seconds on CPU. Unlike the beam search in Jet-Nemotron, this covers the full cost range rather than a handful of points. Selecting the expansion order by model evidence is also a sensible guard against overfitting when evaluations are limited. The ranking-stability results add practical value: Spearman ρ exceeds 0.98 from about 3% of training at both 0.5B and 15B, which suggests the search can be run early. I expect this method to be reusable beyond this particular model, which is why it carries weight in my assessment.

Critical weaknesses

1. The serving claim in the title is not what is measured. The paper's main pitch is that one checkpoint can switch between speed presets at serving time. However, every headline speedup (2.0×–10.7×) comes from single-preset mode, where only one placement is loaded and the rest sits on CPU. Changing presets in that mode means moving weights and recapturing CUDA graphs, which takes 5–15 s. Training one supernet instead of several hybrids is a real saving, but at serving time this mode offers little over storing a few separately converted models. The supernet mainly saves disk space for the shared FFNs, at the cost of ~67% extra parameters. When two placements are actually kept on GPU (Table 7), the speedup over all-FA drops to 1.36–1.80×. The ordering also flips: the cheaper presets (Idealized|All-6, Idealized|Lklhd-6) end up slower than Idealized|All-18. Per-request routing is described as still under development. Since runtime flexibility is the stated motivation, this gap between claim and evidence is the issue that weighs most on my assessment.

2. The quality-retention numbers are likely optimistic, and a key control is missing. Placement search optimizes the log-likelihood of math traces, and the reported averages (Table 1, Fig. 5) leave out MMLU and NIAH. NIAH is exactly where the hybrids break down. It drops from 100 for all-FA to 25–57 for presets at 4.8× and above, and RULER falls from 79.4 to 48.6 for the fastest preset. The "97%–77% retention" figure therefore mostly reflects math and short-context tasks. Several dev tasks were also used for preset selection. In addition, AIME scores come from single greedy runs on 30 problems, where one question moves the score by 3.3 points. Finally, there is no comparison against a dedicated hybrid with the same layout trained on the same budget. The single-placement SFT run uses a different SFT mix, so it does not settle this. Without that control it is hard to tell how much quality the weight sharing costs, and that is what determines whether the supernet approach is worth adopting.

Questions for authors

Clarifications and additional results that could change my assessment

  1. Supernet-mode serving. What sequence length does Table 7 use, and how do these numbers compare to single-preset throughput for the same presets at the same length? Why are the cheaper presets (Idealized|All-6, Idealized|Lklhd-6) slower than Idealized|All-18 in this mode? Is it only reduced KV-cache capacity and batch size? It would also help to see throughput with all 8 presets resident and the actual cost of switching between them. If supernet mode cannot get reasonably close to the single-preset numbers, what is the practical advantage over shipping a few separately converted hybrids?

  2. Cost of weight sharing. Can you compare at least one preset (e.g. the Reg|Lklhd-18 layout) against a dedicated hybrid with the same layout, distilled and fine-tuned on the same token budget? The existing single-placement SFT run uses a different SFT mix (SFT*), so I cannot read the effect of sharing from it.

  3. Full averages and variance. Please report Table 1 averages that include NIAH and MMLU, so the retention percentages reflect long-context retrieval as well. For the dev AIME24/25 results, can you give multi-seed averages or confidence intervals? Many of the gaps between presets are within one or two questions.

  4. Proxy validity. How well does the trace log-likelihood score S(x) correlate with downstream accuracy across placements, beyond the 8 presets? The rank-stability analysis is entirely on the proxy. One example is puzzling: Reg|Lklhd-13 (6.9×) scores 60.2, while Idealized|Lklhd-6 (6.2×) scores 66.8. This suggests that the choice of proxy and cost model affects the frontier more than the paper indicates.

  5. Speculative decoding. Are the speedups in Fig. 12 measured end-to-end in vLLM, or computed from estimated acceptance rates combined with the cost model? What draft length γ is used?

5 comments

We thank the reviewer for the time they spent reviewing our manuscript. We address each of their concerns in the following one by one.

W1: "The serving claim in the title is not what is measured... Since runtime flexibility is the stated motivation, this gap between claim and evidence is the issue that weighs most on my assessment."

We agree the emphasis on serving-time flexibility is miscalibrated (overemphasizes the importance of the supernet serving mode) and will adapt the abstract and the intro accordingly. In abstracts we say "can be switched between requests at serving time," which is about per-request mode (supernet serving mode), while the headline evidence supports deployment-time selection (single-preset mode). We will revise abstract and conclusion and state the claim as "deployment time flexibility” : placement is selected at deployment time from a single supernet checkpoint and can be swapped as deployement constraints (SLA< server load, task type etc.) change --- more on it in Q1. Supernet serving mode (per-request) is a supported extension and should not be load bearing for the contributions, the routing policy for sueprnet serving mode is future work. We note the contribution is unchanged by this: deployment-time flexibility at any granularity first requires obtaining the presets, and obtaining the entire frontier in one training run — the saving the reviewer acknowledges as real — is the paper's central claim.

W2: "The quality-retention numbers are likely optimistic, and a key control is missing."

We answer the averaging and variance parts under Q3, proxy selection under Q4, and the dedicated-hybrid control under Q2. One point here: the dev/held-out split exists to control selection effects — the held-out tasks (MMLU-Pro, GPQA, HLE, LCB, τ²-Bench, IFEval, AIME-NV) were never used for any training or selection decision (Appendix J).

Q1: Supernet-mode serving.

*What sequence length does Table 7 use, and how do these numbers compare to single-preset throughput for the same presets at the same length? *

Table 7 uses the 16k setting from Table 8 (input length 1, 16,000 output tokens, 1,000 prompts). The 926 tok/s all‑FA reference is the 16k single-preset number. We will add this to the caption.

“Why are the cheaper presets (Idealized|All-6, Idealized|Lklhd-6) slower than Idealized|All-18 in this mode?”

In this mode, throughput depends on the union of resident mixer weights, so it depends on how much a preset overlaps with the co-resident all‑FA placement. Idealized|All‑18 keeps 13 FA layers, whose weights it shares with all‑FA. Only 35 of its 48 layers (32 SWA and 3 linear) add extra mixer weights. Idealized|All‑6 and Lklhd‑6 have no FA layers, so all 48 of their layers add mixers, 18 of which are linear. A larger union leaves less memory for the KV cache, which means smaller batches (concurrency level) and lower throughput at saturation.

“If supernet mode cannot get reasonably close to the single-preset numbers, what is the practical advantage over shipping a few separately converted hybrids?”

As discussed under W1, the paper’s main contribution is flexibility at deployment time. Switching per request at serving time (with the supernet mode) can be seen as a secondary feature. Supernet mode implements the runtime preset switching but it should not be seen as the central contribution or claim. Importantly, even with supernet mode, routing each request automatically based on its content (choosing a placement from properties of the prompt or the task type) is not yet solved and is left for future work.

As discussed in W1 we will revise this and clarify the scope.

As for the advantages, compared with separately converted hybrids, the supernet offers:

  • One training run instead of N conversions: Fixed-architecture hybrids such as Apriel‑H1 need a separate conversion run for each variant (i.e., each cost target), and each architecture must be fixed before distillation.

  • Optimal placements for any cost target, chosen for a particular deployment: Supernet offers the ability to extract a placement for a given deployment’s constraints (e.g., SLA, server load, short vs. long context, domain, etc.). No fixed set of converted models can offer this.

  • A way to choose presets for new domains, with or without labeled data for speed of performance maximization. We see preset selection varying along two axes: (i) whether annotated benchmarks exist for the target domain, and (ii) whether the deployment has a quality floor (maximize throughput) or a throughput/latency budget (maximize quality). Both of these features for present selection are supported.

    • (i) With annotated benchmarks that are cheap to evaluate, they enter the search objective directly; if benchmarks are expensive, the search runs on teacher-trace log-likelihood and the top candidates are validated on the benchmark. Without labels, teacher log-likelihood on domain prompts (or speculative-decoding acceptance rate against the all-FA path) serves as the objective.
    • (ii) Throughput-constrained: the DP returns the best placement for all cost budgets. Because a single DP covers every budget, the operating point can be re-selected immediately whenever SLAs or server load etc. change, with no retraining. Quality-constrained: select the fastest placement whose validated quality meets the floor; when zero quality loss is required, serve self-speculative decoding instead — the all-FA path verifies while an optimized efficient placement drafts, theoretically yielding teacher-level outputs from the same weights.
  • Cheaper co-residency when needed: If several operating points must be resident on the GPU at once, the shared-weight union (at most about 25B parameters) is much smaller than N dedicated models (N × 15B).

Q2 "Can you compare at least one preset against a dedicated hybrid with the same layout..? The existing single-placement SFT run uses a different SFT mix, so I cannot read the effect of sharing from it.

We agree the existing single-placement SFT run does not settle this question. The weight-sharing cost is investigated is the 0.5B ablation (Sec. 3.3, Fig. 9), where data and steps are held equal: fully stochastic training matches and eventually surpasses targeted training restricted to the selected preset set (not a single one) while also improving 1,000 held-out canary placements. Hence: a marginal token of isolated preset training (not in the Fig. 9) would improve only that preset; a marginal token of stochastic training improves every placement simultaneously, marginal token of targeted preset set training also improves (weakly) presets beyond the selected set (even the held-out canaries improve, which share some parameters with the targeted set). Since concentrating on a preset set already fails to beat stochastic sharing on 0.5B, concentrating further on a single architecture is unlikely to reverse the conclusion. We decided not to run a dedicated 15B ablation due to limited compute.

Q3 "Please report Table 1 averages that include NIAH and MMLU... multi-seed averages or confidence intervals for AIME."

A table with recomputed all-tasks average with both excluded tasks added back is in the answer to reviewer A3gT. Any preset's retention shifts by at most 4 points. We will add these to the revised manuscript. Context on the original exclusions: NIAH scores are somewhat bimodal (either very good or very bad), while RULER, the graded long-context retrieval benchmark is included; dev MMLU was excluded because its post-SFT scores seemed unreliable in our evals. We included held-out MMLU-Pro results.

AIME variance: we agree that dev AIME24/25 are single greedy runs and are potentially noisy. The variance-controlled AIME signal is held-out AIME (NV), a 10-seed average (as stated in Table 10, τ²-Bench, LCB, and IFEval are 3-seed averages); its preset ordering is consistent with the frontier (82.7 all-FA → 82.2 at 2.0× → 80.0 at 2.9× → 74.4 at 4.8×).

Q4: Proxy validity — "How well does S(x) correlate with downstream accuracy... Reg|Lklhd-13 (6.9×) scores 60.2, while Idealized|Lklhd-6 (6.2×) scores 66.8."

The two presets originate from searches under different cost models (regression vs. idealized, Appendix E) — each was a frontier candidate at its predicted cost, and their measured speedups then landed close together (6.9× vs. 6.2×), exposing some fine-grained proxy+cost-model imperfection (which can be resolved upon evaluation on expensive downstream tasks if available for a given deployment scenario) .

As for broader question of “How well does S(x) correlate with downstream accuracy”:

  • Selecting proxi tasks s.t. they approximate the downstream evals as good as possible is essential to make this method work in production deployment setups
  • Overall, in our preliminary experiments we measured the correlation on 171 placements of the 15B supernet: cheap proxy evaluations (dev-task scores over MMLU, FDA, SWDE, RULER) predict expensive generative evaluation on the real targets (AIME24/25, MATH-500, GSM8K), over 171 placements selected to cover the full performance range on dev tasks, Spearman ρ was 0.93, confirming high correlation.

Q5: "Are the speedups in Fig. 12 measured end-to-end in vLLM, or computed from estimated acceptance rates combined with the cost model? What draft length γ is used?"

The latter, Fig. 12 speedups are projections composing (a) acceptance rates estimated exactly from draft/target log-probabilities on precomputed traces via the unbiased estimator in Appendix F (no generation involved), and (b) the validated per-token cost model. We will make it very clear in the updated PDF and refrain from calling speculative decoding speedup “end-to-end”. The acceptance rates themselves are measured quantities; what the projection omits is serving-stack overhead — integrating self-speculative decoding into the serving stack is a systems contribution in its own right, scoped as future work. We use γ=8\gamma=8, and will make it explicit in the final version. We note this analysis is a supplementary benefit of weight sharing and is not load-bearing for the paper's central claims.

Comments on the paper

1 comment

We thank the reviewers again for their valuable reviews and the time spent doing this work.

We kindly note that we have revised the submitted PDF and marked the changes we made in blue. This includes most of the problems and misunderstandings raised during the rebuttal. Here is an overview of changes we made:

  • we made clear that the main motivation of the paper is "deployment-time" flexibility and that "per-request switching" is an extension which is not load bearing (abstract, intro, conclusion)
  • all results averages are aggregation over all tasks now (also Figure 5 was updated)
  • we made clear that speculative decoding results are projected
  • we added a new appendix subsection "Deployment scenarios" describing the envisioned deployment scenarios for the supernet
  • we adapted "surrogate-optimal" / "surrogate-predicted" terminology where necessary

We hope these edits could addresses the reviewer’s concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.