acceptodds
Under review as a conference paper at ICLR 2027

Prefix-LM: Prefix-Invariant Transformers for Width-Adaptive Progressive Inference

Abstract

LLMs require a significant amount of resources during autoregressive generation, but not all tokens are equal. Token prediction is a task of varying difficulty, but monolithic architectures allocate the same compute regardless of the difficulty of prediction. Progressive inference, where representations are incrementally computed and refined as needed, can address this challenge. This has traditionally been achieved through early-exiting methods that operate along the depth dimension of the network. Elastic networks that operate along the width dimension can select a different computational budget per input, but the resulting outputs cannot be progressively refined. We identify the missing property as prefix invariance: the intermediate representations of smaller submodels are prefixes of larger ones. To this end, we propose Prefix-LM: an architectural design that achieves prefix invariance by enforcing block-triangular maps, enabling KV cache sharing and full reuse of intermediate computations. We apply Prefix-LM at the 1B scale, demonstrating practical speedups and establishing prefix invariance as a useful foundation for adaptive computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
adaptive inference, prefix invariance, many-in-one models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6sfGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

This paper studies computation reuse in language models with variable widths. Submodels of different widths can produce different intermediate representations even when they share parameters, requiring recomputation when the model is widened. The authors introduce prefix invariance and enforce block-triangular mappings, together with hierarchical normalization, attention head grouping, and a cumulative output projection. These changes make the hidden states of smaller models exact prefixes of those of larger models, allowing existing computation and KV cache entries to be reused. The paper also introduces two training techniques, HSD and MNG, and strategies for handling missing historical KV entries. Experiments use approximately 1B-parameter Llama and Nemotron architectures, with large-batch decoding throughput evaluated on a single H200 GPU.

Critical strengths

Prefix invariance captures a key requirement for reusing computation as model width increases. Sections 2 and 3.1 clearly explain why parameter sharing alone is insufficient and describe the conditions that each Transformer component must satisfy. In particular, the changes to normalization and attention keep existing features and KV entries valid when the model is widened. I consider this the paper's most valuable contribution.

Critical weaknesses
  1. HSD and MNG lack ablation results on language models. Section 3.2 introduces these two training techniques, and Appendix D.1 states that both are used in the main language model experiments. However, the HSD comparison in Figure 3 and Appendix D.2 is conducted on low-rank linear regression. I could not find language model results with HSD or MNG removed. This makes it difficult to assess which model widths benefit from each technique and by how much. Such a comparison would help clarify the role of these training techniques.
  2. The presentation of the experimental results makes the performance gains difficult to assess. Some axis labels and legends in Figures 6 and 7 are too small to read comfortably at normal viewing size. The discussion of Figures 5 and 6 is also mostly qualitative, leaving readers to estimate the gains over the baselines from the curves. I would like the authors to select a few representative settings and report the accuracy improvement at the same compute budget, or the computation saved at comparable prediction agreement levels. Appendix Table 3 already provides throughput speedup figures, but the main text offers little quantitative comparison or discussion of these results.
Questions for authors
  1. Are language model ablation results available for removing HSD, replacing HSD with distillation from the full model, and removing MNG?
  2. With the training data and nested widths held fixed, how does each setting affect the quality of the submodels and the full model?
2 comments

Response to W1, Q1 & Q2 (Language Model Ablations for HSD and MNG):

Thank you for pointing this out. We agree these ablations were absent from our manuscript. We have now added them for GPT-2 (Table 2). Setup. We follow standard practice in knowledge distillation (Hinton et al., 2015) and train every variant on a weighted combination of cross-entropy on the labels and a distillation term. The two variants differ only in the teacher: either the full model (CE + KD) or the next-larger submodel (CE + HSD). The mixing weight is tuned separately for each, so that both are compared at their best setting.

HSD. HSD is not required for Prefix-LM to train, but is our choice of distillation target. Each submodel is distilled from the next larger one instead of from the full model, which costs nothing here because prefix invariance yields all K outputs in a single pass. With the cumulative classifier, the teacher terms of the HSD term cancel in the summed gradient (Sec. A.1), so it acts as stagewise label supervision: each block learns to correct the prediction of the prefix below it. With CE + HSD, the full model achieves a loss of 3.461, compared to 3.494 with CE + KD (+0.033), while the submodels are within 0.5% of CE + KD. In the SGD experiment below, HSD also converges markedly faster than KD across all widths (new Fig. 15). Finally, HSD has a systems benefit: its backward pass requires only one hard-label loss per block, which, in the same setup, yields 7% higher training throughput and 15% lower peak memory than KD.

MNG. In the language-model experiments, turning MNG off changes the loss by at most 0.12% at any width. This is expected, and we should have stated it: MNG multiplies each block's gradient by a constant, and all our language models are trained with AdamW, which normalizes every coordinate's update by the running magnitude of its own gradient. A constant per-block scale therefore cancels out, except for the effect of ε and clipping, and MNG is effectively inactive in these runs. It matters for optimizers without this invariance:

  • SGD: We added a synthetic experiment on nested low-rank regression trained with plain SGD (Fig. 15). MNG converges faster for both distillation objectives, and the gain grows with width, as Eq. (7) predicts.
  • Muon: Muon (Jordan et al., 2024), which is increasingly used for language model training, orthogonalizes the updates of each weight matrix as a whole. It is therefore invariant under a uniform scale of the matrix, but not under different per-block scales within a single matrix that MNG corrects, so we expect MNG to be relevant there as well.

We have revised Sec. 3.2 to state the role of both techniques in these terms.


Response to W2 (Presentation Improvements & Quantitative Comparisons):

Thank you for the suggestion. We have polished the plots and discussion so that the results showcase the gains our method can achieve.

To this end, we present a single downstream-task plot (revised Figure 5) comparing the quality-compute trade-off. We report average accuracy across six different benchmarks described in Section C.2.1. We train a router for both architectures, Prefix-LM and Slimmable, and separate this comparison from CALM and LayerSkip. For example, on Llama-3.2-1B, Prefix-LM achieves 45.02% accuracy at 0.946 GFLOPs/token, compared with 44.58% at 3.846 GFLOPs/token for Slimmable cascading; on Qwen3.5-2B, Prefix-LM achieves 48.92% at 2.476/token, using estimated FLOP counts.

We have now refined the presentation of the quality-performance tradeoff, and now report speedup in Figure 6 and FLOPs comparison in Figure 7, in a matched setting. For example, Figure 7 shows that Prefix-LM with the masking strategy requires only 70.2% of the FLOPs of the fixed-width medium model to match its performance, a 1.42x theoretical compute speedup. Highlighting the efficiency of our method, we report that Prefix-LM with cascading achieves a higher MT-Bench evaluation score of 1.90, compared to 1.85, at the same latency.

Dear Reviewer,

Thank you again for your feedback. As the discussion deadline approaches, we would appreciate hearing whether our response and revised manuscript address your concerns. We hope you will consider raising your score in light of the revisions. We are happy to clarify any point that needs further explanation.

Reviewer_WNHzOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PREFIX-LM, a width-adaptive architecture in which narrower submodels produce exact prefixes of wider-model representations at every layer. This property enables token-level progressive inference that reuses activations, logits, and existing KV-cache entries when additional width is activated. The paper develops corresponding training and cache-management methods and demonstrates 1B-scale cost-quality trade-offs and large-batch decoding speedups.

Critical strengths
  • The paper provides a coherent end-to-end design for width-progressive autoregressive inference.

Prefix invariance is enforced consistently across linear maps, normalization, attention, residual paths, and the output classifier, allowing a token to be refined by activating only additional representation blocks rather than restarting a wider model. The paper also distinguishes the exact reuse of existing KV blocks from the separate problem of historical entries that were never computed, and provides both exact and approximate policies for the latter. This is important because it turns width adaptivity into a concrete progressive-decoding mechanism rather than merely providing several nested models from which inference must choose independently.

Critical weaknesses
  • The strict nesting that enables exact reuse also imposes a potentially consequential optimization and representation constraint across all model widths.

Early representation blocks must simultaneously support every deployed submodel, while later blocks are only allowed to append new information and can never revise features computed by earlier blocks. Consequently, the full model is not simply a dense model with optional computation removed: it is restricted to a hierarchical representation in which each earlier prefix must remain independently predictive. HSD and MNG address supervision and gradient-scaling issues arising from joint training, but they do not remove this underlying coupling among the width-specific objectives.

The truncated-SVD result shows that this restriction is cost-free in a linear setting, but the paper explicitly notes that simultaneous optimality should not be expected in the nonlinear case. Thus, the principal trade-off of PREFIX-LM is structural: exact progressive reuse may come at the cost of the best attainable representations for either the smallest prefixes, the full model, or both. The reported 1B-scale competitiveness is encouraging, but it does not eliminate this limitation or establish how the trade-off behaves as the number of prefixes, model scale, or task complexity increases.

Questions for authors

1. Relationship to Fully Nested Transformers/StairFormer

Fully Nested Transformers/StairFormer appears to obtain a closely related layer-wise nesting property using block lower-triangular maps, prefix-preserving normalization, and head-aligned attention.

Could the authors clarify whether StairFormer’s filtration-preservation property is mathematically equivalent to prefix invariance under the partitions considered here? Which parts of the architectural construction and theoretical results are new relative to that work, and which contributions should instead be understood as extensions specific to autoregressive progressive inference, such as cumulative logits, per-token escalation, missing-history KV policies, HSD/MNG, and the wall-clock decoder?

A precise comparison would materially affect my assessment of architectural novelty.

2. Effect of increasing the number of deployed prefixes

The architectural constraint appears to become stronger as more intermediate width boundaries are introduced: more prefixes must remain independently useful, more blocks receive different combinations of losses, and each refinement stage has less freedom to revise earlier computation.

How should the method be expected to behave as K becomes substantially larger? Is the small number of deployed widths in the main experiments primarily an engineering choice, or does it reflect an optimization or capacity trade-off intrinsic to the method? This affects how broadly the approach can support fine-grained adaptive computation.

3. Necessity and stability of hierarchical self-distillation

Because only the largest model receives direct label supervision in Eq. (6), every smaller prefix learns from a teacher that is itself changing during training and, except at the final level, is also trained by distillation rather than labels.

What prevents errors or over-smoothing from propagating down this teacher chain? Is hierarchical self-distillation necessary for PREFIX-LM to train successfully, or is it one possible way to manage the conflicting objectives of jointly training all prefixes? Please also clarify whether detached adjacent teachers create materially different fixed points from directly supervising every prefix with the ground-truth label.

4 comments
AuthorsOpenReview ↗

Response 1/2

Response to W1, W2 & Q2 (Structural Constraints):

Structural constraints. Thank you for raising this. There are indeed two distinct aspects, one concerns nested submodels, while the other concerns the triangular connectivity enforced in Prefix-LM.

Regarding the former, the intuition is correct. In the general case, there is no guarantee that a chain of nested submodels constitutes an optimally refinable frontier. This is a broader limitation of parameter sharing, and affects other nested model solutions too. Exact reuse, which sets Prefix-LM apart from Slimmable, is enabled not just by parameter nesting but also by the triangular connectivity across layers. Also, allowing later blocks to append only new information does not imply that features computed by earlier blocks are an informational bottleneck for later ones. They are optimized to be both predictive as-is and within the submodel chain. This aspect is shared broadly with other nested model training methods (Yu et al., 2019; Cai et al., 2020; Zaccone et al., 2026). The restriction of nesting parameters provably does not impact representability only in the linear case, and Thm. 3.4 extends this for Prefix-LM (i.e., including block triangular weights). As such, this concern must be treated as a no-free-lunch issue that applies to the entire research line, not just our work.

What is really relevant is the impact of enforcing triangular connectivity wrt a slimmable architecture, not the nested training by itself. In our revision, we added an ablation study on GPT-2 in Sec. 4.2 to answer this question. In this setting, we use the maximum number of submodels allowed by the architecture, as defined by the number of KV groups. To isolate architectural effects from joint-training effects, we train Prefix-LM and Slimmable models independently at the same width. We then decompose their performance gap into contributions from the parameter-count mismatch and triangular connectivity alone (Fig. 10(a)).

In practice, training iso-token and up to 8x Chinchilla, we observe a gap of just 5% (relative) in the validation loss, wrt a dense parameter-matched architecture. Reporting this on average over downstream performance, the difference wrt the dense architecture is typically ~2% (iso-token) (Fig. 5 aggregates over tasks, full results in Fig. 14). The same Prefix-LLM-Llama3.2-1B model reaches the performance dense baseline (iso-FLOPs) (see Fig. 13(a)).

Scaling: number of submodels. In model elasticity, the ability to instantiate smaller models without retraining from scratch (Yu et al., 2019) motivates additional training, provided that the shared effort remains competitive with training those models separately (Cai et al., 2020). That is, a method proves itself effective when the training shows similar performance among independently trained submodels across the total number of tokens.

The reviewer's observation is correct. Having more submodels reduces the number of active parameters in Prefix-LM, and we have already addressed the structural cost noted above. That aside, scalability wrt the number of submodels follows the same argument regarding iso-flops training mentioned above.

Fig. 9(a-b) shows the scaling behavior per submodel. Both Prefix-LM and Slimmable continue improving with additional training and reach the reference with less than the overall training budget (iso-FLOPs). When compared to independently trained submodels (Fig. 8(c-d)), Prefix-LM shows better scaling than Slimmable, thanks to more cost-effective optimization directly connected to joint optimization in a single pass, a feature allowed by prefix invariance.

Scaling: model scale. In Fig. 5, we added results for Qwen3.5-2B to the Llama3.2-1B originally proposed in the study. As pretraining bigger models is economically expensive and needs access to advanced dedicated infrastructure, we believe that the results shown on a 2B architecturally different model should mediate concerns related to generalisability to other models.

Scaling: task complexity. We omit evaluation on more complex tasks like math and code tasks because strong performance requires training budgets far beyond ours, even for standard non-nested models: Allal et al. (2025) train Llama-3.2-1B for 11T tokens, approximately 367× our 30B budget. Our decision not to evaluate these tasks therefore reflects training constraints rather than an established limitation of nested models. To assess optimization in these domains, Fig. 12 reports loss across widths and training budgets on (a) our held-out test set, which resembles the training mixture, (b) GSM8K (math), and (c) MBPP (code). Slimmable and Prefix-LM show similar scaling trends across all three datasets, with their full-width models approaching the dense reference as training increases.

AuthorsOpenReview ↗

Response 2/2

Clarification about deployed models in main exp (Q2). The choice of submodels, consistent across Prefix-LM and Slimmable, is primarily intended to provide a comparable range of submodel sizes relative to early-exit methods and to enable a fair comparison in an iso-token regime. Allowing more submodels would make Prefix-LM and CALM/LayerSkip almost not overlap on the GFLOPs/token x-axis.


Response to Q1 (Comparison with StairFormer / Fully Nested Transformers):

Thank you for raising this point. The mentioned manuscript is a non-archival workshop paper. Under the conference policy, non-archival work is not treated as prior work when assessing a submission's novelty, and therefore we have not provided a direct comparison in our manuscript. We consulted the Program Chairs on how to proceed and, on their advice, have raised the matter with the Area Chair, who can advise on the appropriate handling. We kindly ask you to follow up with the AC on the particulars of this point.

Having said that, we do agree that the components the reviewer has identified, i.e., cumulative logits, per-token escalation, missing-history KV policies, HSD/MNG, and the wall-clock decoder, together with the necessary-and-sufficient characterization of prefix invariance, are all novel contributions of this submission. We hope this positioning is sufficient to justify our differentiation from StairFormers.


Response to Q3 (Necessity, Stability, and Fixed Points of HSD):

Is HSD necessary? No, and we acknowledge the text did not make this clear. Prefix-LM also trains successfully with standard distillation from the full model (KD) or without distillation (Table 3). HSD changes only the teacher: each submodel learns from the next larger one, motivated by evidence that distillation degrades when the teacher–student capacity gap is large (Mirzadeh et al., 2020). Under prefix invariance, this is free, since all K outputs come from a single forward pass. In the new GPT-2 ablation (Sec. C.5.4, Table 3), HSD gives the lowest loss for the full model (3.461 vs. 3.494 with KD) and on average over the 12 submodels (3.863 vs. 3.867), with 7% higher training throughput and 15% lower peak memory than KD. What prevents errors or over-smoothing from propagating down the chain? No prefix ends up being trained toward another prefix’s prediction. With the cumulative classifier, the teacher terms in Eq. (6) cancel when the gradients of all prefixes sharing a block are summed (Sec. A.1), leaving L/zr=(pr-y)/K. Each block is therefore trained on the ground-truth label to correct the prediction of the prefix below it. We also use no temperature (T = 1 in Eq. (6)), so targets are not softened from level to level. Do the fixed points differ from direct label supervision? HSD is a stagewise form of label supervision, so it is not identical to cross-entropy on every prefix, where block r would also receive the label residuals of all larger prefixes. The two share their fixed points when the per-width optima are nested. In the nested low-rank regression of Sec. C.3, the optimum at every width is the truncated SVD under direct supervision, SD and HSD alike (Eckart–Young–Mirsky; Lemma B.2), and Figs. 3 and 15 show HSD approaching it faster than SD. In the nonlinear case they can differ. Prefix invariance keeps them close, since a larger submodel refines the features of the smaller one and never revises them, but we do not claim they coincide (Sec. 3.5). By contrast, slimmable networks need separate batch-normalization layers per width (Yu et al., 2019), whereas Prefix-LM uses a single normalization layer for all widths (Eq. 3).

References

Dear Reviewer,

Thank you again for your feedback. As the discussion deadline approaches, we would appreciate hearing whether our response and revised manuscript address your concerns. We hope you will consider raising your score in light of the revisions. We are happy to clarify any point that needs further explanation.

I appreciate the authors' efforts to address many of my concerns in the rebuttal. I understand that the architectural limitations I raised are not specific to PREFIX-LM, but are shared by nested model approaches in general. I still think the proposed method has considerable overlap with StairFormer, but I will leave the assessment of this issue to the AC. So, I will keep my current positive score.

Reviewer_v95HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors proposed a progressive inference method along model width. The key novelty is that the hidden states of a smaller submodel can match prefix of the larger model at each layer, and only extend the narrow submodel to wide model when needed. This paper showed the property is valid if and only if each layer is block lower triangular. It contributes a transformer with this property, covering linear layers, normalization, attention, the output head and Mamba-2.

When the model is widen, it reuses earlier KV cache and activations, but needs recompute the missing history or use an approximation policy. In training, the author uses a forward pass with hierarchical self-distillation and a gradient rescaling. Experiments on Llama3.2-1B and Nemotron-Flash-1B trained with 30B tokens . The result is tested with zero-shot accuracy. And decode throughput is up to 4.6x vanilla Llama3.2-1B on H200

Critical strengths
  • The core idea is simple but effective. The author contributed a complete transformer block with the idea of Proposition 2.2. ("prefix invariant if and only if each output block can be written as"). The paper applies this rule to all model layers so "Switching budgets during decoding, therefore, never invalidates existing cache entries".

  • The method works well with lower computing budget. Fig. 5 shows PREFIX-LM close to Slimmable, and "Both outperform the early-exit methods at low FLOPs budgets". Prefix invariance enables "one full-width forward pass yields the logits z1,...,zK of every submodel"

Critical weaknesses
  • The paper does not report or discuss the quality ceiling of the model with constrained resources. It only discussed linear case but not transformer "zero out a large part of every hidden weight, and a natural worry is whether much is lost" For example in Figure 5, on HellaSwag, the largest PREFIX-LM model is at 40 with 2 GFLOPs , but full-depth models CALM and LayerSkip is at 45 with 2.6 GFLOPs.

  • In the paper experiment, the throughput numbers are not matched to quality. The proposed model uses approximate masking policy ("decoding uses masking as the KV cache strategy"), but it does not report the impact on quality and accuracy in table 3. In Table 3, τ = 1 always uses the full width ("with endpoints selecting budgets 1 and K"), but it is already 1.29x-1.63x faster than vLLM. So, at least part of the performance gain are due to the sparser model and custom engine. It would be better to discuss the trade-off between speed and quality

Questions for authors

Clarifications:

  • Could you provide a dense Llama3.2-1B trained with the same data and tokens, and its gap to full-width PREFIX-LM?

  • Table 1 reports quality for masking "at a router threshold of 0.2". Is this the same top-2 margin τ used in Table 3? Please discuss agreement or accuracy for the other τ values in Table 3 as well.

2 comments

Response to W1 & Q1 (Dense Llama3.2-1B Baseline & Quality Ceiling):

Thanks for raising this. Figure 5 now includes the requested dense Llama3.2-1M model reference trained with the same data and tokens (iso-token training). Iso-token is one way to fairly compare the methods, and it is the one we used to present the main results.

Regarding the quality ceiling, particularly when comparing a nested method with a monolithic baseline, the more appropriate comparison would be to present iso-FLOPs curves, i.e., training for the same number of ops while accounting for the efficiency of the triangular structure. To this end, we additionally provide such an iso-FLOPs comparison in Figure 13(a). Results show that both Prefix-LM and Slimmable (full-width) can match the dense baseline. We have added complete ablations on scaling and the representation bottleneck in Sec. 4.2. In particular, Figure 10(b) shows the cost of triangular connectivity (W1). In practice, training iso-token and up to 8x Chinchilla, we observe at most a 5% relative gap in validation loss compared to a dense, parameter-matched architecture.

For downstream tasks, the difference wrt the dense architecture is typically ~2% (iso-token) (Fig. 5; full results in Fig. 14). The same Prefix-LLM-Llama3.2-1B matches the dense baseline (iso-FLOPs), even at smaller scales (see Fig. 13(a)). Compared to other progressive methods (Figure 8), the full-width Prefix-LM scores ~46.5% vs. 48% of the dense baseline (which is bigger) and matches a similarly-sized CALM model, with an overall favorable trade-off between FLOPs and accuracy, both when looking at individual submodels (marks) and progressive inference (lines).


Response to W2 & Q2 (Throughput vs. Quality Trade-off):

Thank you for the suggestion. We have now refined the presentation of the quality-performance trade-off and report speedup in the new Figure 6, and FLOPs comparison in Figure 7, in a matched setting. To simplify the results, the experiments report all the thresholds τ\tauand use the same KL divergence router (Q2). Generation quality is evaluated based on performance on MT-Bench and mapped to speedups, visualizing the performance-cost curves in these experiments (W2). Figure 6 introduces speed baselines showing that speedups are attributable to progressive inference rather than to a faster inference engine; parameter-matched baselines using vLLM are 6.85% faster than the full-size Prefix-LM model. The structured sparsity contributes to speedups, and we note that the block structure of this sparsity is particularly well aligned with hardware. Prefix-LM with block-triangular sparsity is 9.7% faster than a dense vLLM baseline, whereas unstructured sparsity is not well-aligned to hardware and cannot take advantage of this low level of sparsity. To demonstrate this, we benchmark a vLLM baseline using PyTorch’s default kernels for unstructured sparsity, and show that it performs 40% slower than the dense baseline despite the sparsity. Block-triangular sparsity is easier to take advantage of than unstructured sparsity.

Dear Reviewer,

Thank you again for your feedback. As the discussion deadline approaches, we would appreciate hearing whether our response and revised manuscript address your concerns. We hope you will consider raising your score in light of the revisions. We are happy to clarify any point that needs further explanation.

Comments on the paper

1 comment

We sincerely thank the reviewers for their time and thoughtful feedback. We are pleased that the simplicity and effectiveness of this paper's core contribution were appreciated, i.e., the characterization of prefix-invariance as a key requirement for width-based progressive inference.

In response to the feedback, we have revised and improved the manuscript as follows.

Additions

Scaling experiments. We added additional experiments on GPT-2 (sec. 4.2), which scale training up to 8x the Chinchilla budget, along with the optimization dynamics when training a larger number of profiles (max. 13 due to the number of KV groups). These ablations establish successful nested training trends (Fig. 9(a-c)) and show favorable results compared to Slimmable Networks (Fig. 8(b-d). In particular, we added an in-depth analysis of the triangular models (Fig. 10a), showing that triangular weights impact the final performance by roughly 5% (relative) once compared at an equal number of learnable parameters (Fig. 10b) - a modest price compared to enabling progressive inference. We also illustrate that this allows for a more cost-effective optimization procedure wrt Slimmable (Fig. 10c). As further evidence for larger models, we have also extended the Llama.2-1B model training, showcasing that the scaling behavior holds as expected from nested submodel training practices (Fig. 12(a)).

Increased model scale. Our initial study included diverse architectures up to 1B parameters. In the current revision, we are including results for Qwen3.5-2B (Fig. 5(c)), following the same Token-Per-Param (TPP) ratio of 20 (Chinchilla scaling). Results are consistent with other architectures and clearly show the advantage of Prefix-LM for progressive inference wrt. slimmable baselines.

Discussion and ablation about loss function and gradient rescaling. We have improved our discussion of HSD as an alternative to KD: in Sec. C.5.4, we point out that it is not a requirement for Prefix-LM to train, but consistently improves KD and admits an efficient implementation (Table 3). In our setting, this has led to a 7% higher training throughput and a 15% lower peak memory usage. We have also clarified the role of gradient rescaling with respect to the optimizer used.


Revisions

Evaluation of progressive inference. Our revised manuscript now presents the comparison on progressive inference in a clearer manner: Fig. 5 now directly compares the quality-compute trade-off under our progressive inference setting for Llama.2-1B, Nemotron-Flash-1B and Qwen3.5-2B, averaged over six different benchmarks (see Section C.2.1). The plots clearly show the advantages compared to slimmable networks.

For further contextualization, comparison with early-exit-based methods is moved to Fig. 8

Demonstration of speedups. Figure 6 now clearly relates the achieved speedup to quality, reporting performance on MT-Bench, a generation task with LLM-as-a-judge. We also included the full model reference and two baselines useful to relate the speedups due to progressive inference to the structural sparsity of Prefix-LM.

Ablation over KV strategies. Figure 7 shows the direct impact of our KV cache strategies on quality and speed: in particular, (a) shows that, as expected, backfilling is the best when given a width ratio, confirming that missing information in previous tokens matters, while masking dominates when keeping time into account as a measure. The two findings are not contradictory; they characterize the key trade-off between quality and speed in cascading: backfilling incurs additional computation, making it inefficient to implement across the context.

We have carefully addressed each concern in our responses to the reviews, providing additional clarification on each matter. Additions and revisions are marked in blue in our revision.

The Author(s)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.