acceptodds
Under review as a conference paper at ICLR 2027

Telescopic Language Models

Abstract

One deployed language model must often serve many compute budgets, yet serving each budget still means a separate training or compression run per point. We train a Telescopic Language Model (TLM) to be that continuum: a nested-capacity Transformer supervised by stochastic prefix supervision with a full anchor. At every step, one randomly truncated prefix of the capacity axis is trained against the full next-token target, alongside one full-capacity pass, so the trained artifact is a valid language model at every depth. Two forward-backward passes per step, no architectural change, nothing extra at inference. Fixed-exit suites such as Matryoshka Language Model Suites (MLMS) occupy one point in this design space, and the point has a cost: supervising only a few fixed exits leaves the nested model at chance level everywhere else (perplexity 10^2-10^5 in our baselines). On a 200M proxy suite (20B FineWeb-Edu tokens, identical data stream for all methods), a single TLM run is a valid language model at every one of its twenty layer prefixes, in perplexity and on perplexity-sensitive downstream tasks, reducing the area under the quality-budget curve by 43-44% relative to the fixed-exit suites while matching them at full capacity, at  12% lower GPU cost per run. The prefix sampling density is a dial: concentrating it on a few depths recovers fixed-exit quality there at the price of the continuum, so the operating points become a training-time choice rather than an architectural one. These results indicate that the training objective, not the nesting itself, is what makes a model elastic.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, nested models, Matryoshka models, stochastic prefix supervision, elastic inference, adaptive computation, early exiting, model compression, training efficiency, quality-budget frontier

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QDHxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper keeps the MLMS architecture and changes its training objective. At each step, it applies next-token supervision to one randomly sampled prefix and to the full model. The goal is to make every prefix usable at inference, allowing deployment at more depths than the fixed exits trained by MLMS. Experiments on a 200M-parameter suite trained with 20B FineWeb-Edu tokens report substantially better intermediate-prefix quality, similar full-depth quality, and approximately 12% lower training cost. Sampling ablations examine the trade-off between covering all depths and improving a few selected exits.

Critical strengths

The paper demonstrates a useful property: one trained checkpoint can provide usable predictions at every prefix depth. Keeping the architecture and data stream fixed makes the effect of changing the supervision easier to interpret. This could be useful when deployment budgets are not known in advance.

The sampling ablation helps explain the trade-off. Concentrating supervision at selected exits improves those exits while weakening the intermediate prefixes. The authors also acknowledge that MLMS remains better at its smaller trained exits. This makes the results more informative than a comparison based only on aggregate improvements.

Critical weaknesses

I am not yet convinced that the experiments establish enough of an advance over existing intermediate-layer supervision. Much of the AULB improvement comes from evaluating MLMS at depths it was never trained to use. That demonstrates the benefit of training those prefixes, but leaves open whether TLM's particular recipe is better than established alternatives. LayerSkip already trains intermediate predictions and uses a rotating exit-loss curriculum, so supervising many depths without evaluating every exit at every step is not unique to TLM. A comparison on the same architecture and data, with a matched training budget, is needed to establish the benefit of random-prefix supervision plus the full-model anchor. The paper refers to dense all-prefix and LayerDrop controls in Appendix B, but those results are absent from the supplied 12-page PDF. They are central to this assessment.

The training-cost claim depends on a baseline implementation that needs further justification. Section 4.2 attributes MLMS's higher cost to a separate backward pass for each exit through the shared graph. The exit losses can instead be summed and differentiated in one backward pass; separate backward traversals are not required by the objective. The authors acknowledge that implementation choices could change the gap, but the approximately 12% saving is still a headline result. Without an efficient MLMS implementation, I cannot tell how much of that saving comes from the proposed method. This matters because training efficiency is a substantial part of the case for an otherwise simple change to the objective.

Questions for authors

Clarification: Is there a separate supplement containing Appendices A–H? In particular, please point to the dense all-prefix, LayerDrop, readout-calibration, and compute-matched anchor controls referenced in the main text. These results could materially change my assessment.

Baseline implementation: Why are the MLMS exit losses backpropagated separately? Does the reported training-cost advantage remain when the losses are combined into one backward pass, with the same hardware and training settings?

Comparison needed to assess the contribution: Under a matched training budget, how does TLM compare with dense or rotating intermediate-exit supervision adapted to the same cascade? The relevant quantities are the full prefix-quality curve, full-depth quality, and total training cost. If the referenced controls already address this, please explain what they establish.

Reviewer_gNwsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors introduce an updated objective function for the Matryoshka Language Model (MLM) framework. Whereas conventional MLM trains a single transformer stack to exit at a sparse set of predefined layer depths (e.g., layers 20, 40, 60, 80, and 100 of a 100-layer stack), the proposed method generalizes this objective to supervise every layer exit, effectively yielding LL nested sub-models of monotonically increasing capacity. The authors validate this on a 200M-parameter model, demonstrating that arbitrary layer exits can generate coherent text. Crucially, however, the proposed model underperforms vanilla MLM at the specific checkpoint layers for which the baseline was optimized. The dense objective outperforms the baseline strictly at intermediate, off-grid layers where vanilla MLM received no direct supervision.

Critical strengths
  • The study investigates a very interesting question: can we take the residual stream from any layer in the LLM and generate a response? Answering this question has a very high practical importance, especially for controlling the inference cost.
  • The authors show that with their proposed objective, output from each layer in a language model can be used to generate responses with fairly stable perplexity values. This is equivalent to having multiple LLMs, each with its unique depth and parameter count. With regular pre-training, this would take num_layers separate pre-training runs, one for each model.
Critical weaknesses
  • The manuscript has not been properly reviewed by the authors. The writing is jargon-heavy, sometimes hand-wavy, and overall very confusing.
  • The improvements with the proposed objective are realized only for the layers where MLMS is not explicitly trained
  • Experiments are with only one 200M model with one seed
  • The difference (line-234: delta_full) between the vanilla language model and the full forward pass of the trained model under the proposed objective is associated with architectural changes by the authors. The delta_full does reflect both the effect of architectural changes and the effect of objective changes.
  • Table-1: results are incomplete
Questions for authors
  • line-15: no architectural chang compared to what? Line-158 mentions that experiments are done with Matryoshka Language Model Suite (MLMS) architecture, which is not the vanilla architecture. Reader who is not familiar with MLMS architecture (such as myself), will have a very different impression of this line. It might be better clearly mention that no arch changes to MLMS.
  • Line-237: Unexplained metric AUC_fps
  • table-1: why the vanilla model is not evaluated for AULB/LODA?

Where I was confused while reading:

  • Line-12: Thought/suggestion: Until line-21, I was under an impression that prefix has something to do with how prompts are formed (data is processed) for the objective function. The authors do use depth in some parts, which might be a better word to describe the work. Also, I don't know if full anchor is commonly used for a full forward pass but, I did not understand it. Overall, this phrase supervised by stochastic prefix supervision with a full anchor confused me quite a bit.
  • Line-12: confusing truncated prefix of the capacity axis
  • Line-14: confusing full next-token target, alongside one full-capacity pass. The 'full' term here seems to refer to different things and because 'full' was used for full forwrd pass, I could not wrap my head around what is 'full next-token target' and why is it different from 'full-capacity'.
  • Line-22: what is perplexity-sensitive downstream tasks? Are there any tasks that are not sensitive?
  • line-174: I don't understand why two forward-backward passes are required per step. Can you help me understand?
  • line-183: "k, which denotes a generic or evaluation depth", I though k was set of fixed depths (line-167)? what is eval/generic depth, how is it different from fixed depth? or is it the same?
  • Line-184-188: This one sentence has >50 words. Please consider splitting it in multuple sentences.
  • Line-192: What is "anchor gradient"?
  • Line-204: What is "head-only repair"
  • Line-224: why two names for the curve?
  • Line-237: what is operating points?
  • Line-240: what is model throttling? what is " max-from-right step function"?
Reviewer_vRrmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a simple and interesting training objective that enables language models to operate at different depths. The results demonstrate improved flexibility over fixed-exit models. However, experiments are limited to 200M parameters, and the novelty relative to existing nested-model approaches is incremental. Larger-scale experiments and stronger baselines would strengthen the contribution.

Critical strengths

The objective is simple and operationally appealing. On a 200M-parameter, 20B-token experiment, the authors report valid language modeling at all 20 layer prefixes, 43–44% lower area under their quality–budget curve than fixed-exit suites, and 131 versus 149 GPU-hours for the compared training runs. The work directly investigates an interesting limitation of fixed-exit supervision.

Critical weaknesses

-The central observation that unsupervised exits collapse is partly explained by readout miscalibration, which the paper itself acknowledges. This weakens the broader claim that fixed-exit models fundamentally lack intermediate capabilities.

-The strongest evidence is at 200M parameters. It is unclear whether the same trade-offs hold for multi-billion-parameter models.

-A uniform prefix sampler sacrifices performance at important deployment exits. The practical value of intermediate operating points needs stronger demonstration.

Questions for authors

How does the method compare with a Matryoshka model using calibrated intermediate readout heads? Does the benefit persist at 1B–3B scale? How many intermediate operating points are practically useful after accounting for quality thresholds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.