- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper keeps the MLMS architecture and changes its training objective. At each step, it applies next-token supervision to one randomly sampled prefix and to the full model. The goal is to make every prefix usable at inference, allowing deployment at more depths than the fixed exits trained by MLMS. Experiments on a 200M-parameter suite trained with 20B FineWeb-Edu tokens report substantially better intermediate-prefix quality, similar full-depth quality, and approximately 12% lower training cost. Sampling ablations examine the trade-off between covering all depths and improving a few selected exits.
Critical strengths
The paper demonstrates a useful property: one trained checkpoint can provide usable predictions at every prefix depth. Keeping the architecture and data stream fixed makes the effect of changing the supervision easier to interpret. This could be useful when deployment budgets are not known in advance.
The sampling ablation helps explain the trade-off. Concentrating supervision at selected exits improves those exits while weakening the intermediate prefixes. The authors also acknowledge that MLMS remains better at its smaller trained exits. This makes the results more informative than a comparison based only on aggregate improvements.
Critical weaknesses
I am not yet convinced that the experiments establish enough of an advance over existing intermediate-layer supervision. Much of the AULB improvement comes from evaluating MLMS at depths it was never trained to use. That demonstrates the benefit of training those prefixes, but leaves open whether TLM's particular recipe is better than established alternatives. LayerSkip already trains intermediate predictions and uses a rotating exit-loss curriculum, so supervising many depths without evaluating every exit at every step is not unique to TLM. A comparison on the same architecture and data, with a matched training budget, is needed to establish the benefit of random-prefix supervision plus the full-model anchor. The paper refers to dense all-prefix and LayerDrop controls in Appendix B, but those results are absent from the supplied 12-page PDF. They are central to this assessment.
The training-cost claim depends on a baseline implementation that needs further justification. Section 4.2 attributes MLMS's higher cost to a separate backward pass for each exit through the shared graph. The exit losses can instead be summed and differentiated in one backward pass; separate backward traversals are not required by the objective. The authors acknowledge that implementation choices could change the gap, but the approximately 12% saving is still a headline result. Without an efficient MLMS implementation, I cannot tell how much of that saving comes from the proposed method. This matters because training efficiency is a substantial part of the case for an otherwise simple change to the objective.
Questions for authors
Clarification: Is there a separate supplement containing Appendices A–H? In particular, please point to the dense all-prefix, LayerDrop, readout-calibration, and compute-matched anchor controls referenced in the main text. These results could materially change my assessment.
Baseline implementation: Why are the MLMS exit losses backpropagated separately? Does the reported training-cost advantage remain when the losses are combined into one backward pass, with the same hardware and training settings?
Comparison needed to assess the contribution: Under a matched training budget, how does TLM compare with dense or rotating intermediate-exit supervision adapted to the same cascade? The relevant quantities are the full prefix-quality curve, full-depth quality, and total training cost. If the referenced controls already address this, please explain what they establish.