- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an interesting approach to multitoken prediction which is computationally lightweight and thus applicable during pretraining. The approach functions by building a lightweight predictor of intermediate future states at intermediate positions. Empirical results show that this approach does not degrade performance much (in comparison to facebook's mtp solution for example) while yielding good information about the future.
Critical strengths
The approach is lightweight (~5% overhead) and plausibly could be applied in pretraining to significant effect on the learned representations of the transformer. It's significantly more lightweight than the Gloeckle et al. (2024) MTP style paper and avoids NTP degradation while performing well.
Critical weaknesses
The results seem significantly incomplete so that it's hard to judge the meaningfulness between "Yes we should do it" and "Better approaches exist".
- Experiments do not vary the strength of the cosine loss relative to NTP loss. Given this, the results are not strongly indicative of the limits of different approaches or even of this one.
- This also means that the amount of gradient norm relative to NTP gradient norm is uncontrolled. Is the LightMTP approach simply succeeding because there is less gradient norm than for other approaches? Cosine loss tends to have a small gradient near the optimum and the implied gradient might be very low if there is a large fixed component.
- It's odd to mention the Next Latent paper as the most relevant baseline and then not compare with it. That paper is also predicting future latents except with a particular choice (the last latent) and with a token embedding added, also seeing similar-sounding results.
Overall, there's an interesting technique here, but the experiments to validate it as truly useful aren't yet there. The amount of experiments it would take to fix these issues is substantial, but may be doable.
Questions for authors
It's not clear that cosine loss is the right approach. Perhaps a contrastive predictive coding style (as per https://arxiv.org/pdf/1807.03748) would be helpful instead? This might more effectively pull information out of a signal which is mostly the same but with spiky differences since cosine loss can be very low in those settings.
4 comments
I'm skeptical that we should imagine nextlatent as having a 19.7% overhead. For example, a trivial variant relative to LightMTP is simply setting the KL loss to zero in nextlatent.
Structurally, I only see two differences here: cosine vs. huber loss on the latents and whether or not to condition on the token in predicting the latents. Maybe there are better choices here? But the experiments do not yet support that.
We added NextLat as a baseline following the reviewer's suggestion, and implemented it as published (Teoh et al., 2025). Our implementation follows the NextLat paper closely and uses a Smooth L1 (i.e., Huber) next-latent loss plus a KL loss, both with weight 1 as in the authors’ FineWeb-Edu pretraining setup, but a dynamics model built like LightMTP’s MLP head (2-layer instead of 3-layer MLP).
Where the 19.7% compute overhead comes from:
Our d24 models have , a vocabulary of 32,768 and a multi-token horizon . NextLat’s training FLOPs per token break down as follows:
KL term: +16.3% of NTP. Each of the rolled-out latents must be decoded through the (frozen) output head, to compute the KL against the model’s own next-token distribution.
Dynamics MLP: +3.4% of NTP. The MLP takes the concatenation as input and is applied recursively, times per token. LightMTP’s head only takes as input and is applied once per token: +0.6% for LightMTP-S and +1.4% for LightMTP-C.
If anything, our estimate is favorable for NextLat. Following standard FLOP counting, we do not charge NextLat for the additional softmax and KL computations over the full vocabulary that its objective requires per token. The estimate is also consistent with the NextLat authors’ own measurements. Their FineWeb-Edu training-speed comparison reports 3.09 steps/s for NTP, against 2.79 steps/s for NextLat at and 1.73 steps/s at . That is a wall-clock overhead of 11% and 79%, respectively. The authors themselves note that the KL loss “introduces an additional forward and backward pass through the output head, which can be computationally expensive in language models due to large vocabulary sizes.”
On the KL-free variant. The NextLat paper defines the objective with and uses for FineWeb-Edu pretraining. It evaluates the KL-free variant only as an ablation, concluding that “NextLat’s effectiveness arises from the combination of Smooth L1 and KL losses”. We followed the reviewer's suggestion to include NextLat as a baseline in good faith and evaluated the method in the form its authors propose. We respectfully believe that comparing against other variants ablated in the original paper, on the chance that one of them outperforms LightMTP, goes beyond what a fair baseline comparison requires. The same request could be made of any baseline, and would amount to redesigning NextLat on its authors’ behalf. Moreover, even without the KL term, NextLat’s overhead (3.4%) would remain 2.4–6× that of LightMTP.
There are more structural differences:
Besides the latent loss (Smooth L1 vs. cosine on hidden states) and token conditioning, the methods differ in three further ways:
-
NextLat performs a teacher-forced recursive rollout, conditioning step on its previous prediction and the ground-truth token . LightMTP predicts from alone, in a single step.
-
NextLat regresses each future final-layer state individually. LightMTP predicts a summary (-S) or a concatenation (-C) of future intermediate-layer (layer 16) states.
-
NextLat adds KL self-distillation through the output head. LightMTP has no token-space auxiliary loss.
Whether some of NextLat’s design choices could also benefit LightMTP is an interesting direction that we leave to future work.
Dear Reviewer uoDc,
Thank you again for engaging with our NextLat results. As the discussion period is drawing to a close, we wanted to check whether our last response clarified the question regarding NextLat’s compute overhead: most of NextLat’s 19.7% compute cost comes from its KL term, which matches the slowdowns its authors report themselves. The KL-free variant appears in their paper only as an ablation. We would be glad to answer any remaining questions. We believe that we have addressed all of your concerns in-depth. If something is left unresolved we would appreciate your feedback on what is still missing. Otherwise, we would be grateful if you would consider updating your score.
Dear Reviewer,
We are happy to report that as a response to your review, we have now added NextLat as an additional baseline. The results can be found in Appendix B.9. As a short summary: NextLat incurs moderate anticipation cost, degrades on general language modeling along with all other MTP methods except LightMTP, shows moderate planning improvements compared to NTP on Countdown, and falls off on GSM8K against NTP. We thank you for suggesting NextLat as a baseline. It further substantiates that LightMTP is the most effective MTP method at matched compute.
For the sake of time and to include NextLat in the compute-matched results Reviewer 5tx5 asked for, we have only evaluated NextLat in the iso-FLOPs setting for now, and left out the separate code pretraining and finetuning. We will include results for coding and the main paper's parameter-dependent training budget in the camera-ready version (training is currently undergoing).
Together with our last comment, we have addressed all your points: We have addressed the choice of cosine loss as our loss function (W1). We have explained why the results show that a potentially small gradient norm is not why LightMTP performs well (W2). And we have added NextLat as an additional baseline (W3). If you feel that your comments were sufficiently answered, please consider raising your score. Once again, we wish to thank you for helping to improve the paper.