acceptodds
Under review as a conference paper at ICLR 2027

LightMTP: Lightweight Latent Multi-Token Prediction

Abstract

Next-token prediction (NTP) is the standard pretraining objective for large language models, yet it provides an explicit training signal only for the immediate next token, which can lead models to exploit local patterns instead of capturing longer-range structure and ideas. Multi-token prediction (MTP) addresses this by training models to predict several future tokens. However, existing MTP methods often introduce a large computational overhead with limited improvements in downstream performance. Latent MTP approaches address this efficiency issue by encoding future tokens into a vector representation. However, these approaches usually rely on external helper models for future token encoding. We propose LightMTP, a lightweight, i.e., parameter-efficient, latent MTP approach that bootstraps the future token representations from the model's own hidden states. Our two LightMTP variants extend supervision to more future tokens without requiring the additional computational overhead of conventional MTP nor the external supervision latent MTP normally relies on. LightMTP adds at most 1% extra parameters, retains better performance on general language modeling benchmarks, and achieves similar gains in planning, coding, and reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi token prediction, pretraining, LLM, language modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uoDcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an interesting approach to multitoken prediction which is computationally lightweight and thus applicable during pretraining. The approach functions by building a lightweight predictor of intermediate future states at intermediate positions. Empirical results show that this approach does not degrade performance much (in comparison to facebook's mtp solution for example) while yielding good information about the future.

Critical strengths

The approach is lightweight (~5% overhead) and plausibly could be applied in pretraining to significant effect on the learned representations of the transformer. It's significantly more lightweight than the Gloeckle et al. (2024) MTP style paper and avoids NTP degradation while performing well.

Critical weaknesses

The results seem significantly incomplete so that it's hard to judge the meaningfulness between "Yes we should do it" and "Better approaches exist".

  1. Experiments do not vary the strength of the cosine loss relative to NTP loss. Given this, the results are not strongly indicative of the limits of different approaches or even of this one.
  2. This also means that the amount of gradient norm relative to NTP gradient norm is uncontrolled. Is the LightMTP approach simply succeeding because there is less gradient norm than for other approaches? Cosine loss tends to have a small gradient near the optimum and the implied gradient might be very low if there is a large fixed component.
  3. It's odd to mention the Next Latent paper as the most relevant baseline and then not compare with it. That paper is also predicting future latents except with a particular choice (the last latent) and with a token embedding added, also seeing similar-sounding results.

Overall, there's an interesting technique here, but the experiments to validate it as truly useful aren't yet there. The amount of experiments it would take to fix these issues is substantial, but may be doable.

Questions for authors

It's not clear that cosine loss is the right approach. Perhaps a contrastive predictive coding style (as per https://arxiv.org/pdf/1807.03748) would be helpful instead? This might more effectively pull information out of a signal which is mostly the same but with spiky differences since cosine loss can be very low in those settings.

4 comments

Dear Reviewer,

We are happy to report that as a response to your review, we have now added NextLat as an additional baseline. The results can be found in Appendix B.9. As a short summary: NextLat incurs moderate anticipation cost, degrades on general language modeling along with all other MTP methods except LightMTP, shows moderate planning improvements compared to NTP on Countdown, and falls off on GSM8K against NTP. We thank you for suggesting NextLat as a baseline. It further substantiates that LightMTP is the most effective MTP method at matched compute.

For the sake of time and to include NextLat in the compute-matched results Reviewer 5tx5 asked for, we have only evaluated NextLat in the iso-FLOPs setting for now, and left out the separate code pretraining and finetuning. We will include results for coding and the main paper's parameter-dependent training budget in the camera-ready version (training is currently undergoing).

Together with our last comment, we have addressed all your points: We have addressed the choice of cosine loss as our loss function (W1). We have explained why the results show that a potentially small gradient norm is not why LightMTP performs well (W2). And we have added NextLat as an additional baseline (W3). If you feel that your comments were sufficiently answered, please consider raising your score. Once again, we wish to thank you for helping to improve the paper.

I'm skeptical that we should imagine nextlatent as having a 19.7% overhead. For example, a trivial variant relative to LightMTP is simply setting the KL loss to zero in nextlatent.

Structurally, I only see two differences here: cosine vs. huber loss on the latents and whether or not to condition on the token in predicting the latents. Maybe there are better choices here? But the experiments do not yet support that.

We added NextLat as a baseline following the reviewer's suggestion, and implemented it as published (Teoh et al., 2025). Our implementation follows the NextLat paper closely and uses a Smooth L1 (i.e., Huber) next-latent loss plus a KL loss, both with weight 1 as in the authors’ FineWeb-Edu pretraining setup, but a dynamics model built like LightMTP’s MLP head (2-layer instead of 3-layer MLP).

Where the 19.7% compute overhead comes from:

Our d24 models have d=1536d{=}1536, a vocabulary of 32,768 and a multi-token horizon k=4k{=}4. NextLat’s training FLOPs per token break down as follows:

KL term: +16.3% of NTP. Each of the kk rolled-out latents must be decoded through the (frozen) output head, to compute the KL against the model’s own next-token distribution.

Dynamics MLP: +3.4% of NTP. The MLP takes the concatenation [h^t+i−1;emb(xt+i)][\hat h_{t+i-1};emb(x_{t+i})] as input and is applied recursively, kk times per token. LightMTP’s head only takes h^t+i−1\hat h_{t+i-1} as input and is applied once per token: +0.6% for LightMTP-S and +1.4% for LightMTP-C.

If anything, our estimate is favorable for NextLat. Following standard FLOP counting, we do not charge NextLat for the kk additional softmax and KL computations over the full vocabulary that its objective requires per token. The estimate is also consistent with the NextLat authors’ own measurements. Their FineWeb-Edu training-speed comparison reports 3.09 steps/s for NTP, against 2.79 steps/s for NextLat at d=2d{=}2 and 1.73 steps/s at d=8d{=}8. That is a wall-clock overhead of 11% and 79%, respectively. The authors themselves note that the KL loss “introduces an additional forward and backward pass through the output head, which can be computationally expensive in language models due to large vocabulary sizes.”

On the KL-free variant. The NextLat paper defines the objective with λKL>0\lambda_\text{KL} > 0 and uses λKL=1\lambda_\text{KL}{=}1 for FineWeb-Edu pretraining. It evaluates the KL-free variant only as an ablation, concluding that “NextLat’s effectiveness arises from the combination of Smooth L1 and KL losses”. We followed the reviewer's suggestion to include NextLat as a baseline in good faith and evaluated the method in the form its authors propose. We respectfully believe that comparing against other variants ablated in the original paper, on the chance that one of them outperforms LightMTP, goes beyond what a fair baseline comparison requires. The same request could be made of any baseline, and would amount to redesigning NextLat on its authors’ behalf. Moreover, even without the KL term, NextLat’s overhead (3.4%) would remain 2.4–6× that of LightMTP.

There are more structural differences:

Besides the latent loss (Smooth L1 vs. cosine on hidden states) and token conditioning, the methods differ in three further ways:

  1. NextLat performs a teacher-forced recursive rollout, conditioning step ii on its previous prediction and the ground-truth token xt+ix_{t+i}. LightMTP predicts from hth_t alone, in a single step.

  2. NextLat regresses each future final-layer state ht+ih_{t+i} individually. LightMTP predicts a summary (-S) or a concatenation (-C) of future intermediate-layer (layer 16) states.

  3. NextLat adds KL self-distillation through the output head. LightMTP has no token-space auxiliary loss.

Whether some of NextLat’s design choices could also benefit LightMTP is an interesting direction that we leave to future work.

Dear Reviewer uoDc,

Thank you again for engaging with our NextLat results. As the discussion period is drawing to a close, we wanted to check whether our last response clarified the question regarding NextLat’s compute overhead: most of NextLat’s 19.7% compute cost comes from its KL term, which matches the slowdowns its authors report themselves. The KL-free variant appears in their paper only as an ablation. We would be glad to answer any remaining questions. We believe that we have addressed all of your concerns in-depth. If something is left unresolved we would appreciate your feedback on what is still missing. Otherwise, we would be grateful if you would consider updating your score.

Reviewer_5tx5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LightMTP, an auxiliary training objective that encourages Transformers to predict intermediate hidden representations associated with future token positions alongside the standard NTP loss. The authors propose two variants: LightMTP-S, which predicts a single future hidden representation, and LightMTP-C, which predicts multiple future hidden representations. They evaluate both variants against NTP and several existing MTP baselines. Their experiments suggest that LightMTP can improve planning and code generation performance over NTP while largely preserving general language modeling performance and introducing minimal additional training overhead.

Critical strengths

For me, the main strength of this paper is the simplicity of the proposed approach. LightMTP leverages the model's own intermediate hidden states for future-token supervision, eliminating the need for external teacher models or complex auxiliary architectures. It introduces minimal additional parameters and training overhead while leaving inference unchanged. Another strength is the breadth of the experimental evaluation, which goes beyond standard benchmark scores to investigate different aspects of model behavior, including future-token anticipation, planning capabilities, and structural consistency in generated outputs.

Critical weaknesses

The results do not convince me that LightMTP should be preferred over other methods such as Meta-L. The claimed gains are often small relative to the reported standard deviations. LightMTP achieves slightly better perplexity, but CORE accuracy is very similar to Meta-L. In Table 2, Meta-L outperforms LightMTP. In Table 3, LightMTP shows a slight improvement over Meta-L. In Table 4, both methods achieve broadly comparable coding performance. Also, the results seem somewhat cherry-picked. Looking at Appendix Tables 10 and 11, LightMTP does not consistently outperform NTP or other methods across different model sizes. I would have framed the paper more around training-time efficiency rather than improvements in benchmark numbers that are often marginal. Currently, some of the efficiency results are presented only in Appendix Table 6. To convincingly demonstrate a compute-efficient training technique, the authors should compare performance at matched training FLOPs.

Furthermore, I am not sure I buy the extra parameter argument. If the extra parameters are discarded at inference time, and if they constitute a relatively small fraction of the total parameters, then I would not claim a significant advantage based on that alone. What matters more is how these extra parameters affect training cost. I would have framed the paper more around training-time efficiency rather than improvements in benchmark numbers that are often marginal. Currently, some of the efficiency results are presented only in Appendix Table 6. To convincingly demonstrate a compute-efficient training technique, the authors should compare performance at matched training FLOPs and provide a clearer analysis of training efficiency.

Other minor issues:

  • It is not clear how Figure 3 is generated and what exactly it measures. More details about how target reachability is evaluated after each equation would be helpful.
  • Some experimental details are missing. For example, it is not clear whether fine-tuning in Table 4 is performed using the LightMTP or NTP objective.
Questions for authors

Besides the concerns raised above:

  • For the “Anticipation Cost” metric, how is ΔAcck>1\Delta \mathrm{Acc}_{k>1} aggregated across different values of kk?
  • How does LightMTP's accuracy compare to that of other methods under the same training compute budget (e.g., matched training FLOPs)?
2 comments

Dear Reviewer 5tx5,

Since the discussion period is ending in a few hours, we wanted to briefly follow up. Following your request for matched-FLOP results on downstream tasks, we have added Countdown, GSM8K and MBPP results for all methods at NTP’s compute budget (Appendix B.9). At matched compute, LightMTP achieves the best scores across all downstream tasks and outperforms Meta-L. With these results, we believe we have comprehensively addressed all of your listed concerns. Please let us know if anything is still missing. Otherwise, we kindly ask you to consider raising your score.

We just missed your update. Thank you for your response.

Reviewer_PCwgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LightMTP trains a small auxiliary head to predict the model’s own future hidden states. It compares single-target and parallel multi-target prediction without external teachers or extra vocabulary projections. Experiments across three model sizes show planning and coding gains while largely preserving general language performance. The auxiliary head is removed after training, leaving inference unchanged. Also, the paper includes a broad range of experiments.

Critical strengths
  1. The method is simple and practical. It reuses states from the existing forward pass and adds little training overhead.

  2. The experiments cover several relevant settings. They include language benchmarks, planning, and coding, alongside comparisons of prediction horizons, target layers, and regularization.

Critical weaknesses
  1. The novelty relative to NITP is unclear. NITP [1] already uses a small projector, same-model intermediate targets, stop-gradient, and cosine loss. LightMTP predicts farther and multiple future positions, but does not discuss or compare with this close precedent. The paper needs to show what these changes add beyond the existing approach.

  2. The DeepSeek comparison uses a substantially different recipe. The baseline has three auxiliary modules and a loss weight of 1, rather than one module and the original 0.3-to-0.1 schedule [2]. Its MBPP pass@5 drops to 0.19%, compared with 5.02% for NTP. Without checking a configuration closer to the original, it is unclear whether this failure comes from the objective or the adapted recipe.

  3. The cost discussion misses part of the trade-off. LightMTP saves vocabulary-projection work but must retain intermediate targets and may transfer them across pipeline stages. Token-level MTP can reduce logit memory through careful execution [3] and can also accelerate inference through speculative decoding. The practical choice therefore depends on target-handling costs and the inference workload, not just auxiliary parameter count and training time.

[1] Zhang et al. NITP: Next Implicit Token Prediction for LLM Pre-training. 2026.

[2] DeepSeek-AI. DeepSeek-V3 Technical Report.

[3] Gloeckle et al. Better & Faster Large Language Models via Multi-token Prediction. ICML, 2024.

Questions for authors
  1. What does multi-step LightMTP add over NITP, and which results best demonstrate that difference?

  2. Does the DeepSeek-style baseline still degrade with one auxiliary module and the original loss-weight schedule?

  3. When is LightMTP cheaper overall after accounting for target storage, cross-stage communication, and the speculative-decoding benefits of token-level MTP?

2 comments

Dear Reviewer,

We are happy to report that rerunning DeepSeek's MTP approach with the original paper's loss schedule has made its results more stable. Thank you again for pointing this out and thereby strengthening the paper.

We have rerun DeepSeek as part of the FLOPs-matched set of experiments Reviewer 5tx5 asked for (full results in Appendix B.9). To summarize, we found that DeepSeek at k=2 retains more language modeling performance than other MTP methods except LightMTP, but falls off at k=4. Its anticipation cost at k=2 is the runner-up to our two LightMTP variants and comes in third place. At k=4 it again falls off. On Countdown DeepSeek's planning abilities declined compared to our original setup without the loss schedule and improve less over NTP than most other MTP methods. On GSM8K it was previously the worst method and has now improved but still performs worse than NTP and LightMTP on the strict metric. Collectively, these results substantiate LightMTP's place ahead of any other MTP method in terms of effectiveness while adding only a negligible amount of compute overhead.

Due to time restrictions we have not retrained DeepSeek on the separate code pretraining data, but will include this in the camera-ready version, as training is already in progress.

We believe that our previous comment together with this update has addressed all of your points: We have incorporated NITP into the paper (W1), added a dedicated section discussing LightMTP's cost beyond just how many parameters it adds (W3), and now updated DeepSeek's results with the better performing original loss schedule (W2). We hope you will consider raising your score and we once again thank you for helping us to improve the paper.

Dear Reviewer PCwg,

As the discussion period ends in a few hours, we wanted to briefly follow up. In our two responses, we addressed each of your points: 1) we added NITP to the related work and showed that the multi-token horizon (k>1) drives LightMTP’s gains over the NITP-like k=1 case. 2) We reran DeepSeek with its original single-module, 0.3→0.1 loss schedule (Appendix B.9). 3) And we added a dedicated discussion of target storage, pipeline transfer and speculative decoding.

We believe we have comprehensively addressed all three of your listed concerns. If anything is still missing, please let us know what it is. Otherwise, we kindly ask you to consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of the Discussion and Revisions

Dear Area Chairs and Reviewers,

We thank all three reviewers for their constructive feedback. Below we summarize the strengths they identified and how we addressed their concerns.

Strengths highlighted by the reviewers:

  1. LightMTP is simple and practical, with minimal overhead. All three reviewers single out LightMTP’s simplicity and efficiency: “simple and practical” (PCwg), “the main strength of this paper is the simplicity” (5tx5), and “significantly more lightweight” than Gloeckle et al. (2024) (uoDc).
  2. LightMTP preserves language modeling. PCwg and 5tx5 note gains “while largely preserving general language performance”. uoDc notes that LightMTP “avoids NTP degradation while performing well”.
  3. LightMTP’s evaluations are comprehensive and the writing clear. PCwg and 5tx5 praise the breadth of the evaluation, which goes “beyond standard benchmark scores” (5tx5). All three rated the presentation “clear and easy to follow”.

Our revised paper shows:

At NTP’s training-FLOP budget (new Appendix B.9, including the new DeepSeek-schedule and NextLat baselines) LightMTP is:

  1. Cheapest: LightMTP adds +0.6% (S) and +1.4% (C) training FLOPs per token, against +10.5% to +31.9% for all other methods.
  2. No language-modeling degradation: LightMTP is the only MTP method that matches NTP on FineWeb-Edu perplexity (1.684 vs. 1.683). It beats NTP on LAMBADA (15.92/15.95 vs. 16.00), and LightMTP-S has the best CORE score (40.81 vs. 40.43). Every other MTP method is worse than NTP on both perplexities.
  3. Best look-ahead: LightMTP has the lowest anticipation cost and the best Countdown accuracy (ahead of both Meta variants). LightMTP-S also has the best strict GSM8K score. On MBPP (coding), LightMTP makes fewer structural errors than NTP and outperforms Meta-L.
  4. Every other method trades something away. Token-level methods degrade language modeling at 10–32% extra compute, Future Summaries needs a separately pretrained teacher, and NextLat costs 19.7% more and falls behind LightMTP on downstream experiments. LightMTP is the only method in our comparison that improves look-ahead planning, answer-format adherence and code structure without measurable cost to general language modeling, at under 1.5% extra training compute.

How we addressed the reviewers’ concerns:

  1. PCwg: We added NITP to the related work and showed that the multi-token horizon beyond the NITP-like k=1 case drives the gains (Countdown 50.6% → 56.7% from k=1 to k=8). We also reran DeepSeek with its original single-module, 0.3→0.1 loss schedule, which stabilizes it but still leaves it behind LightMTP, and we added a dedicated cost discussion of target storage, pipeline transfer and speculative decoding. As of this time, Reviewer PCwg has not responded to the additional experiments we added to address their concerns.
  2. 5tx5: We provided the requested matched-FLOP comparison for all methods, including downstream tasks, where LightMTP-S surpasses Meta-L on Countdown, strict GSM8K and language modeling. We also added the missing experimental details. We thank Reviewer 5tx5 for raising their score in response to the matched-compute results.
  3. uoDc: We added NextLat as a baseline, and explained that its 19.7% overhead stems mostly from its KL term (consistent with the slowdowns measured in their paper). We also explained the choice of cosine loss (L2 and contrastive losses performed worse in early experiments) and showed that LightMTP’s effect is not explained by a small gradient: Future Summaries uses the same loss with a different target and has a two-orders-of-magnitude higher anticipation cost.

We are grateful to the reviewers, whose suggestions have substantially strengthened the paper. We hope our revisions and the discussion are reflected in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.