- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how multi-token prediction (MTP) changes contextual cue selection in a one-layer attention model with two prediction heads. Building on a two-stage max-margin analysis previously developed for next-token prediction, the model first learns token-to-future readouts and then freezes them while training a shared attention matrix. Under strong structural assumptions, MTP assigns leading logarithmic score growth only to cues that jointly predict both future tokens, whereas NTP may allocate equal or larger leading coefficients to cues compatible only with the next token. The resulting asymptotic rules yield a constructed recombination where MTP's first head succeeds, and NTP fails. The experimental part is limited to a small model in a simple setting.
Critical strengths
-
The paper gives a clean and technically meaningful account of how an additional prediction horizon changes the margin constraints governing shared attention. The MTP/NTP comparison is unusually well controlled: the first-head readout, initialization, data, and training protocol are matched.
-
The theoretical predictions are tested directly in the synthetic setting, including score growth, alignment with the max-margin direction, and predictions on unseen recombinations. The paper is also commendably explicit about several boundaries of the analysis.
Critical weaknesses
-
The central theory is too far removed from practical MTP training to support the paper's mechanistic motivation.
-
The conceptual delta over the closest theoretical predecessor is narrower than the presentation may suggest. [1] already develop the one-layer, collocation-based, two-stage max-margin framework for NTP. The main new ingredient here is the analysis of the additional constraints induced by a second prediction head sharing attention. This is meaningful, but it requires stronger empirical evidence to establish sufficient significance.
-
The empirical evaluation is insufficient for the claimed mechanistic relevance. The TinyStories study uses a single-seed, one-layer, 128-dimensional attention-only model. The reported qualitative examples do not have held-out perplexity. Consequently, the paper does not show that the proposed mechanism explains real MTP gains.
[1]: Ruiquan Huang, Yingbin Liang, and Jing Yang. “Non-asymptotic Convergence of Training Transformers for Next-token Prediction.” NeurIPS 2024.
Questions for authors
My concern is mainly about the experiments. Can the authors identify full/near/far-only cues in an end-to-end trained multi-layer Transformer and show that the score changes predicted by the theory quantitatively mediate MTP's held-out next-token improvement?
6 comments
W2: The conceptual delta over the closest theoretical predecessor is narrower than the presentation may suggest. Huang et al. [3] already develop the one-layer, collocation-based, two-stage max-margin framework for NTP. The main new ingredient here is the analysis of the additional constraints induced by a second prediction head sharing attention. This is meaningful, but it requires stronger empirical evidence to establish sufficient significance.
Response: We thank the reviewer for this comparison. We agree that [3] is the closest predecessor and that we build on its framework. We would like to clarify what the second head changes, the technical challenges it raises and how we address them, and what empirical evidence the revision adds.
-
What we share with [3]. We adopt the collocation model and the two-stage protocol of [3], and, as there, we describe the outcome of attention training through a max-margin problem. The reviewer's summary of the new ingredient is accurate: it is the effect of a second prediction head that shares attention with the first.
-
What the second head changes. Under NTP, each training prefix constrains only the sum of the coefficients of its next-token-compatible cues, so the solution depends on how these cues overlap across prefixes. The second head turns this into a lower bound on each full cue's coefficient. As a result, the MTP solution is nonzero only on full cues, whereas NTP can give a near cue as much weight as a full cue, or more (Theorem 4.1, Corollary 4.2). This difference has a consequence that an NTP analysis cannot show: on a recombined context, the two objectives can make opposite next-token predictions, even when MTP uses only its first head (Corollary 5.3).
-
Technical challenges. Adding a second head is not a routine extension of [3], for two reasons. First, the two heads are coupled through the same scores. A near cue receives a positive update from the first head and a negative update from the second, so the sign of its net update can change during training (Lemma 3.3). The two losses therefore cannot be analyzed separately, and the analysis of a single head in [3] does not carry over. Second, this coupling extends across prefixes that share a query. Each of these prefixes contributes gradients from both heads to the same score vector. Determining the eventual cue ordering therefore requires analyzing the combined updates across those prefixes.
-
Our technical contribution. To address these challenges, we derive a Lyapunov-type inequality for the normalized scores that controls every cue's score at the same time (Lemma C.5). It gives for each cue's score, under both objectives and for arbitrary overlap among training prefixes. This yields an explicit rate at which full cues separate from the other cues under MTP (Remark 3.6), and it is what allows us to compare the two objectives and to derive the prediction rules in Section 5.
-
Empirical evidence. In synthetic experiments, the measured coefficients of are for the full cue and for the near cue under MTP, and and under NTP, in line with the two max-margin solutions (Figure 1). The revision adds experiments in jointly trained Transformers on TinyStories, with four paired seeds and one- and two-layer models. There, the second head favors the full-like cue in of matched pairs in the one-layer model and in and of pairs in the two layers of the two-layer model (see W1 and Q1).
In summary, we build on the framework of [3], and our contribution is to show how a second head that shares attention changes which cues are selected, to establish this through a new analysis of the coupled dynamics of the two heads, and to support it with synthetic and real-data experiments.
W3: The empirical evaluation is insufficient for the claimed mechanistic relevance. The TinyStories study uses a single-seed, one-layer, 128-dimensional attention-only model. The reported qualitative examples do not have held-out perplexity. Consequently, the paper does not show that the proposed mechanism explains real MTP gains.
Response: We thank the reviewer for these concrete suggestions. We have extended the evaluation in the three directions the reviewer points to: several seeds, a multi-layer Transformer, and held-out performance.
-
What we added. We now train matched NTP and MTP models with four paired seeds, for the one-layer model and for a two-layer Transformer with residual connections, LayerNorm, and MLPs. For both, we report an aggregate test of the mechanism on held-out prefixes (Figure 2, Appendices G.2.1 and G.3; see Q1) and held-out prediction performance (Section 6, Table 3, and Appendix G.3).
-
Held-out performance. In the two-layer model, the first head's validation perplexity is for NTP and for MTP, and its accuracy at the final query is and . For the second target, MTP's direct second head reaches , against for NTP's two-step rollout. The one-layer model shows the same pattern: the first head's validation loss is for NTP and for MTP, and second-target accuracy is for NTP's rollout and for MTP's second head.
-
How we read these results. In these small models, MTP predicts the second token better and the next token slightly worse. This agrees with earlier reports that MTP gains grow with model size [1]. The mechanism we study is nevertheless present: the second head's training signal favors the cue that supports both targets, in both layers and across all seeds (Q1). Our experiments therefore test the mechanism itself, namely how second-token supervision acts on shared attention, in models where it can be measured directly.
We thank the reviewer for the careful reading and thoughtful comments. We address the reviewer's questions below and have revised the paper accordingly. The changes are highlighted in blue in our revision. We hope our responses address the reviewer's concerns.
W1: The central theory is too far removed from practical MTP training to support the paper's mechanistic motivation.
Response: We thank the reviewer for raising this point. We agree that our model is much simpler than a practical MTP system. Below we explain what it shares with practical MTP, why we made the simplifications, and how we checked whether its prediction survives without them.
What our model shares with practical MTP. In parallel MTP [1], several heads predict different future tokens from the same prefix, and their gradients meet in a shared trunk. The same design is used in [2]. In both [1] and [2], the training loss is −∑t∑i=1klogP(xt+i∣x1,…,xt): the sum, over positions and heads, of the cross-entropy for predicting a future token from the same prefix. Our model keeps this structure in its simplest form: two heads predict the next two tokens from the same prefix through one shared attention layer, and our Stage 2 objective is this loss with k=2, taken at one prefix per training example. The loss of the second head therefore reaches the attention scores that the first head uses. This shared supervision is what we study.
Why we simplify. We make these simplifications to keep the analysis tractable, which is common practice in this line of theoretical work. Each choice follows prior analyses. A single attention layer followed by a prediction module is the standard setting in theoretical work on next-token prediction [3, 4, 5]. Linear attention with frozen readouts makes every margin of both heads linear in the attention scores, which allows us to solve the MTP and NTP max-margin problems in closed form; it is used for similar reasons in many theoretical studies of Transformers [6, 7, 8]. Training the readouts first and then freezing them lets us trace any difference between NTP and MTP to the second-token loss, and such staged training is a common way to make Transformer dynamics tractable [3, 9, 10].
Checking the prediction without these simplifications. We trained matched NTP and MTP models end to end on TinyStories with four paired seeds (Appendices G.2 and G.3). These models use softmax attention and learned embeddings, and they train the readouts jointly with attention; one of them is a two-layer Transformer with residual connections, LayerNorm, and MLPs. In each validation prefix, we pair a full-like cue and a near-like cue: both support the next target about equally, and only the full-like cue supports the second target. Our linear model predicts that the second head pushes attention toward the full-like cue, while the first head is indifferent. The jointly trained models behave in this way. The second head favors the full-like cue in 91.14% of pairs in the one-layer model, and in 71.01% and 62.09% of pairs in layers 1 and 2 of the two-layer model, compared with 31.91%, 48.53%, and 48.48% for the first head (four-seed means; see Q1 for the setup). These results suggest that the mechanism in our theory is still at work without the simplifications.
What the theory offers for practice. The purpose of our analysis is to give insight into how MTP works, and we believe it does so in three ways. First, it gives a concrete explanation of how MTP can improve next-token prediction even when the auxiliary head is discarded at inference [11]: the second head changes which cues the shared attention selects, and the first head inherits this selection (Corollary 5.3). Second, it predicts when MTP helps and when it does not: it helps when some cue supports both targets, and it can hurt when the two heads favor different cues (Remark C.7), which offers one explanation for why the reported benefits of MTP vary across settings [1, 2]. Third, it predicts the direction in which each head pushes attention between two cues, which we measured in jointly trained Transformers and found as predicted.
In summary, our model keeps the shared supervision that defines parallel MTP, its simplifications follow common practice for tractability, and its central prediction is observed in jointly trained Transformers. We therefore believe that the theory, although simplified, gives useful insight into how MTP shapes attention in practice.