acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Attend to: Training Dynamics of Multi-Token Prediction

Abstract

Multi-token prediction (MTP) has improved next-token prediction in several large-scale settings, but how its additional supervision changes contextual cue selection remains unclear. In this work, we study a one-layer linear-attention model with two prediction heads sharing a contextual representation. We analyze a two-stage procedure that learns token-to-future associations in the readouts, then freezes the readouts and trains shared attention by gradient flow. Under explicit structural assumptions, we characterize the max-margin solutions governing attention-score growth. We show that MTP assigns leading logarithmic growth only to cues jointly predictive of both future tokens. In contrast, next-token prediction (NTP) assigns leading coefficients according to the occurrence and overlap of next-token compatible cues, allowing ambiguous cues to receive as much or more leading growth as jointly predictive ones. We derive the resulting prediction rules on recombined contexts. An illustrative example shows that, as training time grows, the correct next-token probability tends to one under MTP and to zero under NTP on the same context, even when inference uses only MTP's first head. Synthetic experiments support the predicted cue preferences, max-margin directions, and prediction separation, while TinyStories experiments illustrate learned token-to-future associations and attention differences between NTP and MTP.

Primary area:
learning theory
Keywords:
Multi-token Prediction, Attention Dynamics, Implicit Bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LTuPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how multi-token prediction (MTP) changes contextual cue selection in a one-layer attention model with two prediction heads. Building on a two-stage max-margin analysis previously developed for next-token prediction, the model first learns token-to-future readouts and then freezes them while training a shared attention matrix. Under strong structural assumptions, MTP assigns leading logarithmic score growth only to cues that jointly predict both future tokens, whereas NTP may allocate equal or larger leading coefficients to cues compatible only with the next token. The resulting asymptotic rules yield a constructed recombination where MTP's first head succeeds, and NTP fails. The experimental part is limited to a small model in a simple setting.

Critical strengths
  1. The paper gives a clean and technically meaningful account of how an additional prediction horizon changes the margin constraints governing shared attention. The MTP/NTP comparison is unusually well controlled: the first-head readout, initialization, data, and training protocol are matched.

  2. The theoretical predictions are tested directly in the synthetic setting, including score growth, alignment with the max-margin direction, and predictions on unseen recombinations. The paper is also commendably explicit about several boundaries of the analysis.

Critical weaknesses
  1. The central theory is too far removed from practical MTP training to support the paper's mechanistic motivation.

  2. The conceptual delta over the closest theoretical predecessor is narrower than the presentation may suggest. [1] already develop the one-layer, collocation-based, two-stage max-margin framework for NTP. The main new ingredient here is the analysis of the additional constraints induced by a second prediction head sharing attention. This is meaningful, but it requires stronger empirical evidence to establish sufficient significance.

  3. The empirical evaluation is insufficient for the claimed mechanistic relevance. The TinyStories study uses a single-seed, one-layer, 128-dimensional attention-only model. The reported qualitative examples do not have held-out perplexity. Consequently, the paper does not show that the proposed mechanism explains real MTP gains.

[1]: Ruiquan Huang, Yingbin Liang, and Jing Yang. “Non-asymptotic Convergence of Training Transformers for Next-token Prediction.” NeurIPS 2024.

Questions for authors

My concern is mainly about the experiments. Can the authors identify full/near/far-only cues in an end-to-end trained multi-layer Transformer and show that the score changes predicted by the theory quantitatively mediate MTP's held-out next-token improvement?

6 comments

We thank the reviewer for the careful reading and thoughtful comments. We address the reviewer's questions below and have revised the paper accordingly. The changes are highlighted in blue in our revision. We hope our responses address the reviewer's concerns.

W1: The central theory is too far removed from practical MTP training to support the paper's mechanistic motivation.

Response: We thank the reviewer for raising this point. We agree that our model is much simpler than a practical MTP system. Below we explain what it shares with practical MTP, why we made the simplifications, and how we checked whether its prediction survives without them.

  • What our model shares with practical MTP. In parallel MTP [1], several heads predict different future tokens from the same prefix, and their gradients meet in a shared trunk. The same design is used in [2]. In both [1] and [2], the training loss is −∑t∑i=1klog⁡P(xt+i∣x1,…,xt)-\sum_{t}\sum_{i=1}^{k}\log P(x_{t+i}\mid x_1,\ldots,x_t): the sum, over positions and heads, of the cross-entropy for predicting a future token from the same prefix. Our model keeps this structure in its simplest form: two heads predict the next two tokens from the same prefix through one shared attention layer, and our Stage 2 objective is this loss with k=2k=2, taken at one prefix per training example. The loss of the second head therefore reaches the attention scores that the first head uses. This shared supervision is what we study.

  • Why we simplify. We make these simplifications to keep the analysis tractable, which is common practice in this line of theoretical work. Each choice follows prior analyses. A single attention layer followed by a prediction module is the standard setting in theoretical work on next-token prediction [3, 4, 5]. Linear attention with frozen readouts makes every margin of both heads linear in the attention scores, which allows us to solve the MTP and NTP max-margin problems in closed form; it is used for similar reasons in many theoretical studies of Transformers [6, 7, 8]. Training the readouts first and then freezing them lets us trace any difference between NTP and MTP to the second-token loss, and such staged training is a common way to make Transformer dynamics tractable [3, 9, 10].

  • Checking the prediction without these simplifications. We trained matched NTP and MTP models end to end on TinyStories with four paired seeds (Appendices G.2 and G.3). These models use softmax attention and learned embeddings, and they train the readouts jointly with attention; one of them is a two-layer Transformer with residual connections, LayerNorm, and MLPs. In each validation prefix, we pair a full-like cue and a near-like cue: both support the next target about equally, and only the full-like cue supports the second target. Our linear model predicts that the second head pushes attention toward the full-like cue, while the first head is indifferent. The jointly trained models behave in this way. The second head favors the full-like cue in 91.14%91.14\% of pairs in the one-layer model, and in 71.01%71.01\% and 62.09%62.09\% of pairs in layers 1 and 2 of the two-layer model, compared with 31.91%31.91\%, 48.53%48.53\%, and 48.48%48.48\% for the first head (four-seed means; see Q1 for the setup). These results suggest that the mechanism in our theory is still at work without the simplifications.

  • What the theory offers for practice. The purpose of our analysis is to give insight into how MTP works, and we believe it does so in three ways. First, it gives a concrete explanation of how MTP can improve next-token prediction even when the auxiliary head is discarded at inference [11]: the second head changes which cues the shared attention selects, and the first head inherits this selection (Corollary 5.3). Second, it predicts when MTP helps and when it does not: it helps when some cue supports both targets, and it can hurt when the two heads favor different cues (Remark C.7), which offers one explanation for why the reported benefits of MTP vary across settings [1, 2]. Third, it predicts the direction in which each head pushes attention between two cues, which we measured in jointly trained Transformers and found as predicted.

In summary, our model keeps the shared supervision that defines parallel MTP, its simplifications follow common practice for tractability, and its central prediction is observed in jointly trained Transformers. We therefore believe that the theory, although simplified, gives useful insight into how MTP shapes attention in practice.

W2: The conceptual delta over the closest theoretical predecessor is narrower than the presentation may suggest. Huang et al. [3] already develop the one-layer, collocation-based, two-stage max-margin framework for NTP. The main new ingredient here is the analysis of the additional constraints induced by a second prediction head sharing attention. This is meaningful, but it requires stronger empirical evidence to establish sufficient significance.

Response: We thank the reviewer for this comparison. We agree that [3] is the closest predecessor and that we build on its framework. We would like to clarify what the second head changes, the technical challenges it raises and how we address them, and what empirical evidence the revision adds.

  • What we share with [3]. We adopt the collocation model and the two-stage protocol of [3], and, as there, we describe the outcome of attention training through a max-margin problem. The reviewer's summary of the new ingredient is accurate: it is the effect of a second prediction head that shares attention with the first.

  • What the second head changes. Under NTP, each training prefix constrains only the sum of the coefficients of its next-token-compatible cues, so the solution depends on how these cues overlap across prefixes. The second head turns this into a lower bound on each full cue's coefficient. As a result, the MTP solution is nonzero only on full cues, whereas NTP can give a near cue as much weight as a full cue, or more (Theorem 4.1, Corollary 4.2). This difference has a consequence that an NTP analysis cannot show: on a recombined context, the two objectives can make opposite next-token predictions, even when MTP uses only its first head (Corollary 5.3).

  • Technical challenges. Adding a second head is not a routine extension of [3], for two reasons. First, the two heads are coupled through the same scores. A near cue receives a positive update from the first head and a negative update from the second, so the sign of its net update can change during training (Lemma 3.3). The two losses therefore cannot be analyzed separately, and the analysis of a single head in [3] does not carry over. Second, this coupling extends across prefixes that share a query. Each of these prefixes contributes gradients from both heads to the same score vector. Determining the eventual cue ordering therefore requires analyzing the combined updates across those prefixes.

  • Our technical contribution. To address these challenges, we derive a Lyapunov-type inequality for the normalized scores that controls every cue's score at the same time (Lemma C.5). It gives sq(t)=cq⋆log⁡t+O(log⁡t)s_q(t)=c_q^\star\log t+O(\sqrt{\log t}) for each cue's score, under both objectives and for arbitrary overlap among training prefixes. This yields an explicit rate at which full cues separate from the other cues under MTP (Remark 3.6), and it is what allows us to compare the two objectives and to derive the prediction rules in Section 5.

  • Empirical evidence. In synthetic experiments, the measured coefficients of log⁡t\log t are 1.531.53 for the full cue and 00 for the near cue under MTP, and 0.400.40 and 0.690.69 under NTP, in line with the two max-margin solutions (Figure 1). The revision adds experiments in jointly trained Transformers on TinyStories, with four paired seeds and one- and two-layer models. There, the second head favors the full-like cue in 91.14%91.14\% of matched pairs in the one-layer model and in 71.01%71.01\% and 62.09%62.09\% of pairs in the two layers of the two-layer model (see W1 and Q1).

In summary, we build on the framework of [3], and our contribution is to show how a second head that shares attention changes which cues are selected, to establish this through a new analysis of the coupled dynamics of the two heads, and to support it with synthetic and real-data experiments.

W3: The empirical evaluation is insufficient for the claimed mechanistic relevance. The TinyStories study uses a single-seed, one-layer, 128-dimensional attention-only model. The reported qualitative examples do not have held-out perplexity. Consequently, the paper does not show that the proposed mechanism explains real MTP gains.

Response: We thank the reviewer for these concrete suggestions. We have extended the evaluation in the three directions the reviewer points to: several seeds, a multi-layer Transformer, and held-out performance.

  • What we added. We now train matched NTP and MTP models with four paired seeds, for the one-layer model and for a two-layer Transformer with residual connections, LayerNorm, and MLPs. For both, we report an aggregate test of the mechanism on held-out prefixes (Figure 2, Appendices G.2.1 and G.3; see Q1) and held-out prediction performance (Section 6, Table 3, and Appendix G.3).

  • Held-out performance. In the two-layer model, the first head's validation perplexity is 11.6911.69 for NTP and 12.6812.68 for MTP, and its accuracy at the final query is 40.90±0.18%40.90\pm0.18\% and 39.72±0.69%39.72\pm0.69\%. For the second target, MTP's direct second head reaches 23.03±0.66%23.03\pm0.66\%, against 20.79±0.97%20.79\pm0.97\% for NTP's two-step rollout. The one-layer model shows the same pattern: the first head's validation loss is 4.344.34 for NTP and 4.694.69 for MTP, and second-target accuracy is 4.43%4.43\% for NTP's rollout and 9.39%9.39\% for MTP's second head.

  • How we read these results. In these small models, MTP predicts the second token better and the next token slightly worse. This agrees with earlier reports that MTP gains grow with model size [1]. The mechanism we study is nevertheless present: the second head's training signal favors the cue that supports both targets, in both layers and across all seeds (Q1). Our experiments therefore test the mechanism itself, namely how second-token supervision acts on shared attention, in models where it can be measured directly.

Reviewer_hKUnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how multi-token prediction (MTP) changes which contextual cues attention selects, using a one-layer linear-attention model with one-hot tokens and two prediction heads whose readouts are trained first and then frozen. Under structural assumptions (for example, each training sample contains exactly one "full" cue that predicts both future tokens), gradient flow on the attention scores converges in direction to explicit max-margin solutions in which only full cues grow at leading order under MTP, whereas under next-token prediction (NTP) cues that predict only the next token can grow at least as fast. Consequently, on recombined contexts whose target is defined by the full cue, MTP assigns probability tending to one to the correct token while NTP's probability can tend to zero, which synthetic experiments confirm and examples from TinyStories illustrate.

Critical strengths
  1. The mechanism is clear and the results are explicit. The second head requires each jointly predictive cue's coefficient to be at least 1/G individually, whereas NTP constrains only sums over cues compatible with the next token, and this yields explicit max-margin directions, growth rates and prediction rules. I went through the main proofs (Prop. 3.4, Thm. C.6, Lemma D.2, Thm. D.3 and Cor. D.4) and they appear correct to me.

  2. The paper is candid about where its conclusions fail (Remarks C.7 and C.8), and the synthetic experiments agree closely with the theory, including the predicted growth rate and a softmax-attention variant.

Critical weaknesses
  1. The simplifications are strong enough that much of the analysis is close to known results. With one-hot tokens, a merged key-query matrix, linear attention and frozen readouts, the second training stage is a linearly separable multiclass logistic regression, so convergence to the hard-margin direction follows from existing implicit-bias results (Soudry et al., 2018, Ravi et al., 2024), which typically give sharper residuals than the O(log⁡t)O(\sqrt{\log t}) obtained here. The new contribution is mainly the solution of two short max-margin programs within the collocation framework of Huang et al. (2024). The conclusions also depend on knife-edge conditions such as exactly equal readout gaps across heads and a single full cue per sample, whereas MTP in practice trains readouts and attention jointly, so it is unclear which conclusions carry over.

  2. The advantage on next-token prediction is largely built into the setup and is not tested on real data. In Cor. 5.3, Example 5.4 and Table 1, the correct continuation of a recombined context is defined by its full cue. Under an equally plausible data model in which the continuation follows the majority of cues compatible with the next token, NTP would be the correct one, so the result demonstrates a different inductive bias rather than better next-token prediction, and the abstract invites the broader reading. The TinyStories evidence is qualitative (a single seed, four hand-selected fragments and a jointly trained model that violates the assumptions), and in two of the four fragments MTP attends to tokens whose readouts rank the observed next token very low. An aggregate evaluation over several seeds, for example attention mass on jointly predictive versus next-only tokens together with first-head validation loss, would make the connection to practice considerably more convincing.

Questions for authors
  1. Can Theorems 3.5 and 4.1 be derived directly from multiclass implicit-bias results such as Soudry et al. (2018) or Ravi et al. (2024)? If so, what does the Lyapunov argument add, and is the O(log⁡t)O(\sqrt{\log t}) residual tight?
  2. Does the preference for full cues survive joint training of readouts and attention, or naturally unequal gaps between the heads? Even a synthetic experiment would be informative.
7 comments

We thank the reviewer for the careful reading and thoughtful comments. We address the reviewer's questions below and have revised the paper accordingly. The changes are highlighted in blue in our revision. We hope our responses address the reviewer's concerns.

W1: With one-hot tokens, a merged key-query matrix, linear attention, and frozen readouts, Stage 2 is a linearly separable multiclass logistic regression, so convergence to the hard-margin direction follows from existing implicit-bias results [1, 2], which typically give sharper residuals than our O(log⁡t)O(\sqrt{\log t}). The new contribution is mainly the solution of two max-margin programs within the framework of [3]. The conclusions also depend on knife-edge conditions, such as exactly equal readout gaps and a single full cue per sample, whereas practical MTP trains readouts and attention jointly.

Response: We thank the reviewer for the careful reading of our proofs. We agree that Stage 2 is a convex, linearly separable problem, and that convergence in direction to a max-margin solution follows from known implicit-bias theory, as we note in Section 3.3. Our contribution is to identify which solutions the two objectives select and to quantify the convergence for each cue's score.

  • The max-margin solutions and their comparison. As in prior work [1, 2, 3], we describe the outcome of training through a max-margin problem. What is new is how the second head changes this problem: it turns a constraint on the sum of next-token-compatible cue coefficients into a lower bound on each full cue's coefficient. As a result, the MTP solution is nonzero only on full cues, whereas the NTP solution depends on how compatible cues overlap across training prefixes and has no closed form in general (Theorem 4.1, Corollary 4.2). We then derive what this difference implies for predictions on new contexts (Section 5).

  • How our setting differs from the cited theorems, and what our analysis adds. In [1, 2], each class has its own weight vector, and all classes share the features of an example. In our model, a single vector of attention scores is shared by all classes and by both heads, and the frozen readouts make the features differ by class. This changes the analysis in three ways.

    • The two heads are coupled through the same scores. A near cue receives a positive update from the first head and a negative update from the second, so the sign of its net update can change during training (Lemma 3.3). The two losses therefore cannot be analyzed separately.
    • The same cue appears in many training prefixes. Each cue has one score per query, and that score accumulates updates from every prefix with that query. Our NTP result holds for arbitrary overlap among these prefixes.
    • Our data cannot be assumed to be generic. The sharper residuals in the cited works rely on a condition on the dataset. In [1], the bounded residual holds for almost all datasets, that is, for data in general position. For multiclass problems, [2, Theorem 3.4] imposes such a condition as an assumption (Assumption 4.1). Our data are not in general position: the margin constraints are determined by the cue roles, so many of them are identical, and the two heads often impose the same constraint. We therefore need an argument that does not rely on such a condition.

    Our technical contribution addresses these points. We derive a Lyapunov-type inequality for the normalized scores that controls every cue's score at the same time (Lemma C.5). It gives sq(t)=cq⋆log⁡t+O(log⁡t)s_q(t)=c_q^\star\log t+O(\sqrt{\log t}) for each cue's score, for both objectives and for every dataset satisfying our assumptions, without the additional condition of [2]. General results for homogeneous models [4] show that the score vector converges in direction to a max-margin solution. They do not give a rate for this directional convergence, and hence no explicit residual for individual scores. Our expansion supplies one, and with it an explicit rate at which full cues separate from the other cues (Remark 3.6) and at which the predictions in Section 5 converge.

  • The residual. Our Lyapunov inequality bounds the squared deviation of the scores from the max-margin direction by the growth of the average full-cue score, which is of order log⁡t\log t; taking the square root gives the O(log⁡t)O(\sqrt{\log t}) residual. It is probably not tight in general: for binary classification, [1] obtains an O(log⁡log⁡t)O(\log\log t) residual on degenerate datasets through a refined analysis. Extending that analysis to our multiclass setting with shared scores is beyond what our Lyapunov argument gives. We note that our cue-selection and prediction results depend only on the leading coefficients, so they do not rely on a sharper residual.

Our response to W1 continues in the next text box.

Response to W1 (continued)

  • The conditions described as knife-edge. The conclusions hold beyond the exact conditions the reviewer mentions.
    • Unequal readout gaps. Remark C.8 shows that the MTP solution remains nonzero only on full cues whenever G1≥G2>0G_1\geq G_2>0. Equal gaps are the boundary of this range rather than an isolated point. When G1<G2G_1<G_2, near cues can enter the solution, and the remark gives an example.
    • Several full cues. The new Remark C.9 relaxes the unique-full-cue condition to at least one full cue per training prefix. The MTP solution remains nonzero only on full cues, and the weights among them depend on how they co-occur.
    • Joint training. Please see our response to Q2, where we report the cue preference in jointly trained models.

W2: The next-token advantage is largely built into the setup: in Corollary 5.3, Example 5.4, and Table 1, the correct continuation of a recombined context is defined by its full cue. Under a data model in which the continuation follows the majority of next-token-compatible cues, NTP would be correct, so the result shows a different inductive bias rather than better next-token prediction, and the abstract invites the broader reading. The TinyStories evidence is qualitative. An aggregate evaluation over several seeds, for example attention mass on jointly predictive versus next-only tokens together with first-head validation loss, would be more convincing.

Response: We thank the reviewer for the careful reading. We agree that which prediction counts as correct depends on how the continuation is defined, and we have revised the abstract to state this condition. We would like to clarify that the difference between the two predictions is not built into the setup. It is produced by training on the same data, as we explain below. Following the reviewer's suggestion, we now report a four-seed evaluation of cue-level training signals and held-out next-token loss.

  • What the recombination result shows. Both models are trained on the same prefixes and the same first targets, and both fit the training data. They differ in which cues they rely on, and this becomes visible when familiar cues appear in a new combination. In Example 5.4, the leading first-head logit coefficients on the recombined prefix are 1/21/2 for \texttt{bake} and 2/32/3 for \texttt{play} under NTP, and 11 and 00 under MTP. These predictions are determined by training, before any continuation is chosen. If the continuation instead followed \texttt{keyboards} and \texttt{backstage}, NTP's prediction would be the correct one.

  • Why we evaluate against the full cue. Corollary 5.3 answers a specific question: when the cue that predicted both training targets is the one that determines the continuation, which objective follows it? We think this case is of interest because the full cue is the only cue whose associations agree with both observed targets. We agree that it is one evaluation rule among several, and we have made this clear in the abstract.

  • Aggregate evaluation over four seeds. For the one-layer cue-gradient analysis, we select full-like and near-like pairs separately using each seed's final MTP checkpoint. This lets the cue labels reflect the associations learned in each run. The four seeds yield 859, 859, 869, and 839 pairs, respectively. We hold each seed's pairs fixed across its checkpoints and average the four within-seed fractions with equal weight (Appendix G.2.1).

    At the final checkpoint, the second head's loss locally favors increasing the full-like cue's score relative to the near-like cue's score in 91.14%91.14\% of pairs, compared with 31.91%31.91\% for the first head. The joint loss favors the same transfer in 69.46%69.46\% of pairs (Figure 2). Across the four paired NTP and MTP runs, first-head validation NLL is 4.34±0.024.34\pm0.02 for NTP and 4.69±0.014.69\pm0.01 for MTP (Table 3).

  • How we read these numbers. The second target supplies a training signal that favors cues associated with both targets, and that preference remains visible in the joint loss. MTP nevertheless has higher held-out next-token loss here. Our theory gives a setting where cue selection can improve next-token prediction, namely controlled recombination in which the full cue determines the continuation and the two objectives make different predictions (Corollary 5.3 and Example 5.4).

Reviewer_32LzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper analyses the multi-token prediction setting in LLMs, where two different readout heads make predictions with a shared attention head on the two last tokens. The paper describes a data and algorithmic setting that makes the differences between the multi-toekn prediction and next-token prediction clear to observe. The paper then proves theoretically the parameter dynamics of these two setting under a gradient descent like algorithm.

Critical strengths

The data setting and the assumptions are clearly thought out to bring about the desired distinction in behaviour elegantly.

The analysis looks reasonable and clear. The theorems are believable (have not gone through the appendix to check proofs) and are motivated and explained well.

Critical weaknesses

The paper makes a chain of assumptions/simplifications which are all perhaps individually reasonable for the purpose of simplification. But put together it gives the impression of too artificial a construction. Simplification chain, as I see it is below.

  1. Single head linear attention without residual connections.
  2. data assumptions A1, which essentially makes no sequential dependence etc, and clearly designed to make head1 and head2 perform distinctlive roles.
  3. Two stage training dynamics with readouts trained first and frozen
  4. first stage requiring an entirely different training set and actually knowing the tokens n_1(x) and n_2(x) for all x, and not just the standard next token data [x,y_1, y_2].
  5. The next token head not being trained to predict y_2 as well. (i.e.) the natural option would three loss terms with the first head predicting y_1 with x, and y_2 with [x,y_1] as contexts, and second head predicting y_2 with x. But there are only two loss terms.

Even if 1,2 and 3 above are acceptable due to the complexity of the full problem, 4 and 5 seem to go over the threshold for me. Making this too artificial and seems constructed to create the conclusions already made.

It is possible some of these can be fixed (especially, point 4 above). But it would be a significantly different paper.

Questions for authors

Questions:

Am I misunderstanding something and entirely off-base somewhere?

Future work suggestions:

Can you address points 4 and 5 above in the weaknesses above with the same setting?

10 comments

We thank the reviewer for the careful reading and thoughtful comments. We address the reviewer's questions below and have revised the paper accordingly. The changes are highlighted in blue in our revision. We hope our responses address the reviewer's concerns.

W1: The theoretical model uses a single attention layer with linear attention and no residual connection. Does the predicted cue preference depend on these simplifications?

Response: Thank you for this comment. We agree that, taken together, these simplifications make our model much simpler than a practical Transformer, and we appreciate the reviewer's note that each of them is reasonable on its own. Below we explain why we made each choice and how we checked whether the predicted cue preference survives without them.

  • Why one attention layer. We wanted both prediction heads to act on the same attention scores, so that we could see directly how the second-token loss changes them. A single attention layer followed by a prediction module is also the standard setting in theoretical work on next-token prediction [1, 2, 3], which lets us compare our MTP results with that line of work.

  • Why linear attention. Once the readouts are frozen, linear attention makes every output logit, and hence every margin of both heads, linear in the attention scores. This is what allows us to solve the MTP and NTP max-margin problems in closed form. Linear attention is used for similar reasons in many theoretical studies of Transformers, for example in analyses of in-context learning [4, 5, 6, 7] and of why Adam trains Transformers faster than SGD [8].

  • Why no residual connection. In our model, adding a residual connection changes only the initialization. Because the query qq is itself in the attention support Xi\mathcal X_i and tokens are one-hot, the attention output is ∑x∈Xisx,qex\sum_{x\in\mathcal X_i}s_{x,q}e_x, and adding the residual term eqe_q is the same as raising sq,qs_{q,q} by one. In other words, the model with a residual connection at WkqW_{kq} is our model at Wkq+IW_{kq}+I, trained from 11⊤+I\mathbf 1\mathbf 1^\top+I instead of 11⊤\mathbf 1\mathbf 1^\top. Concretely:

    • Stage 1. A singleton input [x][x] now gives 2ex2e_x rather than exe_x. All readout columns still follow the same recurrence, so both heads still learn the collocations with a common gap GG; only its value changes.
    • Stage 2. The score dynamics keep the same form; each query's own score simply starts at 22 instead of 11.
    • Max-margin solutions. These do not depend on the initialization. MTP still assigns 1/G1/G to each full cue and 00 to every other cue (Proposition 3.4), and the NTP problem keeps the same constraints with the new GG (Section 4).
    • Comparison. NTP and MTP start from the same shifted point, so the comparison stays matched.

    So a residual connection changes where training starts, but not which cues the max-margin solutions select.

  • Checking the prediction without these simplifications. To see whether the cue preference survives in more realistic models, we trained matched NTP and MTP models end to end on TinyStories with four paired seeds (Appendices G.2 and G.3). These models use softmax attention and learned embeddings, and they train the readouts jointly with attention. We study a one-layer model and a two-layer Transformer with residual connections, LayerNorm, and MLPs. Since these models have no specified collocations, we identify full-like and near-like cues from each seed's final MTP model, using how strongly a single token predicts each target relative to the target's frequency: Sh(x,y(h))=log⁡ph(y(h)∣[x])−log⁡pref,h(y(h)),S_h(x,y^{(h)})=\log p_h(y^{(h)}\mid[x])-\log p_{\mathrm{ref},h}(y^{(h)}), where [x][x] is the singleton input and pref,hp_{\mathrm{ref},h} is the smoothed target frequency in training. In each validation prefix, we pair two earlier positions jj and kk whose tokens support the next target about equally (S1S_1 positive and within 0.250.25 nats), while only xjx_j supports the second target (S2(xj,y(2))>0≥S2(xk,y(2))S_2(x_j,y^{(2)})>0\geq S_2(x_k,y^{(2)})). We then ask which way each head would like to move attention between them: Th=(∂sj−∂sk)log⁡ph(y(h)∣X),T_h=(\partial_{s_j}-\partial_{s_k})\log p_h(y^{(h)}\mid X), computed at the pre-softmax scores of the final query. A positive ThT_h means that shifting a little score from kk to jj raises the probability of the head-hh target.

Our response to W1 continues in the next text box.

Response to W1 (continued)

  • What we found. Our linear model predicts that this shift leaves the first head unchanged (T1=0T_1=0) and helps the second head (T2>0T_2>0), so the preference for full cues should come from the second head. The jointly trained models behave in this way (four-seed means at the final checkpoint; Figure 2 and Appendices G.2.1 and G.3).

    • Second head. T2>0T_2>0 in 91.14%91.14\% of pairs in the one-layer model, and in 71.01%71.01\% and 62.09%62.09\% of pairs in layers 1 and 2 of the two-layer model.
    • First head. T1>0T_1>0 in 31.91%31.91\% of pairs in the one-layer model, and in 48.53%48.53\% and 48.48%48.48\% in the two-layer model. The two-layer model shows no consistent first-head preference, close to the predicted T1=0T_1=0, whereas the one-layer first head leans toward the near-like cue.
    • Full MTP objective. The equally weighted loss favors the full-like cue in 69.46%69.46\% of pairs in the one-layer model, and in 63.11%63.11\% and 56.33%56.33\% in layers 1 and 2 of the two-layer model. The second head's pull usually outweighs the first head's.
  • What we take from this. These results suggest that the mechanism in our theory is still at work without the simplifications: among cues that support the next token, second-token supervision moves attention toward the cue that also predicts the following token. The quantity ThT_h is the empirical counterpart of head hh's term in our score dynamics (Lemma 3.3), so it shows directly which way each loss pushes attention between the two cues.

W2: Assumption 1 (Section 2.3) seems to remove sequential dependence and to be designed so that the two prediction heads play different roles. Is it too restrictive, and what happens when it is relaxed?

Response: We understand why Assumption 1 can look tailored to our conclusion, and we agree that it is a strong structural condition. In short, it does not remove dependence between tokens and targets, and it does not predetermine which cues the heads rely on. Below we explain what it imposes, which part of it the result really needs, and which parts can be relaxed.

  • What Assumption 1 restricts. Its conditions concern which prefix tokens support each target; they say nothing about the order of tokens in the prefix. The two targets are also not independent, since the full cue determines both of them.

  • Assumption 1 does not predetermine what the two heads learn. The two heads have different targets by the definition of MTP: the first predicts y(1)y^{(1)} and the second predicts y(2)y^{(2)}. Assumption 1 does not assign them any further roles. If anything, it makes them compatible. Prefix consistency requires every token that supports y(2)y^{(2)} to also support y(1)y^{(1)}, so the cues favored by the second head, the full cues, also support the first head's target. What Assumption 1 does fix is the role of each cue relative to the targets, namely full, near, or uninformative. It does not fix which cues attention selects. That is the outcome of training, and it differs between the two objectives. Under the same assumption and the same data, NTP may give near cues as much leading growth as full cues, or more (Corollary 4.2), whereas MTP gives leading growth only to full cues (Theorem 3.5). The difference therefore comes from the second-token loss, not from the assumption.

  • Prefix consistency and where the result stops holding. Prefix consistency excludes far-only cues, which support the second target but not the first. We impose it because it keeps the two heads compatible: every cue that the second head favors also helps the first head, so the second-token loss can only refine the choice among next-token-compatible cues. We do not expect this to hold for every prefix in natural text, where a token may well be informative about a later token without supporting the next one. Remark C.7 shows what can happen in that case. When a prefix has a near cue and a far-only cue but no full cue, the two heads ask for opposite orderings of the same scores, and MTP's next-token probability stays below 1/21/2 for all tt while NTP's tends to one. We view this as part of the result rather than a gap in it: the same analysis that explains when second-token supervision helps, namely when some cue supports both targets, also identifies when it can hurt, namely when the heads disagree and no cue reconciles them.

  • The unique full cue can be relaxed to at least one full cue. Several full cues are already allowed at test time: Corollary E.6 shows that MTP's first head then predicts the plurality of their first-horizon collocations. We have since found that the training condition can be relaxed in the same way. Suppose each training prefix contains at least one full cue, with the other conditions unchanged. Then the MTP max-margin solution still puts weight only on full cues, so near and uninformative cues still receive no leading growth. The reason is simple. Full cues support both targets, so raising their scores helps every margin of both heads. Meanwhile, the second head's comparison with an inactive class already requires the full-cue coefficients in each prefix to add up to at least 1/G1/G. If we take any solution and set the coefficients of all other cues to zero, every margin is still satisfied, and the solution becomes shorter. The minimum-norm solution therefore has no weight outside the full cues. What changes with several full cues is how the weight is shared among them: this now depends on how the full cues appear together across training prefixes, and some full cue may end up with zero weight. We have added this result as Remark C.9.

  • The remaining conditions are mainly technical.

    • Distinct tokens. With repeated tokens, each token's contribution to a margin is simply multiplied by its count, so the constraints become count-weighted.
    • Inactive class. This holds whenever the vocabulary has more entries than the prefix has distinct associated outputs, in particular whenever V>LiV>L_i, which is the usual situation in language modeling.
    • Role consistency. This lets a single score per token--query pair describe each token's role. Theorem F.1 covers the case without role consistency, prefix consistency, or a unique full cue. For general frozen readouts, it gives the score dynamics and a condition under which one cue's score stays above another's: its readout margins are at least as large in every training prefix, for both heads and every competitor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.