acceptodds
Under review as a conference paper at ICLR 2027

Maximum-Margin Classification with Jointly Trained Softmax Attention

Abstract

We study the implicit bias of joint training in a single-layer, single-head softmax attention model for binary classification, with trainable query, key, value, output, and classifier factors. For exponential and logistic losses, we show that, once the training loss falls below a specified threshold, gradient flow and sufficiently small fixed-step gradient descent drive the loss to zero and monotonically increase a smoothed normalized margin. We further prove that the effective classifier converges in direction. More precisely, after normalization to unit margin, it converges to the unique Euclidean minimum-norm separator of all limiting signed features generated by the training trajectory. Under additional geometric conditions on the data and initialization, we prove for both losses that attention selects prescribed tokens, identifying the limiting classifier as the hard-margin SVM on those tokens. These results establish a maximum-margin characterization for joint attention and classifier training even though the model is not globally homogeneous. Experiments on MNIST illustrate normalized-margin growth and attention concentration, while a controlled example demonstrates token selection and classifier alignment with the predicted SVM.

Primary area:
learning theory
Keywords:
softmax attention, implicit bias, directional convergence, maximum margin

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qCJ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies the implicit bias of a jointly trained softmax attention head under a binary classification problem. The authors first establish a directional convergence result showing that the effective classifier VWaVWa converges to the solution of a max-margin problem with constraints encoded by Z∞Z_\infty, depending on the full learning dynamics. They complete their result by characterizing Z∞Z_\infty under additional geometric assumptions. Then, they illustrate their results with numerical experiments.

Critical strengths

The core contribution is the directional convergence of jointly trained attention to the solution of a max-margin problem.

Critical weaknesses

The implicit bias characterization depends critically on Z∞Z_\infty, a quantity that depends on the gradient-flow trajectory. So the result by itself is not useful for characterizing the generalization properties of the reached solution. This limitation is partially addressed in Section 5, where, under additional geometric conditions, the authors show that the attention mass is localized in one preferred token. However, the deterministic assumptions (G1)-(G4) seem to be very strong: by (G1), the relevant token is immediately identifiable from the data; (G2) is a strong alignment assumption between all the sign tokens, and (G3) is uniform over all ζ\zeta to guarantee that the flow preserves the geometry. It is not even clear that there exists a natural data distribution that satisfies all these assumptions. Hence, the results in Section 5 don't provide a sufficiently broad setting where one can characterize Z∞Z_\infty. As a consequence, it is unclear how useful Theorem 4.4 is.

Questions for authors

Can the authors provide a natural statistical data model under which assumptions (G1)–(G4) hold with high probability? It would considerably strengthen the results to provide a statistical data model for which the generalization performance of the jointly trained attention classifier can be analyzed.

4 comments

We thank the reviewer for recognizing the contribution of our directional convergence result and for the detailed comments on the scope of our assumptions.

The revised Appendix I gives noisy data models in which the selected tokens and the limiting unit-margin classifier can be specified before training. It verifies G1–G4 for one family and proves selection without G2–G3 in an ordered-token setting with a prescribed initialization.

We first discuss the trajectory dependence, then the statistical models and their generalization properties. All page references below refer to the revised manuscript.

1. The role of Theorem 4.4.

Under the training assumptions of Part I, Theorem 4.4 proves convergence of the unit-margin classifier to the unique minimum-norm separator of all limiting signed features. The constraint set is trajectory-dependent in general. Part II identifies these features under additional geometric conditions. For the models below, this determines the limiting SVM from the sampled data before training begins.

2. A statistical model satisfying G1–G4.

Proposition I.1 (p. 35) considers repeated noisy measurements of a binary signal:

\qquad \xi_{ij}\overset{\mathrm{iid}}{\sim} \operatorname{Unif}([-\varepsilon,\varepsilon]^d).$$ The labels are i.i.d. uniform on $\\{-1,1\\}$ and independent of the noise. Assume $0 < \varepsilon < \beta_1 < \cdots < \beta_T$ and $\beta_{j+1}-\beta_j > 2\varepsilon$. Using the token mean as the query and the initialization specified in the proposition, G1–G4 hold almost surely for any sample size and token count $T\geq2$. The finite G3 certificate is verified directly, ensuring covariance control for every nonnegative scoring vector. This is an idealized model of repeated measurements with different signal strengths; the noise bounds preserve their ordering. G1 makes the strongest token identifiable from the data. The selection theorems show that joint training from uniform attention drives every logit gap over its competitors to infinity. **3. The scope of G2 and G3.** Proposition I.2 (pp. 36–37) proves selection and convergence of the unit-margin classifier to the selected-token SVM when signed tokens have a full coordinatewise ordering and positive means. It uses joint gradient flow from $Q_0=I_d$, $K_0=V_0=0$, and $W_0,a_0\geq0$ with $W_0a_0\neq0$, for either exponential or logistic loss. The noisy shared-background example on p. 37 can violate both G2 and G3. The head stays nonnegative, so the proof uses covariance control on nonnegative directions. G3 requires control for every direction in the cone generated by the signed tokens and initial value columns. Thus the additional ordering and specified initialization allow selection without G2–G3. **4. Generalization in these models.** Appendix I.3 (p. 37) analyzes independent test bags from both distributions. For both losses and the stated gradient-flow initializations, it proves, almost surely over the training data $\mathcal D$, $$\Pr\\!\left(Yf_{\Theta(t)}(X)\leq0\mid\mathcal D\right)=0 \qquad\text{for every finite }t > 0.$$ The proof gives a positive margin on every test bag in the support, along the same gradient-flow trajectories for which selection and the SVM limit are established.

Thank you again for your constructive review. In our response and revised manuscript, we clarified when the limiting SVM in Theorem 4.4 can be identified before training and discussed the role of G2 and G3 in token selection. In Appendix I, we give a noisy repeated-measurement model in which G1–G4 hold almost surely under the stated initialization and analyze the generalization performance of the jointly trained attention classifier in this setting.

As the discussion period closes on November 18, we wanted to ask whether our response and revisions have addressed your concerns. We would be happy to clarify any remaining points or provide further details. Thank you for your time!

I thank the authors for their rebuttal. The statistical model proposed shows that the assumptions G1-G4 are not vacuous and can be satisfied. However, as I understand it, the model is trivial in that, at initialization, the untrained classifier already perfectly classifies the data. Consequently, I do not find it particularly convincing from a statistical perspective.

Thank you for pointing this out. You are right that the classifier in Proposition I.1 already has zero test error at initialization. We used that example to identify which tokens attention selects from uniform initial weights and which SVM the classifier converges to. To see whether training can also improve prediction, we considered a version of the repeated-measurement model in which even the best classifier has positive test error.

A step-response model with nonzero initial test error. Suppose a sensor records an upward or downward step, with the two directions Y=+1Y=+1 and Y=−1Y=-1 equally likely. Gaussian readout noise makes the class distributions overlap, so even the best classifier has positive test error. A first-order model for the response amplitude is g(t)=A(1−e−t/τ)g(t)=A(1-e^{-t/\tau}) [1, Eq. (6)]. We take an odd number T≥3T\geq3 of readings at equally spaced times after the step and use the middle reading as the query. The dd channels in reading jj have the form

Xj=Ygj1+B1+ξj,gj=g(tj).X_j=Yg_j\mathbf{1}+B\mathbf{1}+\xi_j,\qquad g_j=g(t_j).

The baseline offset B∼N(0,σB2)B\sim N(0,\sigma_B^2) is shared by all readings from one sample. The readout noises ξj∼N(0,σr2Id)\xi_j\sim N(0,\sigma_r^2 I_d) vary independently across readings. The label, offset, and readout noises are mutually independent.

When do G1–G4 hold? We take K0=0K_0=0, Q0=W0=IdQ_0=W_0=I_d, a0=1a_0=\mathbf{1}, and draw the entries of V0V_0 independently and uniformly from [−ρ,ρ][-\rho,\rho]. Changing the sign of V0V_0 reverses every prediction, so the initial test error averages 50% over these draws. For fixed training size, the signed readings YXjYX_j are positive and strictly increasing in every channel with high probability when g1/σBg_1/\sigma_B is large and σr\sigma_r is small relative to the increments gj+1−gjg_{j+1}-g_j. This gives G1 before training begins.

G2 and G3 also involve the columns of VV at the checkpoint. On these ordered data, nonnegative value columns suffice for both conditions. In particular, every token-pair coefficient in G3 is nonnegative. Starting from V0=0V_0=0, all five factors have positive entries at any fixed positive time. By continuity in the initial weights, they are also positive at that time with high probability for sufficiently small ρ\rho. These signs persist, and gradient flow drives the loss below the threshold in G4. For any fixed training size and any 0<δ<10<\delta<1, joint gradient flow in this model reaches such a checkpoint with probability at least 1−δ1-\delta when the baseline noise, readout noise, and initialization radius are sufficiently small. On this event, Theorem 5.1 guarantees that attention concentrates on the measurement Xi,TX_{i,T} taken at the last sampling time in each training sequence ii. Corollary 5.2 then gives convergence of the unit-margin classifier to the hard-margin SVM fitted to (Xi,T,Yi)i=1n\\{(X_{i,T},Y_i)\\}_{i=1}^n.

Comparison with the best classifier based on the time average. At initialization, attention gives each reading the same weight. We therefore use the best classifier that observes only their average as a baseline. Let gmg_m be the middle response and gˉ\bar g the average response. The best classifier given only the average reading has test error

Φ!(−gˉσB2+σr2/(Td)),\Phi\\!\left(-\frac{\bar g}{\sqrt{\sigma_B^2+\sigma_r^2/(Td)}}\right),

where Φ\Phi is the standard normal distribution function. Now consider a test sample with YB>−gmYB>-g_m. For small enough readout perturbations, the signed query is positive and the signed measurement at the last sampling time exceeds all earlier signed measurements in every channel. On the training event above, KQ⊤KQ^\top stays nonnegative and the sum of its entries grows without bound by Theorem 5.1. The test logit gaps therefore also diverge, and attention concentrates on that measurement, giving the correct prediction.

As σr\sigma_r tends to zero, the resulting upper bound on asymptotic test error approaches Φ(−gm/σB)\Phi(-g_m/\sigma_B), the error from the middle reading alone in this limit. For any positive, strictly increasing, strictly concave response on the observation interval, gm>gˉg_m>\bar g. On the training event above, the trained classifier therefore has strictly lower asymptotic test error than the best classifier using the time average for sufficiently small positive readout noise.

We appreciate your raising this question and hope you will consider this additional analysis in your final assessment.

Reference

[1] Magdalena Jaremkiewicz. Accurate measurement of unsteady state fluid temperature. Heat and Mass Transfer 53, 887–897 (2017).

Reviewer_pkWwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies implicit bias in a single-layer, single-head softmax attention classifier with all five factors Q, K, V, W and a trained jointly. After a low-loss checkpoint, gradient flow and sufficiently small fixed-step GD increase a smooth normalized margin and drive the exponential/logistic loss to zero. The effective classifier converges in direction and, after unit-margin normalization, solves a minimum-Euclidean-norm separation problem over all limiting signed features generated by the trajectory. Additional cooperative geometry and a finite covariance certificate identify these features as prescribed tokens. MNIST and a certified three-token example illustrate the predictions.

Critical strengths
  1. The main contribution addresses a genuine coupled-training difficulty: ordinary gradient flow and fixed-step GD train all five factors while attention features move, despite the absence of global homogeneity. The finite-length argument in Appendices C-D goes beyond a squared-motion budget and supports directional convergence. This is a meaningful advance over simply restating a fixed-head attention result.

  2. The characterization distinguishes the general trajectory-dependent limiting-feature SVM from the stronger selected-token SVM under additional conditions. Appendix E supplies a finite-support optimality certificate, and Appendix H.4 gives an explicit nonzero covariance certificate and a verifiable SVM construction. This makes the theoretical scope and illustrative example unusually concrete.

Critical weaknesses
  1. The general SVM characterization has limited predictive power: Z_infinity in Theorem 4.4 is generated by the training trajectory, so it does not determine the classifier from the data alone or imply initialization independence. Identifying prescribed tokens requires the stronger positivity, acute-cone, reward-gap and covariance-certificate conditions G1-G4. These are acknowledged sufficient conditions, not a hidden assumption or a counterexample to the theorem, but they substantially limit the range of attention geometries explained.

  2. The main empirical support for general directional convergence remains indirect. MNIST verifies smooth-margin growth and includes useful current-feature SVM diagnostics (Appendix H.3), but the instantaneous-feature SVM differs from the SVM over all limiting features in Theorem 4.4. Direct convergence toward a predetermined target is demonstrated chiefly in the certified four-example construction. Thus the experiments illustrate the mechanisms without establishing that the generic-data runs have reached the claimed asymptotic regime. These limitations keep my recommendation at weak acceptance rather than clear acceptance; I did not identify a confirmed fatal error in the checked main proof chain.

Questions for authors
  1. Beyond the constructed example, what simple, nontrivial data families satisfy G3, and what common geometries fail it? A short explanation of this boundary would materially clarify the significance of Part II.

  2. In the existing MNIST setup, how do classifier angular changes and the gap to the current-feature SVM behave with longer training? This would distinguish continuing feature evolution from the directional-convergence regime. I am not requesting a larger-model benchmark.

  3. Can you sharpen the comparison of assumptions and conclusions with near-homogeneous-network implicit-bias results and joint training of a direct attention matrix and decoder? The existing related-work discussion is useful; a concise comparison would establish the precise new guarantee.

My current recommendation is weak acceptance, mainly for the five-factor directional-convergence result and its fixed-step GD extension. The selected-token assumptions and finite-time evidence are the principal reservations.

2 comments

We thank the reviewer for the careful assessment of our joint-training result and the concrete suggestions. We have clarified the assumptions and proof overview in Section 5 (pp. 6–7) and added ordered-token models and selection results in Appendix I (pp. 35–37) of the revised manuscript.

We address your three questions in turn below.

1. When does G3 hold, and what makes it fail?

G3 supplies the covariance signs that keep attention updates nonnegative as the classifier evolves. Its finite certificate ensures that each signed-token coordinate has nonnegative covariance with the token score g⊤sg^\top s, for every head direction g∈Cg\in C and every nonnegative attention scoring vector. This is a condition on the joint token geometry; class separability alone does not imply it.

Noisy repeated measurements of a binary signal provide one sufficient family. Proposition I.1 (p. 35) allows independent bounded noise in every token and coordinate, with signal strengths separated enough to preserve coordinatewise ordering. With the nonnegative generators in that construction, every token-pair covariance contribution is nonnegative. The proposition verifies G1–G4 almost surely for arbitrary sample size, dimension, and token count T≥2T\ge2, using mean queries and its specified initialization.

The restriction becomes visible when different coordinates favor different competitors. For signed tokens (7,7),(6,1),(1,6)(7,7),(6,1),(1,6), the first token dominates both competitors, but the uniform-attention covariance satisfies [Σ(0)(1,6)⊤]1=−16/9[\Sigma(0)(1,6)^\top]_1=-16/9. Since (1,6)(1,6) is a cone generator, G3 fails; this negative action persists under small perturbations. Such competition can also be compatible with G3: Appendix H.4 gives an example where positive pair contributions compensate negative ones through the finite flow certificate.

2. What longer MNIST training shows.

We examined the saved 100,000-update trajectories of the paired seed-23 runs in Figure 1, with unchanged data and step size, and recomputed their current-feature SVMs.

Query scaleAngle to current SVM: 20k → 100kAttained margin fraction: 20k → 100k
cQ=3c_Q=324.58∘→22.54∘24.58^\circ\to22.54^\circ0.7635→0.80050.7635\to0.8005
cQ=30c_Q=3030.84∘→29.23∘30.84^\circ\to29.23^\circ0.7186→0.73370.7186\to0.7337

Over the equal intervals 20k–40k and 40k–60k, classifier rotations decrease from 0.970∘0.970^\circ to 0.497∘0.497^\circ for cQ=3c_Q=3, and from 1.230∘1.230^\circ to 0.660∘0.660^\circ for cQ=30c_Q=30. The current-feature SVM rotates more than the classifier in each interval.

This comparison shows why small classifier rotations alone are insufficient to diagnose maximum-margin convergence: the reference SVM is still moving. Alignment improves, but substantial angular gaps remain at 100k. Theorem 4.4 identifies the infinite-time limit without a finite-time angular error bound, so the iteration count alone does not certify proximity to the limiting SVM. The MNIST measurements show how classifier alignment and representation evolution interact over finite training; the controlled experiment complements them by tracking alignment with a predetermined SVM.

3. What our result adds to related implicit-bias results.

Cai et al. [1] prove directional convergence under global near-homogeneity and strong separability, with a parameter-space KKT characterization under additional regularity. For exact softmax attention, our proof separately controls attention growth and establishes finite length of the normalized head trajectory. This yields convergence to a unique convex SVM in feature space, even when the learned features have multiple accumulation points.

For joint attention–decoder training, Vasudeva et al. [2] use block-normalized updates on a Gaussian two-token model and obtain a constant-factor decoder-margin guarantee (Theorem 4); Julistiono et al. [3] characterize joint norm-constrained minimizers along a regularization path (Theorem 31). Our theorem identifies the limit of the unit-margin classifier along ordinary gradient flow and sufficiently small fixed-step GD on all five factors, from a low-loss checkpoint, for both exponential and logistic losses. Factorization matters here: the induced updates of KQ⊤KQ^\top and VWaVWa differ from direct-matrix and decoder updates. Part I allows these features to depend on the trajectory; Part II identifies them under additional geometry.

References

[1] Cai et al. Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks. ICML, 2025.

[2] Vasudeva et al. Implicit Bias and Fast Convergence Rates for Self-attention. TMLR, 2025.

[3] Julistiono et al. Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection. JMLR, 27(36):1–61, 2026.

Thank you for the clarifications and additional experiments. The sufficient family and counterexample make G3 more interpretable, and the related-work comparison clarifies the contribution.

The longer MNIST runs show improving alignment, but substantial gaps remain at 100k updates. This limits the finite-time evidence without contradicting the asymptotic theorem.

My main concerns are therefore partly addressed. Given the remaining geometric restrictions and limited empirical evidence for the asymptotic result, I maintain my recommendation of weak acceptance (3/4).

Reviewer_18HhOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2
Read review
Paper summary

The authors analyse the convergence and implicit bias of 1 layer attention+linear classifier models in binary classification, in which both the classifier and the attention are jointly trained. They prove convergence of the classifier to the max margin solution for the limiting features obtained by the attention layer; subject to additional assumptions, they also prove convergence of the attention to a dirac spike on a prescribed assignment of a distinguished token to each training datum.

Critical strengths

The problem of determining the dynamics and implicit bias of jointly-trained attention and classifier is an important, difficult and, to my knowledge, open, problem. This project has the potential to shed new light on the problem.

Critical weaknesses
  1. The paper is poorly written, with many instances of notation and terminology being used before being defined. While a lot can be inferred from context, it shouldn't be necessary to work this hard as a reader to understand the paper.
  2. The theory appears to split into two parts: one for the implicit bias of the classifier given the attractor of the features, and another for the implicit bias of the features. The former section was relatively easy to follow. The latter section appears to be overly complicated. Assumptions G1-G4 are not clearly defined as to their meaning, realism, or geometric intuition. I got the impression they are primarily technical and serve for proving the desired theorem rather than corresponding to something realistic or intuitive. Also, the proof mechanism for Theorem 5.1 is not easy to follow since it uses terms such as "reward and covariance signs", "clock", "attention drive" and "variance identity" which are never clearly defined.
  3. Theorem 5.3 seems quite strong. It seems to guarantee that prescribed tokens may be assigned arbitrarily to datapoints and the model will always be able to select the prescribed token. This seems like it should require a lot of capacity from the model, but there's no explicit indication in the assumptions of where this capacity is coming from.
Questions for authors

While the topic is important and the work has great potential, unfortunately I do not think it is ready to be published in its current form. It is essential that the paper be made clearer: notation and terminology should be carefully defined before use, and a higher level picture of the results must be painted for the reader.

The paper was sufficiently hard to follow that I am not sure I properly understand the picture being communicated by Part II, and cannot recommend the paper be accepted without this understanding. I have the following concrete questions.

  1. How realistic are the assumptions G1-G4?

  2. Am I correct in thinking that the non-negativity assumption here is maintained throughout training?

  3. Am I correct in reading the fitting problem as being "worst-case", covering an arbitrary assignment of a distinguished token to each datum?

  4. Is the rather opaque Assumption G.3 what's providing the capacity necessary to fit the problem?

  5. Could the assumptions possibly be relaxed in the case of an easier learning problem?

3 comments

We thank the reviewer for pointing out where the presentation was unclear. We have revised Section 5 to explain the role of each assumption and define the quantities used in the proof overview: token rewards, covariance, rescaled time, and the attention update matrix (pp. 6–7). The overview now separates preservation of the head's token preference from the argument that the attention logit gaps diverge.

Part I proves directional convergence under joint training from a low-loss checkpoint, with a sufficiently small fixed step size for GD. Part II adds geometric conditions that identify the limiting features as individual tokens.

We address your five questions in turn below. All page references refer to the revised manuscript.

1. How realistic are G1–G4?

An interpretable setting is repeated noisy measurements of a binary signal at different strengths. The new Proposition I.1 (p. 35) considers

\qquad \xi_{ij}\overset{\mathrm{iid}}{\sim}\operatorname{Unif}([-\varepsilon,\varepsilon]^d),$$ where $Y_i\in\\{-1,1\\}$, $0 < \beta_1 < \cdots < \beta_T$, $\beta_1 > \varepsilon > 0$, and $\beta_{j+1}-\beta_j > 2\varepsilon$. Using the token mean as the query and the initialization specified in the proposition, all four conditions hold almost surely, for any sample size and token count $T\geq2$. Token positions may be permuted across samples. The restriction is that the signal gaps dominate the noise, so the measurements remain ordered and support the same class. The common-cone and covariance requirements are strong sufficient conditions for preserving a consistent token preference during training. We therefore present this as an idealized setting for analyzing selection. Attention starts uniform in this construction; the selection theorems then give concentration on the strongest token and convergence of the unit-margin classifier to the selected-token SVM. **2. Is non-negativity maintained throughout training?** Yes, from the checkpoint satisfying G1–G4. $Q,K,W,a$ remain entrywise nonnegative, while each column of $V$ remains in the cone $C$. $V$ itself may have negative entries. This invariance follows from the gradient updates without projection, for both gradient flow and GD under the stated step-size condition. Section 5.1 now makes this distinction explicit (p. 7). **3. Does the theorem cover arbitrary assignments of distinguished tokens?** No. For fixed data, G1 requires the distinguished signed token $y_i x_{i j_i^\star}$ to exceed every competitor in every coordinate. An arbitrary assignment need not satisfy this requirement. Theorem 5.3 applies only to assignments satisfying G1–G4; Section 5.1 now explains this restriction immediately after stating the conditions (p. 6). **4. Is G3 providing the capacity to fit the assignment?** G1 already makes the prescribed rankings representable by a single shared attention map. Let $D_{ij}=y_i(x_{i j_i^\star}-x_{ij})$ and $t_i=y_iq_i$. Choosing $KQ^\top=I_d$ gives the logit gaps $$\Delta_{ij}=\frac{D_{ij}^{\top}t_i}{\sqrt d} > 0$$ simultaneously for every sample and competitor. Scaling this map makes all these gaps arbitrarily large. G3 controls the optimization: together with the other assumptions, its covariance condition keeps the attention updates nonnegative as the head evolves. Section 5.1 defines this covariance and explains its role (p. 7). **5. Could the assumptions be relaxed for an easier learning problem?** Yes, when the tokens have additional ordering structure. Proposition I.2 (pp. 36–37) assumes that the signed tokens within each sample have the same strict ordering in every coordinate, and that their mean is coordinatewise positive. Individual tokens may have negative coordinates and negative pairwise inner products. From $Q_0=I_d$, $K_0=V_0=0$, and $W_0,a_0\geq0$ with $W_0a_0\neq0$, joint gradient flow on all five factors selects the largest token, drives the loss to zero, and yields convergence to the selected-token SVM after unit-margin normalization. This holds for both exponential and logistic losses. The ordering allows this conclusion without G2–G3; the noisy example on p. 37 can violate both. The result replaces those conditions with a full ordering of the tokens and the specified initialization. It is proved for gradient flow; the original theorems also cover GD and allow unordered competitors.

Thank you for your clarifications. I have a better understanding of the result now and have raised my score to a 2. I remain of the opinion that much of the paper is difficult to read and would benefit from a thorough review and rewriting by the authors.

Thank you for your follow-up comments. We are glad our response helped clarify the result.

We believe the revised manuscript makes the main arguments in both parts of the paper easier to follow. Section 4.4 now explains why the initial estimate alone does not establish directional convergence and how the finite-length argument completes that step. Section 5 clarifies which tokens can be prescribed under the assumptions and defines the quantities used to show why attention concentrates on those tokens during training.

We appreciate the opportunity to discuss these points with you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.