- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies the implicit bias of a jointly trained softmax attention head under a binary classification problem. The authors first establish a directional convergence result showing that the effective classifier converges to the solution of a max-margin problem with constraints encoded by , depending on the full learning dynamics. They complete their result by characterizing under additional geometric assumptions. Then, they illustrate their results with numerical experiments.
Critical strengths
The core contribution is the directional convergence of jointly trained attention to the solution of a max-margin problem.
Critical weaknesses
The implicit bias characterization depends critically on , a quantity that depends on the gradient-flow trajectory. So the result by itself is not useful for characterizing the generalization properties of the reached solution. This limitation is partially addressed in Section 5, where, under additional geometric conditions, the authors show that the attention mass is localized in one preferred token. However, the deterministic assumptions (G1)-(G4) seem to be very strong: by (G1), the relevant token is immediately identifiable from the data; (G2) is a strong alignment assumption between all the sign tokens, and (G3) is uniform over all to guarantee that the flow preserves the geometry. It is not even clear that there exists a natural data distribution that satisfies all these assumptions. Hence, the results in Section 5 don't provide a sufficiently broad setting where one can characterize . As a consequence, it is unclear how useful Theorem 4.4 is.
Questions for authors
Can the authors provide a natural statistical data model under which assumptions (G1)–(G4) hold with high probability? It would considerably strengthen the results to provide a statistical data model for which the generalization performance of the jointly trained attention classifier can be analyzed.
4 comments
Thank you again for your constructive review. In our response and revised manuscript, we clarified when the limiting SVM in Theorem 4.4 can be identified before training and discussed the role of G2 and G3 in token selection. In Appendix I, we give a noisy repeated-measurement model in which G1–G4 hold almost surely under the stated initialization and analyze the generalization performance of the jointly trained attention classifier in this setting.
As the discussion period closes on November 18, we wanted to ask whether our response and revisions have addressed your concerns. We would be happy to clarify any remaining points or provide further details. Thank you for your time!
I thank the authors for their rebuttal. The statistical model proposed shows that the assumptions G1-G4 are not vacuous and can be satisfied. However, as I understand it, the model is trivial in that, at initialization, the untrained classifier already perfectly classifies the data. Consequently, I do not find it particularly convincing from a statistical perspective.
Thank you for pointing this out. You are right that the classifier in Proposition I.1 already has zero test error at initialization. We used that example to identify which tokens attention selects from uniform initial weights and which SVM the classifier converges to. To see whether training can also improve prediction, we considered a version of the repeated-measurement model in which even the best classifier has positive test error.
A step-response model with nonzero initial test error. Suppose a sensor records an upward or downward step, with the two directions and equally likely. Gaussian readout noise makes the class distributions overlap, so even the best classifier has positive test error. A first-order model for the response amplitude is [1, Eq. (6)]. We take an odd number of readings at equally spaced times after the step and use the middle reading as the query. The channels in reading have the form
The baseline offset is shared by all readings from one sample. The readout noises vary independently across readings. The label, offset, and readout noises are mutually independent.
When do G1–G4 hold? We take , , , and draw the entries of independently and uniformly from . Changing the sign of reverses every prediction, so the initial test error averages 50% over these draws. For fixed training size, the signed readings are positive and strictly increasing in every channel with high probability when is large and is small relative to the increments . This gives G1 before training begins.
G2 and G3 also involve the columns of at the checkpoint. On these ordered data, nonnegative value columns suffice for both conditions. In particular, every token-pair coefficient in G3 is nonnegative. Starting from , all five factors have positive entries at any fixed positive time. By continuity in the initial weights, they are also positive at that time with high probability for sufficiently small . These signs persist, and gradient flow drives the loss below the threshold in G4. For any fixed training size and any , joint gradient flow in this model reaches such a checkpoint with probability at least when the baseline noise, readout noise, and initialization radius are sufficiently small. On this event, Theorem 5.1 guarantees that attention concentrates on the measurement taken at the last sampling time in each training sequence . Corollary 5.2 then gives convergence of the unit-margin classifier to the hard-margin SVM fitted to .
Comparison with the best classifier based on the time average. At initialization, attention gives each reading the same weight. We therefore use the best classifier that observes only their average as a baseline. Let be the middle response and the average response. The best classifier given only the average reading has test error
where is the standard normal distribution function. Now consider a test sample with . For small enough readout perturbations, the signed query is positive and the signed measurement at the last sampling time exceeds all earlier signed measurements in every channel. On the training event above, stays nonnegative and the sum of its entries grows without bound by Theorem 5.1. The test logit gaps therefore also diverge, and attention concentrates on that measurement, giving the correct prediction.
As tends to zero, the resulting upper bound on asymptotic test error approaches , the error from the middle reading alone in this limit. For any positive, strictly increasing, strictly concave response on the observation interval, . On the training event above, the trained classifier therefore has strictly lower asymptotic test error than the best classifier using the time average for sufficiently small positive readout noise.
We appreciate your raising this question and hope you will consider this additional analysis in your final assessment.
Reference
[1] Magdalena Jaremkiewicz. Accurate measurement of unsteady state fluid temperature. Heat and Mass Transfer 53, 887–897 (2017).
We thank the reviewer for recognizing the contribution of our directional convergence result and for the detailed comments on the scope of our assumptions.
The revised Appendix I gives noisy data models in which the selected tokens and the limiting unit-margin classifier can be specified before training. It verifies G1–G4 for one family and proves selection without G2–G3 in an ordered-token setting with a prescribed initialization.
We first discuss the trajectory dependence, then the statistical models and their generalization properties. All page references below refer to the revised manuscript.
1. The role of Theorem 4.4.
Under the training assumptions of Part I, Theorem 4.4 proves convergence of the unit-margin classifier to the unique minimum-norm separator of all limiting signed features. The constraint set is trajectory-dependent in general. Part II identifies these features under additional geometric conditions. For the models below, this determines the limiting SVM from the sampled data before training begins.
2. A statistical model satisfying G1–G4.
Proposition I.1 (p. 35) considers repeated noisy measurements of a binary signal:
\qquad \xi_{ij}\overset{\mathrm{iid}}{\sim} \operatorname{Unif}([-\varepsilon,\varepsilon]^d).$$ The labels are i.i.d. uniform on $\\{-1,1\\}$ and independent of the noise. Assume $0 < \varepsilon < \beta_1 < \cdots < \beta_T$ and $\beta_{j+1}-\beta_j > 2\varepsilon$. Using the token mean as the query and the initialization specified in the proposition, G1–G4 hold almost surely for any sample size and token count $T\geq2$. The finite G3 certificate is verified directly, ensuring covariance control for every nonnegative scoring vector. This is an idealized model of repeated measurements with different signal strengths; the noise bounds preserve their ordering. G1 makes the strongest token identifiable from the data. The selection theorems show that joint training from uniform attention drives every logit gap over its competitors to infinity. **3. The scope of G2 and G3.** Proposition I.2 (pp. 36–37) proves selection and convergence of the unit-margin classifier to the selected-token SVM when signed tokens have a full coordinatewise ordering and positive means. It uses joint gradient flow from $Q_0=I_d$, $K_0=V_0=0$, and $W_0,a_0\geq0$ with $W_0a_0\neq0$, for either exponential or logistic loss. The noisy shared-background example on p. 37 can violate both G2 and G3. The head stays nonnegative, so the proof uses covariance control on nonnegative directions. G3 requires control for every direction in the cone generated by the signed tokens and initial value columns. Thus the additional ordering and specified initialization allow selection without G2–G3. **4. Generalization in these models.** Appendix I.3 (p. 37) analyzes independent test bags from both distributions. For both losses and the stated gradient-flow initializations, it proves, almost surely over the training data $\mathcal D$, $$\Pr\\!\left(Yf_{\Theta(t)}(X)\leq0\mid\mathcal D\right)=0 \qquad\text{for every finite }t > 0.$$ The proof gives a positive margin on every test bag in the support, along the same gradient-flow trajectories for which selection and the SVM limit are established.