- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies context–signal correspondence learning in balanced orthogonal mixtures. Similar setting has been analyzed for the case with two mixture components (K = 2), and its finite width guarantee uses a sufficient width that grows as the initialization scale decreases. Under the same training setting, this paper shows high-probability convergence of logistic risk to zero for every fixed number K ≥ 2 of mixture components in a finite-width two-head softmax Transformer.
Critical strengths
The paper contains comprehensive numerical results.
Critical weaknesses
- The paper is very difficult to read. No proper discussion about the motivation for the setup. From the paper, it is difficult to understand the importance of the problem studied. The introduction directly jumps to the considered setup, without elaborating why this setup is important and how this simple setup is connected to the practice.
- The theorems are presented without proper discussion. It is very difficult to parse what these theorems mean. Not clear what is new and what challenges the authors have to face to extend K=2 to K≥ 2. No insights were provided.
Questions for authors
- Why this setup is important? How is this setup connected to practice? What insights can we obtain by studying this setup?
- What is the meaning of these theorems?
- What are the main challenges when extending K=2 to K≥ 2? What is the novelty in the proof?
2 comments
Thanks for the clarifications. However, the issues still remain and the paper still needs significant changes. Hence, I maintain my rating.
We appreciate your observation that motivation and theorem interpretation need clearer exposition. We will open with the example, explain the inherited model and protocol together, and state the insights before the technical machinery.
(1) Why this setting matters
How does training make one token relevant to another? In pronoun resolution or selecting information for a question, context identifies what matters. Our setting isolates this operation: each input has context, its matching classification signal, and a distractor.
For example, [c1,+v1,−v2] has label +1, whereas [c2,−v2,+v1] has label −1. Context identifies the relevant signal. Balance and orthogonality remove frequency and geometric-correlation effects, making the formation of this correspondence accessible to a complete training analysis.
The proof explains how the representation develops. Stage 1 aligns readouts with signal directions. Stage 2 freezes those actual readouts and trains key/query factors on conflict-conditioned population loss. Signal queries acquire context-key scores that give different context contributions in the two heads, creating separating features. Stage 3 freezes these features; remaining inactive gates return, and the readout converges in direction to the hard-margin classifier determined by the learned representation.
The sufficient width stays fixed as initialization shrinks, even though the attention-training interval grows logarithmically. The learned-feature margin determines the exact coefficient in asymptotic risk decay. The analysis also identifies a transient geometric decrease followed by sustained growth. These results explain what attention training changes and how the representation governs the classifier.
(2) What the four theorems mean
Theorem 4.1: At every fixed K≥2, one finite-width Gaussian initialization learns through all three prescribed stages with high probability. For fixed bias, an admissible Stage 1 duration, and confidence, a single sufficient width works at each sufficiently small positive initialization scale. It also stays independent of the target risk.
Theorem 4.2: Features produced by attention determine the limiting classifier. If v∗ is the minimum-norm point of the labeled feature hull and γ=∥v∗∥, then W(t)/logt→v∗/γ2, tR3(t)→γ−2, and tW˙(t)→v∗/γ2. Feature geometry determines the readout direction and asymptotic speed.
Theorem 4.3: For fixed K≥3, the joint learning event yields signed endpoint attention comparisons. First-, second-, and third-order gaps resolve the token interactions behind separation.
Theorem 4.4: Successful learning can begin with a negative initial right derivative of the augmented feature margin on an event of positive probability. On the learning theorem's high-probability event, this margin grows after a width-dependent delay.
(3) What changes beyond two components
At the leading reference scores, context-attention probabilities for the two signal queries on a positive conflict satisfy p+,1∣2(0)−p+,1∣3(0)=2(K−2)z/(9K)+O(z2). Here z is the reference contrast. The first-order term vanishes at K=2 and is present for K≥3. The separation argument combines the distinct query contributions. The top Gram-growth space also has dimension K(K−1)/2 per head, so fluctuations occupy several fastest-growing directions.
The full joint-Gram analysis retains both self-Grams. At fixed K and confidence, the Stage 1 readout error is O(ω2/m). The full Gram error is O(ω2eΛKt/m) over Stage 2, where ΛK=2μ/[9(K−1)] and μ is the reference alignment. At reference contrast c0, this gives endpoint error O(c0/m) and initialization-uniform width. Directed gate-normal bounds handle asynchronous returns and connect separation to permanent activation. We will organize the proof overview around these obstacles and estimates.
(4) Evidence supporting the mechanism
Across 200 trajectories with paired seeds, two widths, and five initialization scales, normalized Stage 1 errors have medians 0.078 and 0.074. Every relative Gram-error maximum on the 257-point grid occurs at launch; the endpoint residual after linear transport is at most 0.285% of the transported initial error. A complete K=3, width-4096 certificate validates the path from initialization to invariant-tail entry and vanishing risk. We will present these findings in the numerical section. The supplement contains per-seed data, technical notes, and the runnable certificate.