acceptodds
Under review as a conference paper at ICLR 2027

Towards Understanding How Deep Transformers Perform Classification with Unlabeled Context

Abstract

Transformers achieve remarkable performance in various tasks in natural language processing and computer vision. Despite the empirical success of transformers, the impact of model depth on performance remains not fully understood. In this paper, we consider a classification task with an unlabeled context, and theoretically investigate the performance of a deep transformer model trained by gradient descent in learning this task. Our analysis demonstrates that deeper transformer layers can produce features with higher signal-to-noise ratio (SNR), and as a result, linear classification based on features of deeper layers can yield higher prediction accuracy. Experiments on both synthetic and real-world datasets are conducted to validate our theory. Our study sheds light on the benefits and impact of transformer model depth, and provides insights into how unlabeled context can enhance transformer performance on supervised tasks.

Primary area:
learning theory
Keywords:
deep transformer, unlabeled context, signal-to-noise ratio

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_sBtjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies supervised binary classification with unlabeled context using a deep residual attention-only transformer. Under an orthogonal Gaussian-noise model, zero initialization, and population gradient descent, it claims small population loss after the second update and increasing layer-wise signal-to-noise ratios. The proposed mechanism is learned within-class aggregation. A downstream maximum-margin analysis is used to derive depth-dependent classification risk bounds, and synthetic, noisy-image, and task-trained GPT-2 experiments illustrate related layer-wise trends.

The objective of connecting training dynamics, denoising, and downstream representation quality is interesting. However, the main theoretical guarantees currently depend on unresolved distributional and derivational steps under the stated data model.

Critical strengths
  1. Connecting the training dynamics of deep softmax attention to layer-wise signal/noise decomposition and downstream classification is a worthwhile theoretical objective. Tracking signal and noise coefficients provides an interpretable framework for examining within-class aggregation.

  2. The paper attempts to analyze a trained multilayer model rather than only exhibiting hand-constructed denoising weights. The detailed appendices and experiments across synthetic data, noisy MNIST/CIFAR, and richer task-trained architectures make the proposed phenomenon concrete, although they do not resolve the proof issues below.

Critical weaknesses
  1. The core training and denoising proofs use calculations inconsistent with the stated independent-label model. On p.29, step (i) replaces the sample-dependent logistic derivative with a constant −r_1. Let q=(2^{L+1}−1)/N and a=η(1+q)/2. The paper’s first-step result and uniform attention give h_N=x_N+q∑_j x_j and v^(1)=aμ, hence the query margin is a||μ||²(1+q y_N∑_j y_j). Under independent Rademacher labels, this is random, so the derivative cannot simply be pulled out of the expectation as the stated constant. Separately, p.43 replaces both class counts with N/2 and treats exact balance as without loss of generality under token reordering. Reordering cannot change class counts. Conditioning on balance also changes the first-step expected gradient, so adding a balance assumption alone does not preserve the existing formulas. These issues affect Lemma D.1 and the subsequent analysis built on it, making the central guarantees unestablished in the current version.

  2. The concentration and downstream-risk arguments require additional justification. Lemma G.4 assumes constant coefficients, whereas the coefficients used in Eq. E.27 are generated by attention and depend on the same noise vectors. A fixed-coefficient Gaussian argument does not directly apply; a uniform concentration result or another valid treatment is needed. Appendix F also needs to verify the full assumptions of the cited maximum-margin theorem for the transformed features, rather than relying only on a sub-Gaussian tail claim. Finally, decreasing risk upper bounds do not prove monotonicity of actual risks. These are substantive gaps in the paper’s main theoretical contribution, beyond the first weakness.

Questions for authors

The following questions could materially change my assessment:

  1. Can you correct the second-step derivative calculation on p.29 and the exact-balance substitution on p.43, specify the label distribution for the revised guarantees, and trace the corrections through Lemma D.1 and the subsequent training/SNR analysis? Please account for the fact that conditioning on exact balance changes the first-step gradient.

  2. Can you provide concentration bounds valid for noise-dependent attention coefficients and verify the full assumptions of the external maximum-margin theorem for the transformed features? Please distinguish a decreasing risk upper bound from an ordering of actual classification risks.

  3. What evidence isolates the benefit of unlabeled context from depth alone? A focused no-context or context-replacement control would clarify the mechanism. Please also specify the GPT-2 attention mask and token layout so that access to context by the probed token is clear.

My recommendation is rejection in the current version because the main theoretical contribution rests on the unresolved steps above. I would reconsider substantially if the proofs are repaired under a clearly stated distribution and the conclusions are appropriately scoped.

2 comments
AuthorsOpenReview ↗

Response 1


W1:The core training and denoising proofs use calculations inconsistent with the stated independent-label model. On p.29, step (i) replaces the sample-dependent logistic derivative with a constant −r_1. Let q=(2^{L+1}−1)/N and a=η(1+q)/2. The paper’s first-step result and uniform attention give h_N=x_N+q∑_j x_j and v^(1)=aμ, hence the query margin is a||μ||²(1+q y_N∑_j y_j). Under independent Rademacher labels, this is random, so the derivative cannot simply be pulled out of the expectation as the stated constant. Separately, p.43 replaces both class counts with N/2 and treats exact balance as without loss of generality under token reordering. Reordering cannot change class counts. Conditioning on balance also changes the first-step expected gradient, so adding a balance assumption alone does not preserve the existing formulas. These issues affect Lemma D.1 and the subsequent analysis built on it, making the central guarantees unestablished in the current version.

Q1.Can you correct the second-step derivative calculation on p.29 and the exact-balance substitution on p.43, specify the label distribution for the revised guarantees, and trace the corrections through Lemma D.1 and the subsequent training/SNR analysis? Please account for the fact that conditioning on exact balance changes the first-step gradient.

AW1&AQ1: For step (i) on p.29, the calculation of the second-step logistic derivative is based on the balanced population assumption. At the second iteration, the logistic derivative is ℓ′(yNf(X)eN)\ell'(y_Nf(X)e_N) and we have that $y_Nf(X) e_N= y_N {v^{(1)}}^{\top}(x_N+\frac{2^{L}-1}{N}\sum_{j=1}^{N} x_j) = {v^{(1)}}^{\top}\mu+ {v^{(1)}}^{\top}\mu y_N\sum_{j=1}^{N} y_j= {v^{(1)}}^{\top}\mu $ where the first equality is by the first step training results that Wl(1)=0d×d,l=0,…,LW_{l}^{(1)}= \boldsymbol{0}_{d\times d},l=0,\dots,L and the last equality is by the balanced population assumption gives that ∑j=1Nyj=0\sum_{j=1}^{N}y_j=0. Therefore, we have ℓ′(yNf(X)eN)=ℓ′(v(1)⊤μ)\ell'(y_N f(X)e_N)= \ell'( {v^{(1)}}^{\top}\mu), thus this term is a constant that can be pulled out of the expectation.

For the joint sampling scheme, the balanced population assumption will be applied to the NN context tokens. The query token will be re-indexed as xN+1x_{N+1}, with an independent Rademacher label yN+1y_{N+1}. The query token is masked from the intermediate context-processing layers and enters only through the final query readout. Thus, the balance assumption applies only to the context tokens and does not constrain the query label.

Regarding the reordering of the token in Appendix E.2 (p.43), we agree that reordering alone cannot give equal class counts. The equation yi=1,i=1,...,N/2;yi=−1,i=N/2,...,Ny_i = 1, i = 1,...,N/2; \quad y_i = −1, i = N/2,...,N follows the balanced population assumption, which gives two label groups (+1/−1+1/-1) of equal size N/2N/2. The reordering step is introduced only after this assumption, for notational convenience in the proof. We will revise the proof to distinguish clearly between the balance assumption and the subsequent reordering, and we will clarify that this theorem is proved under a balanced group setting.


W2: The concentration and downstream-risk arguments require additional justification. Lemma G.4 assumes constant coefficients, whereas the coefficients used in Eq. E.27 are generated by attention and depend on the same noise vectors. A fixed-coefficient Gaussian argument does not directly apply; a uniform concentration result or another valid treatment is needed.

AW2: Thanks for pointing this out. The description in Lemma G.4 is not accurate for the case where the coefficients also depend on the noise vectors, and we corrected the Lemma G.4 that can be used in Equation(E.27) as follows: For ξ‾i=∑k=1Nβi,kξk\overline{\xi}_i=\sum_{k=1}^{N}\beta_{i,k}\xi_k, since βi,k\beta_{i,k} are bounded and ξk\xi_k i.i.d follows N(0,σnoise2I)N(\mathbf{0}, \sigma_{\mathrm{noise}}^2 \mathbf{I}), then ξ‾i\overline{\xi}_i is sub Gaussian random variable. Therefore, ∥∣ξ‾i∥∣2\||\overline{\xi}_i\||^2 is a sub-exponential random variable, and Bernstein inequality for a sub-exponential random variable can therefore be applied to it.


AuthorsOpenReview ↗

Response 2


Q2.Can you provide concentration bounds valid for noise-dependent attention coefficients and verify the full assumptions of the external maximum-margin theorem for the transformed features? Please distinguish a decreasing risk upper bound from an ordering of actual classification risks.

W2 Appendix F also needs to verify the full assumptions of the cited maximum-margin theorem for the transformed features, rather than relying only on a sub-Gaussian tail claim. Finally, decreasing risk upper bounds do not prove monotonicity of actual risks. These are substantive gaps in the paper’s main theoretical contribution, beyond the first weakness.

AQ2&AW2: Regarding the condition verification for applying Theorem 3.1 in Cao et al. (2021) in Appendix F, Theorem 3.1 of Cao et al. (2021) assumes the data point is from a mixture of two sub-Gaussian distributions with means μ\mu and −μ−\mu respectively and the same covariance matrix Σ\Sigma.

In Appendix F, we prove that for every layer ll, the last-column token hl,N,m=αl,N,mμ+∑k=1Nβl,k,N,mξk,mh_{l,N,m}=\alpha_{l,N,m} \mu +\sum_{k=1}^{N}\beta_{l,k,N,m}\xi_{k,m} used in the downstream logistic regression satisfies this data model: Specifically, our data can be decomposed as hl,N,m=yNE[∣αl,N,m∣]μ+(αl,N,m−yNE[∣αl,N,m∣])μ+∑k=1Nβl,k,N,mξk,mh_{l,N,m}=y_N\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu+(\alpha_{l,N,m}-y_N\mathbb{E}[|\alpha_{l,N,m}|])\mu+\sum_{k=1}^{N}\beta_{l,k,N,m}\xi_{k,m}. Here the first term is the signal part, and the second and third terms together form the noise part h~l,N,m=(αl,N,m−yNE[∣αl,N,m∣])μ+∑k=1Nβl,k,N,mξk,m\tilde{h}_{l,N,m}=(\alpha_{l,N,m}-y_N \mathbb{E} \big[|\alpha_{l,N,m}|\big])\mu+\sum_{k=1}^{N} \beta_{l,k,N,m}\xi_{k,m}, which has zero mean and covariance matrix Σl=Cov(h~l,N,m,h~l,N,m)\Sigma_l={\rm Cov}\big(\tilde{h}_{l,N,m}, \tilde{h}_{l,N,m}\big). Given αl,N,m,βl,k,N,m\alpha_{l,N,m},\beta_{l,k,N,m} are all bounded and ξk,m\xi_{k,m} are i.i.d Gaussian random variables for all mm, it follows that the noise term is sub-Gaussian even if they are not independent. Hence, hl,N,mh_{l,N,m} is also sub-Gaussian random variable. The two class means are E[∣αl,N,m∣]μ\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu and −E[∣αl,N,m∣]μ-\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu with the same covariance matrix Σl\Sigma_l. Therefore, all conditions of Thm 3.1 in Cao et al. (2021) are satisfied, and the proof in Appendix F remains valid.

Our theory provides an upper bound on the population risk, showing that the increase of SNR across layers corresponds to features with more effectively captured signals, resulting in improved performance on the downstream tasks. Our experimental results support this conclusion and can help address your concern. In particular, it is observed that hidden states from deeper transformer layers achieve higher test accuracy on the downstream task, suggesting that the bound is informative in practice.


Q3:What evidence isolates the benefit of unlabeled context from depth alone? A focused no-context or context-replacement control would clarify the mechanism. Please also specify the GPT-2 attention mask and token layout so that access to context by the probed token is clear.

AQ3: We would like to emphasize that our attention-only deep transformer in Eq. (2.1) already provides a natural context-free control for studying the denoising effect of the unlabeled context. After the first training step, the vector vv covers the direction of the signal μ\mu, but this trained linear classifier vv is not involved in our denoising effect analysis or in the downstream evaluation. Without such a classifier vv, we use only the hidden state hl,N,m,l=0,…,Lh_{l,N,m}, l=0,\dots, L as features in the downstream classification task.

Moreover, in this attention-stacked transformer model (2.1), if all context tokens are masked, the output hidden state reduces to the query token xNx_N itself. Therefore, the model (2.1) naturally provides a baseline without context, and the effect of the context can be assessed by comparing the hidden layer features with the original input token xNx_N.

Our experiments on attention-only transformers (2.1) in Sec. 5 show that, compared with the input token, utilizing the unlabeled context enables the features to capture more signal, resulting in improved downstream prediction accuracy.

Therefore, both our theoretical framework and our experiments already include a control for the unlabeled context, and they both support the conclusion that the self-attention mechanism leverages the unlabeled context to improve feature quality.

For GPT-2 experiments, we have updated our GPT-2 experiment results in the revised paper PDF (Appendix C.5-C.7, highlighted in blue in the manuscript). All experiments now use GPT-2’s standard causal mask. We also explicitly specify the query token positions used (the last few tokens) in the downstream task in the detailed description of each experimental setting. For example, in the MNIST experiments with GPT-2, the downstream classifier uses the eighth token, hl,8,mh_{l,8,m} from layer l,l=0,…,12l, l=0,\dots, 12.


Reviewer_zwjmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how unlabeled context helps deep attention-only transformers learn representations that are easier to classify. In a binary Gaussian mixture with a fixed signal direction, the authors analyze full-layer gradient descent and give a training guarantee, a layer-wise lower bound on the signal-to-noise ratio (SNR), and a downstream risk bound for a max-margin classifier that decreases with depth. They explain the effect as within-class averaging by the trained attention. Experiments on synthetic data, MNIST, CIFAR-10, and a small GPT-2 model show layer-wise gains in SNR and probe accuracy.

Critical strengths

Important question and an ambitious analysis. The paper studies why each layer of a deep transformer can improve the representation, and connects full-layer training of multi-layer softmax attention, the layer-wise SNR, and downstream risk in one framework. This goes beyond prior work that gives constructions without training guarantees or trains only the first layer.

Clear and interpretable mechanism. The trained attention puts more weight on same-class tokens, and the residual connections amplify the signal relative to the noise. The sorted attention maps (Figures 2 and 5) show this block structure, and the intuition is easy to follow.

Critical weaknesses

W1. The training guarantee in Theorem 3.2 rests on a closed-form second update that appears to be incorrect. The proof (Lemma D.1, step (i) on p. 29) replaces the logistic derivative with a constant and moves it outside the expectation. After the first update, attention is uniform, so the margin depends on the sum of the context labels; it is therefore random and correlated with the feature that the derivative multiplies. My exact calculation for independent labels (N = 64, L = 6, unit step size and signal norm) gives a second-step coefficient of about -3.40, whereas the formula in the paper gives 2.04. Since the subsequent training and denoising arguments build on this closed form, the current proof does not establish Theorem 3.2.

W2. The main denoising result, the layer-wise SNR bound in (3.1), requires a depth restriction that is not stated. The paper claims that "every self-attention layer contributes to noise reduction" (p. 2). However, the final step of the proof (p. 58) requires roughly 4l≲N4^l \lesssim \sqrt{N}, and Appendix F uses 4l≤N/2+14^l \le N/2+1, whereas Condition 3.1 only states L=Θ(log⁡N)L=\Theta(\log N), without the constants needed to imply either restriction. This is consistent with Figure 1a, where the SNR flattens after about five layers. Two further steps also require justification: the proof assumes exactly half of the tokens in each class "without loss of generality" (p. 43), although reordering tokens does not change the random class counts, and it applies Lemma G.4, which assumes fixed coefficients, to noise-dependent attention weights (p. 55). The claim should be stated with an explicit depth condition, and these two steps should be justified.

W3. The downstream claim in Theorem 3.3 is not fully justified. The paper states that deeper features lead to higher prediction accuracy (p. 5). The proof (Appendix F) only argues that the noise is sub-Gaussian and then applies Theorem 3.1 of Cao et al. (2021), which also imposes conditions on the noise structure and covariance. These conditions are not verified for features whose noise depends on the input through attention. In addition, a smaller risk upper bound at deeper layers does not imply a smaller actual risk. The statement on p. 5 should be limited to the improvement in the bound.

W4. The contribution of the unlabeled context, which is central to the title and the conclusion, is not isolated. Under Definition 2.1, the noise is orthogonal to the signal direction μ\mu, so a classifier that knows this direction classifies the query without error at layer 0, and the first population gradient step already recovers this direction. Unlabeled samples may still help a downstream probe that does not know μ\mu, but the experiments do not separate this effect from the effects of depth or supervised training. A control that keeps the query fixed and removes the context, or replaces it with tokens that carry no class signal, would address this. For GPT-2, the probe uses the first token, but the attention mask is not specified; under a standard causal mask, this token cannot attend to the context.

W5. The experiments do not fully support the claims made for them in the main text. The paper states that the synthetic experiments "adhering to our settings confirm our theoretical results" (p. 2), but these experiments (Appendix C.1) use d = 100 with N = 50, momentum, gradient clipping, balanced classes, and a loss on all tokens. They illustrate the proposed mechanism but do not directly test the quantitative bound in (3.1). The conclusion also states that the GPT-2 experiments confirm the findings "in broader scenarios" (p. 9). However, the GPT-2 probes on CIFAR-10 and 10-class MNIST (Appendix C) use early stopping based on test loss and test accuracy (pp. 25–26), and no separate validation set is described, so the reported test sets may also have been used for model selection. No variation across repeated runs or error bars are reported.

Minor

  • CIFAR-10 center filtering also removes test examples (pp. 20–21). Results without filtering would clarify the scope of the reported improvements.
  • The "among the first" convergence claim (p. 2) should be discussed in relation to Gao et al. (2024), with the differences in setting made explicit. Work on token clustering and rank collapse, such as Dong et al. (2021) and Geshkovski et al. (2023), is also relevant.
  • Equation (2.1) contains one more attention block than the stated number of layers.
Questions for authors

Q1. Can you recompute the second update in Lemma D.1 while keeping the dependence of the logistic derivative on the context labels? Can the proof of Theorem 3.2 be repaired, and what additional conditions on L and N are needed?

Q2. For what explicit depth range does the bound in (3.1) hold? Is the intended label distribution independent Rademacher or exactly balanced? If it is balanced, how do the first-step parameters in Lemma D.1 change?

Q3. How do the layer features satisfy the conditions of Theorem 3.1 in Cao et al. (2021), given that their noise depends on the input through attention? If these conditions do not hold, can you provide another justification for Theorem 3.3, or limit the accuracy claim to an improvement in the risk bound?

Q4. With the query kept fixed, how do the layer-wise SNR and probe accuracy change when the context tokens are replaced with tokens that carry no class signal, or removed? Using the same trained checkpoints and a matched-length replacement control would help assess the contribution of context at inference time.

Q5. The GPT-2 experiments state that the "first column token" of each layer output is probed (e.g., p. 22). Is this correct, and which attention mask is used? Are the sets used for early stopping (pp. 25–26) different from the reported test sets? If not, do the trends in Figures 19 and 21 hold with a separate validation set?

4 comments
AuthorsOpenReview ↗

Response 1


W1. The training guarantee in Theorem 3.2 rests on a closed-form second update that appears to be incorrect. The proof (Lemma D.1, step (i) on p. 29) replaces the logistic derivative with a constant and moves it outside the expectation. After the first update, attention is uniform, so the margin depends on the sum of the context labels; it is therefore random and correlated with the feature that the derivative multiplies. My exact calculation for independent labels (N = 64, L = 6, unit step size and signal norm) gives a second-step coefficient of about -3.40, whereas the formula in the paper gives 2.04. Since the subsequent training and denoising arguments build on this closed form, the current proof does not establish Theorem 3.2.

Q1. Can you recompute the second update in Lemma D.1 while keeping the dependence of the logistic derivative on the context labels? Can the proof of Theorem 3.2 be repaired, and what additional conditions on L and N are needed?

Q2: Is the intended label distribution independent Rademacher or exactly balanced? If it is balanced, how do the first-step parameters in Lemma D.1 change?

AW1&AQ1&AQ2: For step (i) on p.29, the calculation of the second-step logistic derivative is based on the balanced population assumption. At the second iteration, the logistic derivative is ℓ′(yNf(X)eN)\ell'(y_Nf(X)e_N) and we have that $y_Nf(X) e_N= y_N {v^{(1)}}^{\top}(x_N+\frac{2^{L}-1}{N}\sum_{j=1}^{N} x_j) = {v^{(1)}}^{\top}\mu+ {v^{(1)}}^{\top}\mu y_N\sum_{j=1}^{N} y_j= {v^{(1)}}^{\top}\mu $ where the first equality is by the first step training results that Wl(1)=0d×d,l=0,…,LW_{l}^{(1)}= \boldsymbol{0}_{d\times d},l=0,\dots,L and the last equality is by the balanced population assumption gives that ∑j=1Nyj=0\sum_{j=1}^{N}y_j=0. Therefore, we have ℓ′(yNf(X)eN)=ℓ′(v(1)⊤μ)\ell'(y_N f(X)e_N)= \ell'( {v^{(1)}}^{\top}\mu), thus this term is a constant that can be pulled out of the expectation.

For the joint sampling scheme, the balanced population assumption will be applied to the NN context tokens. The query token will be re-indexed as xN+1x_{N+1}, with an independent Rademacher label yN+1y_{N+1}. The query token is masked from the intermediate context-processing layers and enters only through the final query readout. Thus, the balance assumption applies only to the context tokens and does not constrain the query label.

Regarding the reordering of the token in Appendix E.2 (p.43) mentioned in W2, we agree that reordering alone cannot give equal class counts. The equation yi=1,i=1,...,N/2;yi=−1,i=N/2,...,Ny_i = 1, i = 1,...,N/2; \quad y_i = −1, i = N/2,...,N follows the balanced population assumption, which gives two label groups (+1/−1+1/-1) of equal size N/2N/2. The reordering step is introduced only after this assumption, for notational convenience in the proof. We will revise the proof to distinguish clearly between the balance assumption and the subsequent reordering, and we will clarify that this theorem is proved under a balanced group setting.


W2. The main denoising result, the layer-wise SNR bound in (3.1), requires a depth restriction that is not stated. The paper claims that "every self-attention layer contributes to noise reduction" (p. 2). However, the final step of the proof (p. 58) requires roughly 4l≤N4^l \leq \sqrt{N}, and Appendix F uses 4l≤N/2+14^l \leq N/2+1, whereas Condition 3.1 only states L=Θ(log⁡N)L=\Theta(\log N), without the constants needed to imply either restriction. This is consistent with Figure 1a, where the SNR flattens after about five layers. Two further steps also require justification: the proof assumes exactly half of the tokens in each class "without loss of generality" (p. 43), although reordering tokens does not change the random class counts, and it applies Lemma G.4, which assumes fixed coefficients, to noise-dependent attention weights (p. 55). The claim should be stated with an explicit depth condition, and these two steps should be justified.

Q2: For what explicit depth range does the bound in (3.1) hold?

AW2&AQ2: We clearly defined the meaning of Θ(⋅)\Theta(\cdot) in the Notation Section (Appendix A). Both depth restrictions 4l≤N4^l \leq \sqrt{N} for (p. 58) and 4l≤N/2+14^l \leq N/2+1 for Appendix F are indeed consistent with the order L=Θ(log⁡N)L=\Theta(\log N) given in Condition 3.1. It is a common way of representing the order of variables and hiding constants in theoretical analysis. Our theoretical results that every self-attention layer contributes to noise reduction are obtained under Condition 3.1, where the model depth is of order log(N)log(N), ensuring that the model depth is not extremely large. Our theorem does not claim SNR improvement beyond this regime.

(continued in next response)

AuthorsOpenReview ↗

Response 2

AW2&&AQ2 (Continued): For Lemma G.4, thanks for pointing this out. The description in Lemma G.4 is not accurate for the case where the coefficients also depend on the noise vectors, and we corrected the Lemma G.4 that can be used in Equation(E.27) as follows: For ξ‾i=∑k=1Nβi,kξk\overline{\xi}_i=\sum_{k=1}^{N}\beta_{i,k}\xi_k, since βi,k\beta_{i,k} are bounded and ξk\xi_k i.i.d follows N(0,σnoise2I)N(\mathbf{0}, \sigma_{\mathrm{noise}}^2 \mathbf{I}), then ξ‾i\overline{\xi}_i is sub Gaussian random variable. Therefore, ∥∣ξ‾i∥∣2\||\overline{\xi}_i\||^2 is a sub-exponential random variable, and the Bernstein inequality for sub-exponential random variables can therefore be applied to it.

For the explanation of the balanced population assumption, please see AW1&AQ1&AQ2.


W3. The downstream claim in Theorem 3.3 is not fully justified. The paper states that deeper features lead to higher prediction accuracy (p. 5). The proof (Appendix F) only argues that the noise is sub-Gaussian and then applies Theorem 3.1 of Cao et al. (2021), which also imposes conditions on the noise structure and covariance. These conditions are not verified for features whose noise depends on the input through attention. In addition, a smaller risk upper bound at deeper layers does not imply a smaller actual risk. The statement on p. 5 should be limited to the improvement in the bound.

Q3. How do the layer features satisfy the conditions of Theorem 3.1 in Cao et al. (2021), given that their noise depends on the input through attention? If these conditions do not hold, can you provide another justification for Theorem 3.3, or limit the accuracy claim to an improvement in the risk bound?

AQ3&AW3: Regarding the condition verification for applying Theorem 3.1 in Cao et al. (2021) in Appendix F, Theorem 3.1 of Cao et al. (2021) assumes the data point is from a mixture of two sub-Gaussian distributions with means μ\mu and −μ−\mu respectively and the same covariance matrix Σ\Sigma.

In Appendix F, we prove that for every layer ll, the last-column token hl,N,m=αl,N,mμ+∑k=1Nβl,k,N,mξk,mh_{l,N,m}=\alpha_{l,N,m} \mu +\sum_{k=1}^{N}\beta_{l,k,N,m}\xi_{k,m} used in the downstream logistic regression satisfies this data model: Specifically, our data can be decomposed as hl,N,m=yNE[∣αl,N,m∣]μ+(αl,N,m−yNE[∣αl,N,m∣])μ+∑k=1Nβl,k,N,mξk,mh_{l,N,m}=y_N\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu+(\alpha_{l,N,m}-y_N\mathbb{E}[|\alpha_{l,N,m}|])\mu+\sum_{k=1}^{N}\beta_{l,k,N,m}\xi_{k,m}. Here the first term is the signal part, and the second and third terms together form the noise part h~l,N,m=(αl,N,m−yNE[∣αl,N,m∣])μ+∑k=1Nβl,k,N,mξk,m\tilde{h}_{l,N,m}=(\alpha_{l,N,m}-y_N \mathbb{E} \big[|\alpha_{l,N,m}|\big])\mu+\sum_{k=1}^{N} \beta_{l,k,N,m}\xi_{k,m}, which has zero mean and covariance matrix Σl=Cov(h~l,N,m,h~l,N,m)\Sigma_l={\rm Cov}\big(\tilde{h}_{l,N,m}, \tilde{h}_{l,N,m}\big). Given αl,N,m,βl,k,N,m\alpha_{l,N,m},\beta_{l,k,N,m} are all bounded and ξk,m\xi_{k,m} are i.i.d Gaussian random variables for all mm, it follows that the noise term is sub-Gaussian even if they are not independent. Hence, hl,N,mh_{l,N,m} is also sub-Gaussian random variable. The two class means are E[∣αl,N,m∣]μ\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu and −E[∣αl,N,m∣]μ-\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu with the same covariance matrix Σl\Sigma_l. Therefore, all conditions of Thm 3.1 in Cao et al. (2021) are satisfied, and the proof in Appendix F remains valid.

Our theory provides an upper bound on the population risk, showing that the increase in SNR across layers corresponds to features with more effectively captured signals, resulting in improved performance on the downstream tasks. Our experimental results support this conclusion and can help address your concern. In particular, it is observed that hidden states from deeper transformer layers achieve higher test accuracy on the downstream tasks, suggesting that the bound is informative in practice.


AuthorsOpenReview ↗

Response 3


W4. The contribution of the unlabeled context, which is central to the title and the conclusion, is not isolated. Under Definition 2.1, the noise is orthogonal to the signal direction μ\mu, so a classifier that knows this direction classifies the query without error at layer 0, and the first population gradient step already recovers this direction. Unlabeled samples may still help a downstream probe that does not know μ\mu, but the experiments do not separate this effect from the effects of depth or supervised training. A control that keeps the query fixed and removes the context, or replaces it with tokens that carry no class signal, would address this. For GPT-2, the probe uses the first token, but the attention mask is not specified; under a standard causal mask, this token cannot attend to the context.

Q4. With the query kept fixed, how do the layer-wise SNR and probe accuracy change when the context tokens are replaced with tokens that carry no class signal, or removed? Using the same trained checkpoints and a matched-length replacement control would help assess the contribution of context at inference time.

AQ4&AW4: The benefit of unlabeled context we studied in this work is not to obtain an optimal classifier. We focus on a different regime: how the self-attention mechanism can leverage unlabeled context to capture more signal in features and improve the signal-to-noise ratio (SNR) of the outputs across layers.

We would like to emphasize that our attention-only deep transformer in Eq. (2.1) already provides a natural context-free control for studying the denoising effect of the unlabeled context. After the first training step, the vector vv covers the direction of the signal μ\mu, but this trained linear classifier vv is not involved in our denoising effect analysis or in the downstream evaluation. Without such a classifier vv, we use only the hidden state hl,N,m,l=0,…,Lh_{l,N,m}, l=0,\dots, L as features in the downstream classification task.

Moreover, in this attention-stacked transformer model (2.1), if all context tokens are masked, the output hidden state reduces to the query token xNx_N itself. Therefore, the model (2.1) naturally provides a baseline without context, and the effect of the context can be assessed by comparing the hidden layer features with the original input token xNx_N.

Our experiments on attention-only transformers (2.1) in Sec. 5 show that, compared with the input token, utilizing the unlabeled context enables the features to capture more signal, resulting in improved downstream prediction accuracy.

Therefore, both our theoretical framework and our experiments already include a control for the unlabeled context, and they both support the conclusion that the self-attention mechanism leverages the unlabeled context to improve feature quality.


W5. The experiments do not fully support the claims made for them in the main text. The paper states that the synthetic experiments "adhering to our settings confirm our theoretical results" (p. 2), but these experiments (Appendix C.1) use d = 100 with N = 50, momentum, gradient clipping, balanced classes, and a loss on all tokens. They illustrate the proposed mechanism but do not directly test the quantitative bound in (3.1).

AW5: Thanks for pointing out this mismatch between our wording and the experimental setup. The synthetic-data experiments in Sec C.1 are based on the exact data model (Def 2.1) and the deep transformer (2.1) used in our theoretical analysis. These experiments verify our theoretical insights that the signal-to-noise ratio increases across self-attention layers in deep transformers. While the gradient clipping and momentum are not explicitly covered by our theoretical results, they do not change the self-attention dynamics on which the layer-wise denoising analysis is based, and our experiments still test our theoretical insights.

Since these choices are not the exact theoretical setting, the experiments would be more appropriately interpreted as evidence that our theoretical results hold in similar and broader settings. We will carefully revise our wording to avoid misunderstanding.

For the test of the quantitative bound in (3.1), our experiments in fact support the conclusion in (3.1) rather than contradict it. Since our result (3.1) is a lower bound, it is not intended to predict the exact empirical multiplicative factors. Instead, it characterizes the trend that SNR increases across layers. The experimental observations confirm this trend and do not contradict with the theory. In this sense, we believe it is unfair to require the experiments to verify the exact multiplicative factors in the analysis.


AuthorsOpenReview ↗

Response 4


M1:CIFAR-10 center filtering also removes test examples (pp. 20–21). Results without filtering would clarify the scope of the reported improvements.

AM1: The 95% center filtering is a commonly used preprocessing step for complex real-world datasets, and it removes only the most extreme 5% of outlier examples in each group. We have updated our latest CIFAR-10 experiments (Appendix C.6) without center filtering in the revised paper PDF (highlighted in blue). The results show that, even without this filtering step, deeper layers still achieve higher test accuracies on the downstream logistic regression task.


M2:The "among the first" convergence claim (p. 2) should be discussed in relation to Gao et al. (2024), with the differences in setting made explicit. Work on token clustering and rank collapse, such as Dong et al. (2021) and Geshkovski et al. (2023), is also relevant.

AM2: We summarize the related papers you mentioned as follows:

Gao et al. (2024) analyze global optimization of large-scale Transformers through a mean-field limit and show that gradient flow converges to the Wasserstein gradient flow as the model width and depth go to infinity.

Dong et al. (2021) study the rank collapse in the forward representations of self-attention networks, and prove that token representations converge to a rank-one matrix at a doubly exponential rate.

Geshkovski et al. (2023) model tokens as interacting particles under self-attention dynamics with fixed weights and study what happens as the dynamics’ time tends to infinity. They also prove that in the one-dimensional case, the self-attention matrix converges to a low-rank Boolean matrix.

Although these papers are on related topics, their insights are different from ours. Our paper provides insights into how attention mechanisms can leverage unlabeled context to improve performance in supervised learning tasks and provides the convergence guarantee for the analyzed transformers. We will carefully tune our claims to avoid misunderstanding and include these papers in the related work.


M3: Equation (2.1) contains one more attention block than the stated number of layers.

AM3: There are a total of L+1 attention layers in the defined transformer due to the existence of the final output layer in the first line of Eq (2.1). We present the final output layer separately for clarity, and this does not influence the correctness of the theorem. This is simply a matter of notation. Equivalently, one may introduce a new notation L~:=L+1\tilde{L}:=L+1 to denote the number of layers, so that the transformer is described as having exactly L~\tilde{L} layers.


W5: The conclusion also states that the GPT-2 experiments confirm the findings "in broader scenarios" (p. 9). However, the GPT-2 probes on CIFAR-10 and 10-class MNIST (Appendix C) use early stopping based on test loss and test accuracy (pp. 25–26), and no separate validation set is described, so the reported test sets may also have been used for model selection. No variation across repeated runs or error bars are reported.

Q5. The GPT-2 experiments state that the "first column token" of each layer output is probed (e.g., p. 22). Is this correct, and which attention mask is used? Are the sets used for early stopping (pp. 25–26) different from the reported test sets? If not, do the trends in Figures 19 and 21 hold with a separate validation set?

AW5&AQ5: We have updated our GPT-2 experiment results in the revised paper PDF (Appendix C.5-C.7, highlighted in blue in the manuscript).

In our experiments, we split the source training data into disjoint training and validation sets. The hyperparameters were selected on the validation set. Early stopping is based on the validation loss and accuracy. The independent test set is only used to report the test accuracy and test SNR and it is not accessed by the model beforehand. We also repeated the experiments across multiple random seeds and report the mean ± standard deviation across seeds in the figures. Please refer to our updated GPT-2 experiments in the revised paper PDF (colored in blue). A similar trend is still observed here: features from deeper transformer layers yield better performance in the downstream classification tasks.

All experiments now use GPT-2’s standard causal mask. We also explicitly specify the query token positions used (the last few tokens) in the downstream task in the detailed description of each experimental setting. For example, in the MNIST experiments with GPT-2, the downstream classifier uses the eighth token, hl,8,mh_{l,8,m} from layer l,l=0,…,12l, l=0,\dots, 12.


Reviewer_cg1sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how depth affects classification with unlabeled context in an attention-only transformer. The authors claim that, under suitable dimension and depth conditions, training reaches a small population logistic loss and the learned representations exhibit a multiplicative increase in signal-to-noise ratio across layers. The proposed mechanism is that learned attention preferentially aggregates tokens from the same class. A subsequent analysis seeks to translate this representation improvement into a decreasing population-risk upper bound for a separately trained linear classifier. Experiments on synthetic data, noisy MNIST and CIFAR-10, and task-trained GPT-2 architectures examine layer-wise SNR, attention patterns, and linear-probe accuracy.

Critical strengths
  • The most interesting aspect is the attempt to explain how training produces attention weights that support denoising.

  • The paper examines training loss, representation SNR, class-sorted attention matrices, and downstream linear-probe performance. These measurements address different parts of the proposed explanation. In particular, including downstream probes is a useful attempt to assess whether the representation changes have consequences for prediction. The additional experiments with image-derived inputs and GPT-2 architectures broaden the investigation, even though their interpretation requires stronger controls.

  • The decomposition part of the method provides a useful framework for tracking signal amplification and noise aggregation separately. The connection between a signal-aligned attention matrix, preferential attention to same-class tokens, and improved representation SNR is intuitive and relevant to the stated problem. Retaining softmax attention and residual connections also makes the analysis more informative than an exclusively linear-attention calculation.

Critical weaknesses
  • The training and denoising proofs use inconsistent sampling assumptions. Appendix D.1 (p.29) replaces the second-step logistic derivative by a constant −r1-r_1. However, with a=(2L+1−1)/Na=(2^{L+1}-1)/N and the stated v(1)=cμv^{(1)}=c\mu, the margin is
yNf(1)(X)eN=c∥μ∥22(1+a yN∑j=1Nyj),c=η(1+a)2,y_N f^{(1)}(X)e_N = c\|\mu\|_2^2 \left(1+a\,y_N\sum_{j=1}^{N}y_j\right), \qquad c=\frac{\eta(1+a)}{2},

which generally depends on the context labels. Exact balance would remove this dependence, but would also change Eq.(D.15) to v(1)=ημ/2v^{(1)}=\eta\mu/2. Moreover, token reordering does not justify imposing equal class counts in Appendix E.2 (p.43). Balanced sampling is specified for the synthetic experiments, but the theoretical calculations need to be reconciled under one explicit joint sampling model.

  • The probabilistic arguments need further justification. Equation(E.27) applies LemmaG.4, proved for deterministic coefficients, to attention coefficients depending on the same noise vectors. AppendixF establishes sub-Gaussianity of the transformed features, but does not verify all distributional and covariance assumptions. Both applications require additional arguments to establish the claimed SNR and risk guarantees.

  • The GPT-2 experiments need to establish access to context. Appendices C.4--C.7 probe the first token without specifying whether causal masking is removed. If the standard causal mask is retained, that token cannot access the remaining context. The mask and query position should be clarified before these experiments are interpreted as evidence of context-based denoising.

  • The evaluation protocol needs an independent test set. Appendices C.6--C.7 explicitly describe early stopping based on test loss and test accuracy. If these are also the final evaluation sets, model selection affects the reported results. The authors should distinguish validation from testing and report uncertainty across training seeds and downstream sample selections.

Questions for authors

please see my detailed comments above

2 comments
AuthorsOpenReview ↗

Response 1

W1 The training and denoising proofs use inconsistent sampling assumptions. Appendix D.1 (p.29) replaces the second-step logistic derivative by a constant. However, with and the stated, the margin is yNf(1)(X)eN=c∥∣μ∥∣22(1+a,yN∑j=1Nyj),c=η(1+a)2y_N f^{(1)}(X)e_N = c\||\mu\||2^2 \left(1+a,y_N\sum_{j=1}^{N}y_j\right), \quad c=\frac{\eta(1+a)}{2}, which generally depends on the context labels. Exact balance would remove this dependence, but would also change Eq.(D.15) to v(1)=ημ/2v^{(1)}=\eta \mu/2. Moreover, token reordering does not justify imposing equal class counts in Appendix E.2 (p.43). Balanced sampling is specified for the synthetic experiments, but the theoretical calculations need to be reconciled under one explicit joint sampling model.

AW1:For step (i) on p.29, the calculation of the second-step logistic derivative is based on the balanced population assumption. At the second iteration, the logistic derivative is ℓ′(yNf(X)eN)\ell'(y_Nf(X)e_N) and we have that $y_Nf(X) e_N= y_N {v^{(1)}}^{\top}(x_N+\frac{2^{L}-1}{N}\sum_{j=1}^{N} x_j) = {v^{(1)}}^{\top}\mu+ {v^{(1)}}^{\top}\mu y_N\sum_{j=1}^{N} y_j= {v^{(1)}}^{\top}\mu $ where the first equality is by the first step training results that Wl(1)=0d×d,l=0,…,LW_{l}^{(1)}= \boldsymbol{0}_{d\times d},l=0,\dots,L and the last equality is by the balanced population assumption gives that ∑j=1Nyj=0\sum_{j=1}^{N}y_j=0. Therefore, we have ℓ′(yNf(X)eN)=ℓ′(v(1)⊤μ)\ell'(y_N f(X)e_N)= \ell'( {v^{(1)}}^{\top}\mu), thus this term is a constant that can be pulled out of the expectation.

For the joint sampling scheme, the balanced population assumption will be applied to the NN context tokens. The query token will be re-indexed as xN+1x_{N+1}, with an independent Rademacher label yN+1y_{N+1}. The query token is masked from the intermediate context-processing layers and enters only through the final query readout. Thus, the balance assumption applies only to the context tokens and does not constrain the query label.

Regarding the reordering of the token in Appendix E.2 (p.43), we agree that reordering alone cannot give equal class counts. The equation yi=1,i=1,...,N/2;yi=−1,i=N/2,...,Ny_i = 1, i = 1,...,N/2; \quad y_i = −1, i = N/2,...,N follows the balanced population assumption, which gives two label groups (+1/−1+1/-1) of equal size N/2N/2. The reordering step is introduced only after this assumption, for notational convenience in the proof. We will revise the proof to distinguish clearly between the balance assumption and the subsequent reordering, and we will clarify that this theorem is proved under a balanced group setting.


W2: The probabilistic arguments need further justification. Equation(E.27) applies Lemma G.4, proved for deterministic coefficients, to attention coefficients depending on the same noise vectors. Appendix F establishes sub-Gaussianity of the transformed features, but does not verify all distributional and covariance assumptions. Both applications require additional arguments to establish the claimed SNR and risk guarantees.

AW2: Thanks for pointing this out. The description in Lemma G.4 is not accurate for the case where the coefficients also depend on the noise vectors, and we corrected the Lemma G.4 that can be used in Equation(E.27) as follows: For ξ‾i=∑k=1Nβi,kξk\overline{\xi}_i=\sum_{k=1}^{N}\beta_{i,k}\xi_k, since βi,k\beta_{i,k} are bounded and ξk\xi_k i.i.d follows N(0,σnoise2I)N(\mathbf{0}, \sigma_{\mathrm{noise}}^2 \mathbf{I}), then ξ‾i\overline{\xi}_i is sub Gaussian random variable. Therefore, ∥∣ξ‾i∥∣2\||\overline{\xi}_i\||^2 is a sub-exponential random variable, and Bernstein inequality for sub-exponential random variables can therefore be applied to it.

(continued in next response)

AuthorsOpenReview ↗

Response 2

AW2 (Continued): Regarding the condition verification for applying Theorem 3.1 in Cao et al. (2021) in Appendix F, Theorem 3.1 of Cao et al. (2021) assumes the data point is from a mixture of two sub-Gaussian distributions with means μ\mu and −μ−\mu respectively and the same covariance matrix Σ\Sigma.

In Appendix F, we prove that for every layer ll, the last-column token hl,N,m=αl,N,mμ+∑k=1Nβl,k,N,mξk,mh_{l,N,m}=\alpha_{l,N,m} \mu +\sum_{k=1}^{N}\beta_{l,k,N,m}\xi_{k,m} used in the downstream logistic regression satisfies this data model: Specifically, our data can be decomposed as hl,N,m=yNE[∣αl,N,m∣]μ+(αl,N,m−yNE[∣αl,N,m∣])μ+∑k=1Nβl,k,N,mξk,mh_{l,N,m}=y_N\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu+(\alpha_{l,N,m}-y_N\mathbb{E}[|\alpha_{l,N,m}|])\mu+\sum_{k=1}^{N}\beta_{l,k,N,m}\xi_{k,m}. Here the first term is the signal part, and the second and third terms together form the noise part h~l,N,m=(αl,N,m−yNE[∣αl,N,m∣])μ+∑k=1Nβl,k,N,mξk,m\tilde{h}_{l,N,m}=(\alpha_{l,N,m}-y_N \mathbb{E} \big[|\alpha_{l,N,m}|\big])\mu+\sum_{k=1}^{N} \beta_{l,k,N,m}\xi_{k,m}, which has zero mean and covariance matrix Σl=Cov(h~l,N,m,h~l,N,m)\Sigma_l={\rm Cov}\big(\tilde{h}_{l,N,m}, \tilde{h}_{l,N,m}\big). Given αl,N,m,βl,k,N,m\alpha_{l,N,m},\beta_{l,k,N,m} are all bounded and ξk,m\xi_{k,m} are i.i.d Gaussian random variables for all mm, it follows that the noise term is sub-Gaussian even if they are not independent. Hence, hl,N,mh_{l,N,m} is also sub-Gaussian random variable. The two class means are E[∣αl,N,m∣]μ\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu and −E[∣αl,N,m∣]μ-\mathbb{E}\big[|\alpha_{l,N,m}|\big]\mu with the same covariance matrix Σl\Sigma_l. Therefore, all conditions of Thm 3.1 in Cao et al. (2021) are satisfied, and the proof in Appendix F remains valid.


W3: The GPT-2 experiments need to establish access to context. Appendices C.4--C.7 probe the first token without specifying whether causal masking is removed. If the standard causal mask is retained, that token cannot access the remaining context. The mask and query position should be clarified before these experiments are interpreted as evidence of context-based denoising.

AW3: We have updated our GPT-2 experiment results in the revised paper PDF (Appendix C, highlighted in blue in the manuscript). All experiments now use GPT-2’s standard causal mask. We also explicitly specify the query token positions used (the last few tokens) in the downstream task in the detailed description of each experimental setting. For example, in the MNIST experiments with GPT-2, the downstream classifier uses the eighth token, hl,8,mh_{l,8,m} from layer l,l=0,…,12l, l=0,\dots, 12.


W4: The evaluation protocol needs an independent test set. Appendices C.6--C.7 explicitly describe early stopping based on test loss and test accuracy. If these are also the final evaluation sets, model selection affects the reported results. The authors should distinguish validation from testing and report uncertainty across training seeds and downstream sample selections.

AW4: We have updated our GPT-2 experiment results in the revised paper PDF (Appendix C). In our experiments, we split the source training data into disjoint training and validation sets. The hyperparameters were selected on the validation set. Early stopping is based on the validation loss and accuracy. The independent test set is only used to report the test accuracy and test SNR and it is not accessed by the model beforehand. We also repeated the experiments across multiple random seeds and report the mean ± standard deviation across seeds in the figures. Please refer to our updated GPT-2 experiments in the revised paper PDF (colored in blue).

Reviewer_Ft4HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how an attention-only Transformer uses unlabeled context for binary classification. Under specific data and model assumptions, the authors connect three aspects: gradient-based training achieves a small population loss; the ratio of SNRs between consecutive hidden layers admits a lower bound supporting layer-wise improvement, explained through a within-class averaging mechanism; and linear classifiers trained on deeper representations admit tighter population risk upper bounds. Experiments on synthetic data, preprocessed and noise-corrupted MNIST and CIFAR-10, and GPT-2-style architectures report generally increasing trends in SNR or generalized SNR and linear-probe accuracy with depth.

Critical strengths

S1. Clear and well-motivated research question. The paper investigates how trained multi-layer Softmax attention uses unlabeled context to form within-class aggregations. Tracking layer-wise SNR provides a concrete, analyzable perspective on representation quality and the role of depth.

S2. Clear connection between optimization, representation quality, and downstream classification. The paper brings together training dynamics, layer-wise SNR analysis, and downstream linear classification. The population risk analysis gives the representation-level SNR results a task-level interpretation, complemented by linear-probing experiments.

Critical weaknesses

W1. Inconsistent treatment of label distributions leaves a gap in the core training analysis. Eq. (D.15) uses vanishing cross-token label correlations, as obtained under independent, zero-mean label sampling. However, the second gradient update replaces the sample-dependent loss derivative with the fixed value −r1-r_1, although the classification margin depends on the context labels through yN∑jyjy_N\sum_j y_j. Section E.2 subsequently assumes exactly balanced classes “without loss of generality,” which changes the label distribution rather than merely reordering the tokens. If exact balance is imposed consistently, the first update yields v(1)=ημ/2v^{(1)}=\eta\mu/2, rather than the expression derived in Eq. (D.15). Please reconcile these assumptions and rederive the parameter expressions underlying Lemma D.1, clarifying whether the subsequent training loss bounds and SNR guarantees remain valid.

W2. The benefit of unlabeled context needs clarification because the data model permits perfect context-free classification. The noise is orthogonal to μ\mu, so μ⊤x=y∥μ∣22\mu^\top x=y\|\mu|_2^2, and the Bayes error based on a single query is zero. Moreover, under the population logistic-loss objective, a context-free linear classifier initialized at zero obtains a positive multiple of μ\mu after one gradient step. This does not rule out benefits for particular finite-sample learners, but it limits what the results establish about the necessity or advantage of context. Please identify the metric or learning regime in which context provides a benefit and include a corresponding context-free baseline.

Questions for authors

Please address the questions raised in W1–W2.

2 comments
AuthorsOpenReview ↗

Response 1

W1. Inconsistent treatment of label distributions leaves a gap in the core training analysis. Eq. (D.15) uses vanishing cross-token label correlations, as obtained under independent, zero-mean label sampling. However, the second gradient update replaces the sample-dependent loss derivative with the fixed value −r1-r_1, although the classification margin depends on the context labels through yN∑jyjy_N\sum_j y_j. Section E.2 subsequently assumes exactly balanced classes “without loss of generality,” which changes the label distribution rather than merely reordering the tokens. If exact balance is imposed consistently, the first update yields v(1)=ημ/2v^{(1)}=\eta\mu /2, rather than the expression derived in Eq. (D.15). Please reconcile these assumptions and rederive the parameter expressions underlying Lemma D.1, clarifying whether the subsequent training loss bounds and SNR guarantees remain valid.

AW1: For the second-step gradient update (step (i) on p.29), the calculation of the second-step logistic derivative is based on the balanced population assumption. At the second iteration, the logistic derivative is ℓ′(yNf(X)eN)\ell'(y_Nf(X)e_N) and we have that $y_Nf(X) e_N= y_N {v^{(1)}}^{\top}(x_N+\frac{2^{L}-1}{N}\sum_{j=1}^{N} x_j) = {v^{(1)}}^{\top}\mu+ {v^{(1)}}^{\top}\mu y_N\sum_{j=1}^{N} y_j= {v^{(1)}}^{\top}\mu $ where the first equality is by the first step training results that Wl(1)=0d×d,l=0,…,LW_{l}^{(1)}= \boldsymbol{0}_{d\times d},l=0,\dots,L and the last equality is by the balanced population assumption gives that ∑j=1Nyj=0\sum_{j=1}^{N}y_j=0. Therefore, we have ℓ′(yNf(X)eN)=ℓ′(v(1)⊤μ)\ell'(y_N f(X)e_N)= \ell'( {v^{(1)}}^{\top}\mu), thus this term is a constant that can be pulled out of the expectation.

For the joint sampling scheme, the balanced population assumption will be applied to the NN context tokens. The query token will be re-indexed as xN+1x_{N+1} with an independent Rademacher label yN+1y_{N+1}. The query token is masked from the intermediate context-processing layers and enters only through the final query readout. Thus, the balance assumption applies only to the context tokens and does not constrain the query label.

Regarding the reordering of the token in Appendix E.2 (p.43), we agree that reordering alone cannot give equal class counts. The equation yi=1,i=1,...,N/2;yi=−1,i=N/2,...,Ny_i = 1, i = 1,...,N/2; \quad y_i = −1, i = N/2,...,N follows the balanced population assumption, which gives two label groups (+1/−1+1/-1) of equal size N/2N/2. The reordering step is introduced only after this assumption, for notational convenience in the proof. We will revise the proof to distinguish clearly between the balance assumption and the subsequent reordering, and we will clarify that this theorem is proved under a balanced group setting.


AuthorsOpenReview ↗

Response 2

W2. The benefit of unlabeled context needs clarification because the data model permits perfect context-free classification. The noise is orthogonal to μ\mu, so μTx=y∥μ∥22\mu^T x = y\|\mu\|_2^2, and the Bayes error based on a single query is zero. Moreover, under the population logistic-loss objective, a context-free linear classifier initialized at zero obtains a positive multiple μ\mu of after one gradient step. This does not rule out benefits for particular finite-sample learners, but it limits what the results establish about the necessity or advantage of context. Please identify the metric or learning regime in which context provides a benefit and include a corresponding context-free baseline.

AW2: The benefit of unlabeled context we studied in this work is not to obtain an optimal classifier. We focus on a different regime: how the self-attention mechanism can leverage unlabeled context to capture more signal in features and improve the signal-to-noise ratio (SNR) of the outputs across layers.

We would like to emphasize that our attention-only deep transformer in Eq. (2.1) already provides a natural context-free control for studying the denoising effect of the unlabeled context. After the first training step, the vector vv covers the direction of the signal μ\mu, but this trained linear classifier vv is not involved in our denoising effect analysis or in the downstream evaluation. Without such a classifier vv, we use only the hidden state hl,N,m,l=0,…,Lh_{l,N,m}, l=0,\dots, L as features in the downstream classification task.

Moreover, in this attention-stacked transformer model (2.1), if all context tokens are masked, the output hidden state reduces to the query token xNx_N itself. Therefore, the model (2.1) naturally provides a baseline without context, and the effect of the context can be assessed by comparing the hidden layer features with the original input token xNx_N.

Our experiments on attention-only transformers (2.1) in Sec. 5 show that, compared with the input token, utilizing the unlabeled context enables the features to capture more signal, resulting in improved downstream prediction accuracy.

Therefore, both our theoretical framework and our experiments already include a control for the unlabeled context, and they both support the conclusion that the self-attention mechanism leverages the unlabeled context to improve feature quality.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.