- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper views two LLM behaviors, ICL and repetitive generation under a common lens. They analyze softmax attention in the long-context setting and show that under stationary inputs with elliptical marginals, attention output depends on input mean and covariance. This extends existing results by relaxing independence and Gaussianity assumptions.
When applied to ICL, this shows a softmax attention layer can asymptotically implement a gradient descent step for in-context linear regression under Gaussian assumptions. When applied to repetitive generation, they derive an asymptotically first-order Markov description of generation, which they relate to repetitive behavior.
Experiments are conducted to demonstrate the covariance-readout approximation on natural-language embeddings. They give evidence of the emergence of gradient descent behavior in trained attention models on synthetic regression tasks, and the similarity of hidden representations for repeated tokens during repetitive generation.
Critical strengths
(1) The paper provides a clean mathematical analysis. It gives a meaningful extension of existing softmax analysis by relaxing the independence and Guassianity assumptions to stationary ergodic sequences and elliptical marginals. The application to the repetition setting is particularly interesting.
(2) The extension to the multilayer setting is cool. Deriving an asymptotically first order Markov description of generation provides insight potentially beyond just repetitive behavior in language models.
Critical weaknesses
(1) I think the authors do not sufficiently distinguish their novelty from existing work. The covariance-readout formulation is extremely similar to the results of Boursier and Boyer, particularly Lemma 2.1. While relaxing the independence and Gaussianity assumptions is a genuine theoretical contribution, the novelty of the overall framework seems somewhat overstated. Also, the assumptions are largely stated without justification. Are these assumptions commonly used? Are they representative of trained language models?
(2) The ICL experiments are relevant, although similar gradient-descent-like behavior in trained softmax attention models has already been explored in prior work it's a different setting so that's still fine. My main concern is with the repetition experiments. Table 1 shows high similarity between representation of repeated occurrences of the same token. However, it shows the similarity is high even in the original prompt. In fact, it was slightly higher for the prompt than in the repeated sections. It is unclear if the similarity is due to the covariance readout mechanism. I think a more direct comparison of context dependence between repetitive and non-repetitive generation would be necessary to support this interpretation.
Questions for authors
(1) What exactly is the novelty relative to the paper by Boursier and Boyer, particularly lemma 2.1? Just looking purely at the ICL section, what parts are new?
(2) Could the authors provide further justification for their distributional assumptions?
(3) Is my assessment of the repetition experiment fair? Could the authors compare context dependence between repetitive and non-repetitive generations?
6 comments
Response to Reviewer mzbF (Part 2)
Weakness 2 and Question 3: Insufficient evidence from the repetition experiment.
Our original experiment aimed to show that, for prompts leading to repetition, the final hidden state is almost entirely determined by the current token. The high similarity in the prompt tail was intended to suggest that reduced context dependence was already present before generation, with repetition emerging when autoregressive generation begins.
We agree that the original experiment does not sufficiently support this interpretation. Shared embeddings carried through residual connections can produce such similarity. Moreover, similar hidden-state directions do not imply that next-token predictions are insensitive to earlier context, since small representation differences may still affect output probabilities. A direct comparison of context dependence between repetitive and non-repetitive generation is therefore needed.
We have replaced the original experiment with the following:
We empirically examine whether repetitive generation is associated with reduced context dependence using the following two metrics.
Metrics.
Attention Weight Dispersion is the standard deviation of an attention head's weights across input positions. For a multilayer language model, attention weight dispersion is averaged across heads and then layers. Lower values indicate more uniformly distributed attention weights, with less selective emphasis on individual context tokens.
Perturbation Impact is the total variation distance between next-token distributions before and after modifying the context by masking selected positions. Lower values indicate greater robustness to context perturbations. We average this metric over multiple random masks to reduce sampling variability.
Experimental setup.
We evaluate six pretrained TinyStories models on 1,024 randomly sampled MiniPile documents. To control for context length, we truncate each prompt to its first 1,024 tokens using the corresponding tokenizer and assess whether repetition occurs immediately. For prompts without repetition, both metrics are evaluated at the last token of the prompt. For those with repetition, we average over the first period. For perturbation impact, we sample 10 masks, each selecting 64 positions uniformly at random from the original prompt. We then average the prompt-level metrics separately for prompts with and without repetition.
Results.
Table 1 shows that the repetition group has lower attention weight dispersion and perturbation impact across all six models. These results associate repetitive generation with lower context dependence, consistent with the context-forgetting mechanism suggested by our analysis.
Table 1: Attention weight dispersion and perturbation impact across six pretrained TinyStories models. Each entry reports group means in the order repetition / no repetition.
Model Attention Weight Dispersion Perturbation Impact TinyStories-3M 0.00523 / 0.00598 0.07554 / 0.09952 TinyStories-8M 0.00511 / 0.00537 0.09208 / 0.10964 TinyStories-28M 0.00510 / 0.00514 0.09665 / 0.12573 TinyStories-33M 0.00734 / 0.00738 0.10141 / 0.15526 TinyStories-1Layer-21M 0.00443 / 0.00540 0.05815 / 0.06662 TinyStories-2Layers-33M 0.00713 / 0.00762 0.06820 / 0.08057
These changes have been incorporated into Section 5.2.
Thank you for the detailed response and additional experiments. I think the new perturbation-impact experiment is a meaningful improvement.
I have one remaining concern about the experimental setup in Section 5.2. It seems that non-repetitive cases are evaluated at the end of the original prompt, while repetitive cases are evaluated after generation has begun. Is this correct?
If so, couldn't the lower perturbation impact arise naturally because the model is already relying on its recently generated repetitive sequence rather than the original prompt? Could the authors compare both groups at the end of the original prompt, grouping them according to whether subsequent generation becomes repetitive?
I am open to raising my score given the improvements, but would appreciate clarification on this point.
Response to Reviewer mzbF (Part 1)
Thank you for your insightful feedback!
Weakness 1 and Question 1: The novelty from existing work.
Our covariance-readout formulation is indeed closely related to Lemma 2.1 of Boursier and Boyer. We relax the independence and Gaussianity assumptions through a different proof approach, while recovering their Gaussian result as a special case. We acknowledge that this overlap limits the novelty of the formulation itself.
The key distinction lies in how we use the theory to study ICL. Boursier and Boyer (2025) use ICL as a training setting in which to apply their theory. They analyze training dynamics and establish risk guarantees connecting finite-prompt models to their infinite-prompt counterparts, with the latter dependent on the data distribution. Our goal is to explain how softmax attention can perform learning within the context. We construct explicit, task-agnostic attention parameters whose forward pass asymptotically implements a gradient-descent step for the regression task encoded in the prompt. Our contribution is therefore a concrete algorithmic mechanism for ICL under standard softmax attention.
Weakness 1 and Question 2: The assumptions are stated without justification.
These assumptions are commonly used in probability and statistical analysis. Stationarity and ergodicity are standard conditions for extending laws of large numbers to dependent observations. Elliptical distributions are a classical generalization of multivariate Gaussians. The sub-Gaussian condition is used in prior attention analyses, including Boursier and Boyer (2025), while mixing conditions are standard tools for analyzing dependent sequences (Rio et al., 2017).
For trained language models, these assumptions serve as modeling idealizations rather than an exact characterization of their representations. They provide a tractable setting for identifying the covariance-readout mechanism. Our natural-language experiments examine whether the predicted alignment remains informative in practice.
We have added Remark 3.6 (Distributional assumptions) to the theory section.