acceptodds
Under review as a conference paper at ICLR 2027

Softmax Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition

Abstract

Large language models (LLMs) exhibit two seemingly unrelated behaviors: in-context learning (ICL) and repetitive generation. In both cases, the model appears to summarize the context into a population-level statistic to guide subsequent generation while discarding token-level detail. We investigate whether this "summarization and forgetting" can be derived from the attention mechanism itself. For stationary inputs with elliptical marginals, the softmax attention outputs converge almost surely to in the long-context regime, where and denote the input mean and covariance, respectively, and is a scalar. Beyond the mean contribution, this limit provides a readout of the input’s second-order statistics. Its dependence on the current input token alone leads to two consequences. (i) For in-context linear regression tasks, a residual softmax attention layer can emulate one gradient-descent step. (ii) When propagated across multiple transformer layers, this readout drives the terminal hidden state toward a deterministic function of the current token. Autoregressive generation thus reduces asymptotically to a first-order Markov chain, whose periodic orbits provide a structural explanation for repetitive generation. These two phenomena emerge as facets of a common covariance-readout principle.

Primary area:
learning theory
Keywords:
Softmax attention, in-context learning, repetition

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_mzbFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper views two LLM behaviors, ICL and repetitive generation under a common lens. They analyze softmax attention in the long-context setting and show that under stationary inputs with elliptical marginals, attention output depends on input mean and covariance. This extends existing results by relaxing independence and Gaussianity assumptions.

When applied to ICL, this shows a softmax attention layer can asymptotically implement a gradient descent step for in-context linear regression under Gaussian assumptions. When applied to repetitive generation, they derive an asymptotically first-order Markov description of generation, which they relate to repetitive behavior.

Experiments are conducted to demonstrate the covariance-readout approximation on natural-language embeddings. They give evidence of the emergence of gradient descent behavior in trained attention models on synthetic regression tasks, and the similarity of hidden representations for repeated tokens during repetitive generation.

Critical strengths

(1) The paper provides a clean mathematical analysis. It gives a meaningful extension of existing softmax analysis by relaxing the independence and Guassianity assumptions to stationary ergodic sequences and elliptical marginals. The application to the repetition setting is particularly interesting.

(2) The extension to the multilayer setting is cool. Deriving an asymptotically first order Markov description of generation provides insight potentially beyond just repetitive behavior in language models.

Critical weaknesses

(1) I think the authors do not sufficiently distinguish their novelty from existing work. The covariance-readout formulation is extremely similar to the results of Boursier and Boyer, particularly Lemma 2.1. While relaxing the independence and Gaussianity assumptions is a genuine theoretical contribution, the novelty of the overall framework seems somewhat overstated. Also, the assumptions are largely stated without justification. Are these assumptions commonly used? Are they representative of trained language models?

(2) The ICL experiments are relevant, although similar gradient-descent-like behavior in trained softmax attention models has already been explored in prior work it's a different setting so that's still fine. My main concern is with the repetition experiments. Table 1 shows high similarity between representation of repeated occurrences of the same token. However, it shows the similarity is high even in the original prompt. In fact, it was slightly higher for the prompt than in the repeated sections. It is unclear if the similarity is due to the covariance readout mechanism. I think a more direct comparison of context dependence between repetitive and non-repetitive generation would be necessary to support this interpretation.

Questions for authors

(1) What exactly is the novelty relative to the paper by Boursier and Boyer, particularly lemma 2.1? Just looking purely at the ICL section, what parts are new?

(2) Could the authors provide further justification for their distributional assumptions?

(3) Is my assessment of the repetition experiment fair? Could the authors compare context dependence between repetitive and non-repetitive generations?

6 comments
AuthorsOpenReview ↗

Response to Reviewer mzbF (Part 1)

Thank you for your insightful feedback!

Weakness 1 and Question 1: The novelty from existing work.

Our covariance-readout formulation is indeed closely related to Lemma 2.1 of Boursier and Boyer. We relax the independence and Gaussianity assumptions through a different proof approach, while recovering their Gaussian result as a special case. We acknowledge that this overlap limits the novelty of the formulation itself.

The key distinction lies in how we use the theory to study ICL. Boursier and Boyer (2025) use ICL as a training setting in which to apply their theory. They analyze training dynamics and establish risk guarantees connecting finite-prompt models to their infinite-prompt counterparts, with the latter dependent on the data distribution. Our goal is to explain how softmax attention can perform learning within the context. We construct explicit, task-agnostic attention parameters whose forward pass asymptotically implements a gradient-descent step for the regression task encoded in the prompt. Our contribution is therefore a concrete algorithmic mechanism for ICL under standard softmax attention.

Weakness 1 and Question 2: The assumptions are stated without justification.

These assumptions are commonly used in probability and statistical analysis. Stationarity and ergodicity are standard conditions for extending laws of large numbers to dependent observations. Elliptical distributions are a classical generalization of multivariate Gaussians. The sub-Gaussian condition is used in prior attention analyses, including Boursier and Boyer (2025), while mixing conditions are standard tools for analyzing dependent sequences (Rio et al., 2017).

For trained language models, these assumptions serve as modeling idealizations rather than an exact characterization of their representations. They provide a tractable setting for identifying the covariance-readout mechanism. Our natural-language experiments examine whether the predicted alignment remains informative in practice.

We have added Remark 3.6 (Distributional assumptions) to the theory section.

AuthorsOpenReview ↗

Response to Reviewer mzbF (Part 2)

Weakness 2 and Question 3: Insufficient evidence from the repetition experiment.

Our original experiment aimed to show that, for prompts leading to repetition, the final hidden state is almost entirely determined by the current token. The high similarity in the prompt tail was intended to suggest that reduced context dependence was already present before generation, with repetition emerging when autoregressive generation begins.

We agree that the original experiment does not sufficiently support this interpretation. Shared embeddings carried through residual connections can produce such similarity. Moreover, similar hidden-state directions do not imply that next-token predictions are insensitive to earlier context, since small representation differences may still affect output probabilities. A direct comparison of context dependence between repetitive and non-repetitive generation is therefore needed.

We have replaced the original experiment with the following:

We empirically examine whether repetitive generation is associated with reduced context dependence using the following two metrics.

Metrics.

Attention Weight Dispersion is the standard deviation of an attention head's weights across input positions. For a multilayer language model, attention weight dispersion is averaged across heads and then layers. Lower values indicate more uniformly distributed attention weights, with less selective emphasis on individual context tokens.

Perturbation Impact is the total variation distance between next-token distributions before and after modifying the context by masking selected positions. Lower values indicate greater robustness to context perturbations. We average this metric over multiple random masks to reduce sampling variability.

Experimental setup.

We evaluate six pretrained TinyStories models on 1,024 randomly sampled MiniPile documents. To control for context length, we truncate each prompt to its first 1,024 tokens using the corresponding tokenizer and assess whether repetition occurs immediately. For prompts without repetition, both metrics are evaluated at the last token of the prompt. For those with repetition, we average over the first period. For perturbation impact, we sample 10 masks, each selecting 64 positions uniformly at random from the original prompt. We then average the prompt-level metrics separately for prompts with and without repetition.

Results.

Table 1 shows that the repetition group has lower attention weight dispersion and perturbation impact across all six models. These results associate repetitive generation with lower context dependence, consistent with the context-forgetting mechanism suggested by our analysis.

Table 1: Attention weight dispersion and perturbation impact across six pretrained TinyStories models. Each entry reports group means in the order repetition / no repetition.

ModelAttention Weight DispersionPerturbation Impact
TinyStories-3M0.00523 / 0.005980.07554 / 0.09952
TinyStories-8M0.00511 / 0.005370.09208 / 0.10964
TinyStories-28M0.00510 / 0.005140.09665 / 0.12573
TinyStories-33M0.00734 / 0.007380.10141 / 0.15526
TinyStories-1Layer-21M0.00443 / 0.005400.05815 / 0.06662
TinyStories-2Layers-33M0.00713 / 0.007620.06820 / 0.08057

These changes have been incorporated into Section 5.2.

Thank you for the detailed response and additional experiments. I think the new perturbation-impact experiment is a meaningful improvement.

I have one remaining concern about the experimental setup in Section 5.2. It seems that non-repetitive cases are evaluated at the end of the original prompt, while repetitive cases are evaluated after generation has begun. Is this correct?

If so, couldn't the lower perturbation impact arise naturally because the model is already relying on its recently generated repetitive sequence rather than the original prompt? Could the authors compare both groups at the end of the original prompt, grouping them according to whether subsequent generation becomes repetitive?

I am open to raising my score given the improvements, but would appreciate clarification on this point.

Reviewer_mcn6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary and contributions

The paper proposes that long-context softmax attention retains context statistics while losing sensitivity to individual tokens, connecting in-context learning and repetitive generation. It derives a covariance-directed attention limit for stationary, ergodic inputs with elliptical marginals, gives error bounds under additional mixing assumptions, constructs a softmax head that asymptotically implements a gradient step for Gaussian regression, and extends the analysis to a conditional multilayer approximation yielding approximately first-order Markovian generation.

Validation

Experiments show increasing readout alignment for random attention applied to pretrained embeddings, gradient-descent-like behavior in synthetic regression models, and high same-token hidden-state similarity during repetitive TinyStories generation. These support parts of the analysis, but do not establish that trained language-model attention reaches the proposed averaging regime or that this mechanism causes repetition.

Novelty

The Gaussian ICL contribution is incremental: Boursier–Boyer already establishes Gaussian softmax/linear-attention equivalence and analyzes regression learning, making the proposed gradient-step prediction a straightforward specialization. The clearer additions are elliptical alignment, dependence-aware concentration bounds, and the conditional multilayer connection to repetition. The latter is conceptually distinctive, but its practical significance remains weakly validated.

Critical strengths

The paper connects standard softmax attention to gradient descent in a novel way. Proposition 4.1 gives a task-independent parameterization of a single residual softmax attention head that asymptotically implements one gradient step for Gaussian linear regression. The synthetic experiments support both the predicted behavior and the emergence of the proposed parameter structure through training.

Critical weaknesses

The paper's theory describes convergence as context length tends to infinity for a fixed model under specified distributional assumptions, but it does not establish when trained language models enter this regime. Learned attention can maintain strong preferences for particular tokens over practical context lengths, and the finite-context bounds contain parameter-dependent constants that are not evaluated for the pretrained models. Figure 1 provides useful evidence of covariance-readout alignment, but uses randomly initialized attention applied to pretrained embeddings. Table 1 examines pretrained models, yet high same-token representation similarity does not establish convergence to the proposed context-free architecture: shared embeddings carried through residual connections can produce such similarity while contextual components remain important for prediction. The paper would be strengthened by measuring readout approximation error in pretrained attention layers across context lengths, together with next-token sensitivity to controlled changes in preceding context. Without this evidence, the proposed explanation of repetition remains a conditional asymptotic account whose relevance to observed generation is unclear.

Questions for authors

In Section 5.2, Table 1 reports high cosine similarity between terminal representations of different occurrences of the same token. You interpret this as showing that “the terminal representation is largely determined by the current token, with limited sensitivity to the preceding context.”

Could you clarify how this measurement distinguishes the proposed covariance-readout mechanism from the shared token-embedding component carried through the residual stream? In the GPT-2-style architecture you analyze, same-token occurrences share an input embedding that contributes directly to their terminal representations through residual connections. This could produce high cosine similarity even if smaller context-dependent components substantially affect next-token predictions. Similarities are also already high in the prompt tails and generally do not increase during repetitive generation.

Could you provide a control that isolates contextual influence—for example, holding the current token and position fixed while varying the preceding context, and measuring changes in both terminal representations and next-token distributions? This would help establish whether Table 1 supports reduced context sensitivity and approximately first-order Markovian behavior beyond the token-identity contribution already present in the architecture.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mcn6

Thank you for the detailed suggstions!

Weakness: Figure 1 doesn't use pretrained attention layers.

We have revised the experiment to use pretrained models. For each model, we use its tokenizer and its first self-attention layer with pretrained weights to evaluate covariance-readout alignment. We have also revised the estimation of the mean and covariance to use the current causal prefix, making the experimental setup more consistent with the theoretical setting.

These changes have been incorporated into Section 3.3.

Weakness and Questions regarding the repetition experiment.

Our original experiment aimed to show that, for prompts leading to repetition, the final hidden state is almost entirely determined by the current token. The high similarity in the prompt tail was intended to suggest that reduced context dependence was already present before generation, with repetition emerging when autoregressive generation begins.

We agree that the original experiment does not sufficiently support this interpretation. Shared embeddings carried through residual connections can produce such similarity. Moreover, similar hidden-state directions do not imply that next-token predictions are insensitive to earlier context, since small representation differences may still affect output probabilities. A direct comparison of context dependence between repetitive and non-repetitive generation is therefore needed.

We have replaced the original experiment with the following:

We empirically examine whether repetitive generation is associated with reduced context dependence using the following two metrics.

Metrics.

Attention Weight Dispersion is the standard deviation of an attention head's weights across input positions. For a multilayer language model, attention weight dispersion is averaged across heads and then layers. Lower values indicate more uniformly distributed attention weights, with less selective emphasis on individual context tokens.

Perturbation Impact is the total variation distance between next-token distributions before and after modifying the context by masking selected positions. Lower values indicate greater robustness to context perturbations. We average this metric over multiple random masks to reduce sampling variability.

Experimental setup.

We evaluate six pretrained TinyStories models on 1,024 randomly sampled MiniPile documents. To control for context length, we truncate each prompt to its first 1,024 tokens using the corresponding tokenizer and assess whether repetition occurs immediately. For prompts without repetition, both metrics are evaluated at the last token of the prompt. For those with repetition, we average over the first period. For perturbation impact, we sample 10 masks, each selecting 64 positions uniformly at random from the original prompt. We then average the prompt-level metrics separately for prompts with and without repetition.

Results.

Table 1 shows that the repetition group has lower attention weight dispersion and perturbation impact across all six models. These results associate repetitive generation with lower context dependence, consistent with the context-forgetting mechanism suggested by our analysis.

Table 1: Attention weight dispersion and perturbation impact across six pretrained TinyStories models. Each entry reports group means in the order repetition / no repetition.

ModelAttention Weight DispersionPerturbation Impact
TinyStories-3M0.00523 / 0.005980.07554 / 0.09952
TinyStories-8M0.00511 / 0.005370.09208 / 0.10964
TinyStories-28M0.00510 / 0.005140.09665 / 0.12573
TinyStories-33M0.00734 / 0.007380.10141 / 0.15526
TinyStories-1Layer-21M0.00443 / 0.005400.05815 / 0.06662
TinyStories-2Layers-33M0.00713 / 0.007620.06820 / 0.08057

These changes have been incorporated into Section 5.2.

Reviewer_tiX8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies softmax attention in the long-context regime as a readout of population-level input statistics. It analyzes the attention limit under stationarity, ergodicity, and related conditions, and characterizes covariance-directed alignment for elliptical marginals. The authors then construct a residual softmax attention head that asymptotically emulates one GD step from zero regression weights on specified centered Gaussian regression tasks. A multilayer extension provides a conditional Markovian account of repetitive generation. Experiments examine attention alignment on natural-language embeddings, synthetic regression tasks, and repetitive continuations from small pretrained models.

Critical strengths
  1. The paper provides useful theoretical extensions of long-context attention analysis. It allows temporal dependence, characterizes covariance-directed alignment for elliptical marginals under the stated moment conditions, and provides error bounds reflecting context length and dependence. These results help clarify the relationship between finite-sample attention and population-level statistical readout.
  2. The connection to one GD step has an explicit construction and experimental support from multiple perspectives. The authors specify parameters for a single residual softmax attention head that asymptotically emulates one GD step under the stated Gaussian regression conditions. Figures 2–3 compare predictions, gradients with respect to the test input, and learned weight structure, making the examination of this correspondence more informative than a comparison of prediction errors alone.
Critical weaknesses
  1. The repetition experiments do not directly test whether reduced context dependence is associated with repetition. Table 1 includes only continuations classified as repetitive and reports similarities between hidden representations of occurrences of the same token. These similarities are already high in the prompt tails and decrease slightly during generation for most evaluated models. For example, the value for TinyStories-33M changes from 0.9346 to 0.9231. The results do not establish what specifically connects this representation property to repetition. Moreover, similar hidden-state directions do not imply that next-token predictions are insensitive to earlier context; small representation differences may still affect output probabilities. Without non-repetitive controls and direct measurements of history sensitivity, the evidence does not distinguish ordinary same-token representation similarity from reduced context dependence associated with repetition. This limits the paper's central application claim that statistical readout explains repetitive generation in pretrained models (Section 5.2, Table 1).
  2. The covariance term's additional explanatory value and its applicability to actual pretrained models are not sufficiently validated. Figure 1 measures cosine similarity between the attention output and the full readout (V\mu+\gamma V\Sigma K^\top Qx). If (V\mu) contributes substantially to both vectors, their high similarity could largely reflect a shared mean direction. Without a mean-only baseline and comparisons after subtracting the mean contribution, the covariance term's additional explanatory value remains unclear. The choice or estimation of (\gamma) in this experiment also needs clearer documentation. The experiment combines pretrained input embeddings with a randomly initialized attention layer, providing evidence in a single-layer setting. The subsequent multilayer and generation interpretations concern representations produced by trained networks, but the results do not show how approximation errors between actual layer outputs and the readout change with context length. It therefore remains unclear which layers and context lengths in pretrained models are well described by the approximation, weakening the connection between the theoretical results and actual model behavior (Sections 3.3 and 5.1, Figure 1).
Questions for authors
  1. Does the model actually rely less on earlier context when repetition occurs? Please include non-repetitive controls and directly measure history sensitivity. For example, hold the final token and its position fixed while modifying earlier context, keeping the statistical setting as comparable as possible, and measure changes in the next-token distribution. Does sensitivity differ before repetition, during repetition, and in non-repetitive continuations? This control would more directly test the relationship between context dependence and repetition.
  2. How much explanatory value does the covariance term add beyond the mean alone? Please add a mean-only (V\mu) baseline and compare directions, magnitudes, and approximation errors after subtracting this mean contribution. How much does the covariance term improve the approximation? Please also explain how (\gamma) is determined. This would help establish whether the high similarity in Figure 1 primarily reflects the mean contribution.
  3. Does the readout remain accurate in trained multilayer models? Please evaluate the approximation using the actual (Q,K,V) matrices across layers of at least one of the pretrained models already studied, and report errors at different context lengths. Do shallow and deep layers behave similarly, and which layers and context lengths are best described by the approximation? This would clarify the analysis's explanatory scope for actual pretrained models.
2 comments
AuthorsOpenReview ↗

Response to Reviewer tiX8 (Part 1)

Thank you for the detailed suggstions!

Weakness 1 and Question 1: Insufficient evidence from the repetition experiment.

Our original experiment aimed to show that, for prompts leading to repetition, the final hidden state is almost entirely determined by the current token. The high similarity in the prompt tail was intended to suggest that reduced context dependence was already present before generation, with repetition emerging when autoregressive generation begins.

We agree that the original experiment does not sufficiently support this interpretation. Shared embeddings carried through residual connections can produce such similarity. Moreover, similar hidden-state directions do not imply that next-token predictions are insensitive to earlier context, since small representation differences may still affect output probabilities. A direct comparison of context dependence between repetitive and non-repetitive generation is therefore needed.

We have replaced the original experiment with the following:

We empirically examine whether repetitive generation is associated with reduced context dependence using the following two metrics.

Metrics.

Attention Weight Dispersion is the standard deviation of an attention head's weights across input positions. For a multilayer language model, attention weight dispersion is averaged across heads and then layers. Lower values indicate more uniformly distributed attention weights, with less selective emphasis on individual context tokens.

Perturbation Impact is the total variation distance between next-token distributions before and after modifying the context by masking selected positions. Lower values indicate greater robustness to context perturbations. We average this metric over multiple random masks to reduce sampling variability.

Experimental setup.

We evaluate six pretrained TinyStories models on 1,024 randomly sampled MiniPile documents. To control for context length, we truncate each prompt to its first 1,024 tokens using the corresponding tokenizer and assess whether repetition occurs immediately. For prompts without repetition, both metrics are evaluated at the last token of the prompt. For those with repetition, we average over the first period. For perturbation impact, we sample 10 masks, each selecting 64 positions uniformly at random from the original prompt. We then average the prompt-level metrics separately for prompts with and without repetition.

Results.

Table 1 shows that the repetition group has lower attention weight dispersion and perturbation impact across all six models. These results associate repetitive generation with lower context dependence, consistent with the context-forgetting mechanism suggested by our analysis.

Table 1: Attention weight dispersion and perturbation impact across six pretrained TinyStories models. Each entry reports group means in the order repetition / no repetition.

ModelAttention Weight DispersionPerturbation Impact
TinyStories-3M0.00523 / 0.005980.07554 / 0.09952
TinyStories-8M0.00511 / 0.005370.09208 / 0.10964
TinyStories-28M0.00510 / 0.005140.09665 / 0.12573
TinyStories-33M0.00734 / 0.007380.10141 / 0.15526
TinyStories-1Layer-21M0.00443 / 0.005400.05815 / 0.06662
TinyStories-2Layers-33M0.00713 / 0.007620.06820 / 0.08057

These changes have been incorporated into Section 5.2.

Weakness 2 and Question 2: The covariance term's additional explanatory value remains unclear, and the choice of γ\gamma also needs clearer documentation.

The scalar γ\gamma depends on the input distribution and is difficult to estimate accurately in pretrained attention layers. In our original experiment, we simply set γ=1\gamma=1, which was not adequately justified. We have therefore revised the experiment to measure the cosine similarity between fT(u)−Vμf_T(u)-V\mu and the covariance direction VΣK⊤QuV\Sigma K^\top Qu, without requiring an estimate of γ\gamma. We have also revised the estimation of the mean and covariance to use the current causal prefix, making the experimental setup more consistent with the theoretical setting.

Subtracting VμV\mu removes the shared mean contribution and directly examines whether the covariance term explains the direction of the remaining attention output. This isolates its additional directional explanatory value beyond the mean.

These changes have been incorporated into Section 3.3.

AuthorsOpenReview ↗

Response to Reviewer tiX8 (Part 2)

Weakness 2 and Question 3: Evaluation in pretrained multilayer models.

The distribution-dependent factor γ\gamma is difficult to estimate for pretrained representations and can vary across layers and queries. Without reliable estimates, we cannot currently provide a quantitative evaluation of the full readout's approximation error across layers. We apologize for this limitation.

We have also replaced the randomly initialized attention in Section 3.3 with each model's first pretrained self-attention layer, using its actual Q,K,VQ,K,V matrices. The revised experiment evaluates covariance-readout alignment in pretrained attention across context lengths, although its empirical scope remains limited to the first layer.

These changes have been incorporated into Section 3.3.

Reviewer_nmK9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies Softmax attention in the population regime (the infinite context regime in this case). The conclusion from this limit is that the output of the Softmax attention can be interpreted as a covariance readout: Instead of focusing on the input token, the model examines it through its covariance. These results, and their finite-context counterparts, are then applied to study two empirical phenomena: In context learning and repetition.

Critical strengths
  1. The paper is clearly written and easy to follow from the beginning.
  2. The structure of the paper makes the result transparent: All theoretical results are clearly explained, and real-data experiments are provided after.
Critical weaknesses
  1. Novelty: The covariance readout part of the paper largely follows from the work of Boyer and Boursier (2026). In that sense, the first part of the paper is an extension of the past work, although the analysis following this is genuinely new.
  2. Most of my concerns come from the repetition analysis in Section 5 (See questions section ).
Questions for authors
  1. Proposition 4.1 is stated in the infinite-context limit. My understanding it that there is no finite-context length analysis of this proposition. Is this correct? If that is indeed the case, then claims such as "TRAINED TRANSFORMERS DO EMULATE GRADIENT DESCENT" should be corrected accordingly, separating empirical phenomenon from a proved statement.

  2. Corollay 5.1 needs Assumption 3.3. As such, applying it requires to prove that data is "Elliptical sub-Gaussian with strong mixing". However, in line 441, where it says "Asymptotically Markovian generation", it is assumed that corollary 5.1 holds, even though this assumption is never proved. Is this correct? If so, then the tone of some claims should be corrected, such as the last sentence in this paragraph (line 450-451).

3 comments
AuthorsOpenReview ↗

Response to Reviewer nmK9

Thank you for your insightful feedback!

Question 1: Finite-context analysis of Proposition 4.1.

Proposition 4.1 originally stated only an asymptotic result. We have now added a finite-context analysis showing that, under the stated Gaussian and query conditions, the constructed attention layer approximates one GD step computed from the same finite context with an O(T−1)O(T^{-1}) mean-square error for fixed test queries, together with analogous conditional bounds for demonstration queries. These changes have been incorporated into Appendix A.

Question 2: Assumptions underlying asymptotically Markovian generation.

Thank you for pointing this out. The asymptotically Markovian characterization relies on the assumptions of Corollary 5.1. We have revised the paragraph to state explicitly that the Markovian description is conditional on these assumptions. These changes have been incorporated into Section 5.1.

I thank the authors for their answers. I decide to keep my score.

Thank you for taking the time to consider our responses. We appreciate your thoughtful feedback, which has helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.