acceptodds
Under review as a conference paper at ICLR 2027

How Far Can Sliding Window Attention See?

Abstract

Sliding window attention (SWA) reduces the inference cost of Transformers by restricting each token to attend only to a fixed-size local window. Although the resulting efficiency gains are well understood, it remains unclear how information propagates beyond this window and, consequently, the maximal sentence length a SWA model can effectively manipulate. We provide an intuitive account of information propagation through successive SWA layers and use it to derive a theoretical prediction for the maximal effective length. For a model with window size and layers, our analysis predicts a context length of , where is the -th harmonic number. Experiments on the cramming task across several model families, scales, depths, and window sizes confirm this prediction, with predicted lengths within of the measured limits in all tested configurations. Finally, we give an explicit construction that approximates full attention with SWA.

Primary area:
learning theory
Keywords:
expressivity, approximation, theory, sliding window attention, transformer, cramming

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K2ocOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effective information utilization range of sliding window attention. It proposes that this range grows approximately linearly with the attention window size and logarithmically with model depth, rather than matching the full architectural receptive field. The authors motivate this scaling through an information propagation model, provide theoretical results under specific assumptions, and evaluate the prediction through soft-prompt cramming experiments across several pretrained models. The paper addresses a relevant question for efficient long-context modeling. However, the evidence currently supports a conditional theoretical model and an empirical scaling hypothesis more convincingly than a general characterization of effective context utilization.

Critical strengths

Relevant research question. Understanding how effectively local attention architectures retain information beyond their immediate windows is important. The distinction between architectural reachability and effective information utilization provides a useful perspective on this problem.

A concrete prediction supported by theory and experiments. The paper proposes a simple, testable scaling relationship and investigates it through both theoretical analysis and experiments across multiple models and window sizes. This combination makes the work more informative than a purely qualitative discussion of information propagation.

Potential to motivate further architecture research. The analysis provides a starting point for studying the relative benefits of larger attention windows, greater depth, and mechanisms for direct long-range access.

Critical weaknesses

The central claim is stronger than the theoretical evidence supports. The proposed scaling depends on restrictive assumptions about information degradation and on an assumed proportional relationship between Fisher information and the cramming horizon. These assumptions are not sufficiently established for general pretrained Transformers. Although the derivation may hold within the proposed model, it does not establish a universal law of effective context length. The abstract and conclusion should more clearly distinguish conditional theoretical results from empirical observations.

The connection to practical context utilization is insufficiently developed. Soft-prompt cramming evaluates whether a frozen model can reproduce a target sequence after its continuous input vectors have been directly optimized. This is a useful capacity probe, but it differs substantially from retrieving relevant information, integrating evidence, and performing in-context learning from natural-language inputs. The paper does not demonstrate that the measured cramming horizon reliably characterizes these capabilities. Consequently, interpreting the result as an effective context limit for practical models appears premature.

The propagation model is idealized relative to the behavior of trained models. The analysis abstracts away much of the content-dependent routing and nonlinear processing that can influence information retention. The supporting experiments provide some evidence for the proposed approximation, but do not sufficiently establish its validity across propagation distances and model configurations. Moreover, finite-budget optimization makes it difficult to separate an intrinsic capacity limit from a limitation of the cramming procedure.

The implications for model design remain relatively direct and underdeveloped. Larger windows and occasional global attention are natural ways to reduce the number of local relay steps. However, the paper does not establish when these choices outperform additional depth under matched computational or memory budgets. It also does not provide more specific guidance on allocating window sizes or placing global-attention layers. The current analysis therefore offers limited insight beyond these intuitive design considerations.

Questions for authors
  1. Can the relationship between Fisher information and the cramming horizon be independently validated or theoretically justified, rather than assumed? Which parts of the central conclusion remain valid if this relationship does not hold?

  2. How well does the measured cramming horizon predict performance on practical long-context tasks, particularly retrieval, multi-step reasoning, and in-context learning?

  3. How sensitive are the observed saturation points to the optimization budget, initialization, and target-sequence distribution? What evidence distinguishes model capacity limitations from optimization difficulty?

  4. Does the proposed dependence on depth explain the experimental results better than alternative simple scaling relationships, after accounting for uncertainty in the measured horizons?

  5. Can the framework produce and validate a more specific architectural prediction, such as how to allocate attention windows or global-attention layers under a fixed inference budget?

2 comments

We thank the reviewer for their detailed review and for highlighting the relevance of our research question, the testable prediction supported by theory and experiments, and the potential to motivate further architecture research.

The evidence currently supports a conditional theoretical model and an empirical scaling hypothesis more convincingly than a general characterization of effective context utilization.

We agree that our theory relies on crucial assumptions, especially in order to obtain such a simple formula for such a complicated problem. Knowing the effective context utilization of SWA is a very relevant and intricate problem, and we merely hope that our paper provides a step in the correct direction.

We are encourage by the fact that our formula is way tighter than that of previous work analyzing the cramming task (within a factor of 2 instead of 10 for [1]), and that we also provide theoretical constructions that may be interesting in their own right (as mentioned by Reviewer xGH7).

The abstract and conclusion should more clearly distinguish conditional theoretical results from empirical observations.

We thank the reviewer for pointing this out and have adapted both accordingly.

The connection to practical context utilization is insufficiently developed.

While the cramming task has been studied by other works, it is true that how limitations on this task extend to other tasks of interest is not straightforward. We revised the text, especially the abstract, introduction, and conclusion, to make sure this distinction is explicit.

The propagation model is idealized relative to the behavior of trained models.

That is a valid limitation of our work. Our theoretical analysis relies on crucial simplifying assumptions, as is standard in theoretical works. We tried to justify all assumptions as well as possible, cited other work relying on the same assumptions, and also gave an intuitive view on the origin of the formula. However, we understand how important it is to state that our theoretical analysis relies on those simplifying assumptions and stressed this in the revised manuscript.

The implications for model design remain relatively direct and underdeveloped.

That is entirely correct! Our paper sits in a line of work trying to quantitatively predict the performance of transformers on specific tasks [1-4]. The hope of this line of work is that eventually, precise enough predictions on complicated enough tasks will provide a much deeper understanding of how transformers work. This might in turn lead to interesting practical implications.

However, as this is a relatively new field, our results are modest and the practical implications are not straightforward.

Can the relationship between Fisher information and the cramming horizon be independently validated or theoretically justified, rather than assumed?

We agree that this is the most crucial assumption, but is hard to validate empirically, and we leave this line as future work.

If the relationship does not hold, our intuitive analysis of SWA (Section 3), our theoretical constructions (Section 4), and the empirical scaling law (Section 6) still hold. However, the analysis of Section 5, while correct in predicting the Fisher information, loses its practical relevance. That would leave open the problem of finding a more rigorous proof of the formula we obtain for the cramming horizon.

How well does the measured cramming horizon predict performance on practical long-context tasks, particularly retrieval, multi-step reasoning, and in-context learning?

This is an important question left unanswered by previous work [1,5], which we have no answer for and leave as future work.

How sensitive are the observed saturation points to the optimization budget, initialization, and target-sequence distribution?

We thank the reviewer for raising this point. We added experiments to show that their impact is modest, and refer to our answer Q4 to Reviewer xGH7.

Does the proposed dependence on depth explain the experimental results better than alternative simple scaling relationships, after accounting for uncertainty in the measured horizons?

As said in the answer before, our measured horizons are more than precise enough that we can validate our prediction. There is of course no absolute guarantee our formula is correct. But the intuition we provide, our theoretical justification, as well as our extensive experiments lead us to believe this is the case.

Can the framework produce and validate a more specific architectural prediction?

Our hope is that this work can lead to relevant specific architectural predictions down the line. However, as stated above, this line of work is relatively new. As such, the reviewer correctly identified that our results would require additional refinement before having concrete impact (such as figuring out the relationship between cramming and other tasks of interest).

[1] Meyer, et al. How Many Different Outputs Can a Transformer Generate? ICML'26.

[2] Yang, et al. Knee-Deep in C-RASP: A Transformer Depth Hierarchy. NeurIPS'25.

[3] Merrill, et al. A Little Depth Goes a Long Way: the Expressive Power of Log-Depth Transformers. NeurIPS'25.

[4] Yehudai, et al. When Can Transformers Count to n? UAI'26.

[5] Kuratov, et al. Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity. ACL'25

Reviewer_D5FUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims at understanding how LLMs can leverage stacked SWA layers in order to propagate information beyond the SWA window size. It derives a theoretical prediction for how much information can be compressed into each layer's key-values and tests it on multiple models and window sizes. Empirical results largely validate the theory and show that the effective window size does not scale proportionally to the model's depth, but proportionally to the harmonic series ∑k(1/k)\sum_k(1/k).

Critical strengths
  • The paper identifies a concrete law governing the effective attention span of off-the-shelf pretrained LLMs when used zero-shot in a pure sliding-window-attention setting. This law could be particularly helpful to understand the dynamics of stacked sliding window attention layers and how to design future architectures based on them.
  • The paper introduces a clear methodology to test this effective attention span.
  • They test this law on 8 models and 3 window sizes. The results are all consistent with the law they identified in the first place.
Critical weaknesses

My main concern is the gap between the generality of the claimed effective-context law and the experimental setting used to validate it. The experiments impose small sliding-window masks on frozen pretrained checkpoints—full-attention Qwen models and hybrid local/global-attention Gemma-2 models—without retraining their weights. Although target-specific soft prompts are optimized, these experiments characterize recall after an attention-mask change, rather than models trained to operate with pure sliding-window attention.

This distinction is consequential for two reasons:

• The paper does not systematically establish that predictive performance remains reasonable after conversion. It reports neither perplexity nor broad task evaluations under the imposed masks. The traffic-light example illustrates the distinction: changing the light’s color influences the relative preference between “stop” and “go,” but the top prediction under SWA is “approach” or “decided,” depending on window size. Activation patching demonstrates that information can influence predictions, but does not establish reliable completion accuracy. The measured recall horizon could therefore reflect broader disruption from the mask change, including removal of attention sinks, alongside limitations of information propagation.

• These checkpoints were not trained to rely exclusively on successive local layers to relay distant information. Models trained with pure SWA could learn different encoding and propagation strategies, potentially changing the observed horizon. The relay observations and approximate harmonic scaling are interesting, but the experiments do not establish whether the same scaling holds for models trained from scratch with pure SWA.

Without evaluation on models trained with pure SWA, or a more narrowly scoped claim, I do not consider the empirical evidence sufficient to support the proposed law as a general characterization of SWA’s effective context length.

Questions for authors
  1. Could you test how much the models' performance is impacted by their conversion to pure SWA? In particular, could you analyze in depth the zero-shot results for handwritten prompts such as the traffic light sentence before and after model conversion?
  2. In Appendix H, normalizing by J₁ makes the relative information decay independent of hidden dimension. However, for a fixed tokenizer and target distribution, the information content of the target tokens does not grow with the model’s hidden dimension. Why should the information required per recoverable token scale with J₁, rather than additional dimensions allowing more tokens to be encoded or providing redundancy against interference? The resulting dimension-independent token horizon is counterintuitive and warrants further justification.
3 comments

We thank the reviewer for their careful review and valuable feedback. We believe adding the suggested experiments strongly enhances the soundness of our analysis.

My main concern is the gap between the generality of the claimed effective-context law and the experimental setting used to validate it. The experiments impose small sliding-window masks on frozen pretrained checkpoints—full-attention Qwen models and hybrid local/global-attention Gemma-2 models—without retraining their weights. Although target-specific soft prompts are optimized, these experiments characterize recall after an attention-mask change, rather than models trained to operate with pure sliding-window attention.

We thank the reviewer for identifying this crucial whole in our experimental setting. We ran additional experiments on Ministral-3-3B and Mistral-7B-v0.1, and obtained a factor of at most 1.7. The reviewer is therefore correct in predicting that a pure SWA model performs better than those we studied. We expect this is because Assumption 20 becomes less accurate. Nonetheless, our prediction remains much more accurate than previous work on the cramming task [1], which had a factor of 10.

We added the results to Table 1:

ModelParametersLayersW=16W=32W=64
Ministral-3-3B3B2690/62 [1.46]180/123 [1.46]350/247 [1.42]
Mistral-7B-v0.17B32110/65 [1.69]200/130 [1.54]400/260 [1.54]

We are also happy to evaluate our prediction on further pure SWA models if needed!

In Appendix H, normalizing by J1J_1 makes the relative information decay independent of hidden dimension. [...] The resulting dimension-independent token horizon is counterintuitive and warrants further justification.

That is indeed a counter-intuitive result. It is worth noting that our experiments do seem to validate this trend. For instance, Table 1 shows that both Qwen3-0.6B and Qwen3-1.7B share roughly the same cramming horizon, despite the latter having double the hidden dimension. The same can be said for Qwen3-4B and Qwen3-8B.

PairLayersdmodeld_{\text{model}}Horizon at W = 16 / 32 / 64
Qwen3-0.6B vs Qwen3-1.7B281,024 vs 2,04870/140/270 vs 60/140/290
Qwen3-4B vs Qwen3-8B362,560 vs 4,09670/130/270 vs 70/130/270

We were actually surprised when we realized our very simplistic theoretical formula seemed to hold in practice. That is why we conducted such extensive experiments to validate it. We still find it puzzling that such a simple function of only the number of layers and window size can predict the cramming horizon, independant of vocabulary size, embedding dimension, and number of parameters. That is why we are excited about our paper and especially eager to hear the opinion of other researchers of the field.

Our best guess is that the dimension is not the limiting factor, and as such does not influence the cramming horizon: if all information gets lost past a certain amount of windows, no increase in dimension can fix it. Once again, we are happy to conduct any additional experiment you believe could help us understand this result!

On the size of the sliding windows.

To address this, we extend our experiments to W=512W=512, eight times larger than W=64W=64. The results suggest that the observed agreement with the prediction persists at a substantially larger, practically relevant window size. Due to limited compute we evaluated only Qwen3-0.6B and Gemma-2-2B using M=1000M=1000.

ModelPredicted horizon WHℓWH_\ellObserved bracketDeviation
Qwen3-0.6B2010.71880–1890−6.3%-6.3\%
Gemma-2-2B1973.51820–1830−7.5%-7.5\%

[1] Meyer, et al. How Many Different Outputs Can a Transformer Generate? ICML'26.

Thank you for your answer. Your results on trained SWA models confirm my initial observations, that is that this measurement captures the information propagation strategies that the model has learned during training, impeded by the fact that the attention mask is changed at inference time. As such, I will keep my current score.

We thank the reviewer for pushing us to make our analysis more detailed. We conduct further experiments to show that randomly initializing the Ministral model makes the cramming horizon go back to our theoretical formula (the details are discussed below). In short, our formula is almost exact for randomly initialized models. Training the model without SWA and adding a window at inference only has negligible impact on the cramming horizon. However, training a model specifically on SWA can increase the cramming horizon up to 70%.

We reframed our paper to clearly state our three main contributions:

  • We provide intuition as to the working of SWA.
  • We discuss the optimal parameter scaling to approximate regular attention with SWA.
  • We leverage assumptions to find a very simple formula for the baseline cramming horizon of a SWA model. This formula is almost exact for both randomly initialized models and models trained without SWA. Training a model on SWA allows to improve the cramming horizon from the baseline, by a factor at most 2. Even for those models, our analysis remains far tighter than previous theoretical formulas on the cramming task (factor 2 instead of 10).

We believe that this discussion phase strongly improved the rigour of our analysis and the clarity of our paper. We are extremely thankful to the reviewer for their valuable feedback.

Experimental details.

We reinitialized all decoder blocks in Qwen3-4B and Ministral-3-3B, froze the resulting models, and repeated the cramming experiments at W=16,32,64. Projection weights were initialized from a zero-mean Gaussian with standard deviation 0.02, with normalization parameters reset appropriately. We used three independent weight seeds, M=200, a 5,000-step cap, patience 500, and 100 targets per final bracket endpoint. So only token embeddings, the final normalization layer, and the output head are unchanged.

ModelWindowPrediction WHℓWH_\ellRandomized mean midpointDeviation from the theoretical prediction
Qwen3-4B1666.875+12.3%
Qwen3-4B32133.6135+1.1%
Qwen3-4B64267.2255-4.6%
Ministral-3-3B1661.765+5.4%
Ministral-3-3B32123.3125+1.3%
Ministral-3-3B64246.7231.7-6.1%

In conclusion, these experiments suggest that our theoretical formula is precise for any randomly initialized architecture. Training on SWA allows to increase the cramming horizon of the model up to a factor 1.7 for the Minsitral models.

This is due to the fact that training a model on SWA allows to conserve more information between layers, and Assumption 20 is less valid because of this. As such, measuring how much a model concords with Assumption 20 (which we sketched in Appendix I) is a promising road to evaluate the performance of SWA transformers. We believe this further strengthens our paper, and thank the reviewer for their perspective.

Reviewer_xGH7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how far information can travel through sliding-window attention (SWA). In theory, the attention graph's receptive field scales with window size times depth, but in practice—when measured via soft-prompt cramming—the effective horizon is often shorter. By treating successive layers as noisy replays and linking Fisher information to cramming capacity, the authors derive a predicted horizon of WHℓW H_\ell, where HℓH_\ell is the harmonic number. The authors validated across eight models and three window sizes. In addition, they also include a separate construction showing how deeper, wider SWA networks can approximate full attention.

Critical strengths
  1. Strong empirical design: The experiments highlight the practical gap between what the attention graph allows in theory and what optimized soft prompts can actually pull off. Beyond the primary model and window-size sweeps, it is nice to see thorough sanity checks in the appendix covering depth, recovery thresholds (appendix J), and various noise scales (appendix I.2).
  2. Valuable theoretical construction: showing how causal SWA can explicitly simulate full attention is a nice standalone contribution. The authors go a step further than Yun. et.al (nips 2020) by pinning down concrete depth and width costs, including position-encoding considerations.
Critical weaknesses

I am not fully convinced by how the normalization constant is fixed in Theorem 26. Based on Assumption 25, which states that the cramming horizon is proportional to summed Fisher information, the proof sets the one-layer horizon to W by appealing to Theorem 10, and thereby fixes the proportionality constant at 1/J11/J_1. However, Theorem 10 only constructs a specific transformer that hits this horizon-it does not show that an arbitrary pre-trained one-layer model will achieve it, nor that the same constant holds across different depths. The harmonic Fisher-information sum may hold under the noise assumptions, trying it directly to the cramming horizon of the evaluated checkpoints needs stronger justification. The noise-scale correction in Appendix I.2 addresses a different issue.

Questions for authors
  1. How do you justify using the worst-case/constructed maximum from Theorem 10 to fix the proportionality constant in Theorem 26 for pre-trained models? if bridging this gap requires an extra assumption, it would be nice to state it explicitly; otherwise, how would the main result look if written with an explicit, model-dependent calibration factor?
  2. When evaluating a model/window setup right near the measured plateau, does recovery improve if you give the optimizer a larger budget or run multiple random initializations on the same targets? Even with the helpful controls in appendix J, it is currently hard to tell whether the plateau reflects a true theoretical absence of an accessible soft prompt, or simply optimization failure within a fixed budget.
  3. Definition 9 defines accessible sequences over the entire token vocabulary, whereas the experiments in Section 6.1 sample words from a dictionary before tokenizing. How does your empirical recovery rate map back to the theoretical fraction in Def. 9? Specifically, do you expect the horizon to still track WHtWH_t if targets are sampled uniformly over raw tokens, or should the empirical claims be framed more narrowly around natural word distributions?
4 comments

We are very grateful to the reviewer for their insightful review. Their comments, including the inacuracy they found in the Appendix proof, show a careful reading of our paper, and we believe adressing their comments has strongly improved our paper.

Q1: I am not fully convinced by how the normalization constant is fixed in Theorem 26.

That is correct, our proof is imprecise, and Theorem 26 only provides an upper bound on the normalization constant. We thank the reviewer for spotting this.

We added an experimental verification that the normalization constant is WW in practice: for every model and value of WW, we isolate the first transformer layer and measure its cramming horizon. Our experiments show that it is equal to exactly WW for all models and window sizes of Table 1. This is quite intuitive: WW is typically small, so easy to reach in terms of cramming, and it is a strict upper bound. So we would expect the transformers we study to have this exact cramming horizon when we isolate their first layer. This is exactly what happens in practice.

As for whether the coefficient holds across different depths, we assume this is the case in Assumption 25. We agree that this is an approximation, and we leave to future work the refinement of our theoretical analysis. We are however, encouraged by the fact that our analysis is tight within a factor less than 2, compared to previous work on the cramming task which is only tight up to a factor 10 [1].

We have corrected the manuscript accordingly.

Q2: The harmonic Fisher-information sum may hold under the noise assumptions, trying it directly to the cramming horizon of the evaluated checkpoints needs stronger justification.

Indeed, the main justification we have is ad hoc. Tying both leads to much better prediction results than previous work, which suggests that there may be a link.

We necessarily need to rely on some assumptions to obtain a formula as straightforward as ours. While we believe exploring the relation between Fisher information and the cramming task can indeed be interesting future work, we believe our contributions (highlighted by the reviewer) in terms of added intuition, theoretical construction, and cramming horizon formula (even though the theoretical justification is imperfect), consist in a solid contribution by themselves.

Q3: How do you justify using the worst-case/constructed maximum from Theorem 10 to fix the proportionality constant in Theorem 26 for pre-trained models?

As mentioned above, the model-dependent calibration factor happens to match exactly the constructed maximum from Theorem 10. We forgot to mention it in the paper, and apologize for the mistake.

Q4: When evaluating a model/window setup right near the measured plateau, does recovery improve if you give the optimizer a larger budget or run multiple random initializations on the same targets?

We thank the reviewer for noticing this. We simply used 5000 optimization steps following [1], but this lacked justification. To check we increased the maximum number of steps from 5,000 to 10,000, retaining the same early-stopping patience of 500 steps without accuracy improvement. Using W=64W=64, M=200M=200 the experiments strongly suggest that the cramming horizon does not stem from the optimization budget.

Model5K-steps bracket10K-steps bracketMidpoint shift
Qwen3-0.6B260–270270–280+3.8%
Gemma-2-2B250–260250–2600%

We also repeat the same experiment with five independent initializations per target, confirming that initialization has negligible impact as well.

ModelWOriginal bracketFive-starts bracket
Qwen3-0.6B1660–7060–70
Gemma-2-2B1660–7060–70
Qwen3-0.6B64260–270270–280
Gemma-2-2B64250–260260–270

Q5: Definition 9 defines accessible sequences over the entire token vocabulary, whereas the experiments in Section 6.1 sample words from a dictionary before tokenizing. How does your empirical recovery rate map back to the theoretical fraction in Def. 9?

That's a very good point! Once again, while we followed [1], we would expect the cramming horizon to change depending on how we sample the target words. More precisely, since encoding arbitrary token sequences is harder, the resulting cramming horizon should be lower. We conducted an additionnal experiment that suggests the variation is modest in practice. This could be explained by the fact that both type of sequences are as hard to encode in the context of the cramming task.

ModelRandom-word bracketUniform-token bracket
Gemma-2-2B250–260230–240

[1] Meyer, et al. How Many Different Outputs Can a Transformer Generate? ICML'26.

Thank you for your detailed response with additional experiments. I noticed that you added Assumption 26 setting the proportionality coefficient to 1/J11/J_1, which addresses the issue in the original theorem. Most of my concerns have been addressed now. However, the newly added Mistral-3-3B and Mistral-7B experiments have measured horizons of 1.4-1.7 times WHlWH_l across window sizes, which seems indeed strange. Do they mean the factor is model dependent? The claim that the prediction matches every model seems too strong in my opinion. I will keep my rating of 2 for now, but remain open to raise it if the authors provide further explanation.

We thank the reviewer for their remark. We conducted further experiments to show that randomly initializing the Ministral model makes the cramming horizon go back to our theoretical formula (the details are discussed in our answer to Reviewer D5FU). In short, our formula is almost exact for randomly initialized models. Training the model without SWA and adding a window at inference only has negligible impact on the cramming horizon. However, training a model specifically on SWA can increase the cramming horizon up to 70%.

We reframed our paper to clearly state our three main contributions:

  • We provide intuition as to the working of SWA.
  • We discuss the optimal parameter scaling to approximate regular attention with SWA.
  • We leverage assumptions to find a very simple formula for the baseline cramming horizon of a SWA model. This formula is almost exact for both randomly initialized models and models trained without SWA. Training a model on SWA allows to improve the cramming horizon from the baseline, by a factor at most 2. Even for those models, our analysis remains far tighter than previous theoretical formulas on the cramming task(factor 2 instead of 10).

We believe that this discussion phase strongly improved the rigour of our analysis and the clarity of our paper. We are extremely thankful to the reviewer for their valuable feedback.

Thank you for the additional experiments, which makes the context length WHlWH_l more convincing. However, I will keep my rating because it is still unclear whether training with SWA explains the increased horizon.

Comments on the paper

1 comment

Dear Reviewers,

The authors have finished uploading their responses to reviewers, and the discussion phase is now underway. If you haven't already, please carefully review the authors' responses to understand their perspectives. Please engage in thoughtful and constructive discussions with the authors, and update your review or rating if necessary.

The discussion period lasts until November 18th AoE, which means you have less than five days left to interact with the authors. Please start engaging well before the period ends, so that the authors will have sufficient time to respond to any follow-up questions/comments.

Your active participation and contribution to the ongoing discussion are highly encouraged. Thank you very much for your contribution to ICLR.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.