- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the effective information utilization range of sliding window attention. It proposes that this range grows approximately linearly with the attention window size and logarithmically with model depth, rather than matching the full architectural receptive field. The authors motivate this scaling through an information propagation model, provide theoretical results under specific assumptions, and evaluate the prediction through soft-prompt cramming experiments across several pretrained models. The paper addresses a relevant question for efficient long-context modeling. However, the evidence currently supports a conditional theoretical model and an empirical scaling hypothesis more convincingly than a general characterization of effective context utilization.
Critical strengths
Relevant research question. Understanding how effectively local attention architectures retain information beyond their immediate windows is important. The distinction between architectural reachability and effective information utilization provides a useful perspective on this problem.
A concrete prediction supported by theory and experiments. The paper proposes a simple, testable scaling relationship and investigates it through both theoretical analysis and experiments across multiple models and window sizes. This combination makes the work more informative than a purely qualitative discussion of information propagation.
Potential to motivate further architecture research. The analysis provides a starting point for studying the relative benefits of larger attention windows, greater depth, and mechanisms for direct long-range access.
Critical weaknesses
The central claim is stronger than the theoretical evidence supports. The proposed scaling depends on restrictive assumptions about information degradation and on an assumed proportional relationship between Fisher information and the cramming horizon. These assumptions are not sufficiently established for general pretrained Transformers. Although the derivation may hold within the proposed model, it does not establish a universal law of effective context length. The abstract and conclusion should more clearly distinguish conditional theoretical results from empirical observations.
The connection to practical context utilization is insufficiently developed. Soft-prompt cramming evaluates whether a frozen model can reproduce a target sequence after its continuous input vectors have been directly optimized. This is a useful capacity probe, but it differs substantially from retrieving relevant information, integrating evidence, and performing in-context learning from natural-language inputs. The paper does not demonstrate that the measured cramming horizon reliably characterizes these capabilities. Consequently, interpreting the result as an effective context limit for practical models appears premature.
The propagation model is idealized relative to the behavior of trained models. The analysis abstracts away much of the content-dependent routing and nonlinear processing that can influence information retention. The supporting experiments provide some evidence for the proposed approximation, but do not sufficiently establish its validity across propagation distances and model configurations. Moreover, finite-budget optimization makes it difficult to separate an intrinsic capacity limit from a limitation of the cramming procedure.
The implications for model design remain relatively direct and underdeveloped. Larger windows and occasional global attention are natural ways to reduce the number of local relay steps. However, the paper does not establish when these choices outperform additional depth under matched computational or memory budgets. It also does not provide more specific guidance on allocating window sizes or placing global-attention layers. The current analysis therefore offers limited insight beyond these intuitive design considerations.
Questions for authors
-
Can the relationship between Fisher information and the cramming horizon be independently validated or theoretically justified, rather than assumed? Which parts of the central conclusion remain valid if this relationship does not hold?
-
How well does the measured cramming horizon predict performance on practical long-context tasks, particularly retrieval, multi-step reasoning, and in-context learning?
-
How sensitive are the observed saturation points to the optimization budget, initialization, and target-sequence distribution? What evidence distinguishes model capacity limitations from optimization difficulty?
-
Does the proposed dependence on depth explain the experimental results better than alternative simple scaling relationships, after accounting for uncertainty in the measured horizons?
-
Can the framework produce and validate a more specific architectural prediction, such as how to allocate attention windows or global-attention layers under a fixed inference budget?
2 comments
[1] Meyer, et al. How Many Different Outputs Can a Transformer Generate? ICML'26.
[2] Yang, et al. Knee-Deep in C-RASP: A Transformer Depth Hierarchy. NeurIPS'25.
[3] Merrill, et al. A Little Depth Goes a Long Way: the Expressive Power of Log-Depth Transformers. NeurIPS'25.
[4] Yehudai, et al. When Can Transformers Count to n? UAI'26.
[5] Kuratov, et al. Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity. ACL'25
We thank the reviewer for their detailed review and for highlighting the relevance of our research question, the testable prediction supported by theory and experiments, and the potential to motivate further architecture research.
We agree that our theory relies on crucial assumptions, especially in order to obtain such a simple formula for such a complicated problem. Knowing the effective context utilization of SWA is a very relevant and intricate problem, and we merely hope that our paper provides a step in the correct direction.
We are encourage by the fact that our formula is way tighter than that of previous work analyzing the cramming task (within a factor of 2 instead of 10 for [1]), and that we also provide theoretical constructions that may be interesting in their own right (as mentioned by Reviewer xGH7).
We thank the reviewer for pointing this out and have adapted both accordingly.
While the cramming task has been studied by other works, it is true that how limitations on this task extend to other tasks of interest is not straightforward. We revised the text, especially the abstract, introduction, and conclusion, to make sure this distinction is explicit.
That is a valid limitation of our work. Our theoretical analysis relies on crucial simplifying assumptions, as is standard in theoretical works. We tried to justify all assumptions as well as possible, cited other work relying on the same assumptions, and also gave an intuitive view on the origin of the formula. However, we understand how important it is to state that our theoretical analysis relies on those simplifying assumptions and stressed this in the revised manuscript.
That is entirely correct! Our paper sits in a line of work trying to quantitatively predict the performance of transformers on specific tasks [1-4]. The hope of this line of work is that eventually, precise enough predictions on complicated enough tasks will provide a much deeper understanding of how transformers work. This might in turn lead to interesting practical implications.
However, as this is a relatively new field, our results are modest and the practical implications are not straightforward.
We agree that this is the most crucial assumption, but is hard to validate empirically, and we leave this line as future work.
If the relationship does not hold, our intuitive analysis of SWA (Section 3), our theoretical constructions (Section 4), and the empirical scaling law (Section 6) still hold. However, the analysis of Section 5, while correct in predicting the Fisher information, loses its practical relevance. That would leave open the problem of finding a more rigorous proof of the formula we obtain for the cramming horizon.
This is an important question left unanswered by previous work [1,5], which we have no answer for and leave as future work.
We thank the reviewer for raising this point. We added experiments to show that their impact is modest, and refer to our answer Q4 to Reviewer xGH7.
As said in the answer before, our measured horizons are more than precise enough that we can validate our prediction. There is of course no absolute guarantee our formula is correct. But the intuition we provide, our theoretical justification, as well as our extensive experiments lead us to believe this is the case.
Our hope is that this work can lead to relevant specific architectural predictions down the line. However, as stated above, this line of work is relatively new. As such, the reviewer correctly identified that our results would require additional refinement before having concrete impact (such as figuring out the relationship between cramming and other tasks of interest).