acceptodds
Withdrawn from ICLR 2027

Structure Before Attention: A Zero-Initialized Convolution Recovers Windowed Language-Model Quality

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sliding-window attention reduces long-context language-modeling cost but can worsen perplexity. We study whether a small convolution added to the residual stream before attention can offset this increase, and what contributes to its benefit. Under paired-seed attribution, paired runs share initialization, data order, and evaluation batches, differing only by the initially inactive branch. Under dense attention the branch gives modest gains at 81M and 253M parameters. In 81M models on PG-19 book text with an 8192-token context and a 256-token window in eight of ten layers, it lowers perplexity by 2.1%, against 0.6% under dense attention, and the windowed model with the branch finishes within 0.16% of the dense model with it in each of three seeds. Across contexts and window sizes, the extra gain descriptively tracks the window penalty (the baseline cost of using sliding-window attention). A pointwise (width-one) branch, which mixes no neighboring tokens, recovers most of the gain, and in a single-seed test, removing the branch after 60% of training and training on retains most of its advantage. The intuition that dense attention learns the branch's function while windowed attention cannot motivated an amplitude-sensitivity test whose registered predictions fail; the mechanism behind the amplified benefit remains open.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sliding-window attention, long-context language models, depthwise convolution, paired-seed evaluation, seed variance, architectural ablation, pre-registration, efficient transformers, attention–SSM hybrids

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VUrgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper study whether a small convolution added to the residual stream before attention can alleviate the perplexity increase caused by window attention. Controlled experiments with various seeds are shown to demonstrate the effectiveness. The flow of the logic is messy and hard to follow, likely generated with AI.

Critical strengths

N/A

Critical weaknesses
  1. The study is shallow, there is not much can be learned from reading the work.
  2. The logic flow is messy and hard to follow.
Questions for authors

N/A

Reviewer_qBv5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether a zero-initialized depthwise causal convolution placed in the residual stream before attention can recover the quality lost by sliding-window attention. The experiments are conducted on small models. These results suggest that the gain is not primarily explained by temporal neighbor mixing. The paper reports that its preregistered amplitude-sensitivity predictions fail, leaving the mechanism behind the larger windowed-attention benefit unresolved.

Critical strengths

The method is easy to follow and reimplement, while its improvement is also modest.

Critical weaknesses
  1. There is no 1B+ replication or standard long-context capability benchmark, so the practical generality remains uncertain.

  2. There are many methods to blend the boundary tokens; conv is not the only choice, while the paper lacks comparation of them.

Questions for authors
  1. Can the windowed amplification be replicated at 1B+ scale, on a second long-document corpus, and on a standard long-context capability benchmark rather than aggregate perplexity alone?

  2. Under the optimized implementation, what are the end-to-end training/inference throughput, peak-memory, and quality-compute Pareto curves versus changing the window size or number of dense layers?

  3. Do width-one and width-four branches remain different under matched parameter/FLOP budgets and symmetric insertion/removal experiments that reset or separate optimizer state?

Reviewer_LR3NOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a zero-initialized convolutional branch before attention and reports larger perplexity gains under windowed attention. Paired training runs and several controls examine the contribution of temporal mixing and training history.

Critical strengths
  1. The paired comparisons and multiple-seed experiments provide reasonable confidence in the main effect.

  2. The pointwise and branch-removal controls are informative, and the authors acknowledge the limitations of their mechanistic explanations.

Critical weaknesses
  1. The novelty is limited. The method is a small variation on existing convolutional branches, and the width-one control already captures most of the gain.

  2. The writing could be more focused. The main contribution is difficult to identify among the many ablations and mechanistic discussions.

  3. The main result is demonstrated in a narrow setting. The windowed-attention gain is only shown at 81M on PG-19. The 253M experiment tests dense attention only, and the window-specific benefit becomes smaller in the longer-training run.

Questions for authors
  1. Does the windowed-attention advantage persist at a larger model size and a substantially longer training budget?
  2. How does the proposed branch compare with Canon-A under the same training setup? It would be useful to separate the effects of changing the branch placement and using zero initialization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.