acceptodds
Under review as a conference paper at ICLR 2027

When Can Attention Heads Be Statically Defined?

Abstract

Self-attention recomputes token interactions for each input, although some heads learn similar attention patterns across inputs. Reusing fixed patterns for these heads could reduce training cost by removing query-key score computation and softmax. Through controlled pretraining comparisons, we identify Selective Attention Freezing (SAF), a recipe that selects heads with low attention-pattern variance and fixes their attention weights to fitted post-softmax means halfway through training. We represent these fixed causal patterns with absolute-position and relative-distance preferences, reducing their storage from quadratic to linear in sequence length. A fused kernel reconstructs the patterns during execution and processes ordinary and replaced heads in a single launch. At matched training-token budgets, our preferred 25% replacement setting gives 1.056× faster post-replacement optimiser updates at 124M parameters and 4K context, with a 0.77% increase in perplexity. SAF transfers to 1B parameters at 8K context, giving a 1.066× update speedup with a 0.51% perplexity increase. Replacing 50% of heads extends the speedups to 1.119× and 1.165×, with perplexity increases of 2.49% and 2.03%, respectively. The resulting checkpoints also support downstream finetuning with faster updates for long inputs and faster causal prefill.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Static attention, Attention head selection, Pretraining, GPU kernels

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vSmxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether some attention heads can be replaced with fixed, input-independent attention patterns to reduce the computational cost of language model training. The authors propose Selective Attention Freezing (SAF), which identifies heads with low attention-pattern variance and replaces their attention weights with fitted post-softmax means halfway through training. These fixed patterns are represented using absolute-position and relative-distance preferences, reducing their storage requirements from quadratic to linear in sequence length. A fused kernel is developed to execute ordinary and fixed-pattern attention heads together, avoiding query-key computation and softmax for the replaced heads.

The authors evaluate SAF through controlled pretraining experiments, downstream finetuning, causal prefill benchmarks, and associative recall tasks. At a 25% replacement rate, SAF achieves 1.056× faster post-replacement training updates on a 124M model at 4K context, with a 0.77% perplexity increase. On a 1B model at 8K context, it achieves a 1.068× speedup with a 0.51% perplexity increase. The paper also reports improved generalization to more key-value associations at a fixed sequence length compared with ordinary attention and pruning baselines.

Critical strengths

1. An interesting and well-motivated approach to reducing attention computation.

I like the idea of identifying attention heads whose attention weights are relatively stable across inputs and replacing them with fixed patterns. The approach is quite aggressive, but it is also interesting because it can eliminate query-key computation and softmax for the selected heads, rather than merely reducing the number of attended positions. The combination of variance-based head selection, compact pattern fitting, and fused execution provides a coherent solution to the problem.

2. Careful experimental analysis and implementation.

The paper provides fairly extensive ablations on pattern construction, head selection, replacement rates, and replacement timing. I particularly appreciate the comparisons under matched training-token and wall-clock budgets, as well as the pruning controls. The custom fused kernel and its numerical validation are also useful contributions, showing that the proposed design can translate into measurable runtime improvements rather than only theoretical computational savings.

Critical weaknesses

1. The paper lacks a comparison with static sparse attention that retains input-dependent attention weights.

My main concern is that SAF directly freezes the attention weights, which seems like a rather strong assumption. In conventional static sparse attention, the sparsity pattern can be predetermined, but the attention weights over the retained positions are still computed dynamically from the queries and keys. For example, sliding-window attention or combinations of local and regularly spaced attention positions can reduce computation while preserving some content-dependent flexibility.

I understand why SAF takes a more aggressive approach: by fixing the weights themselves, it can eliminate query-key computation and softmax for the selected heads, potentially offering greater computational savings. However, this also removes their ability to adjust token selection based on the input, which could come at an accuracy cost.

I think a comparison with this intermediate design is important. Such a baseline would help establish whether completely removing input-dependent attention is actually worthwhile, or whether retaining dynamic attention over a sparse set of positions offers a better accuracy–efficiency trade-off. The current pruning comparisons are useful, but they do not fully answer this question.

This is particularly relevant because the reported end-to-end training gains remain relatively modest, and the matched-time pruning baselines achieve better perplexity in some settings (Table 1 and Table 15).

2. The practical benefits and generalization of fixed attention patterns are not yet fully established.

Another concern is how the fixed attention patterns transfer across different sequence lengths and inference workloads. The proposed representation explicitly uses absolute key positions and relative distances, with parameters fitted for a particular context length. Although the paper demonstrates prefill speedups for different input lengths, it is still unclear how well the resulting patterns preserve model quality when the evaluation context goes beyond the calibration length.

More importantly, the experiments do not cover autoregressive, token-by-token decoding with KV caching. I understand that efficient training is the primary objective of this paper, so I would not necessarily require a complete decoding system. Still, given the relevance of these fixed patterns to inference, a clearer discussion or evaluation of this limitation would make the contribution more convincing.

I also think the evaluation on pretrained LLMs could be stronger. The Qwen3-4B experiments are useful, but the quality and prefill benchmarks in Appendix C.6 use different head-selection and calibration settings. As a result, it is difficult to directly assess the accuracy–speed trade-off of a single configuration. Broader evaluation on pretrained models, ideally with matched quality and efficiency measurements, would help establish how general the findings are.

Questions for authors

1. Comparison with static sparse attention.

Have the authors considered comparing SAF against a static sparse attention baseline that retains dynamically computed attention weights over the selected positions? For example, a sliding-window pattern or a combination of local attention and periodically sampled positions could provide a useful comparison. I would be particularly interested in the accuracy–efficiency trade-off under matched training-token or wall-clock budgets. This would help clarify the benefit of completely fixing attention weights instead of only fixing the sparsity pattern.

2. Sequence-length generalization and autoregressive decoding.

Could the authors clarify how the fixed positional patterns are handled when the inference length differs from the calibration length, especially when it exceeds the length used for fitting? In particular, how are the absolute-position and relative-distance parameters extended, and how are the normalization terms handled? The paper reports prefill execution speed beyond the calibration length in Appendix C.6, but does not report the corresponding quality results.

I would also be interested in whether the authors have considered token-by-token decoding with KV caching. Does the fixed-pattern formulation introduce additional implementation constraints or limit the potential efficiency gains in this setting?

3. Evaluation on pretrained LLMs and the accuracy–efficiency trade-off.

Could the authors provide accuracy and prefill efficiency measurements using the same selected head sets and calibration protocol on Qwen3-4B? Appendix C.6 appears to use variance-selected heads for the main zero-shot accuracy evaluation, while the GQA prefill benchmark uses forward-KL-selected heads with a different calibration setup. Since the two selectors produce substantially different accuracy results, evaluating their speed and accuracy under the same configuration would be helpful.

If feasible, an additional evaluation on another pretrained LLM would further strengthen the generality of the conclusions.

4. Functional importance of the selected heads.

I have one further question about the interpretation of the selected heads. The paper argues that low-variance heads can retain useful token mixing after their attention weights are frozen. However, low input dependence does not necessarily imply that a head makes an important contribution to the model. Some heads might simply have limited functional importance and could potentially be pruned instead.

The pruning comparisons partly address this issue, but I would be interested in a more direct characterization of the contribution of the selected heads after continued training. In particular, under what conditions does retaining fixed-pattern value mixing provide a meaningful benefit over removing the heads entirely?

Related work such as MoH: Multi-Head Attention as Mixture-of-Head Attention and Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse provides another perspective on head utilization and redundancy. I wonder whether encouraging more balanced or input-dependent head utilization would change the prevalence of heads that can be replaced by fixed patterns. I see this as an interesting direction for future work.

6 comments
AuthorsOpenReview ↗

Response to Reviewer vSmx (1/2)

Thank you for the constructive review. Please see our per-point response below. The revised manuscript marks all changes in blue.

W1.1 The paper lacks a comparison with static sparse attention that retains input-dependent attention weights. … For example, sliding-window attention or combinations of local and regularly spaced attention positions can reduce computation while preserving some content-dependent flexibility.

W1.2 Such a baseline would help establish whether completely removing input-dependent attention is actually worthwhile, or whether retaining dynamic attention over a sparse set of positions offers a better accuracy–efficiency trade-off.

W1.3 This is particularly relevant because the reported end-to-end training gains remain relatively modest, and the matched-time pruning baselines achieve better perplexity in some settings (Table 1 and Table 15).

Q1 Have the authors considered comparing SAF against a static sparse attention baseline that retains dynamically computed attention weights over the selected positions? … I would be particularly interested in the accuracy–efficiency trade-off under matched training-token or wall-clock budgets.

Thanks for the suggestion! Sliding windows and SAF remove different things. Sliding window limits which positions a head can attend to but keeps content-dependent weights. Freezing attention weights keep every position but remove the dependence on content. We added experiments that restrict the selected heads to a sliding window of the most recent 256 or 1,024 tokens. Following your suggestion, we also tested a 256-token window combined with every 64th earlier token. Each run starts from the same checkpoint and uses the same data order as the corresponding SAF experiment. The table below shows the results at 4K as the average over three seeds of the 124M model. ΔPPL is the perplexity increase over ordinary attention, at the same number of training tokens (matched tokens) or the same training time (matched time). ρ\rho is the relative-distance vector of the frozen pattern (Eq. 3), which can be stored in FP32 or BF16.

On the selected 25% of headsΔPPL, matched tokensChange in total training timeΔPPL, matched time
SAF, ρ\rho in FP32, submitted+0.66%−2.1%+0.46%
SAF, ρ\rho in BF16+0.66%−3.0%+0.34%
Pruning+0.81%−5.2%+0.11%
Sliding window, 256+0.07%+0.6%+0.28%
Sliding window, 1,0240.00%+1.3%+0.44%
Local 256 + stride 64+0.07%+3.4%+0.89%
Sliding window, 1,024, on every head+0.34%−5.9%−0.57%

On these heads, the 256-token sliding window achieves lower perplexity than SAF, both at matched tokens and at matched time. At matched tokens the same holds at 8K, 16K and 1B.

A sliding window computes fewer attention scores, so it should also save time, and this saving grows with context length. At 4K with 25% of heads it did not shorten training (+0.6%), because these heads still compute queries and keys and run in a separate attention call from the ordinary heads of the same layer. With more sliding-window heads we can save more time, at −1.9% for 50% of heads and −5.9% with all the heads.

However, SAF is better in two respects. A forward pass over a full input (prefill) is faster with SAF than with the sliding window on the same heads up to 8K. At 4K it takes 280 ms with SAF, 310 ms with the sliding window. On associative recall (MQAR, the models of Figure 5), SAF reaches 54.4% accuracy, against 34.0% for the sliding window and 26.2% for ordinary attention.

We also want to note that SAF was not designed to compete with sparse attention, and the paper asks a different question. Frozen patterns had been shown to work in translation models and pretrained encoders, but for causal language-model pretraining it was not known which heads can do without input-dependent weights, which patterns works better, or when.

We have added these results to Section 5.1, and Appendix C. On W1.3, please see our reply to CNRA (W1 and W2).

AuthorsOpenReview ↗

Response to Reviewer vSmx (2/2)

W2.1 … it is still unclear how well the resulting patterns preserve model quality when the evaluation context goes beyond the calibration length.

W2.2 … the experiments do not cover autoregressive, token-by-token decoding with KV caching.

Q2.1 Could the authors clarify how the fixed positional patterns are handled when the inference length differs from the calibration length, especially when it exceeds the length used for fitting? In particular, how are the absolute-position and relative-distance parameters extended, and how are the normalization terms handled?

Q2.2 … whether the authors have considered token-by-token decoding with KV caching. Does the fixed-pattern formulation introduce additional implementation constraints or limit the potential efficiency gains in this setting?

Thank you for raising these two settings. The paper asks which heads can be given fixed patterns during pretraining and how much time this saves, and answers with controlled comparisons in which only the selected heads differ. For context length, we train and fit separate models at 4K, 8K and 16K, and the findings hold at each (Section 5.3). Beyond the training length, ordinary attention is not trained, so a change in quality could not be attributed to the fixed patterns and the comparison would no longer be controlled. We therefore consider this as a separate direction.

On Q2.1, the pattern for a shorter input is the top-left block of the fitted pattern, because Z(i)Z(i) in Eq. 3 depends only on positions up to ii. α\alpha and ρ\rho have no fitted values beyond the fitted length, so a model used at a longer length should be fitted at that length. We have added this to Section 3.2.

On Q2.2, token-by-token decoding with a KV cache is an inference optimisation that needs its own kernel, and our efficiency claims focus on training and the forward pass over a full input. At each step a frozen head needs no query and no cached keys, only its row of the frozen pattern and its cached values.

W2.3 … the quality and prefill benchmarks in Appendix C.6 use different head-selection and calibration settings. … Broader evaluation on pretrained models, ideally with matched quality and efficiency measurements, would help establish how general the findings are.

Q3.1 Could the authors provide accuracy and prefill efficiency measurements using the same selected head sets and calibration protocol on Qwen3-4B?

Q3.2 If feasible, an additional evaluation on another pretrained LLM would further strengthen the generality of the conclusions.

Thanks for noticing this. The difference was intended. The prefill benchmark was a timing test of the kernel, because its running time depends mainly on how many heads are frozen and on the input size, not on which heads are chosen or on the values in their patterns. It replaced the same number of random heads in every layer with uniform patterns, which the submission did not describe correctly. With exactly the heads and patterns of the accuracy evaluation (Table 29), prefill is as fast as reported or slightly faster, as we expected. The table gives the prefill speedup over ordinary attention at batch size 4 and 2,048 tokens.

Share of heads replaced10%20%30%
Variance-selected heads with fitted patterns1.030×1.053×1.079×
Random heads with uniform patterns1.011×1.028×1.068×

We have added these results to Appendix C.6.

Q4.1 … under what conditions does retaining fixed-pattern value mixing provide a meaningful benefit over removing the heads entirely?

Keeping the value mixing helps in two conditions. The first is quality at matched tokens with midpoint replacement, where SAF has the smaller perplexity increase at 124M and 1B (Tables 1 and 3). The second is associative recall, where SAF reaches about twice the MQAR accuracy of pruning when heads are replaced or pruned after the full run (Figure 5). Pruning is better at matched time with the same heads, where it completes more training steps, and pruning by gradient-based importance (gate-Taylor) has lower perplexity when applied after the full run (Table 18).

Q4.2 Related work such as MoH … and Attention Sink Forges Native MoE in Attention Layers … I wonder whether encouraging more balanced or input-dependent head utilization would change the prevalence of heads that can be replaced by frozen patterns.

Thank you for this suggestion and for the pointers to MoH and sink-aware training, which we have added to related work. Our view is that the two properties are distinct. These methods change how much each head is used for a given input, whereas SAF selects heads whose attention weights vary little across inputs. A head can therefore be used selectively and still keep a frozen pattern. Whether such training also changes how many heads have low variance is untested, and we see it as interesting future work.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the new sliding-window comparisons.

However, these results also reinforce my main concern: retaining dynamic attention weights generally preserves model quality much better, while the practical efficiency gains of SAF remain relatively modest.

In particular, the results for a 1,024-token window applied to all heads suggest a potentially more favorable accuracy–efficiency trade-off.

Overall, I find the empirical analysis interesting and valuable, but I remain unconvinced that the practical efficiency gains are sufficiently substantial. I feel the work is more compelling as an analysis of attention-head behavior than as a practical efficiency method. I will therefore maintain my original rating.

I hope my earlier suggestions have been helpful in improving the paper, and looking forward to seeing a more complete and refined version in the future.

Reviewer_BoSaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper discusses methods to reduce training cost by avoiding query-key computation and the softmax operation. The authors show that some attention heads learn similar patterns across inputs and thus can be fixed during training to reduce training time. The paper proposes Selective Attention Freezing (SAF), which selects heads with low attention-pattern variance and replaces their attention weights with fitted post-softmax means during training. The proposed method shows comparable performance with reduced number of attention heads.

Critical strengths

Strengths:

  1. The paper is easy to follow and has an interesting research question: if certain heads are input-independent, can we fix those during training to decrease training time?

  2. As input-independent attention heads still require $ O(T^2) $ storage, the authors propose a method to capture absolute position effects using $ O(T) $ storage.

  3. The proposed method shows comparable performance in terms of perplexity (0.5% increase) even at the 1B-scale model, with an improvement in perplexity as 25% of the attention heads are replaced.

Critical weaknesses
  1. The authors need to clearly differentiate and discuss whether the experiments are related to input-independent attention heads or heads whose broad attention patterns can be approximated without affecting downstream task performance.

  2. As input-independent attention heads still require $ O(T^2) $ storage, the authors propose a method to capture absolute position effects using $ O(T) $ storage. However, it is unclear how and when Eq. (4) is used to fit the fixed pattern obtained from Eq. (3). How is the checkpoint selected for this calibration? It also remains unclear how fixing attention heads at intermediate checkpoints affects training dynamics and the learned model. Although the authors report some stability in head selection across checkpoints, this does not establish whether the selected heads would remain input-independent throughout training or how replacement alters optimization.

  3. The paper uses attention variance as a head-selection criterion, but low variance does not necessarily imply that a head can be safely replaced. The functional consequences of fixed attention patterns for selected heads have not been discussed in the paper.

  4. The authors do not discuss why certain heads become input-independent or how the remaining heads compensate after replacement. This is important to understand the phenomenon the paper discusses and can lead to even better head selection criteria.

  5. The end-to-end training speedups are relatively small if we consider the calibration and fitting overhead. The paper does not establish whether these gains justify the additional complexity.

  6. The pretraining experiments consider a single dataset, FineWeb-Edu; thus, it is not clear how the results generalize to other standard language training datasets.

Questions for authors

Questions:

  1. When and how is the calibration step implemented in training to fit Eq. (4)? Suppose we select the heads before/after the current checkpoint; will that change the selected attention heads?

  2. The authors should clearly discuss why the speedups are relatively small when considering calibration and fitting overhead.

  3. Why is attention variance chosen as a head selection criterion? Does the attention variance of the chosen head change during training when no intervention is done?

  4. The authors should discuss the limitations of the work and clarify whether the experiments are related to input-independent attention heads or heads whose broad attention patterns can be approximated without affecting downstream task performance.

5 comments
AuthorsOpenReview ↗

Response to Reviewer BoSa (1/2)

Thank you for the detailed review and for finding the research question interesting. Your questions about the procedure showed us that Section 3 did not explain it clearly enough, and we run extra experiments for several of your other points. The changes in revised manuscript are marked in blue.

W1 The authors need to clearly differentiate and discuss whether the experiments are related to input-independent attention heads or heads whose broad attention patterns can be approximated without affecting downstream task performance.

Q4.2 … and clarify whether the experiments are related to input-independent attention heads or heads whose broad attention patterns can be approximated without affecting downstream task performance.

Thank you for pointing out this distinction. Our experiments focus on the second one, heads whose attention a frozen pattern approximates at a small cost in perplexity and downstream accuracy. No head is strictly input-independent, and all our claims are about this measured cost (Tables 1–3, Figure 5). We have made the introduction in Section 1 consistent with this.

W2.1 … it is unclear how and when Eq. (4) is used to fit the fixed pattern obtained from Eq. (3).

W2.2 How is the checkpoint selected for this calibration?

Q1.1 When and how is the calibration step implemented in training to fit Eq. (4)?

Q1.2 Suppose we select the heads before/after the current checkpoint; will that change the selected attention heads?

Calibration runs once at the midpoint and Algorithm 1 in Appendix A.2 introduces the full procedure. We have now also mentioned this in Section 3 to make it clear. We train with ordinary attention to the midpoint, measure attention on 32 calibration sequences and rank heads by the variance of their attention weights across inputs (Eq. 1). For each selected head we fit the absolute-position vector α\alpha and the relative-distance vector ρ\rho of Eq. 3 to its mean attention pattern by minimising Eq. 4, and then continue training with the same optimiser state.

W2.3 It also remains unclear how fixing attention heads at intermediate checkpoints affects training dynamics and the learned model.

W2.4 … this does not establish whether the selected heads would remain input-independent throughout training or how replacement alters optimization.

W4.2 … or how the remaining heads compensate after replacement.

Q3.2 Does the attention variance of the chosen head change during training when no intervention is done?

The selected heads become low-variance early in training and stay so without any intervention. We measured the attention variance of all 144 heads at four point during the ordinary training (Table 7).

Training progress0%25%50%, selection100%
Mean variance rank of the 36 selected heads, lowest firs97201821
Selected heads among the 36 with lowest variance0–330–313628–33

By quarter-training, 30–31 of the 36 heads are already among the 36 with the lowest variance, so selecting earlier or later would give largely the same heads. After replacement, the gap in training loss to ordinary attention stabilises within a few hundred training steps (Figure 8). Then the magnitude of the frozen heads’ outputs decreases with their norm falls from 0.38 to 0.21, while the attention variance of the other heads barely changes. We have added the variance measurements to Section 5.1 and Appendix B.2, and the measurements of the heads' outputs after replacement to Appendix B.3.

W3.1 … low variance does not necessarily imply that a head can be safely replaced.

Q3.1 Why is attention variance chosen as a head selection criterion?

We agree that low variance alone does not show that a head can be replaced, and we did not rely on that assumption. We chose variance because, after the selected heads were replaced and training continued, it gave lower final perplexity than residual cosine and forward KL (Table 6). These are the two other selection scores we compared. Residual cosine is based on a head's output, and forward KL is the KL divergence between a head's attention and its mean pattern. Residual cosine predicts the cost of replacing one head more closely (Table 5), but not the final result after replacing 36 heads. Variance also predicts the cost of freezing a set of heads before any further training. At the midpoint, mean patterns raise perplexity by 4.5–5.8% on the selected heads, by 11–41% on each of 33 random sets with the same number of heads per layer, and by 77–115% on the highest-variance heads in those layers (added to Appendix B.2). This is expected. Among all frozen patterns, the mean has the lowest squared error to a head's attention, and this error equals the head's variance (Appendix B.1).

AuthorsOpenReview ↗

Response to Reviewer BoSa (2/2)

W3.2 The functional consequences of fixed attention patterns for selected heads have not been discussed in the paper.

W4.1 The authors do not discuss why certain heads become input-independent …

The frozen heads are still necessary to the trained model, but their contribution depends little on the input. At the end of SAF training, setting the outputs of the frozen heads to zero raises perplexity by 20.2%, whereas replacing each output with its mean over inputs raises it by only 0.85% (Table 8 in the revised manuscript). In a frozen head, the attention weights are a frozen pattern and are no longer computed from queries and keys. The value vectors that these weights mix are still computed from the input.

On why these heads become low-variance. What we've found is that the selected heads lie mostly in early layers and spread their attention over many tokens (Section 5.1). Their low variance is acquired through training, since their mean variance rank is 97 of 144 at initialisation and 20 by quarter-training. It is also not a property of the layers alone, because heads that use trained position-only weights in the same layers from step 0 have a larger perplexity increase than SAF (our reply to CNRA, W3).

W5.1 The end-to-end training speedups are relatively small if we consider the calibration and fitting overhead.

W5.2 The paper does not establish whether these gains justify the additional complexity.

Q2 The authors should clearly discuss why the speedups are relatively small when considering calibration and fitting overhead.

The reported savings already include the calibration and fitting overhead. The training times in Table 1 cover the first half with ordinary attention, calibration and fitting. Calibration and fitting take 55 s at 124M. At 1B they take 1,464 s, against about 5,100 s saved by the faster training steps that follow (Tables 15 and 27). The total saving of 2.1-5.3% is smaller than the saving per training step because only the second half of training runs with frozen heads. Storing part of each frozen pattern in BF16 instead of FP32 speeds up training further (our reply to CNRA, W1). Training steps after replacement are then 1.07-1.08× faster at 25% and 1.19-1.22× faster at 50%, and total training time falls by 3.0% and 7.4%.

On W5.2, what the method adds to training is one calibration pass and one fit at the midpoint, with an attention kernel that runs frozen and ordinary heads together. The optimiser, the data order and the learning-rate schedule are unchanged. The same frozen heads then make a forward pass over a full input (causal prefill) 1.09-1.24× faster at 25-50% replacement without being fitted again (Section 5.3). Our key contribution lies in establishing which heads can be frozen, with which patterns, and from when, which was not known for causal language-model pretraining.

W6 The pretraining experiments consider a single dataset, FineWeb-Edu; thus, it is not clear how the results generalize to other standard language training datasets.

Thank you for this suggestion. Though we cannot add pretraining experiment for extra datasets, we have added calibration and evaluation on three other domains, which are Wikipedia (WikiText-103), mathematical web text (OpenWebMath) and Python code (CodeParrot). For the 124M models at 25% replacement, we selected heads on each domain and compared them with the heads selected on FineWeb-Edu. We then evaluated SAF models pretrained on FineWeb-Edu on each domain, with heads selected and patterns fitted on mixed data. Largely the same heads are selected on Wikipedia and mathematical text, and the perplexity increase is about 1% on these two domains and about 4% on code.

DomainShared headsΔPPL
FineWeb-Edu36+0.69%
Wikipedia31+1.04%
Mathematical web text30+0.75%
Python code21+4.03%

ΔPPL is the perplexity increase of SAF over ordinary attention, and shared heads are the ones that were also selected on FineWeb-Edu (36 in total). All the results are the average over three seeds. We have added these results to a new Appendix C.7.

Q4.1 The authors should discuss the limitations of the work …

We have added a Limitations section after the Conclusion. It covers the single pretraining corpus, the model scale, and some settings where pruning is better.

Reviewer_BoSaOpenReview ↗

Response to the Rebuttal

I thank the authors for their careful response to the review. The authors have responded to most of my concerns. The following are my comments based on the reviews and the authors' response:

  1. Could you please clarify the meaning of midpoint, " and with respect to what, as it is not clear from the response. If this with respect to training, how to select it?
Reviewer_CNRAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether attention heads whose patterns vary little across inputs can be replaced with fixed patterns during language-model pretraining, and whether this reduces training cost in practice. Through controlled comparisons at the 124M scale, the authors arrive at a recipe called Selective Attention Freezing (SAF). Midway through training, SAF measures attention on a small set of calibration sequences and ranks all heads by the across-input variance of their attention matrices. It then replaces the lowest-variance heads (25% by default) with their mean post-softmax attention pattern. Replaced heads keep their value and output projections, so their outputs still depend on the input, but they no longer compute query/key projections, attention scores, or softmax. To avoid quadratic storage, each fixed pattern is fitted as a log-linear combination of an absolute key-position vector and a relative-distance vector. A fused Triton kernel then executes ordinary and fixed heads in a single launch alongside FlashAttention-2. The paper compares alternative fixed patterns (sharp mean, Gaussian and Dirichlet samples, and structured random), head-selection scores, replacement rates, replacement timing, and gradual or loss-triggered schedules. It also compares SAF with head pruning and random head selection under matched-token and matched-time budgets. At 124M parameters and 4K context, 25% replacement increases held-out perplexity by 0.77% and makes post-replacement optimizer updates 1.056× faster. At 1B parameters and 8K context, the perplexity increase is 0.51% with a 1.068× speedup on four GPUs. The authors also report small accuracy changes after finetuning on SST-2, BoolQ, and QuALITY, faster long-input finetuning and causal prefill, and better generalization to more key-value pairs on MQAR compared with ordinary attention and two pruning baselines. Finally, they report zero-shot results for replacing heads in a pretrained Qwen3-4B model without further training.

Critical strengths
  1. This paper presents a careful, detailed controlled study. Pattern type, selection score, replacement rate, timing, schedule, and frozen versus trainable patterns are each compared from shared checkpoints with matched data order. In addition, the main settings use three seeds to reduce the randomness.
  2. The authors provide a solid systems implementation. The fused kernel runs ordinary and fixed heads in one launch with FlashAttention-2. It is validated against an eager reference in FP32 and BF16.
  3. The experimental results cover finetuning, causal prefill, associative recall, and zero-shot replacement in Qwen3-4B.
Critical weaknesses
  1. Post-replacement throughput improvements remain minimal, yielding only a 1.056x speedup at 124M and 1.068x at 1B. The authors estimate about 3% savings over the full run. The speedup also shrinks with longer context (1.025x at 16K), even though attention takes a larger share of compute there, and the paper does not explain this. Short-input fine-tuning (SST-2, 0.90x) and batch-one prefill become slower.
  2. Under matched time, same-head pruning has a lower perplexity penalty than SAF (0.11% vs. 0.46%). After later intervention, gate-Taylor pruning beats SAF under both token and time budgets. Pruning also gives larger training and prefill speedups (Tables 19-20), while SAF increases prefill peak memory by 3.75%.
  3. Missing baseline that trains position-only heads from scratch. α\alpha and ρ\rho are already trainable relative/absolute biases. A natural comparison is to make some heads position-only from step 0. This would test whether the midpoint calibration is necessary, and it would give savings over the whole run instead of only the second half.
  4. Several closely related works are not discussed: LazyFormer [1], which reuses attention distributions across layers to cut pretraining cost; the Toeplitz Neural Network [2], which mixes tokens with relative-position weights similar to ρ(i−j)\rho(i-j); and Reservoir Transformers [3], which freeze layers during training to reduce wall-clock time. A discussion/comparison of these works would better position the contribution.

[1] LazyFormer: Self Attention with Lazy Update, arXiv 2021 [2] Toeplitz Neural Network for Sequence Modeling, ICLR 2023 [3] Reservoir Transformers, ACL 2021

Questions for authors

Check the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CNRA (1/2)

Thank you for the careful review and for the positive assessment of the controlled design and the kernel. Your question about why the speedup falls at 16K led us to a further optimisation of the kernel, with which the speedup no longer falls at long contexts. The changes in revised manuscript are marked in blue.

W1.1 Post-replacement throughput improvements remain minimal, yielding only a 1.056x speedup at 124M and 1.068x at 1B. The authors estimate about 3% savings over the full run.

W1.2 The speedup also shrinks with longer context (1.025x at 16K), even though attention takes a larger share of compute there, and the paper does not explain this.

W1.3 Short-input fine-tuning (SST-2, 0.90x) and batch-one prefill become slower.

Thanks for pointing this out! We carefully investigate this. We measured how the time of one training step is divided among its operations. Most of the saving in the fronzen heads came from skipping the computation of query and key vectors. This computation comes before the attention operation, and its cost per token does not grow with context length. The attention operation itself took only slightly less time for frozen heads than for ordinary heads, and its cost per token grows with context length, from 28% of the GPU kernel time of a training step at 4K to 57% at 16K. The part that became faster was therefore a smaller share at longer contexts. Having said that, after our investigation, we found that the attention operation of the frozen heads can be made faster as well. Storing the relative-distance vector ρ\rho of each frozen pattern (Eq. 3) in BF16 instead of FP32 halves the attention time of a layer whose heads are all frozen with SAF.

Speedup of a training step after replacement4K8K16K
25% of heads, FP321.056×1.030×1.025×
25% of heads, BF161.072×1.074×1.076×
50% of heads, FP321.119×1.101×1.070×
50% of heads, BF161.187×1.198×1.216×

The speedup no longer falls with context length with BF16. The perplexity increase at matched tokens (the same number of training tokens) is the same with ρ\rho in either precision, and peak memory stays below that of ordinary attention. We have added an explanation of the trend to Section 5.3, and the time breakdown and these measurements to a new Appendix D.4.

On W1.1, the saving over a full run is limited by when the heads are frozen, not by how fast they run. With ρ\rho in BF16, every training step after replacement is 1.07–1.08× faster at 25% and 1.19–1.22× faster at 50%, but only the second half of training uses frozen heads, so total training time falls by 3.0% and 7.4%. Freezing earlier extends the saving to more of the run, and the perplexity increase rises only from 0.66% to 0.77% when replacement moves from the midpoint to quarter-training (Section 5.2). At inference the frozen heads are used throughout. A forward pass over a full input (causal prefill) is 1.09–1.12× faster at 25% and 1.20–1.24× faster at 50% replacement (Section 5.3).

We would also like to explain how we see the contribution, because the paper is not mainly a proposal for faster training. Earlier work showed that frozen patterns can replace attention in translation models and pretrained encoders. For causal language-model pretraining, it was not known which heads and which patterns to choose, when to replace them, or whether a layer that mixes frozen and ordinary heads is faster than an optimised attention implementation such as FlashAttention. The paper answers these questions with carefully designed controlled comparisons.

On W1.3, short inputs leave little attention computation to save. SST-2 inputs have about 25 tokens, so the frozen overhead of launching each GPU kernel dominates the run time. CUDA Graphs can remove most of this overhead, and SAF is then 1.113× faster at 25% replacement (Appendix D.3).

AuthorsOpenReview ↗

Response to Reviewer CNRA (2/2)

W2.1 Under matched time, same-head pruning has a lower perplexity penalty than SAF (0.11% vs. 0.46%).

W2.2 After later intervention, gate-Taylor pruning beats SAF under both token and time budgets.

W2.3 Pruning also gives larger training and prefill speedups (Tables 19-20), while SAF increases prefill peak memory by 3.75%.

We agree with all three observations. Pruning removes all of a head's computation, so if the only goal is a faster training run at 124M, pruning these heads is the better choice. In the paper, pruning is a control for a different question, which is whether a head with frozen attention weights still contributes something that is lost when the head is removed. With midpoint replacement, keeping the frozen heads provides lower perplexity than pruning them at the same number of training tokens, and after the later intervention (heads replaced or pruned after the full ordinary-attention run, followed by further training) it gives about twice the accuracy on the MQAR associative-recall task.

The matched-token comparisons are in Table 1 for 124M and Table 3 for 1B, and the MQAR comparison is in Figure 5 and Table 20. On W2.1, the gap at matched time becomes smaller when ρ\rho is stored in BF16. The perplexity increase of SAF then falls from 0.46% to 0.34%, against 0.11% for pruning (new, Table 23). On W2.2, the MQAR result comes from the same later-intervention models. There, both pruned models stay at the level of ordinary attention while SAF reaches about twice their accuracy, so the frozen heads contribute something that pruning does not.

The 3.75% increase in prefill peak memory is a constant 143 MB, most of which stores ρ\rho. It does not grow with the batch, so its share of peak memory falls from 13% at batch size 1 to 3.75% at batch size 32.

W3 A natural comparison is to make some heads position-only from step 0. This would test whether the midpoint calibration is necessary, and it would give savings over the whole run instead of only the second half.

Thank you for suggesting this control experiment. We conducted an extra experiment on this. Freezing heads from initialisation gives a larger average perplexity increase than midpoint replacement in every variant we tested. We made 36 heads position-only from initialisation, without query or key projections and with trainable α\alpha and ρ\rho, the absolute-position and relative-distance vectors of the frozen pattern (Eq. 3). The heads were either three per layer (uniform layout) or placed with the per-layer counts that SAF selects (SAF's layer layout). In the table, ΔPPL is the perplexity increase over ordinary attention and total time is the change in total training time.

SettingΔPPLTotal time
SAF+0.66%−2.1%
Position-only from step 0, uniform layout+0.98%+9.1%
Position-only from step 0, SAF's layer layout+1.98%+6.3%
Position-only from step 0, SAF's layer layout, α\alpha and ρ\rho frozen at the midpoint+1.73%+0.3%

Both variants with SAF's layer layout are worse than SAF (3 random seeds). Which heads can be frozen therefore depends on what the model has learned by the midpoint, not only on the layers they are in, and this is why the midpoint calibration is needed. None of the variants saved training time, because in our implementation a training step that also trains α\alpha and ρ\rho is slower than one with ordinary attention.

We trained α\alpha and ρ\rho because an untrained pattern at step 0 is only a uniform average over earlier tokens. Such a pattern would save time over the whole run, as you mentioned. The trainable version can learn any pattern of this form and is already worse than SAF, so we did not run the untrained one. We have added these results to Table 16c and Section 5.2.

W4 Several closely related works are not discussed: LazyFormer [1] … the Toeplitz Neural Network [2] … and Reservoir Transformers [3] … A discussion/comparison of these works would better position the contribution.

Thank you for these references, which we have added and discussed in Section 2. What SAF does differently from all three is to select individual heads during training and freeze only their attention weights.

Thank you for the detailed response and the additional experiments. Most of my concerns have been addressed, in particular the explanation of the context-length trend with the BF16 fix, the position-only baseline trained from step 0, and the added related work. I would be inclined to raise my score by about 0.5, but since this year's scale only has four integer levels, I will keep my positive score of 3.

Thank you for your positive feedback and for taking the time to review our response and additional experiments! We really appreciate your thoughtful comments and suggestions, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.