- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether some attention heads can be replaced with fixed, input-independent attention patterns to reduce the computational cost of language model training. The authors propose Selective Attention Freezing (SAF), which identifies heads with low attention-pattern variance and replaces their attention weights with fitted post-softmax means halfway through training. These fixed patterns are represented using absolute-position and relative-distance preferences, reducing their storage requirements from quadratic to linear in sequence length. A fused kernel is developed to execute ordinary and fixed-pattern attention heads together, avoiding query-key computation and softmax for the replaced heads.
The authors evaluate SAF through controlled pretraining experiments, downstream finetuning, causal prefill benchmarks, and associative recall tasks. At a 25% replacement rate, SAF achieves 1.056× faster post-replacement training updates on a 124M model at 4K context, with a 0.77% perplexity increase. On a 1B model at 8K context, it achieves a 1.068× speedup with a 0.51% perplexity increase. The paper also reports improved generalization to more key-value associations at a fixed sequence length compared with ordinary attention and pruning baselines.
Critical strengths
1. An interesting and well-motivated approach to reducing attention computation.
I like the idea of identifying attention heads whose attention weights are relatively stable across inputs and replacing them with fixed patterns. The approach is quite aggressive, but it is also interesting because it can eliminate query-key computation and softmax for the selected heads, rather than merely reducing the number of attended positions. The combination of variance-based head selection, compact pattern fitting, and fused execution provides a coherent solution to the problem.
2. Careful experimental analysis and implementation.
The paper provides fairly extensive ablations on pattern construction, head selection, replacement rates, and replacement timing. I particularly appreciate the comparisons under matched training-token and wall-clock budgets, as well as the pruning controls. The custom fused kernel and its numerical validation are also useful contributions, showing that the proposed design can translate into measurable runtime improvements rather than only theoretical computational savings.
Critical weaknesses
1. The paper lacks a comparison with static sparse attention that retains input-dependent attention weights.
My main concern is that SAF directly freezes the attention weights, which seems like a rather strong assumption. In conventional static sparse attention, the sparsity pattern can be predetermined, but the attention weights over the retained positions are still computed dynamically from the queries and keys. For example, sliding-window attention or combinations of local and regularly spaced attention positions can reduce computation while preserving some content-dependent flexibility.
I understand why SAF takes a more aggressive approach: by fixing the weights themselves, it can eliminate query-key computation and softmax for the selected heads, potentially offering greater computational savings. However, this also removes their ability to adjust token selection based on the input, which could come at an accuracy cost.
I think a comparison with this intermediate design is important. Such a baseline would help establish whether completely removing input-dependent attention is actually worthwhile, or whether retaining dynamic attention over a sparse set of positions offers a better accuracy–efficiency trade-off. The current pruning comparisons are useful, but they do not fully answer this question.
This is particularly relevant because the reported end-to-end training gains remain relatively modest, and the matched-time pruning baselines achieve better perplexity in some settings (Table 1 and Table 15).
2. The practical benefits and generalization of fixed attention patterns are not yet fully established.
Another concern is how the fixed attention patterns transfer across different sequence lengths and inference workloads. The proposed representation explicitly uses absolute key positions and relative distances, with parameters fitted for a particular context length. Although the paper demonstrates prefill speedups for different input lengths, it is still unclear how well the resulting patterns preserve model quality when the evaluation context goes beyond the calibration length.
More importantly, the experiments do not cover autoregressive, token-by-token decoding with KV caching. I understand that efficient training is the primary objective of this paper, so I would not necessarily require a complete decoding system. Still, given the relevance of these fixed patterns to inference, a clearer discussion or evaluation of this limitation would make the contribution more convincing.
I also think the evaluation on pretrained LLMs could be stronger. The Qwen3-4B experiments are useful, but the quality and prefill benchmarks in Appendix C.6 use different head-selection and calibration settings. As a result, it is difficult to directly assess the accuracy–speed trade-off of a single configuration. Broader evaluation on pretrained models, ideally with matched quality and efficiency measurements, would help establish how general the findings are.
Questions for authors
1. Comparison with static sparse attention.
Have the authors considered comparing SAF against a static sparse attention baseline that retains dynamically computed attention weights over the selected positions? For example, a sliding-window pattern or a combination of local attention and periodically sampled positions could provide a useful comparison. I would be particularly interested in the accuracy–efficiency trade-off under matched training-token or wall-clock budgets. This would help clarify the benefit of completely fixing attention weights instead of only fixing the sparsity pattern.
2. Sequence-length generalization and autoregressive decoding.
Could the authors clarify how the fixed positional patterns are handled when the inference length differs from the calibration length, especially when it exceeds the length used for fitting? In particular, how are the absolute-position and relative-distance parameters extended, and how are the normalization terms handled? The paper reports prefill execution speed beyond the calibration length in Appendix C.6, but does not report the corresponding quality results.
I would also be interested in whether the authors have considered token-by-token decoding with KV caching. Does the fixed-pattern formulation introduce additional implementation constraints or limit the potential efficiency gains in this setting?
3. Evaluation on pretrained LLMs and the accuracy–efficiency trade-off.
Could the authors provide accuracy and prefill efficiency measurements using the same selected head sets and calibration protocol on Qwen3-4B? Appendix C.6 appears to use variance-selected heads for the main zero-shot accuracy evaluation, while the GQA prefill benchmark uses forward-KL-selected heads with a different calibration setup. Since the two selectors produce substantially different accuracy results, evaluating their speed and accuracy under the same configuration would be helpful.
If feasible, an additional evaluation on another pretrained LLM would further strengthen the generality of the conclusions.
4. Functional importance of the selected heads.
I have one further question about the interpretation of the selected heads. The paper argues that low-variance heads can retain useful token mixing after their attention weights are frozen. However, low input dependence does not necessarily imply that a head makes an important contribution to the model. Some heads might simply have limited functional importance and could potentially be pruned instead.
The pruning comparisons partly address this issue, but I would be interested in a more direct characterization of the contribution of the selected heads after continued training. In particular, under what conditions does retaining fixed-pattern value mixing provide a meaningful benefit over removing the heads entirely?
Related work such as MoH: Multi-Head Attention as Mixture-of-Head Attention and Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse provides another perspective on head utilization and redundancy. I wonder whether encouraging more balanced or input-dependent head utilization would change the prevalence of heads that can be replaced by fixed patterns. I see this as an interesting direction for future work.
6 comments
Response to Reviewer vSmx (2/2)
W2.1 … it is still unclear how well the resulting patterns preserve model quality when the evaluation context goes beyond the calibration length.
W2.2 … the experiments do not cover autoregressive, token-by-token decoding with KV caching.
Q2.1 Could the authors clarify how the fixed positional patterns are handled when the inference length differs from the calibration length, especially when it exceeds the length used for fitting? In particular, how are the absolute-position and relative-distance parameters extended, and how are the normalization terms handled?
Q2.2 … whether the authors have considered token-by-token decoding with KV caching. Does the fixed-pattern formulation introduce additional implementation constraints or limit the potential efficiency gains in this setting?
Thank you for raising these two settings. The paper asks which heads can be given fixed patterns during pretraining and how much time this saves, and answers with controlled comparisons in which only the selected heads differ. For context length, we train and fit separate models at 4K, 8K and 16K, and the findings hold at each (Section 5.3). Beyond the training length, ordinary attention is not trained, so a change in quality could not be attributed to the fixed patterns and the comparison would no longer be controlled. We therefore consider this as a separate direction.
On Q2.1, the pattern for a shorter input is the top-left block of the fitted pattern, because in Eq. 3 depends only on positions up to . and have no fitted values beyond the fitted length, so a model used at a longer length should be fitted at that length. We have added this to Section 3.2.
On Q2.2, token-by-token decoding with a KV cache is an inference optimisation that needs its own kernel, and our efficiency claims focus on training and the forward pass over a full input. At each step a frozen head needs no query and no cached keys, only its row of the frozen pattern and its cached values.
W2.3 … the quality and prefill benchmarks in Appendix C.6 use different head-selection and calibration settings. … Broader evaluation on pretrained models, ideally with matched quality and efficiency measurements, would help establish how general the findings are.
Q3.1 Could the authors provide accuracy and prefill efficiency measurements using the same selected head sets and calibration protocol on Qwen3-4B?
Q3.2 If feasible, an additional evaluation on another pretrained LLM would further strengthen the generality of the conclusions.
Thanks for noticing this. The difference was intended. The prefill benchmark was a timing test of the kernel, because its running time depends mainly on how many heads are frozen and on the input size, not on which heads are chosen or on the values in their patterns. It replaced the same number of random heads in every layer with uniform patterns, which the submission did not describe correctly. With exactly the heads and patterns of the accuracy evaluation (Table 29), prefill is as fast as reported or slightly faster, as we expected. The table gives the prefill speedup over ordinary attention at batch size 4 and 2,048 tokens.
| Share of heads replaced | 10% | 20% | 30% |
|---|---|---|---|
| Variance-selected heads with fitted patterns | 1.030× | 1.053× | 1.079× |
| Random heads with uniform patterns | 1.011× | 1.028× | 1.068× |
We have added these results to Appendix C.6.
Q4.1 … under what conditions does retaining fixed-pattern value mixing provide a meaningful benefit over removing the heads entirely?
Keeping the value mixing helps in two conditions. The first is quality at matched tokens with midpoint replacement, where SAF has the smaller perplexity increase at 124M and 1B (Tables 1 and 3). The second is associative recall, where SAF reaches about twice the MQAR accuracy of pruning when heads are replaced or pruned after the full run (Figure 5). Pruning is better at matched time with the same heads, where it completes more training steps, and pruning by gradient-based importance (gate-Taylor) has lower perplexity when applied after the full run (Table 18).
Q4.2 Related work such as MoH … and Attention Sink Forges Native MoE in Attention Layers … I wonder whether encouraging more balanced or input-dependent head utilization would change the prevalence of heads that can be replaced by frozen patterns.
Thank you for this suggestion and for the pointers to MoH and sink-aware training, which we have added to related work. Our view is that the two properties are distinct. These methods change how much each head is used for a given input, whereas SAF selects heads whose attention weights vary little across inputs. A head can therefore be used selectively and still keep a frozen pattern. Whether such training also changes how many heads have low variance is untested, and we see it as interesting future work.
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the new sliding-window comparisons.
However, these results also reinforce my main concern: retaining dynamic attention weights generally preserves model quality much better, while the practical efficiency gains of SAF remain relatively modest.
In particular, the results for a 1,024-token window applied to all heads suggest a potentially more favorable accuracy–efficiency trade-off.
Overall, I find the empirical analysis interesting and valuable, but I remain unconvinced that the practical efficiency gains are sufficiently substantial. I feel the work is more compelling as an analysis of attention-head behavior than as a practical efficiency method. I will therefore maintain my original rating.
I hope my earlier suggestions have been helpful in improving the paper, and looking forward to seeing a more complete and refined version in the future.
Response to Reviewer vSmx (1/2)
Thank you for the constructive review. Please see our per-point response below. The revised manuscript marks all changes in blue.
Thanks for the suggestion! Sliding windows and SAF remove different things. Sliding window limits which positions a head can attend to but keeps content-dependent weights. Freezing attention weights keep every position but remove the dependence on content. We added experiments that restrict the selected heads to a sliding window of the most recent 256 or 1,024 tokens. Following your suggestion, we also tested a 256-token window combined with every 64th earlier token. Each run starts from the same checkpoint and uses the same data order as the corresponding SAF experiment. The table below shows the results at 4K as the average over three seeds of the 124M model. ΔPPL is the perplexity increase over ordinary attention, at the same number of training tokens (matched tokens) or the same training time (matched time). ρ is the relative-distance vector of the frozen pattern (Eq. 3), which can be stored in FP32 or BF16.
On these heads, the 256-token sliding window achieves lower perplexity than SAF, both at matched tokens and at matched time. At matched tokens the same holds at 8K, 16K and 1B.
A sliding window computes fewer attention scores, so it should also save time, and this saving grows with context length. At 4K with 25% of heads it did not shorten training (+0.6%), because these heads still compute queries and keys and run in a separate attention call from the ordinary heads of the same layer. With more sliding-window heads we can save more time, at −1.9% for 50% of heads and −5.9% with all the heads.
However, SAF is better in two respects. A forward pass over a full input (prefill) is faster with SAF than with the sliding window on the same heads up to 8K. At 4K it takes 280 ms with SAF, 310 ms with the sliding window. On associative recall (MQAR, the models of Figure 5), SAF reaches 54.4% accuracy, against 34.0% for the sliding window and 26.2% for ordinary attention.
We also want to note that SAF was not designed to compete with sparse attention, and the paper asks a different question. Frozen patterns had been shown to work in translation models and pretrained encoders, but for causal language-model pretraining it was not known which heads can do without input-dependent weights, which patterns works better, or when.
We have added these results to Section 5.1, and Appendix C. On W1.3, please see our reply to CNRA (W1 and W2).