acceptodds
Under review as a conference paper at ICLR 2027

Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels

Abstract

Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging. We identify an important source of approximation error overlooked by standard linearization methods: a small number of outlier dimensions contribute disproportionately to query–key dot products and account for a large fraction of the approximation error. Motivated by this heterogeneity, we propose Outlier-Aware Attention Linearization (OAL), a dimension-aware method for constructing quadratic attention approximations. Starting from a second-order Taylor approximation, we introduce a Hadamard-product formulation that makes individual query–key contributions explicit. This formulation relaxes globally shared coefficients to support dimension-specific linear terms and cross-dimensional quadratic interactions while preserving separable evaluation. To retain this flexibility with fewer parameters, OAL groups dimensions with similar contribution scales to share coefficients, while allowing distinct treatment of the few outliers. The resulting positive quadratic kernel is calibrated against pretrained attention outputs and admits linear-time evaluation with respect to sequence length for fixed head dimensions. On Qwen2.5-0.5B, OAL reduces frozen-attention output relative MSE by approximately 8.9% compared with DiJiang. Replacing middle-layer attention modules and jointly fine-tuning the kernel coefficients with LoRA further lowers WikiText-103 perplexity from 31.71 to 27.50 compared with a learned global quadratic, while improving accuracy on PIQA, HellaSwag, and ARC-E. These results demonstrate the value of outlier-aware dimension grouping for both attention approximation and pretrained-model conversion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention linearization, Dimensional outliers, Adaptive dimension grouping, Quadratic kernels, Pretrained Transformer conversion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5TBzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Outlier-Aware Attention Linearization (OAL) for linearizing pretrained softmax attention. The authors first analyze dimension-wise contributions of query-key interactions in pretrained attention and find that attention logits are highly concentrated in a small subset of dimensions. Approximating only these high-contribution dimensions (≈5.5% of head dimensions) retains a substantial fraction of full-head attention-output MSE (37.8%–64.8% retained-error ratio, consistently above Gaussian controls, p < 0.001 against 1,000 cardinality-matched random-coordinate controls). Motivated by this, the authors introduce a dimension-aware quadratic kernel with per-dimension linear coefficients b_r and per-dimension-pair quadratic coefficients C_rt, evaluated exactly in O(N) via a Hadamard-diagonal identity. To control parameter count, dimensions are sorted by contribution score and partitioned into 2–5 groups via bootstrap stability of adjacent contribution gaps, with coefficients shared within groups and group pairs; a lower-triangular factor L (M = LL^T) enforces positive kernel weights. Coefficients are calibrated by output matching on frozen Q/K/V and then jointly optimized with LoRA while groups stay fixed. On Qwen2.5-0.5B and Llama-3.2-1B, OAL achieves lower frozen-attention MSE/RelMSE than Performer, SpectraFormer, DiJiang, and TaylorShift, and better WikiText-103 PPL and downstream accuracy than those converted methods and a learned global quadratic (G=1) baseline, though it remains behind the LoRA-adapted softmax reference.

Critical strengths
  1. The diagnostic analysis is convincing and well controlled. The authors show not only that dimension-wise contributions are concentrated, but also that this concentration cannot be explained by dimensionality or coordinate selection, using two independent controls (isotropic Gaussian replacement and 1,000 cardinality-matched random-coordinate subsets), and further verify stability across held-out inputs (Spearman ρ = 0.991) and sequence lengths. This diagnostic is independently useful to the linear-attention community.

  2. The method is engineering-sound and deployable. The Hadamard-diagonal identity enables exact recurrent evaluation of the quadratic kernel without materializing an N×N matrix, and the M = LL^T parameterization guarantees a positive normalization denominator. Grouping reduces learned parameters from O(d_h²) to O(G²), and at G=5 the runtime and peak memory are comparable to the G=1 baseline (Figure 3), indicating the added flexibility incurs little operator overhead.

Critical weaknesses
  1. The central "outlier" framing is not substantiated, and the link between χ_r and outlier dimensions is never established. The paper is titled and motivated around "Outlier-Aware," yet χ_r is a custom causal average of absolute coordinate products, and the paper does not connect it to the stricter statistical criteria used in the outlier-dimensions literature (e.g., top-1% absolute activation, extreme median across at least 50% of inputs, average z-score ≥ 8 standard deviations). More importantly, prior work shows that "large logit contribution" and "functionally important" are distinct: in some models, outlier dimensions have near-zero logit contribution when predicting their preferred tokens and can be ablated with almost no accuracy change, suggesting they may be byproducts of the softmax normalization constraint rather than information carriers. OAL only measures contribution magnitude and performs no functional validation (e.g., whether ablating high-contribution dimensions actually degrades attention output or downstream accuracy). Whether the "outlier-aware" label is warranted is therefore not established by the paper itself. This is a framing-level issue rather than a presentational one.

  2. A key ablation is missing, so the source of the gain—grouping versus learnable coefficients per se—cannot be isolated. The paper compares G=1 and G=5, which is an ablation on the number of groups, not on "learnable vs. fixed" coefficients. TaylorShift uses fixed Taylor coefficients, Hedgehog learns MLP feature maps, and DiJiang's treatment of kernel parameters during conversion is not clearly described. OAL does not answer: how much performance is lost if coefficients are fixed to some estimate and excluded from backpropagation? What happens to performance and cost if dimension-specific parameters are fully unshared (G = d_h) and trained? Without these two controls, the effectiveness of the paper's central contribution—contribution-aware grouping—cannot be isolated.

  3. Cross-dataset and cross-model stability of group boundaries is not validated. Prior work shows that the location and function of outlier dimensions vary substantially across models (in some models they relate to high-frequency token prediction; in others they have almost no functional effect). OAL computes χ_r only on a WikiText-103 calibration set, and group boundaries are determined only on that set via bootstrap (Appendix B). There is no cross-dataset (e.g., code, multilingual, dialogue) or cross-distribution comparison to verify that group boundaries are stable. If group boundaries depend on the calibration distribution, the transferability of the method is questionable.

Questions for authors
  1. What is the relationship between χ_r and established statistical definitions of outlier dimensions (e.g., absolute activation ranking, z-score thresholds)? Is there any experiment showing the overlap between dimensions selected by χ_r and those selected by standard OD criteria?

  2. Is there any functional validation—e.g., does ablating high-contribution dimensions significantly degrade attention-output error or downstream accuracy? If ablating these dimensions leaves model behavior largely unchanged, how does that affect the appropriateness of the "outlier-aware" framing?

  3. Please add a control on whether coefficients participate in backpropagation: how much do PPL and downstream tasks differ between fixing the calibrated coefficients (excluding them from joint LoRA optimization) and the current joint optimization? What are the performance and memory/time costs when dimension-specific parameters are fully unshared (G = d_h, no grouping)?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5TBz: R3.1–R3.2

Thank you for the detailed review. We address each concern below.

R3.1 — Contribution scores and statistical definitions of outlier dimensions

Concern (paraphrased): How does χ_r relate to established statistical definitions of outlier dimensions, and is there evidence about the overlap between the dimensions selected by these criteria?

Relationship between activation outliers and QK contributions

Conventional outlier-dimension criteria identify dimensions exhibiting unusually large activation magnitudes, whereas our contribution score χr\chi_r quantifies each dimension's contribution to QK attention logits. Since OAL focuses on attention approximation, we investigate this relationship directly in pretrained Q/K representations, using activation magnitude and within-head z-score as two outlier-selection criteria.

Our results demonstrate a strong empirical correspondence between activation outliers and high-contribution QK dimensions, supporting the outlier-aware motivation of OAL. Across all 336 attention heads of Qwen2.5-0.5B, the Top-5 dimensions selected by average Q/K activation magnitude exhibit 83.0% overlap with those selected by χr\chi_r, corresponding to 4.15 shared dimensions per head on average. Furthermore, among dimensions identified as activation outliers using a within-head z-score threshold of z≥2z\geq 2, 90.4% also belong to the corresponding χr\chi_r, Top-5.

These findings establish substantial agreement between activation-based outlier criteria and contribution-based identification under the evaluated settings. Together with the approximation-error concentration analysis in Section 3.2 and the functional ablations in R3.2, they support our central motivation: identifying and treating high-contribution QK dimensions differently can improve attention approximation.

R3.2 — Functional validation and the “outlier-aware” framing

Concern (paraphrased): Does ablating high-contribution dimensions affect attention outputs or downstream accuracy, and what does the available functional evidence imply for the “outlier-aware” framing?

We added functional ablations in the original softmax model, comparing the removal of Top-k contribution dimensions with randomly selected dimensions. The number of removed dimensions and all other attention computations are kept the same.

Attention-output effects

Across both Qwen2.5-0.5B and Llama-3.2-1B, masking high-contribution dimensions consistently produces substantially larger attention-output errors than masking randomly selected dimensions. As shown in Appendix G.3, Top-k masking yields 4.05×–21.91× larger MSE on Qwen and 5.30×–10.50× larger MSE on Llama, providing functional evidence for the importance of the identified dimensions.

modelkTop-k MSE ↓Random-k MSE ↓Ratio
Qwen2.5-0.5B10.056380.00257≈21.9×
Qwen2.5-0.5B30.063120.00963≈6.56×
Qwen2.5-0.5B50.061940.01528≈4.05×
Llama-3.2-1B10.014380.00137≈10.50×
Llama-3.2-1B30.022480.00285≈7.89×
Llama-3.2-1B50.023040.00435≈5.30×

Model-level effects

We further evaluate the functional importance of high-contribution dimensions by applying Top-5 and Random-5 masking to three sampled layers (11, 16, and 22) of the original Qwen2.5-0.5B softmax model separately. This ablation uses fewer evaluation samples than the model-conversion experiments, with an unmasked baseline PPL of 10.9329. Relative to this baseline, Top-5 masking increases PPL by 3.22 on average, compared with 0.24 for Random-5 masking. This substantially larger degradation demonstrates that high-contribution dimensions play an important role in preserving language-modeling performance, providing further functional evidence for the outlier-aware motivation of OAL.

AuthorsOpenReview ↗

Response to Reviewer 5TBz: R3.3–R3.4

R3.3 — Coefficient optimization and fully unshared parameters

Concern (paraphrased): How do PPL and downstream scores change when calibrated coefficients are kept fixed during LoRA versus jointly optimized? How does optimizing fully unshared dimension-specific parameters (G = d_h) affect quality, memory use, and runtime?

Fixed versus jointly optimized coefficients

On Qwen2.5-0.5B, jointly optimizing the calibrated kernel coefficients and LoRA yields modest improvements in all four evaluated metrics compared with keeping the coefficients fixed during LoRA adaptation.

AdaptationTest PPL ↓PIQA ↑HellaSwag ↑ARC-Easy ↑
Calibrated kernel coefficients fixed, LoRA only27.964363.76%35.41%49.41%
Joint optimization of LoRA and kernel coefficients27.502064.69%37.20%50.38%

Fully unshared parameters

The representative-head analysis on Qwen2.5-0.5B shows diminishing approximation gains over the tested range of G=1G=1 to 1616, motivating compact grouping (Figure 2(b)). An end-to-end comparison of quality, memory, and runtime at G=dhG=d_h has not been conducted.

DiJiang training protocol

In our DiJiang baseline, the feature weights are optimized jointly with LoRA, while the DCT projection remains fixed (Appendix F.1).

R3.4 — Group-boundary stability across calibration distributions and models

Concern (paraphrased): How stable are group boundaries across datasets, distributions, and models when they are determined using WikiText-103 calibration data?

Cross-dataset and cross-model grouping stability

Our additional experiments demonstrate substantial consistency in contribution rankings and group assignments across different calibration distributions on both Qwen2.5-0.5B and Llama-3.2-1B. Following the original grouping procedure, we independently recomputed contribution scores and group assignments on WikiText-103, C4, and CodeParrot, covering all 336 query heads of Qwen and 512 query heads of Llama.

Across the three dataset pairs, contribution rankings achieve Spearman correlations of 0.946–0.983 on Qwen and 0.940–0.981 on Llama, while group partitions achieve Adjusted Rand Index (ARI) values of 0.867–0.925 and 0.855–0.908, respectively. The Top-5 contributing dimensions also exhibit substantial overlap across datasets (85.8%–91.4% on Qwen and 91.7%–95.7% on Llama).

These results indicate that the dominant contribution structure and resulting group assignments remain largely consistent across the evaluated calibration distributions on both backbones, although some individual group boundaries vary. Importantly, each backbone independently identifies its own high-contribution dimensions, demonstrating that the grouping procedure is applicable beyond a single model without requiring identical dimension identities across architectures.

Thank you for your response. I'm glad you added what we discussed to the appendix, and I look forward to this work being further improved. However, I suggest you read more articles in the quantitative field—I think the paper as a whole still needs improvement.

Thank you for following up and for taking the time to consider our additional results. We appreciate your feedback.

If possible, could you point us to one or two representative papers related to the literature you mentioned? We would also be grateful if you could clarify which specific aspects of our work you believe still need improvement, particularly any concerns that remain after the additional experiments.

Your guidance would help us better understand your concerns and we'll do our best to address those points during the rebuttal period.Thank you again for your time and suggestions.

Reviewer_2P5nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Outlier-Aware Attention Linearization (OAL), a grouped quadratic approximation to pretrained softmax attention. Dimensions are grouped by their statistics on query-key contributions, and group-specific coefficients are calibrated by fitting attention outputs. The positive quadratic parameterization ensures valid normalization and the recurrent formulation allows for linear sequence-length complexity. Experiments with frozen-attention approximation and hybrid model conversion with LoRA on Qwen2.5-0.5B and Llama-3.2-1B.

Critical strengths
  • The paper relates an empirical finding on heterogeneous contributions of queries and keys to a concrete kernel design, supported by controlled approximation diagnostics.

  • The positive quadratic parameterization ensures a positive denominator for normalization, and the recurrent formulation preserves quadratic interactions explicitly across dimensions.

  • OAL consistently outperforms the evaluated converted models on both backbones. The learned global quadratic serves as a useful control and common replacement ranges and LoRA budgets increase comparability.

  • The paper separates the complexity at the operator level and the behavior of the full model and admits the remaining quality gap to softmax attention.

Critical weaknesses
  1. Important conversion baselines are missing. Hedgehog [1] and LoLCATs [2] are particularly relevant for preserving pretrained behavior during attention linearization, but are not evaluated. The paper clarifies that it is out of scope for its small-parameter comparison to compare learned MLP feature maps. This is a valid distinction, but it does not answer the larger question of OAL's competitiveness and currently the main weakness of this work. It would be interesting to see comparisons at matched adaptation budgets, reporting the number of parameters. Another related learned quadratic baseline is ReBased [3], which might help to disentangle the advantages of contribution-based grouping from other means of adapting quadratic kernels.

  2. Evaluation at larger scales and longer contexts is limited. We limit model-conversion experiments to 0.5B and 1B backbones, and we evaluate model quality at a 4096-token context length. As attention linearization is most relevant for long sequences, it would be useful to evaluate longer-context language modeling and retrieval to check whether OAL preserves useful attention behavior in the regime it is intended for. Experiments on larger models would also test whether contribution-based grouping is effective across scales. Moreover, adaptation is based on a single training seed, leaving the stability of the reported improvements unknown.

  3. A better characterization of the unique contribution compared to the existing quadratic attention is needed. TaylorShift [4] offers efficient evaluation of second-order polynomial attention, and ReBased [3] proposes learnable modifications to quadratic kernels. The Hadamard-diagonal identity is an exact algebraic rearrangement of quadratic features, so its role should be distinguished from a fundamentally new linearization mechanism. Potentially the distinctive contribution is the contribution-guided sharing of quadratic coefficients for pre-trained attention. Stronger controlled evidence for this element would make the novelty more convincing.

  4. There is still a conversion gap, and this should be investigated. OAL achieves a perplexity of 27.50 on Qwen2.5-0.5B and 10.89 on the LoRA-adapted softmax reference, and 32.99 on Llama-3.2-1B and 8.19 on the latter. HellaSwag accuracy drops from 51.38% to 37.20% and from 62.84% to 35.42% respectively. OAL improves over the evaluated alternatives, but substantial pretrained capability is still lost. A sweep over an adaptation budget would tell us whether this gap is largely a matter of limited training, or of limitations of the quadratic approximation.

  5. The experiments do not separate the advantage of outlier-aware grouping from additional parameter flexibility. The primary controlled comparison is against a single-group quadratic. This shows that a more flexible kernel helps, but does not show that contribution-based grouping explains the improvements shown in the results. The group-budget experiment changes the parameter count as well, and tests a single representative head. The central contribution would be more strongly supported if comparisons were made with random groups of the same size, or with other grouping strategies, all under the same calibration budget.

References

[1] Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Ré. The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry. ICLR, 2024. arXiv:2402.04347.

[2] Michael Zhang et al. LoLCATs: On Low-Rank Linearizing of Large Language Models. ICLR, 2025. arXiv:2410.10254.

[3] Yaroslav Aksenov et al. Linear Transformers with Learnable Kernel Functions are Better In-Context Models. ACL, 2024. arXiv:2402.10644.

[4] Tobias Christian Nauen, Sebastian Palacio, and Andreas Dengel. TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) using Taylor-Softmax. ICPR, 2024. arXiv:2403.02920.

Questions for authors
  1. If dimensions are randomly assigned to groups with the same group sizes as OAL, how does performance change? This would help determine whether the gains come from identifying outliers or simply adding more trainable coefficients.

  2. Is there a setting where OAL makes the converted model faster or more memory-efficient than the softmax model while maintaining similar accuracy? Please report full-model inference and token-generation measurements.

  3. How does OAL compare with Hedgehog, LoLCATs, and ReBased when given the same training data and adaptation budget?

  4. How much GPU time and peak memory do grouping, coefficient calibration, and LoRA adaptation require? How do these costs change for larger models?

  5. After LoRA changes the query and key projections, do the dimensions identified as outliers remain the same? Do the fixed groups also work on longer sequences and data from other domains?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2P5n: R2.1

Thank you for the detailed feedback. We respond to your questions below.

R2.1 — Comparisons with relevant attention-conversion methods

Concern (paraphrased): How does OAL compare with Hedgehog, LoLCATs, and ReBased under matched training data and adaptation budgets, with parameter counts reported?

Comparison design

Approximating pretrained attention with compact kernels is an established approach to model conversion, exemplified by DiJiang (Chen et al., 2024). Our comparisons examine whether pretrained QK contribution statistics provide useful guidance for allocating and sharing approximation parameters under a common conversion and adaptation protocol. The parameter comparison concerns kernel-specific parameters, excluding the shared LoRA adapters (Appendix F.1).

Relation to related linear-attention approaches

Hedgehog learns MLP feature maps through attention-weight distillation (Zhang et al., 2024). ReBased uses normalization and learnable affine transformations of queries and keys to adapt its quadratic feature map for associative recall and language modeling (Aksenov et al., 2024). OAL uses pretrained QK contribution statistics to guide the allocation and sharing of approximation coefficients across dimensions for attention conversion. LoLCATs combines learned linear attention with sliding-window softmax attention and uses attention transfer followed by LoRA adaptation (Zhang et al., 2025). Windowed attention and contribution-guided parameter sharing address different aspects of attention design and are potentially complementary. Comparing OAL with the complete LoLCATs pipeline would also involve differences in windowed attention and attention-transfer training, so such a comparison would not by itself isolate the contribution of the grouping criterion.

Controlled evidence

Alongside Performer, SpectraFormer, DiJiang, and TaylorShift, the learned-global quadratic control examines global versus grouped coefficient sharing. The size-matched random-grouping control further holds group sizes and coefficient counts fixed. Under matched adaptation settings, contribution-guided grouping achieves lower perplexity and higher accuracy on all three evaluated downstream tasks on both backbones (R2.2). These results support the value of contribution-guided dimension assignments beyond parameter count alone.

AuthorsOpenReview ↗

Response to Reviewer 2P5n: R2.2

R2.2 — Distinctive contribution and controlled evidence for grouping

Concern (paraphrased): What distinguishes OAL from existing quadratic attention, including TaylorShift and ReBased? Do the gains arise from contribution-guided grouping or from additional parameter flexibility?

Distinctive contribution

OAL structures the approximation around unequal contributions across pretrained query–key dimensions. Dimensions with different contribution scales can receive different parameters, while grouping keeps the parameterization compact. We use a learnable grouped quadratic kernel as a concrete realization of this dimension-aware approximation principle. The HD identity is an algebraic reformulation used in implementing this kernel. The controlled comparisons below show that contribution-guided grouping improves attention approximation and converted-model quality under matched settings.

Grouping control

Figure 2(b) shows that increasing the number of groups improves attention approximation with diminishing returns as the coefficient count grows, motivating compact parameter sharing.

To further isolate the contribution of our grouping criterion, we conducted a size-matched random-grouping ablation, keeping the group counts, group sizes, coefficient counts, and calibration budgets identical.

On Qwen2.5-0.5B, we evaluated frozen attention approximation across 42 heads from three layers using 16 held-out sequences of length 4,096. Compared with random grouping, contribution-guided OAL reduces attention-output MSE by 2.50% before the output projection WoW_o (0.078 vs. 0.080) and 6.06% after WoW_o (0.031 vs. 0.033). OAL also achieves lower relative MSE in both settings.

Model-conversion control

Under matched adaptation settings, contribution-guided grouping achieves lower test perplexity and higher accuracy on all three downstream tasks than size-matched random grouping on both backbones. With group sizes and coefficient counts held fixed, these results support the value of contribution-guided dimension assignments beyond parameter count alone.

Within each backbone, all three methods use the same replacement layout and adaptation settings, with training seed 42, a sequence length of 4,096, and 512 adaptation steps. Random grouping preserves each head's group count, group sizes, and coefficient count. For both grouped variants, coefficients are calibrated for their respective groups and jointly updated with LoRA while group assignments remain fixed. The tables compare Learned Global with two OAL variants: Random (size-matched random grouping) and Guided (contribution-guided grouping), reporting test perplexity and normalized downstream accuracies.

Qwen2.5-0.5B

MethodTest PPL ↓PIQA ↑HellaSwag ↑ARC-Easy ↑
Learned Global31.705961.15%32.43%41.41%
Random29.170862.57%34.55%48.40%
Guided27.502064.69%37.20%50.38%

Llama-3.2-1B

MethodTest PPL ↓PIQA ↑HellaSwag ↑ARC-Easy ↑
Learned Global36.027557.40%35.02%33.46%
Random34.027461.09%34.86%37.96%
Guided32.989663.66%35.42%41.67%
AuthorsOpenReview ↗

Response to Reviewer 2P5n: R2.3–R2.4

R2.3 — Adaptation budget and full-model efficiency

Adaptation budget and conversion quality

Concern (paraphrased): Would increasing the adaptation budget substantially reduce the conversion gap, helping distinguish limited training from limitations of the quadratic approximation?

We use a fixed LoRA budget, matched across methods within each backbone, to evaluate kernel designs within our available computational resources. This comparison does not isolate the contributions of adaptation budget and quadratic approximation to the conversion gap. At the evaluated budget, LoRA substantially improves OAL’s perplexity and downstream accuracy, and contribution-guided grouping improves both measures over size-matched random grouping on both backbones (R2.2). These results support dimension-aware parameter sharing as a useful component for further work on model conversion.

Full-model inference efficiency

Concern (paraphrased): Is there a measured setting where OAL improves full-model inference speed or memory use over softmax while maintaining similar accuracy, including token generation?

The runtime and memory results evaluate OAL at the operator level. For fixed head dimensions, the operator admits linear-time evaluation and sequence-independent recurrent-state size (Section 4.4).

Recent work such as Kimi Linear reports competitive quality and decoding-efficiency gains in a hybrid model trained from scratch, illustrating the broader potential of linear-attention components (Kimi Team et al., 2025). OAL contributes to this direction by studying how unequal QK dimension contributions can improve attention approximation, with the resulting model-quality benefits supported by the matched controls in R2.2.

R2.4 — Cost of grouping, calibration, and adaptation

Concern (paraphrased): How much GPU time and peak memory do grouping, coefficient calibration, and LoRA adaptation require, and how do these costs scale with model size?

Overall computational cost

Our measurements indicate that OAL's offline grouping and coefficient calibration incur modest one-time computational overhead, requiring only a few minutes on a consumer-grade laptop GPU, while the subsequent LoRA adaptation constitutes the substantially longer training stage.

In the profiled Qwen2.5-0.5B setting, contribution analysis, grouping, and initial coefficient calibration take a few minutes in total using an RTX 4060 Laptop GPU (8 GB). Coefficient calibration peaks at 1.89 GiB allocated GPU memory. These stages are performed once before LoRA adaptation; the measurements reuse cached Q/K/V tensors and a fitted global-kernel initialization.

LoRA adaptation cost

For the reported 512-step runs under their respective hardware configurations:

ModelHardwareTraining-step timeReported peak GPU memory
Qwen2.5-0.5B1 × A800≈78 min≈9.3 GiB allocated
Llama-3.2-1B1 × RTX 4090≈166 min≈22 GiB allocated

Costs at larger model scales have not been systematically profiled.

Reviewer_V6cMOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

OAL replaces the softmax attention kernel with a learned positive quadratic kernel. It groups query and key dimensions by their contribution to attention scores and shares coefficients within groups and group pairs. The kernel can be evaluated with recurrent statistics, with linear cost in sequence length for fixed head dimensions. The paper studies attention output error and converts middle layers of Qwen2.5-0.5B and Llama-3.2-1B. It then adapts the converted models with LoRA for 512 updates.

Critical strengths

The paper identifies a clear source of error in attention conversion. A small set of query and key dimensions can have large contributions, so allowing different coefficients across dimension groups is a reasonable idea. The learned global quadratic baseline helps isolate the benefit of this grouping under the tested setup.

Critical weaknesses
  • The paper does not establish a useful quality and efficiency trade-off. Hybrid models already use local or linear attention to reduce the cost of full attention. Sparse attention methods such as DeepSeek Sparse Attention (DSA) also offer a relevant route for adapting pretrained models. These alternatives are not fully addressed by comparisons with kernel approximations alone. The paper needs to explain when OAL is a useful choice and what benefit it offers over these routes. The current results are not convincing on this point. After the same LoRA budget, OAL has PPL 27.50 versus 10.89 for softmax on Qwen, and 32.99 versus 8.19 on Llama. It also trails softmax on every reported downstream task. At the same time, the paper states that FlashAttention is faster and uses less memory throughout the tested operator benchmark. Thus, the paper shows a better approximation than some tested alternatives, but does not show a practical setting where its large quality loss is justified by measured efficiency gains. This substantially limits the importance of the result.

  • Important experiments needed to interpret the conversion results are missing. The paper converts layers 3–20 in Qwen and layers 1–14 in Llama. It motivates these choices using activation statistics, but does not test whether these ranges are better than other choices. A common range across methods within each backbone is a useful control, but it does not establish how sensitive the result is to the number or position of replaced layers. The paper also omits the performance of the untouched pretrained models. Its softmax reference has already received LoRA updates. Without the original model and the converted model before LoRA, it is hard to separate the loss from conversion and the recovery from adaptation. These gaps prevent a clear assessment of the method's value and support my score of 1.

Questions for authors
  • At what measured operating point does OAL provide a useful quality and efficiency trade-off over softmax? Please report full-model latency and memory together with quality, including the cost of retained softmax layers. How does this use case differ from hybrid attention and pretrained sparse-attention conversion such as DSA?

  • How were the exact replacement ranges selected for each backbone? Can you show how quality and efficiency change with the number and position of replaced layers?

  • What are the PPL and downstream scores of the untouched pretrained models and the calibrated OAL models before LoRA? Does a larger adaptation budget substantially close the gap to the softmax reference?

4 comments
AuthorsOpenReview ↗

Response to Reviewer V6cM: R1.1

Thank you for your careful review and for recognizing the dimension-wise source of approximation error and the role of the learned-global control. Below, we explain how the diagnostics and conversion experiments assess this kernel-design contribution.

R1.1 — Practical quality–efficiency trade-off and alternative conversion approaches

Concern (paraphrased): At what measured operating point does OAL offer a useful quality–efficiency trade-off over softmax, including the cost of retained softmax layers? How does this use case compare with hybrid attention and sparse conversion approaches such as DSA?

Role of the linear-attention component

OAL studies how dimension-wise contribution information can improve a compact linear-attention kernel. Our primary focus is on improving attention approximation at the operator level, rather than developing a fully optimized end-to-end inference system. This component-level focus is relevant to hybrid architectures: Kimi Linear, for example, combines KDA and global MLA layers at a 3:1 ratio (Kimi Team et al., 2025). Hybridization specifies where linear and full attention are used, while OAL addresses the approximation within the converted layers. Our comparisons keep the retained softmax layers and replacement layout the same within each backbone, allowing us to examine the kernel choice in a common architecture. In contrast, sparse attention approaches such as DSA reduce computation through selective token attention, whereas OAL investigates structured kernel approximation for pretrained attention conversion. As an early-stage exploration, our goal is to establish the potential of this dimension-aware approximation principle at the component level, providing a foundation for future scaling and integration into larger, more optimized attention architectures.

Complexity and current implementation

Section 4.4 and Appendix D establish linear-time evaluation and sequence-independent recurrent-state size for fixed head dimensions, and our implementation realizes this computation without forming an N×NN\times N attention matrix. Figure 3 shows that the (G=5) and (G=1) attention operators have comparable runtime and peak memory under the same HD backend. These are operator-level measurements; a full-model serving operating point has not been established. We acknowledge that practical end-to-end efficiency depends not only on the attention formulation but also on the retained softmax layers and the level of kernel and serving-system optimization. Consequently, further engineering optimization and full-model evaluation would be necessary to establish a favorable quality–efficiency trade-off in deployment. With group sizes and coefficient counts held fixed, contribution-guided grouping improves attention approximation and converted-model quality relative to random grouping.

AuthorsOpenReview ↗

Response to Reviewer V6cM: R1.2–R1.4

R1.2 — Selection of replacement layers and sensitivity

Concern (paraphrased): How were the replacement ranges selected for each backbone, and how sensitive are quality and efficiency to the number and position of the replaced layers?

Selection rationale

The replacement range is a configuration hyperparameter in our pretrained-model conversion experiments, rather than an intrinsic component of OAL. Our objective is not to eliminate softmax attention entirely. Hybrid attention architectures, such as Kimi Linear, demonstrate the relevance of combining linear and full attention components. In our pretrained-model conversion setting, we retain selected softmax layers while investigating the effectiveness of OAL in the converted layers.

As shown in Figure 7 of the appendix, pronounced massive activations persist across the intermediate layers of both backbones. Motivated by this observation, we select these intermediate regions as representative targets for attention replacement: layers 3–20 for Qwen2.5-0.5B and layers 1–14 for Llama-3.2-1B (zero-based indexing).

Sensitivity to the replacement range

The replacement layout is a configuration choice in our model-conversion evaluation, rather than a proposed layer-selection strategy. We use the same layout across methods within each backbone to assess the dimension-aware kernel design under controlled conditions. The reported relative gains are established under these shared layouts; sensitivity to alternative layer counts and positions is not assessed in the present experiments.

R1.3 — Quality before conversion and before LoRA adaptation

Concern (paraphrased): What are the PPL and downstream scores of the untouched pretrained models and the calibrated OAL models before LoRA, and how do these compare with the adapted models?

The tables below report the untouched pretrained models and calibrated OAL models before adaptation, alongside their results after 512 adaptation steps.

Qwen2.5-0.5B

Model stateTest PPL ↓PIQA ↑HellaSwag ↑ARC-Easy ↑
Softmax, before LoRA12.089370.29%52.01%58.08%
OAL, before LoRA1142.1060.28%29.78%42.09%
Softmax, after LoRA10.886170.18%51.38%62.63%
OAL, after LoRA27.502064.69%37.20%50.38%

Llama-3.2-1B

Model stateTest PPL ↓PIQA ↑HellaSwag ↑ARC-Easy ↑
Softmax, before LoRA9.0674.86%62.85%60.56%
OAL, before LoRA3561.2155.88%26.96%33.08%
Softmax, after LoRA8.194774.92%62.84%59.43%
OAL, after LoRA32.989663.66%35.42%41.67%

Across both backbones, OAL achieves substantially lower perplexity and higher accuracy on all three evaluated downstream tasks after LoRA adaptation.

For OAL, adaptation jointly updates the LoRA parameters and kernel coefficients while keeping the dimension groups fixed.

R1.4 — Adaptation budget and the role of model conversion

Concern (paraphrased): Does a larger adaptation budget substantially close the gap to the softmax reference?

R1.3 shows that LoRA adaptation substantially improves OAL’s perplexity and downstream accuracy on both backbones. Within our available computational resources, we use the same LoRA budget across methods within each backbone to compare kernel designs under controlled conditions. A larger adaptation budget may provide further recovery, but its extent is not quantified by this evaluation.

Under these controlled conditions, contribution-guided grouping achieves lower perplexity and higher downstream accuracy than size-matched random grouping on both backbones. These results support using heterogeneous QK dimension contributions to guide parameter sharing in attention approximation.

Thank you for the additional experiments and clarifications. The size-matched random-grouping control strengthens the evidence for contribution-guided grouping, and the newly reported results before LoRA help clarify the conversion loss.

However, my main concerns remain. The converted models still suffer substantial quality degradation, without a measured full-model efficiency benefit to justify this trade-off. Sensitivity to the replacement layout and the effect of a larger adaptation budget also remain untested. Additionally, the controlled grouping experiments do not replace comparisons with closely related conversion methods such as Hedgehog, LoLCATs, or ReBased.

I appreciate the narrower proof-of-concept positioning and the improved evidence for the grouping principle. These additions lead me to raise my rating from 1 to 2, but the remaining gaps prevent me from supporting acceptance.

AuthorsOpenReview ↗

Clarification of Scope

Thank you for reconsidering our submission and recognizing the additional evidence. We would like to clarify the research question and the scope of our comparisons.

Our central contribution is to use heterogeneous contributions across pretrained query–key dimensions to guide attention approximation. OAL realizes this principle through a learnable grouped quadratic kernel. Our experiments examine whether this dimension-wise information improves attention approximation and whether those improvements translate into quality gains under a common conversion protocol.

The requested comparisons with Hedgehog, LoLCATs, and ReBased fall outside the scope of our study. Our comparison focuses on fixed feature maps and kernels with a small number of trainable parameters, excluding the shared LoRA adapters. Hedgehog’s dense MLP feature maps introduce substantially more trainable kernel parameters. LoLCATs incurs a separate training cost for learning feature maps through attention transfer before LoRA adaptation, whereas OAL calibrates a small set of grouped kernel coefficients. LoLCATs’ sliding window attention is potentially complementary to dimension-wise approximation. ReBased’s reported results involve the cost of training entire language models from scratch. Our contribution concerns how pretrained dimension contributions guide parameter sharing, with the quadratic form serving as the realization evaluated here.

The adaptation budget is held fixed across methods to assess the effect of the approximation design under matched conditions. Increasing that budget would address further recovery of the converted models, but our study focuses on whether contribution-guided approximation improves attention fidelity and model quality within the same conversion setting.

Regarding replacement layouts, as explained in R1.2, the layout is an experimental configuration rather than a proposed selection strategy. The intermediate replacement regions were motivated by the activation profiles in Figure 7, and all methods use the same layout setting within each backbone.

We believe the demonstrated value of using pretrained dimension-wise contributions to organize approximation parameters provides a meaningful methodological contribution to attention linearization and a basis for further research.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Scope and Contribution

We thank the reviewers for their careful reading and constructive feedback. We summarize our contribution and the additional evidence below, followed by detailed reviewer-specific responses.

Scope and contribution

Our work begins with an important empirical observation: the contributions of query–key dimensions to pretrained attention are highly heterogeneous, with a small subset of outlier dimensions accounting for disproportionately large contributions and approximation errors. This phenomenon is observed across the pretrained language models studied, supporting its relevance beyond a single model.

Our central novelty lies in bringing this dimension-wise heterogeneity into the design of linear attention approximations, a connection that, to the best of our knowledge, has not been explicitly explored in prior attention-linearization methods. Specifically, OAL uses the unequal contributions of QK dimensions to guide how approximation parameters are assigned and shared across dimensions.

We position OAL as a proof-of-concept study demonstrating the potential of this direction through a compact, learnable grouped quadratic kernel. Frozen-attention diagnostics assess approximation fidelity directly, while controlled model-conversion experiments evaluate its effects on language modeling and downstream performance under limited adaptation. Our primary goal is to establish the value of this dimension-aware approximation principle through controlled kernel-design comparisons. Establishing deployment-level quality–efficiency trade-offs requires further engineering optimization and full-model evaluation.

Recognition of our contributions

We appreciate that the reviewers recognize complementary aspects of our work. Reviewer V6cM notes that our paper “identifies a clear source of error in attention conversion,” while Reviewer 2P5n highlights how it “relates an empirical finding on heterogeneous contributions of queries and keys to a concrete kernel design.” Reviewer 5TBz further describes our diagnostic analysis as “convincing and well controlled.” These comments recognize the empirical observation, its connection to kernel design, and the controlled diagnostic evidence.

AuthorsOpenReview ↗

General Response: Additional Evidence

Addressing the main concerns

Building on these contributions, we summarize additional evidence on (1) the value of contribution-guided grouping under matched settings, (2) the empirical basis and functional importance of high-contribution dimensions, and (3) the stability of contribution statistics and grouping. We then clarify the evaluation setting and computational scope.

1. Controlled evidence for contribution-guided grouping

Under matched adaptation settings and with group sizes and coefficient counts held fixed, contribution-guided grouping achieves lower test perplexity and higher accuracy on all three evaluated downstream tasks than size-matched random grouping on both backbones. Test PPL is 27.50 vs. 29.17 on Qwen2.5-0.5B and 32.99 vs. 34.03 on Llama-3.2-1B (Guided vs. Random; R2.2). These results support the value of contribution-guided dimension assignments beyond parameter count alone.

2. Functional importance of high-contribution dimensions

Across both backbones, masking high-contribution dimensions produces larger attention-output errors than masking the same number of randomly selected dimensions. In a separate Qwen model-level ablation, Top-5 masking increases PPL by 3.22 on average, compared with 0.24 for Random-5 masking, each relative to the same unmasked baseline (R3.2). On Qwen, activation-based and contribution-based Top-5 selections also overlap by 83.0% (R3.1). Together, these results connect the outlier motivation with the functional relevance of dimension-wise QK contributions.

3. Stability of contribution statistics and grouping

Across WikiText-103, C4, and CodeParrot, independently recomputed contribution rankings and group assignments show substantial consistency on both backbones (R3.4). Qwen also shows substantial grouping consistency across sequence lengths of 512–4,096 tokens (R2.5). After adaptation on Qwen, the original highest-contribution dimension remains top-ranked in 96.83% of heads in the converted layers; the mean Top-3 overlap across these heads is 73.28% (R2.5). One additional OAL adaptation run on each backbone yields similar results to the original seed (R2.6).

Evaluation setting and computational scope

Within each backbone, model-conversion comparisons use the same replacement layout, training steps, and adaptation protocol. Softmax provides a reference for retained model quality; the learned-global control examines global versus grouped coefficient sharing, while the size-matched random-grouping control assesses the value of contribution-guided dimension assignments (R1.3, R1.4, R2.2, R2.3). For fixed head dimensions, the OAL operator permits linear-time evaluation and sequence-independent recurrent-state size (Section 4.4). Figure 3 reports comparable operator runtime and peak memory for G=5G=5 and G=1G=1 under the shared HD backend. Implementation and full-model efficiency considerations are addressed in R1.1 and R2.3.

Revised manuscript

We have uploaded a revised PDF incorporating the additional experiments discussed above, primarily in Appendix G, with the new and updated content highlighted in blue for ease of reference.

We sincerely thank all three reviewers again for their valuable feedback and constructive suggestions. We welcome further questions and discussion, and would be happy to provide additional clarification where needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.