- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Outlier-Aware Attention Linearization (OAL) for linearizing pretrained softmax attention. The authors first analyze dimension-wise contributions of query-key interactions in pretrained attention and find that attention logits are highly concentrated in a small subset of dimensions. Approximating only these high-contribution dimensions (≈5.5% of head dimensions) retains a substantial fraction of full-head attention-output MSE (37.8%–64.8% retained-error ratio, consistently above Gaussian controls, p < 0.001 against 1,000 cardinality-matched random-coordinate controls). Motivated by this, the authors introduce a dimension-aware quadratic kernel with per-dimension linear coefficients b_r and per-dimension-pair quadratic coefficients C_rt, evaluated exactly in O(N) via a Hadamard-diagonal identity. To control parameter count, dimensions are sorted by contribution score and partitioned into 2–5 groups via bootstrap stability of adjacent contribution gaps, with coefficients shared within groups and group pairs; a lower-triangular factor L (M = LL^T) enforces positive kernel weights. Coefficients are calibrated by output matching on frozen Q/K/V and then jointly optimized with LoRA while groups stay fixed. On Qwen2.5-0.5B and Llama-3.2-1B, OAL achieves lower frozen-attention MSE/RelMSE than Performer, SpectraFormer, DiJiang, and TaylorShift, and better WikiText-103 PPL and downstream accuracy than those converted methods and a learned global quadratic (G=1) baseline, though it remains behind the LoRA-adapted softmax reference.
Critical strengths
-
The diagnostic analysis is convincing and well controlled. The authors show not only that dimension-wise contributions are concentrated, but also that this concentration cannot be explained by dimensionality or coordinate selection, using two independent controls (isotropic Gaussian replacement and 1,000 cardinality-matched random-coordinate subsets), and further verify stability across held-out inputs (Spearman ρ = 0.991) and sequence lengths. This diagnostic is independently useful to the linear-attention community.
-
The method is engineering-sound and deployable. The Hadamard-diagonal identity enables exact recurrent evaluation of the quadratic kernel without materializing an N×N matrix, and the M = LL^T parameterization guarantees a positive normalization denominator. Grouping reduces learned parameters from O(d_h²) to O(G²), and at G=5 the runtime and peak memory are comparable to the G=1 baseline (Figure 3), indicating the added flexibility incurs little operator overhead.
Critical weaknesses
-
The central "outlier" framing is not substantiated, and the link between χ_r and outlier dimensions is never established. The paper is titled and motivated around "Outlier-Aware," yet χ_r is a custom causal average of absolute coordinate products, and the paper does not connect it to the stricter statistical criteria used in the outlier-dimensions literature (e.g., top-1% absolute activation, extreme median across at least 50% of inputs, average z-score ≥ 8 standard deviations). More importantly, prior work shows that "large logit contribution" and "functionally important" are distinct: in some models, outlier dimensions have near-zero logit contribution when predicting their preferred tokens and can be ablated with almost no accuracy change, suggesting they may be byproducts of the softmax normalization constraint rather than information carriers. OAL only measures contribution magnitude and performs no functional validation (e.g., whether ablating high-contribution dimensions actually degrades attention output or downstream accuracy). Whether the "outlier-aware" label is warranted is therefore not established by the paper itself. This is a framing-level issue rather than a presentational one.
-
A key ablation is missing, so the source of the gain—grouping versus learnable coefficients per se—cannot be isolated. The paper compares G=1 and G=5, which is an ablation on the number of groups, not on "learnable vs. fixed" coefficients. TaylorShift uses fixed Taylor coefficients, Hedgehog learns MLP feature maps, and DiJiang's treatment of kernel parameters during conversion is not clearly described. OAL does not answer: how much performance is lost if coefficients are fixed to some estimate and excluded from backpropagation? What happens to performance and cost if dimension-specific parameters are fully unshared (G = d_h) and trained? Without these two controls, the effectiveness of the paper's central contribution—contribution-aware grouping—cannot be isolated.
-
Cross-dataset and cross-model stability of group boundaries is not validated. Prior work shows that the location and function of outlier dimensions vary substantially across models (in some models they relate to high-frequency token prediction; in others they have almost no functional effect). OAL computes χ_r only on a WikiText-103 calibration set, and group boundaries are determined only on that set via bootstrap (Appendix B). There is no cross-dataset (e.g., code, multilingual, dialogue) or cross-distribution comparison to verify that group boundaries are stable. If group boundaries depend on the calibration distribution, the transferability of the method is questionable.
Questions for authors
-
What is the relationship between χ_r and established statistical definitions of outlier dimensions (e.g., absolute activation ranking, z-score thresholds)? Is there any experiment showing the overlap between dimensions selected by χ_r and those selected by standard OD criteria?
-
Is there any functional validation—e.g., does ablating high-contribution dimensions significantly degrade attention-output error or downstream accuracy? If ablating these dimensions leaves model behavior largely unchanged, how does that affect the appropriateness of the "outlier-aware" framing?
-
Please add a control on whether coefficients participate in backpropagation: how much do PPL and downstream tasks differ between fixing the calibrated coefficients (excluding them from joint LoRA optimization) and the current joint optimization? What are the performance and memory/time costs when dimension-specific parameters are fully unshared (G = d_h, no grouping)?
4 comments
Response to Reviewer 5TBz: R3.3–R3.4
R3.3 — Coefficient optimization and fully unshared parameters
Concern (paraphrased): How do PPL and downstream scores change when calibrated coefficients are kept fixed during LoRA versus jointly optimized? How does optimizing fully unshared dimension-specific parameters (G = d_h) affect quality, memory use, and runtime?
Fixed versus jointly optimized coefficients
On Qwen2.5-0.5B, jointly optimizing the calibrated kernel coefficients and LoRA yields modest improvements in all four evaluated metrics compared with keeping the coefficients fixed during LoRA adaptation.
| Adaptation | Test PPL ↓ | PIQA ↑ | HellaSwag ↑ | ARC-Easy ↑ |
|---|---|---|---|---|
| Calibrated kernel coefficients fixed, LoRA only | 27.9643 | 63.76% | 35.41% | 49.41% |
| Joint optimization of LoRA and kernel coefficients | 27.5020 | 64.69% | 37.20% | 50.38% |
Fully unshared parameters
The representative-head analysis on Qwen2.5-0.5B shows diminishing approximation gains over the tested range of to , motivating compact grouping (Figure 2(b)). An end-to-end comparison of quality, memory, and runtime at has not been conducted.
DiJiang training protocol
In our DiJiang baseline, the feature weights are optimized jointly with LoRA, while the DCT projection remains fixed (Appendix F.1).
R3.4 — Group-boundary stability across calibration distributions and models
Concern (paraphrased): How stable are group boundaries across datasets, distributions, and models when they are determined using WikiText-103 calibration data?
Cross-dataset and cross-model grouping stability
Our additional experiments demonstrate substantial consistency in contribution rankings and group assignments across different calibration distributions on both Qwen2.5-0.5B and Llama-3.2-1B. Following the original grouping procedure, we independently recomputed contribution scores and group assignments on WikiText-103, C4, and CodeParrot, covering all 336 query heads of Qwen and 512 query heads of Llama.
Across the three dataset pairs, contribution rankings achieve Spearman correlations of 0.946–0.983 on Qwen and 0.940–0.981 on Llama, while group partitions achieve Adjusted Rand Index (ARI) values of 0.867–0.925 and 0.855–0.908, respectively. The Top-5 contributing dimensions also exhibit substantial overlap across datasets (85.8%–91.4% on Qwen and 91.7%–95.7% on Llama).
These results indicate that the dominant contribution structure and resulting group assignments remain largely consistent across the evaluated calibration distributions on both backbones, although some individual group boundaries vary. Importantly, each backbone independently identifies its own high-contribution dimensions, demonstrating that the grouping procedure is applicable beyond a single model without requiring identical dimension identities across architectures.
Thank you for your response. I'm glad you added what we discussed to the appendix, and I look forward to this work being further improved. However, I suggest you read more articles in the quantitative field—I think the paper as a whole still needs improvement.
Thank you for following up and for taking the time to consider our additional results. We appreciate your feedback.
If possible, could you point us to one or two representative papers related to the literature you mentioned? We would also be grateful if you could clarify which specific aspects of our work you believe still need improvement, particularly any concerns that remain after the additional experiments.
Your guidance would help us better understand your concerns and we'll do our best to address those points during the rebuttal period.Thank you again for your time and suggestions.
Response to Reviewer 5TBz: R3.1–R3.2
Thank you for the detailed review. We address each concern below.
R3.1 — Contribution scores and statistical definitions of outlier dimensions
Relationship between activation outliers and QK contributions
Conventional outlier-dimension criteria identify dimensions exhibiting unusually large activation magnitudes, whereas our contribution score χr quantifies each dimension's contribution to QK attention logits. Since OAL focuses on attention approximation, we investigate this relationship directly in pretrained Q/K representations, using activation magnitude and within-head z-score as two outlier-selection criteria.
Our results demonstrate a strong empirical correspondence between activation outliers and high-contribution QK dimensions, supporting the outlier-aware motivation of OAL. Across all 336 attention heads of Qwen2.5-0.5B, the Top-5 dimensions selected by average Q/K activation magnitude exhibit 83.0% overlap with those selected by χr, corresponding to 4.15 shared dimensions per head on average. Furthermore, among dimensions identified as activation outliers using a within-head z-score threshold of z≥2, 90.4% also belong to the corresponding χr, Top-5.
These findings establish substantial agreement between activation-based outlier criteria and contribution-based identification under the evaluated settings. Together with the approximation-error concentration analysis in Section 3.2 and the functional ablations in R3.2, they support our central motivation: identifying and treating high-contribution QK dimensions differently can improve attention approximation.
R3.2 — Functional validation and the “outlier-aware” framing
We added functional ablations in the original softmax model, comparing the removal of Top-k contribution dimensions with randomly selected dimensions. The number of removed dimensions and all other attention computations are kept the same.
Attention-output effects
Across both Qwen2.5-0.5B and Llama-3.2-1B, masking high-contribution dimensions consistently produces substantially larger attention-output errors than masking randomly selected dimensions. As shown in Appendix G.3, Top-k masking yields 4.05×–21.91× larger MSE on Qwen and 5.30×–10.50× larger MSE on Llama, providing functional evidence for the importance of the identified dimensions.
Model-level effects
We further evaluate the functional importance of high-contribution dimensions by applying Top-5 and Random-5 masking to three sampled layers (11, 16, and 22) of the original Qwen2.5-0.5B softmax model separately. This ablation uses fewer evaluation samples than the model-conversion experiments, with an unmasked baseline PPL of 10.9329. Relative to this baseline, Top-5 masking increases PPL by 3.22 on average, compared with 0.24 for Random-5 masking. This substantially larger degradation demonstrates that high-contribution dimensions play an important role in preserving language-modeling performance, providing further functional evidence for the outlier-aware motivation of OAL.