- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper highlights oversmoothing problem in linear attention, which causes redundant information and obscures critical visual cues. Based on the observation, the paper argues that smooth attention weights naturally capture redundant patterns, which can be used to model visual redundancy. The authors propose Asymmetric Differential Linear Attention (ADLA) to model such visual redundancy. ADLA consists of two branches: the saliency branch extracts discriminative features via group-weighted key-value summarization, and the redundancy branch provides the standard linear attention. ADLA subtracts the redundancy branch from the saliency branch to achieve attention sharpening. Extensive experiments show that ADLA outperforms the baseline methods across various computer vision tasks including classification, object detection, semantic segmentation and image generation.
Critical strengths
- The paper is well-structured and clear. The paper follows a good logical structure that guides readers through the concepts and findings seamlessly.
- The paper provides clear motivations and provides a different perspective on oversmoothing issue in linear attention. Instead of directly addressing the smoothness of linear attention, the paper proposes ADLA to suppress redundant information.
- The framework design is reasonable and well-motivated. ADLA uses saliency branch to obtain the discriminative features to achieve attention sharpness, and GWS is used in the saliency branch to group queries on informative regions.
- Experiments are conducted across different types of computer vision tasks and validate the effectiveness. The results show consistent improvements and extensive studies show the practical efficiency.
Critical weaknesses
- The redundancy statement needs stronger evidence. The paper highlights that oversmoothing in linear attention can be regarded as a prior for visual redundancy, but it is not actually validated. Smoothed attention may not correspond to irrelevant information. Although Figure 1 visualizations are provided to outline the phenomenon, the smoothing issue is still an observation and can not support it is harmful to the model.
- The role of the redundancy branch is not clear. ADLA subtracts the redundancy branch output from the saliency output, but it does not show the redundancy branch learns redundant visual information or it is necessary to remove the redundancy.
Questions for authors
Please see the weaknesses section above.
2 comments
Response to Reviewer MmD2 (part 2/2)
W2. The role of the redundancy branch is unclear. What evidence shows that subtracting its output is useful for suppressing visual redundancy and improving the model?
Thank you for raising this point. We address the two sub-questions separately.
1) Evidence that the redundancy branch captures redundant visual information.
We draw this conclusion from its response pattern and behavior. Structurally, it retains vanilla linear attention, and natural images contain substantial spatial redundancy, with many tokens providing overlapping evidence for visual decisions [1, 2], which is consistent with its smoothness property. Its response is the most diffuse: rank 28.6 and magnitude entropy 5.06, versus rank 150.5 and entropy 4.53 for the saliency branch; the top- measurements in W1 further show that is the least concentrated map and is the most concentrated. More importantly, in the redundancy-token experiment, our approach of subtracting the redundancy branch is the most robust to redundant-information damage among all measured models (W1), which is sufficient to show that our design is reasonable, achieves the redundancy branch’s ability to model visual redundancy, and effectively removes it.
2) Evidence that removing this redundancy is necessary.
Likewise, in the ablations and the supplementary redundancy-token experiment, our asymmetric differential model performs best, which is sufficient to show the necessity of removing redundancy.
We hope the above responses have adequately addressed your concerns. Please feel free to contact us if you have any further questions. If not, we would greatly appreciate your consideration for a higher score.
References
[1] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. “DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.” Advances in Neural Information Processing Systems (NeurIPS), 2021.
[2] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. “Masked Autoencoders Are Scalable Vision Learners.” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Response to Reviewer MmD2 (part 1/2)
Thank you for the time and effort devoted to reviewing our paper, and for the constructive comments. We have carefully considered all the concerns and address them point‑by‑point below.
Thank you for raising this point. We do not claim that smooth attention and redundancy correspond to irrelevant information. Rather, we use “redundant” in a task-relative sense: a response may carry information while contributing little additional evidence for a particular prediction. Natural images contain substantial spatial redundancy, with many tokens providing overlapping evidence for visual decisions [1, 2]. Therefore, the smooth response of linear attention is well suited to modeling such widely distributed, weakly task-discriminative redundant components. We further provide a complete evidence chain comprising quantitative diagnostics, matched ablations, and a controlled redundancy perturbation:
1) Response diagnostics beyond visualization.
Fig. 3 shows the redundancy branch is the most diffuse (rank 28.6, magnitude entropy 5.06), compared to the saliency branch (rank 150.5, entropy 4.53). After differential fusion, rank rises to 163.3 and entropy drops to 4.27. We further measure the top-p magnitude concentration: for each query row, we sort attention weights by absolute magnitude and compute the fraction of total magnitude captured by the largest p%.
D=As−λAr consistently exhibits higher magnitude concentration than both branches. Compared with As, differential subtraction improves the concentration by 6.32–8.45 percentage points. This demonstrates that the redundancy branch exhibits a more diffuse and smoother response than the saliency branch. Subtracting the diffuse redundancy branch concentrates the attention weights on salient tokens.
2) Matched ablations isolate the necessity of redundancy removal.
ADLA achieves 74.0% top-1 (Tab. 5), compared with 73.2% for symmetric GWS (Tab. 5) and 73.1% for full-dimensional non-differential GWS (Tab. 9). The non-differential control is 0.9 points lower; the symmetric control is 0.8 points lower at higher cost. The gain therefore comes specifically from preserving a smooth redundancy branch and removing it asymmetrically, not from extra capacity or a generic differential form.
3) Controlled redundancy perturbation shows a measurable harmful effect.
We added ADLA, symmetric GWS, and non-differential GWS to the boundary-token redundancy experiment in Appendix B.
ADLA shows a smaller accuracy drop than all controls. With 480 added tokens (r=6), ADLA loses 9.7 percentage points, compared with 10.8 for symmetric GWS and 18.7 for non-differential GWS. Thus, redundant tokens progressively amplify the disadvantage of linear attention, and ADLA’s redundancy removal mitigates it. This shows the harmful effect is measurable, not merely observed. These supporting the robustness benefit from the asymmetric design and the proposed redundancy-removal explanation. Using redundant branches to model and eliminating visual redundancy to improve model performance is feasible.
In summary, diagnostics show that the redundancy branch is the most smooth, combined with matched ablations and the redundancy experiment, the evidence chain is complete: ablations show that preserving the smooth branch and removing it asymmetrically is necessary, and the redundancy experiment shows that ADLA can effectively maintain performance by eliminating visual redundancy. Together, these results validate that our redundancy branch effectively models task-relative redundancy and that asymmetric removal improves performance.
Revisions (blue): Appendix H.