acceptodds
Under review as a conference paper at ICLR 2027

Harnessing Smoothness: Asymmetric Differential Linear Attention for Vision Transformers

Abstract

Linear attention reduces the quadratic cost of softmax attention, but its inherent smoothing effect often causes redundant information to obscure critical visual cues in the fused representation, leading to performance degradation. Rather than merely treating this smoothness as a limitation, we argue that it is inherently well-suited to modeling visual redundancy, as smooth attention weights naturally capture broadly distributed and redundant patterns. Based on this insight, we propose Asymmetric Differential Linear Attention (ADLA), built upon an asymmetric dual-branch architecture. Specifically, the saliency branch extracts discriminative features via group-weighted key-value summarization, while the redundancy branch provides an explicit prior of visual redundancy via vanilla linear attention. Through their differential interaction, ADLA suppresses visual redundancy while reinforcing critical details, sharpening the attention focus. We further incorporate a dynamic kernel routing mechanism to refine feature focusing. Extensive experiments across diverse vision tasks and ablation studies demonstrate that ADLA consistently achieves strong performance while preserving linear complexity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Transformers, Linear Attention, Asymmetric Differential Linear Attention, Visual Redundancy, Group‑Weighted KV Summarization, Dynamic Kernel Routing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MmD2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper highlights oversmoothing problem in linear attention, which causes redundant information and obscures critical visual cues. Based on the observation, the paper argues that smooth attention weights naturally capture redundant patterns, which can be used to model visual redundancy. The authors propose Asymmetric Differential Linear Attention (ADLA) to model such visual redundancy. ADLA consists of two branches: the saliency branch extracts discriminative features via group-weighted key-value summarization, and the redundancy branch provides the standard linear attention. ADLA subtracts the redundancy branch from the saliency branch to achieve attention sharpening. Extensive experiments show that ADLA outperforms the baseline methods across various computer vision tasks including classification, object detection, semantic segmentation and image generation.

Critical strengths
  1. The paper is well-structured and clear. The paper follows a good logical structure that guides readers through the concepts and findings seamlessly.
  2. The paper provides clear motivations and provides a different perspective on oversmoothing issue in linear attention. Instead of directly addressing the smoothness of linear attention, the paper proposes ADLA to suppress redundant information.
  3. The framework design is reasonable and well-motivated. ADLA uses saliency branch to obtain the discriminative features to achieve attention sharpness, and GWS is used in the saliency branch to group queries on informative regions.
  4. Experiments are conducted across different types of computer vision tasks and validate the effectiveness. The results show consistent improvements and extensive studies show the practical efficiency.
Critical weaknesses
  1. The redundancy statement needs stronger evidence. The paper highlights that oversmoothing in linear attention can be regarded as a prior for visual redundancy, but it is not actually validated. Smoothed attention may not correspond to irrelevant information. Although Figure 1 visualizations are provided to outline the phenomenon, the smoothing issue is still an observation and can not support it is harmful to the model.
  2. The role of the redundancy branch is not clear. ADLA subtracts the redundancy branch output from the saliency output, but it does not show the redundancy branch learns redundant visual information or it is necessary to remove the redundancy.
Questions for authors

Please see the weaknesses section above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MmD2 (part 1/2)

Thank you for the time and effort devoted to reviewing our paper, and for the constructive comments. We have carefully considered all the concerns and address them point‑by‑point below.

W1. The interpretation of smooth linear attention as a prior for visual redundancy requires stronger evidence. Smooth attention can carry useful context, and the visualizations alone do not establish an adverse effect on model performance.

Thank you for raising this point. We do not claim that smooth attention and redundancy correspond to irrelevant information. Rather, we use “redundant” in a task-relative sense: a response may carry information while contributing little additional evidence for a particular prediction. Natural images contain substantial spatial redundancy, with many tokens providing overlapping evidence for visual decisions [1, 2]. Therefore, the smooth response of linear attention is well suited to modeling such widely distributed, weakly task-discriminative redundant components. We further provide a complete evidence chain comprising quantitative diagnostics, matched ablations, and a controlled redundancy perturbation:

1) Response diagnostics beyond visualization.

Fig. 3 shows the redundancy branch is the most diffuse (rank 28.6, magnitude entropy 5.06), compared to the saliency branch (rank 150.5, entropy 4.53). After differential fusion, rank rises to 163.3 and entropy drops to 4.27. We further measure the top-pp magnitude concentration: for each query row, we sort attention weights by absolute magnitude and compute the fraction of total magnitude captured by the largest pp%.

ModelTop-ppAsA_sArA_rD=As−λArD=A_s-\lambda A_rGain over AsA_s (pp)
DeiT-T-ADLA5%32.09%14.54%38.41%+6.32
10%43.27%24.72%50.31%+7.04
15%51.81%33.26%59.07%+7.26
PVT-T-ADLA5%32.10%22.02%39.22%+7.12
10%43.27%33.24%50.45%+7.18
15%52.31%42.78%59.45%+7.14
Swin-T-ADLA5%42.15%26.97%50.60%+8.45
10%54.35%40.56%62.43%+8.08
15%62.71%50.41%70.59%+7.88

D=As−λArD=A_s-\lambda A_r consistently exhibits higher magnitude concentration than both branches. Compared with AsA_s, differential subtraction improves the concentration by 6.32–8.456.32–8.45 percentage points. This demonstrates that the redundancy branch exhibits a more diffuse and smoother response than the saliency branch. Subtracting the diffuse redundancy branch concentrates the attention weights on salient tokens.

2) Matched ablations isolate the necessity of redundancy removal.

ADLA achieves 74.0% top-1 (Tab. 5), compared with 73.2% for symmetric GWS (Tab. 5) and 73.1% for full-dimensional non-differential GWS (Tab. 9). The non-differential control is 0.9 points lower; the symmetric control is 0.8 points lower at higher cost. The gain therefore comes specifically from preserving a smooth redundancy branch and removing it asymmetrically, not from extra capacity or a generic differential form.

3) Controlled redundancy perturbation shows a measurable harmful effect.

We added ADLA, symmetric GWS, and non-differential GWS to the boundary-token redundancy experiment in Appendix B.

Modelr=0r=1r=2r=3r=4r=5r=6Drop at r=6 (pp)
Softmax attention72.171.469.868.166.064.061.910.2
Linear attention70.268.866.163.159.556.052.617.6
Non-differential GWS (Tab. 9)73.169.363.761.561.557.654.418.7
Symmetric GWS (Tab. 5)73.272.170.868.866.764.562.410.8
ADLA74.073.372.570.668.266.364.39.7

ADLA shows a smaller accuracy drop than all controls. With 480 added tokens (r=6), ADLA loses 9.7 percentage points, compared with 10.8 for symmetric GWS and 18.7 for non-differential GWS. Thus, redundant tokens progressively amplify the disadvantage of linear attention, and ADLA’s redundancy removal mitigates it. This shows the harmful effect is measurable, not merely observed. These supporting the robustness benefit from the asymmetric design and the proposed redundancy-removal explanation. Using redundant branches to model and eliminating visual redundancy to improve model performance is feasible.

In summary, diagnostics show that the redundancy branch is the most smooth, combined with matched ablations and the redundancy experiment, the evidence chain is complete: ablations show that preserving the smooth branch and removing it asymmetrically is necessary, and the redundancy experiment shows that ADLA can effectively maintain performance by eliminating visual redundancy. Together, these results validate that our redundancy branch effectively models task-relative redundancy and that asymmetric removal improves performance.

Revisions (blue): Appendix H.

AuthorsOpenReview ↗

Response to Reviewer MmD2 (part 2/2)

W2. The role of the redundancy branch is unclear. What evidence shows that subtracting its output is useful for suppressing visual redundancy and improving the model?

Thank you for raising this point. We address the two sub-questions separately.

1) Evidence that the redundancy branch captures redundant visual information.

We draw this conclusion from its response pattern and behavior. Structurally, it retains vanilla linear attention, and natural images contain substantial spatial redundancy, with many tokens providing overlapping evidence for visual decisions [1, 2], which is consistent with its smoothness property. Its response is the most diffuse: rank 28.6 and magnitude entropy 5.06, versus rank 150.5 and entropy 4.53 for the saliency branch; the top-pp measurements in W1 further show that ArA_r is the least concentrated map and D=As−λArD=A_s-\lambda A_r is the most concentrated. More importantly, in the redundancy-token experiment, our approach of subtracting the redundancy branch is the most robust to redundant-information damage among all measured models (W1), which is sufficient to show that our design is reasonable, achieves the redundancy branch’s ability to model visual redundancy, and effectively removes it.

2) Evidence that removing this redundancy is necessary.

Likewise, in the ablations and the supplementary redundancy-token experiment, our asymmetric differential model performs best, which is sufficient to show the necessity of removing redundancy.

We hope the above responses have adequately addressed your concerns. Please feel free to contact us if you have any further questions. If not, we would greatly appreciate your consideration for a higher score.

References

[1] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. “DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.” Advances in Neural Information Processing Systems (NeurIPS), 2021.

[2] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. “Masked Autoencoders Are Scalable Vision Learners.” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

Reviewer_MBTFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ADLA, which subtracts a global linear-attention branch from a branch using content-dependent group weighting. It also introduces token-wise kernel routing. Experiments on classification, detection, segmentation, and generation show improvements over the original backbones.

Critical strengths
  1. The method works across several backbones and tasks, with substantial speedups on high-resolution PVT models.
  2. The ablations include symmetric branches, a non-differential baseline, and static group weighting. These controls provide useful evidence for the design.
Critical weaknesses
  1. The redundancy explanation needs stronger evidence. Smooth attention can still carry useful context, and the redundancy test in Appendix B does not include ADLA. Figure 3 also does not explain how entropy is computed for differential weights that can be negative.
  2. The contribution over closely related methods is limited. GWS extends MHLA’s grouped summaries, while DyDiLA already uses dynamic power-kernel routing. The paper acknowledges these connections, but the small or mixed accuracy gains and missing runtime comparisons make the advantage over these methods unclear.
Questions for authors
  1. How is entropy computed for the signed attention weights in Figure 3?
  2. How do ADLA and the non-differential baseline perform in the redundancy experiment in Appendix B?
  3. Under comparable training budgets, how does ADLA compare with the closest methods in accuracy, latency, and memory?
7 comments
AuthorsOpenReview ↗

Response to Reviewer MBTF (part 1/3)

Thank you for the time and effort devoted to reviewing our paper, and for the constructive comments. We have carefully considered all the concerns and address them point‑by‑point below.

W1 & Q1. How is entropy computed for the signed differential attention weights in Fig. 3?

Thank you for raising this point. The differential operator D=As−λArD=A_s-\lambda A_r can contain negative entries. Consequently, DiD_i is not a probability distribution. For each row of DD, we first construct the normalized magnitude distribution πij(D)=∣Dij∣∑k∣Dik∣,\pi_{ij}(D)=\frac{|D_{ij}|}{\sum_k|D_{ik}|}, and then compute Hmag(Di)=−∑jπij(D)log⁡πij(D).H_{\mathrm{mag}}(D_i) =-\sum_j\pi_{ij}(D)\log\pi_{ij}(D).

HmagH_{\mathrm{mag}} is the standard Shannon entropy of that magnitude distribution. For a signed differential operator, this metric measures only how the absolute magnitude of differential attention map is distributed over key tokens: a lower HmagH_{\mathrm{mag}} indicates more concentrated mixing strength, while a higher value indicates a more diffuse magnitude pattern.

Our claim is therefore clarified: subtracting the smooth redundancy branch makes the effective attention magnitude distribution less diffuse and more concentrated, thereby enhancing the ability to focus on key tokens.

This treatment is also supported by closely related literature. Ye et al. [1] interprets the subtraction as canceling common-mode attention noise rather than as constructing another probability distribution. More directly, Kong et al. [2] analyze Differential Transformer by computing the entropy in their Fig. 2(c) from the renormalized absolute values of the final differential attention scores and explicitly interpret it as magnitude concentration, which is consistent with our measurement method.

Revisions (blue): magnitude entropy in Fig. 3 and App. G.3.

References

[1] Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. “Differential Transformer.” International Conference on Learning Representations (ICLR), 2025.

[2] Chaerin Kong, Jiho Jang, and Nojun Kwak. “Understanding Differential Transformer Unchains Pretrained Self-Attentions.” Advances in Neural Information Processing Systems (NeurIPS), 2025.

AuthorsOpenReview ↗

Response to Reviewer MBTF (part 2/3)

W1 & Q2. What supports the redundancy explanation, and how do ADLA and the non-differential baseline perform in the Appendix B redundancy experiment?

Thank you for raising this point. We clarify that our claim is not that the smooth attention doesn't carry useful context. We use “redundant” in a task-relative sense: a response may carry information while contributing little additional evidence for a particular prediction. Our claim is that the smooth response of linear attention can effectively model visual redundancy carrying a small amount of useful information, and that removing such redundancy enables the model to focus on key content, thereby improving task performance. Our evidence is as follows:

1) Response diagnostics and ablations.

Natural images contain substantial spatial redundancy, with many tokens providing overlapping evidence for visual decisions [3, 4]. This widespread, diffuse redundancy naturally aligns with the smoothing effect of linear attention, which motivates us to explicitly model such redundancy using a standard linear attention branch. Fig. 3 shows that weight magnitude is most diffuse in the redundancy branch: magnitude entropy is 5.06 in this branch, 4.53 in the saliency branch, and 4.27 after fusion. We further measure the top-pp magnitude concentration. For each query row, we rank the attention weights by their absolute magnitudes and calculate the fraction of the total absolute magnitude carried by the largest pp% of weights.

ModelTop-ppAsA_sArA_rD=As−λArD=A_s-\lambda A_rGain over AsA_s (pp)
DeiT-T-ADLA5%32.09%14.54%38.41%+6.32
10%43.27%24.72%50.31%+7.04
15%51.81%33.26%59.07%+7.26
PVT-T-ADLA5%32.10%22.02%39.22%+7.12
10%43.27%33.24%50.45%+7.18
15%52.31%42.78%59.45%+7.14
Swin-T-ADLA5%42.15%26.97%50.60%+8.45
10%54.35%40.56%62.43%+8.08
15%62.71%50.41%70.59%+7.88

As shown in the table, D=As−λArD=A_s-\lambda A_r consistently exhibits higher magnitude concentration than both branches across all architectures and all pp. Compared with AsA_s, differential subtraction improves the concentration by 6.32–8.456.32–8.45 percentage points. This provides direct evidence that subtracting the diffuse redundancy branch suppresses broadly distributed components and concentrates the attention weights on salient tokens. Meanwhile, Fig. 1 and Appendix F.1 visualize this sharper focus for individual queries. Consequently, at the task level, ADLA achieves 74.0% top-1 (Tab. 5), compared with 73.2% for symmetric GWS (Tab. 5) and 73.1% for full-dimensional non-differential GWS (Tab. 9). Together, these findings support our claim: fusion concentrates the spatial response and suppresses redundant information that offers limited additional task-relevant evidence, thereby yielding better performance.

2) Boundary-token redundancy experiment.

To evaluate our claim directly and whether ADLA exhibits stronger robustness against redundant information, we added ADLA, symmetric GWS, and non-differential GWS to the boundary-token redundancy experiment in Appendix B. The results are as follows:

Modelr=0r=1r=2r=3r=4r=5r=6Drop at r=6 (pp)
Softmax attention72.171.469.868.166.064.061.910.2
Linear attention70.268.866.163.159.556.052.617.6
Non-differential GWS (Tab. 9)73.169.363.761.561.557.654.418.7
Symmetric GWS (Tab. 5)73.272.170.868.866.764.562.410.8
ADLA74.073.372.570.668.266.364.39.7

ADLA shows a smaller accuracy drop than all controls. With 480 added tokens (r=6), ADLA loses 9.7 percentage points, compared with 10.8 for symmetric GWS and 18.7 for non-differential GWS. These supporting the robustness benefit from the asymmetric design and the proposed redundancy-removal explanation. Using redundant branches to model and eliminating visual redundancy to improve model performance is feasible.

Revisions (blue): aggregation experiments in Appendix H.2; boundary-token redundancy experiment in Appendix H.3.

References

[3] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. “DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.” Advances in Neural Information Processing Systems (NeurIPS), 2021.

[4] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. “Masked Autoencoders Are Scalable Vision Learners.” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

AuthorsOpenReview ↗

Response to Reviewer MBTF (part 3/3)

W2 & Q3. What is the contribution beyond DyDiLA and MHLA, and how does ADLA compare in accuracy, latency, and memory under comparable training budgets?

Thank you for raising this point. We clarified the differences among methods, and supplemented benchmark results. Metrics include: training latency per epoch, peak GPU memory, inference latency (batch size = 1), and throughput (batch size = 128). Training uses 4× A800 GPUs (global batch size = 1024); inference uses 1× A800 GPU (averaged over 1,000 runs, 100 warmups). For DiT-S/2, inference latency is one denoising step (1-NFE) excluding VAE decoding.

1) Contribution and controlled comparison with DyDiLA.

(i) Different design and focus. DyDiLA emphasizes extensive routing (projectors, kernels, coefficients); ADLA is the first, to our knowledge, to treat linear attention's smoothness as a redundancy prior, and propose an asymmetric-difference paradigm for vision.

(ii) Kernel routing differences. DyDiLA uses four independent routers (Q, K, Q', K'), while ADLA uses one on X, sharing an exponent for Q and K with lower overhead.

ModelKernel Routing SettingParams(M)FLOPs(G)Acc(%)Peak Mem (GB)Training Latency (min/epoch)Inference Latency (ms/image)Throughput (img/s)
DeiT-T (Diff. LA)DyDiLA-style Kernel Routing5.81.171.99.87.06.63262.7
DeiT-T (Diff. LA)Our Dynamic Kernel Routing5.71.172.09.74.45.93581.5

DyDiLA's complex routing increases costs without improving accuracy.

(iii) Task generality. DyDiLA was designed and validated only for diffusion generative models; ADLA, as a general visual backbone attention mechanism, achieves superior performance in image classification, object detection, semantic segmentation, and diffusion generation.

2) Regarding the comparison with MHLA.

Tab. 7 provides a controlled comparison by applying the MHLA to our saliency branch under the same framework and training setting. We report their costs below:

ModelParams(M)FLOPs(G)Acc(%)Peak Mem (GB)Training Latency (min/epoch)Inference Latency (ms/image)Throughput (img/s)
DeiT-T-MHLA (controlled)5.81.275.410.37.67.32770.4
DeiT-T-ADLA5.81.275.910.67.67.32766.8
DeiT-S-MHLA (controlled)224.580.521.217.67.31088.7
DeiT-S-ADLA224.580.821.417.77.31085.2

Under comparable costs, the proposed GWS performs better than MHLA within our architecture.

We further compare against the complete MHLA. MHLA's DeiT models additionally introduce CPE and replace the class token with average pooling over patch tokens. We align the structure of DeiT-ADLA with MHLA and retrain it. MHLA's costs are profiled using the official code (*: our measurement, †: original paper report).

ModelParams(M)FLOPs(G)Acc(%)Peak Mem (GB)Training Latency (min/epoch)Inference Latency (ms/image)Throughput (img/s)
DeiT-T-MHLA5.8*1.2* / 1.1†75.8†10.86.97.03007.7
DeiT-T-ADLA (MHLA-aligned)5.81.276.210.47.57.32777.6
DeiT-S-MHLA22*4.4* / 4.2†81.0†21.615.27.21216.4
DeiT-S-ADLA (MHLA-aligned)224.581.220.717.57.31087.9

In addition, we trained ADLA with CPE and gating for a fair comparison against the complete MHLA DiT-S/2:

ModelSettingParams(M)FLOPs(G)FIDPeak Mem (GB)Training Latency (min/epoch)Inference Latency (ms/image)Throughput (img/s)
DiT-S/2-MHLAw/ CPE33*5.7*64.0†39.717.213.41020.8
DiT-S/2-ADLAw/ CPE336.061.643.819.214.6945.1
DiT-S/2-MHLAw/ CPE+Gating35*6.2*59.8†46.518.815.3934.6
DiT-S/2-ADLAw/ CPE+Gating356.558.050.520.916.8855.1

Results show that, with all other architectural configurations held identical, ADLA outperforms the full MHLA in performance. Although some cost metrics still leave room for improvement, our main contribution lies in being, to our knowledge, the first work to exploit linear attention as a modeling prior for redundant information and to propose an asymmetric architecture, whose superiority we validate, thereby providing a reference for future work in the field of differential attention or linear attention.

Revisions (blue): all comparisons in Appendix I.

We hope the above responses have adequately addressed your concerns. Please feel free to contact us if you have any further questions. If not, we would greatly appreciate your consideration for a higher score.

Reviewer_fhoNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Asymmetric Differential Linear Attention (ADLA) for vision transformers. A saliency branch uses content-dependent group-weighted key–value summaries, while a second linear-attention branch provides a response that is subtracted from the first branch. Token-wise routing selects norm-preserving power kernels to control focusing strength. Choosing the number of groups proportional to the square root of the token count preserves linear complexity in sequence length. Evaluation covers image classification, object detection, semantic segmentation, and class-conditional image generation.

Critical strengths
  1. The component evaluation contains meaningful controlled comparisons. Table 5 excludes depthwise convolution throughout and supports asymmetric GWS over applying GWS to both branches: 74.0% versus 73.2%. Table 9 additionally compares against a non-differential, full-dimensional dynamic-kernel-plus-GWS configuration, which obtains 73.1% despite higher reported cost. Table 7 isolates dynamic group weighting from MHLA-style static mixing. The repeated-run results in Table 6 further support the stability of the reported ADLA accuracies. These experiments provide substantive evidence that the architectural combination is useful within the authors’ framework.

  2. The evaluation is broad and includes practical efficiency measurements. Improvements are reported across several backbone families and four task categories. For example, PVT-S classification improves from 79.8% to 82.2%, while full Mask R-CNN inference latency decreases from 57.56 to 40.11 ms at 1333×800 resolution. Table 8 reports latency, throughput, and training memory across multiple resolutions, including settings where memory consumption increases. This provides a more informative assessment of practical trade-offs than operation counts alone.

Critical weaknesses
  1. The central redundancy-removal interpretation is not adequately established, and its main quantitative diagnostic is underdefined. Equation 7 gives (A=A_s-\lambda A_r). Even when the individual branches are normalized, the difference can contain negative entries and its rows sum to (1-\lambda). Conventional Shannon entropy therefore cannot be applied directly. Nevertheless, Figure 3 interprets the reported entropy of 4.27 as evidence of sharper attention without specifying how negative entries and normalization are handled. In addition, the redundancy branch uses fewer query/key channels by construction, so its lower rank does not establish that it specifically captures task-irrelevant information.
  2. The incremental contribution and practical advantage over the closest methods remain insufficiently resolved. The paper appropriately discusses DyDiLA, which already uses token-specific routed power kernels, and distinguishes its token-level differencing from ADLA’s output-level asymmetric differencing. The main contribution therefore rests substantially on the asymmetric grouped architecture and its demonstrated benefits. Table 7 provides a useful component control, but does not establish an advantage over complete MHLA models: the original MHLA paper reports 81.0% on DeiT-S with 4.2G FLOPs, versus ADLA’s 80.80% with 4.5G. Architectural differences preclude a direct conclusion, but also leave the overall frontier unresolved. Runtime comparisons against original PVT/Swin backbones do not settle this question. For generation, the omitted training duration and sampling configuration further prevent a sufficiently controlled assessment of the reported advantage over prior methods.
Questions for authors
  1. How exactly is the entropy in Figure 3 computed for (A_s-\lambda A_r)? Please define the treatment of negative values and normalization, and report the learned subtraction coefficients or the prevalence of negative attention weights.
  2. Can the existing boundary-token redundancy experiment include ADLA and the symmetric/non-differential controls? This would directly test the proposed redundancy-removal explanation beyond ordinary validation accuracy and selected attention visualizations.
  3. Can you provide the exact generation training budget and sampling settings, together with an aligned accuracy–cost comparison against a complete close method or the Table 7 control? Please distinguish controlled component comparisons from comparisons between complete architectures.
6 comments
AuthorsOpenReview ↗

Response to Reviewer fhoN (part 1/3)

Thank you for the time and effort devoted to reviewing our paper, and for the constructive comments. We have carefully considered all the concerns and address them point‑by‑point below.

W1 & Q1. How is entropy computed for the signed differential attention weights in Fig. 3? Please specify how negative values are handled and how the weights are normalized, and report either the learned subtraction coefficients or the proportion of negative attention weights.

Thank you for raising this point. We first explain how we handle negative values, then report the learned subtraction coefficients and the fraction of negative differential attention weights. Finally, we provide a direct check of whether subtraction concentrates weight magnitude on fewer spatial positions.

1) Interpretation of magnitude entropy

The differential operator D=As−λArD=A_s-\lambda A_r can contain negative entries. When both branches are exactly row-normalized, satisfies ∑jDij=1−λ\sum_jD_{ij}=1-\lambda. Consequently, DiD_i is not a probability distribution. For each row of DD, we first construct the normalized magnitude distribution πij(D)=∣Dij∣∑k∣Dik∣,\pi_{ij}(D)=\frac{|D_{ij}|}{\sum_k|D_{ik}|}, and then compute Hmag(Di)=−∑jπij(D)log⁡πij(D).H_{\mathrm{mag}}(D_i)=-\sum_j\pi_{ij}(D)\log\pi_{ij}(D).

HmagH_{\mathrm{mag}} is the Shannon entropy of that magnitude distribution. This metric measures only how the absolute magnitude of differential attention map is distributed over key tokens: a lower HmagH_{\mathrm{mag}} indicates more concentrated mixing strength, while a higher value indicates a more diffuse magnitude pattern.

This treatment is also supported by related literature. Ye et al. [1] interprets the subtraction as canceling common-mode attention noise rather than as constructing another probability distribution. Kong et al. [2] analyze Differential Transformer by computing the entropy in their Fig. 2(c) from the renormalized absolute values of the final differential attention scores and explicitly interpret it as magnitude concentration, which is consistent with our measurement method.

2) Subtraction coefficient λ\lambda and fraction of negative weights

We analyzed 1,000 ImageNet validation images on DeiT-T-ADLA, PVT-T-ADLA, and Swin-T-ADLA, including every ADLA layer. We average the statistics over the images, attention heads, and queries within each layer, and then give each layer equal weight. We report the mean and range of the subtraction coefficient λ\lambda across layers, the fraction of negative attention weights (Dij<0D_{ij}<0), and the positive-to-negative magnitude ratio R±=∑ijmax⁡(Dij,0)∑ijmax⁡(−Dij,0)R_{\pm}=\frac{\sum_{ij}\max(D_{ij},0)}{\sum_{ij}\max(-D_{ij},0)}:

ModelAverage λ\lambdaλ\lambda rangeFraction of negative weightsR±R_{\pm}
DeiT-T-ADLA0.6751[0.3709, 1.4751]31.83%5.66
PVT-T-ADLA0.6912[0.0713, 1.2565]22.93%8.74
Swin-T-ADLA0.6987[0.1809, 1.3208]30.34%6.51

Although 22.9322.93%–31.8331.83% of the coefficients are negative, their total magnitude is not dominant: the positive magnitude is 5.66×–8.74×5.66\times–8.74\times the negative magnitude.

3) Direct check of spatial concentration.

In order to further demonstrate the response generated by our asymmetric differential method, we further measure the top-pp magnitude concentration. For each query row, we rank the attention weights by their absolute magnitudes and calculate the fraction of the total absolute magnitude carried by the largest pp% of weights.

ModelTop-ppAsA_sArA_rD=As−λArD=A_s-\lambda A_rGain over AsA_s (pp)
DeiT-T-ADLA5%32.09%14.54%38.41%+6.32
10%43.27%24.72%50.31%+7.04
15%51.81%33.26%59.07%+7.26
PVT-T-ADLA5%32.10%22.02%39.22%+7.12
10%43.27%33.24%50.45%+7.18
15%52.31%42.78%59.45%+7.14
Swin-T-ADLA5%42.15%26.97%50.60%+8.45
10%54.35%40.56%62.43%+8.08
15%62.71%50.41%70.59%+7.88

D=As−λArD=A_s-\lambda A_r consistently exhibits higher magnitude concentration than both branches across all architectures and all pp. Compared with AsA_s, differential subtraction improves the concentration by 6.32–8.456.32–8.45 percentage points. This provides direct evidence that subtracting the diffuse redundancy branch suppresses broadly distributed components and concentrates the attention weights on salient tokens.

Revisions (blue): magnitude entropy in Fig. 3 and Appendix G.3; subtraction coefficient and negative weights in Appendix H.1; spatial aggregation experiments in Appendix H.2.

References

[1] Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. “Differential Transformer.” International Conference on Learning Representations (ICLR), 2025.

[2] Chaerin Kong, Jiho Jang, and Nojun Kwak. “Understanding Differential Transformer Unchains Pretrained Self-Attentions.” Advances in Neural Information Processing Systems (NeurIPS), 2025.

AuthorsOpenReview ↗

Response to Reviewer fhoN (part 2/3)

W1 & Q2. The redundancy branch uses fewer query/key channels, so its lower rank alone cannot establish that it captures task-irrelevant information. Can the boundary-token experiment include ADLA and the symmetric/non-differential controls?

Thank you for raising this point. The impact of Q/K channels on rank depends on the specific attention computation. While both ADLA branches receive half of the projected Q/K channels (Sec. 4.3), their behaviors differ: the redundancy branch is bounded by the standard linear attention formulation (Eq. 3), whereas the saliency branch uses GWS to weight local KV summaries separately for each query group (Eq. 11), which makes the spatial weighting depend on the pair of groups as well as the token-level kernel similarity. Consequently, this group-wise interaction circumvents the channel bottleneck, thereby elevating the upper bound of the rank.

Our interpretation of the redundancy branch draws on its structure, the spatial distribution of its response, and the classification ablations. We also extended the boundary-token redundancy experiment in Appendix B to ADLA, symmetric GWS, and the non-differential control to test the redundancy explanation under a controlled perturbation. The details are as follows:

1) Evidence for the redundancy interpretation. We use redundant in a task-relative sense: a response may carry information yet contribute little additional evidence for a particular prediction. In natural images, many tokens contribute little additional evidence to a given visual decision and visual redundancy is pervasive [3] [4]. This widespread, diffuse redundancy naturally aligns with the smoothing effect of linear attention, which motivates us to explicitly model such redundancy using a standard linear attention branch.

To validate this design, our experiments and visualizations reveal distinct response patterns between the two branches and the effect of their subtraction. Fig. 3 shows that weight magnitude is most diffuse in the redundancy branch: magnitude entropy is 5.06 in this branch, 4.53 in the saliency branch, and 4.27 after fusion. The analysis in W1 & Q1 makes this change concrete: on DeiT-T, the largest 10% of weights carry 24.72%, 43.27%, and 50.31% of the total magnitude in these three maps. Fig. 1 and Appendix F.1 visualize this sharper focus for individual queries. Consequently, at the task level, ADLA achieves 74.0% top-1 (Tab. 5), compared with 73.2% for symmetric GWS (Tab. 5) and 73.1% for full-dimensional non-differential GWS (Tab. 9). Together, these findings support the rationale for using the redundancy branch as a subtraction reference: fusion concentrates the spatial response and suppresses redundant information that offers limited additional task-relevant evidence, thereby yielding higher classification accuracy than both controls. This validates two independent aspects of our design: the redundancy branch effectively models redundant information (i.e., content carrying less useful information), and the asymmetric differential architecture effectively removes it, ultimately improving model performance.

2) Boundary-token redundancy experiment. To evaluate whether ADLA exhibits stronger robustness against redundant information, we added ADLA, symmetric GWS, and non-differential GWS to the boundary-token redundancy experiment in Appendix B. The results are as follows:

Modelr=0r=1r=2r=3r=4r=5r=6Drop at r=6 (pp)
Softmax attention72.171.469.868.166.064.061.910.2
Linear attention70.268.866.163.159.556.052.617.6
Non-differential GWS (Tab. 9)73.169.363.761.561.557.654.418.7
Symmetric GWS (Tab. 5)73.272.170.868.866.764.562.410.8
ADLA74.073.372.570.668.266.364.39.7

ADLA shows a smaller accuracy drop than all controls. With 480 added tokens (r=6), ADLA loses 9.7 percentage points, compared with 10.8 for symmetric GWS and 18.7 for non-differential GWS. These supporting the robustness benefit from the asymmetric design and the proposed redundancy-removal explanation. Using redundant branches to model and eliminating visual redundancy to improve model performance is feasible.

Revisions (blue): boundary-token redundancy experiment in Appendix H.3.

References

[3] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. “DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification.” Advances in Neural Information Processing Systems (NeurIPS), 2021.

[4] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. “Masked Autoencoders Are Scalable Vision Learners.” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

AuthorsOpenReview ↗

Response to Reviewer fhoN (part 3/3)

W2 & Q3. What does ADLA contribute beyond DyDiLA and MHLA? How does it compare with a complete close method in accuracy and cost, and under what training and sampling conditions were the generation results obtained?

Thank you for raising this point. We clarified the differences among methods, and supplemented benchmark results. Metrics include: training latency per epoch, peak GPU memory, inference latency (batch size=1), and throughput (batch size=128). Training uses 4×A800 GPUs (global batch size = 1024); inference uses 1×A800 GPU (averaged over 1000 runs, 100 warmups). For DiT-S/2, inference latency is one denoising step (1-NFE) excluding VAE decoding. We address each concern below:

1) Regarding the comparison with DyDiLA.

(i) Different design and focus. DyDiLA emphasizes extensive routing (projectors, kernels, coefficients); ADLA is the first, to our knowledge, to treat linear attention's smoothness as a redundancy prior, and propose an asymmetric-difference paradigm for vision.

(ii) Kernel routing differences. DyDiLA uses four independent routers (Q, K, Q', K'), while ADLA uses one on X, sharing an exponent for Q and K with lower overhead.

ModelKernel Routing SettingParams(M)FLOPs(G)Acc(%)Peak Mem (GB)Training Latency (min/epoch)Inference Latency (ms/image)Throughput (img/s)
DeiT-TDyDiLA-style5.81.171.99.87.06.63262.7
DeiT-TOurs5.71.172.09.74.45.93581.5

DyDiLA's complex routing increases costs without improving accuracy.

2) Regarding the comparison with MHLA.

Tab. 7 provides a controlled comparison by applying the MHLA to our saliency branch under the same framework and training setting. We report their costs below:

ModelParams(M)FLOPs(G)Acc(%)Peak Mem (GB)Training Latency (min/epoch)Inference Latency (ms/image)Throughput (img/s)
DeiT-T-MHLA (controlled)5.81.275.410.37.67.32770.4
DeiT-T-ADLA5.81.275.910.67.67.32766.8
DeiT-S-MHLA (controlled)224.580.521.217.67.31088.7
DeiT-S-ADLA224.580.821.417.77.31085.2

Under comparable costs, the proposed GWS performs better than MHLA within our architecture.

We further compare against the complete MHLA. MHLA's DeiT models introduce CPE and replace the class token with average pooling over patch tokens. We align the structure of DeiT-ADLA with MHLA and retrain it. MHLA's costs are profiled using the official code (*: our measurement, †: original paper report).

ModelParams(M)FLOPs(G)Acc(%)Peak Mem (GB)Training Latency (min/epoch)Inference Latency (ms/image)Throughput (img/s)
DeiT-T-MHLA5.8*1.2* / 1.1†75.8†10.86.97.03007.7
DeiT-T-ADLA (MHLA-aligned)5.81.276.210.47.57.32777.6
DeiT-S-MHLA22*4.4* / 4.2†81.0†21.615.27.21216.4
DeiT-S-ADLA (MHLA-aligned)224.581.220.717.57.31087.9

In addition, we trained ADLA with CPE and gating for a fair comparison against the complete MHLA DiT-S/2:

ModelSettingParams(M)FLOPs(G)FIDPeak Mem (GB)Training Latency (min/epoch)Inference Latency (ms/image)Throughput (img/s)
DiT-S/2-MHLAw/ CPE33*5.7*64.0†39.717.213.41020.8
DiT-S/2-ADLAw/ CPE336.061.643.819.214.6945.1
DiT-S/2-MHLAw/ CPE+Gating35*6.2*59.8†46.518.815.3934.6
DiT-S/2-ADLAw/ CPE+Gating356.558.050.520.916.8855.1

Under identical configurations, ADLA outperforms MHLA in performance. Although some cost metrics still leave room for improvement, our main contribution lies in being, to our knowledge, the first work to exploit linear attention as a modeling prior for redundant information and to propose an asymmetric architecture, whose superiority we validate.

3) Generation training budget and sampling protocol.

Specifically, both DiT baselines and DiT-ADLA were trained from scratch on ImageNet-1K for 400K steps using a global batch size of 256, AdamW, a constant LR of 1×10−41\times10^{-4}, and a linear DDPM with 1,000 timesteps. They share identical settings except for replacing self-attention with ADLA. For sampling, we used a 250-step ancestral DDPM sampler with CFG scales of 1.0 (unguided) and 4.0 (guided). We generated 50K images per model and computed FID-50K, sFID, IS, Precision, and Recall using the official OpenAI ADM evaluation code and ImageNet-256 reference statistics.

Revisions (blue): Appendix I.

We hope the above responses have adequately addressed your concerns. Please feel free to contact us if you have any further questions. If not, we would greatly appreciate your consideration for a higher score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.