acceptodds
Under review as a conference paper at ICLR 2027

Depthwise Channel-Asymmetric Attention: Investigating Capacity, Scaling, and Uses

Abstract

Vision transformers embed image patches by summing over input channels, which assumes every channel measures the same quantity at the same point. This fails for instruments whose channels are separate sensors, such as calorimeter detectors, where electromagnetic and hadronic channels are physically distinct. DepthViT is an architecture which attends along the channel axis instead of the token axis. We extend this model for more general image tasks by restoring the resulting loss of spatial communication with sparse hierarchical attention pooling (HAP) blocks and a structure-preserving head. At a matched 22M budget on the HLS4ML Large Hadron Collider jet benchmark, DepthViT2 reaches 75.34% top-1 against 73.98% for a channel-symmetric ViT-Small, winning every class and using fewer FLOPs; on CIFAR-100 it leads 62.76% to 52.92%. The advantage is budget-dependent: across a ladder from 164K to 22M parameters the comparison changes sign, losing at 164K (-1.56 points), tying at 1M, and winning at 5.4M and 22M (+1.69, +1.36). We trace this to parameter allocation, since the channel operator scales quadratically in the capacity knob k while the feed-forward block is linear, and confirm it with a same-scaffold control showing a parameter-matched symmetric operator matches accuracy, so asymmetry is the cheapest configuration rather than the most accurate one. A scale-adjusted positional encoding further lets one trained model accept many input shapes, beating a fixed-crop ViT that discards an average of 44% of each image. Finally, matched FLOPs coexist with a 4x wall-clock cost that a fused Triton kernel removes at 71x on the isolated operator.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
vision transformers, channel-wise attention, efficient architectures, scientific machine learning, jet classification, variable-resolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kjkgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper extends DepthViT, which applies attention across sensor channels, with spatial communication between image regions. It compares the resulting model with ViTs at matched parameter budgets, finding that its relative performance improves from a disadvantage at small budgets to an advantage at larger budgets. It also investigates channel asymmetry, variable image sizes, and runtime optimization. Validation includes jet classification, natural-image experiments, component tests, and an isolated fused-kernel benchmark.

Critical strengths
  1. The matched-budget evaluation provides useful empirical evidence. Comparing several model sizes under the same training protocol is more informative than comparing a small proposed model against a much larger standard ViT. Repeated runs support the advantages at the larger budgets and show that the relative performance depends on model size.

  2. The same-scaffold controls help qualify the role of channel asymmetry. At 22M parameters, replacing the asymmetric operator with a symmetric alternative yields similar accuracy. This is an informative negative result, and the authors appropriately acknowledge that asymmetry itself does not explain the accuracy advantage at this budget.

Critical weaknesses
  1. The experiments do not sufficiently establish the contribution of the new components or explain the gains. Spatial mixing and the classification head are tested separately on ImageNet, but their contributions are not isolated on the main jet-classification task. In the reported spatial-mixing experiment, the advantage decreases from 3.3 percentage points at epoch 30 to 0.1 at epoch 40. The head's reported 5.3-point improvement is accompanied by limited experimental details. Furthermore, increasing model width changes several components simultaneously, so the parameter-allocation analysis does not establish which changes cause the performance crossover. These gaps limit the evidence for the architectural contributions and their proposed explanation.

  2. The evidence for the claimed practical advantages is incomplete. The full model trains approximately four times slower than ViT, while the reported 71x acceleration concerns an isolated forward operator that has not been integrated into training. The variable-resolution experiment also changes both architecture and preprocessing: the proposed model receives uncropped, bucketed images, while ViT uses fixed crops and different augmentation. Consequently, the experiments establish neither an end-to-end speed advantage nor an architectural advantage under comparable image-processing conditions.

  3. One thing that confuses me is that the authors claims without HAP blocks, the new architecture achieves 49.0% and 56.3% for ImageNet-1K sweep. How come only channel-wise attention without spatial information exchange could perform well for image classification. Or do I missing something in the paper?

Questions for authors
  1. For the classification-head comparison, what encoder configuration, parameter counts, training settings, and number of runs were used? Were all other components held fixed? Are component-wise results available for the main jet-classification task?
  2. Is parameter allocation intended as a possible explanation or an experimentally established mechanism? What evidence distinguishes its effect from the simultaneous changes in spatial mixing and representation width?
  3. Are full-model training measurements available with the fused operator integrated? If not, can the efficiency conclusions be explicitly limited to FLOPs and isolated-operator performance?
  4. Are results available for a ViT using the same image buckets and augmentation pipeline? Without this comparison, what evidence attributes the variable-resolution gain to the architecture rather than the different inputs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer kjkg: isolation control, allocation vs width, efficiency now measured end to end

Thank you for the review. We address each point and mark where we could not run the requested experiment in time.

  1. HAP and head versus channel attention (isolation). New experiment: same embedding, HAP blocks and head, with only the operator swapped for parameter-matched token attention. Channel attention wins at both tiers: 75.12 ± 0.13 vs 74.35 ± 0.11 (5.4M) and 75.34 ± 0.20 vs 74.83 ± 0.27 (22M), three seeds. The scaffold alone does not account for the accuracy (Sec. 3.2).

  2. Allocation versus width. At a fixed 1M budget and fixed residual width, widening only channel attention's inner dimension (paid for by the feed-forward block) raises accuracy from 72.37 ± 0.41 to 74.15 ± 0.10. At 164K it does not help, within seed spread (Sec. 4).

  3. Imagewoof arms differ in preprocessing and augmentation. We agree, and did not have time to rerun with swapped pipelines. The revision now states that the ViT arm was not tuned and keeps the augmentation caveat. Tuning would likely change performance of both. On jets and CIFAR-100, tuning improved ViT results. On CIFAR-100 where we had time to tune DepthViT2 as well, it improved performance. We restate the result as a capability, one trained model accepting five input shapes without crop or positional-table surgery, rather than as an accuracy margin.

  4. HAP and the head are only ablated on ImageNet; HAP gives no gain by epoch 40. Agreed on epoch 40. The paper's claim is faster early learning at an unchanged ceiling, not a higher ceiling, and the paper states this in Sec. 2. Jets ablations of HAP and the head did not fit in the discussion period. The operator swap above partly addresses the related question of whether the scaffold carries the result. We will add full details of the 5.3-point head result (encoder configuration, parameter counts of both heads, epochs, seeds and what was held fixed) to the appendix.

  5. How does a model without HAP reach 49.0 / 56.3% on ImageNet? Without HAP, tokens do not exchange information inside the encoder, but the model is still a bag of local features: each p-by-p patch is embedded, processed by per-token layers, and the head attention-pools over all tokens, so global aggregation happens at readout. Bag-of-local-features models are known to reach this accuracy range on ImageNet (BagNet; Brendel & Bethge, 2019). We will add this explanation to the paper.

  6. Efficiency claims are under-evidenced. We agree the kernel claim was overstated: the original Triton kernel was 71x faster only on the isolated forward operator and never ran in training, so we now measure end-to-end training instead. For C=2 the channel softmax is a sigmoid, so we rewrote the operator in closed form (DepthViT2 Fast Op), which torch.compile fuses into Triton kernels for the forward and backward passes. It matches the original to ≤7e-7 in fp32 and gives matching loss curves on real data. Through our training pipeline on one A100 (300 batches of 128, data loading included), DepthViT2-22M takes 47 s with Fast Op vs 181 s with the original operator and 36 s for ViT-Small: 3.9x faster, and 1.3x the cost of ViT-Small instead of 5.0x. Per GPU step, with both models compiled whole, the gap is 1.7x, narrowing to 1.59x at 1,600 tokens. A second FLOP count (PyTorch FlopCounterMode) gives 4.28 vs 4.30 GFLOPs, matching Table 2. Details are in Sec. 6 and App. F; all accuracy results use the original operator.

  7. Corrections. ViT-Small has 21.41M parameters (DepthViT2 is 3.0% larger at 22M), the 5.4M SGD ViT-Tiny entry is a single seed, and the learning-rate schedule cycles identically for every arm. All three are disclosed in the revision.

We would be grateful if you would reconsider your assessment in light of the new controls and the narrower claims.

Thanks for addressing my raised concerns. We still don't understand why the fully-depth wise model with no HAP blocks could catch up with the pattern (DDD*H)*4 on ImageNet at epoch 40. It seems like the inserted HAP blocks has no impact, which is against my intuition. I will keep the original score.

Reviewer_TRmrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DepthViT2 that extends DepthViT to more general image tasks. It attends to channel axis and efficiently mixes the spatial axis with “hierarchical attention pooling” (HAP) blocks for applications where channels are separate sensors. A structure-preserving attention-pooling head replaces flatten-and-pool, and a 2D positional index into a fixed maximum grid lets one model accept several aspect ratios. Experiments are shown on HLS4ML Large Hadron Collider jet benchmark.

Critical strengths

The proposed method compares at equal parameter budgets across a ladder of sizes which is a plus for niche problems like LHC jet benchmark that uses different sensors for different channels.

Critical weaknesses
  • The paper is hard to read. It uses non-standard terms like “funding the mechanism”, “slot-aware head”, “careful null”, “reported honestly” etc. The HAP formulation (Eq. 4) is hard to follow: it mixes micro and macro shifts, window summaries, and three broadcast terms without a clear figure.

  • Table 1 shows results only using ViT and do not study matched budget-comparisons against Swin, ConvNeXt, or a ResNet-18/ResNet-50 CNNs.

  • The paper is motivated by preserving the identity of physically distinct sensor channels but Table 3 shows that the three different channel treatments don’t differ much.

  • The paper motivates the method for general image tasks but do not compare it on standard Imagenet-1K dataset or object detection and segmentation benchmarks.

  • The proposed method is not novel as channel attention methods have been studied previously. XCiT (El-Nouby et al., NeurIPS 2021) is cited only briefly. DaViT (Ding et al., ECCV 2022) alternates channel attention with spatial window attention, much like the D/H interleaving here. ChannelViT (Bao et al., ICLR 2024) also studies the problem of “channels are separate measurements” setting, with per-channel patch tokens for microscopy and satellite imagery. These are not cited and compared.

  • The efficiency claims in Section 6 are overstated. The abstract says the fused kernel “removes” the 4× wall-clock cost. The kernel is benchmarked only on the isolated operator against an eager baseline running at 2.6 GB/s (under 0.2% of peak bandwidth), and it is not integrated into training, so the end-to-end speedup is unknown. As it stands, DepthViT2-22M trains about 4× slower than ViT-Small at matched FLOPs.

  • Presentation issues. The abstract and conclusion mix single-seed and three-seed numbers. +1.69 at 5.4M is single-seed (the three-seed mean gives +1.82), while +1.36 at 22M is three-seed (single-seed: +0.98). Key findings are spread between the main text and appendices

  • Appendix A reports that ECAL images are 99.9999% zeros (vs. 99.8% for HCAL), so one of the two channels is almost always empty, and the “two physically distinct sensors” setting is close to single-channel in practice. The results in Table 3 are consistent with this.

Questions for authors
  • See weaknesses above.
  • What is the final ImageNet-1K accuracy of DepthViT2 compared with a matched ViT under the same recipe?
  • What is the end-to-end training throughput with the fused kernel integrated?
2 comments
AuthorsOpenReview ↗

Response to Reviewer TRmr: related work, tuned baselines, narrowed scope, DepthViT2 Fast Op timing

Thank you for the review. Several of your points changed the paper.

  1. Novelty and related work. The revision cites DaViT and ChannelViT and states how each differs. DaViT attends across latent channels for computational reasons. ChannelViT keeps input channels separate but as extra tokens under standard token attention. Ours attends across a few physically distinct input streams, each with a private embedding. XCiT is discussed alongside them.

  2. Missing baselines. A parameter-matched ResNet-9 is in App. C (60.40% vs 70.14% at 164K). We have added AdamW-tuned ViTs at every budget. Against them DepthViT2 trails at every jets budget (by 6.06 points at 164K, narrowing to 0.32 at 22M), and we have revised the claims to near-parity at larger budgets. We did not run Swin or ConvNeXt due to limited time and compute resources and instead focused on strengthening the ViT baselines with tuned settings.

  3. Motivated as general-image, but no ImageNet-1K, detection or segmentation results. Agreed. We narrowed the scope to multi-sensor and small-image data and removed the general-image framing. Due to compute limitations, we were unable to run full ImageNet-1K and 22K runs.

  4. Per-channel patch weights and chansum. Agreed. The motivation is softened: a strided convolution keeps per-channel filters before the sum, and chansum shows this costs little here.

  5. ECAL is almost all zeros. Agreed: on this dataset ECAL is extremely sparse, and the chansum result is consistent with it adding little. We did not run an ECAL-zeroed ablation in the discussion period and do not claim that ECAL carries independent signal on this benchmark.

  6. The fused kernel "removes" the cost. Corrected, and we have since closed most of the gap. The original Triton kernel was 71x faster only on the isolated forward operator and never ran in training. For C=2 the channel softmax is two-way, i.e. a sigmoid, so we rewrote the operator in closed form (DepthViT2 Fast Op); torch.compile fuses its forward and backward passes into Triton kernels. It computes the same function: fp32 outputs and gradients of the full 22M model match the original to ≤7e-7, and a 300-batch training run on jets data from the same seed gives matching loss curves. Through our training pipeline on one A100 (300 batches of 128, data loading included), DepthViT2-22M takes 47 s with Fast Op vs 181 s with the original operator and 36 s for ViT-Small: 3.9x faster, and 1.3x the cost of ViT-Small instead of 5.0x. With both models compiled whole, a GPU step is 1.7x slower than ViT-Small's; the remainder comes from DepthViT2's deeper, narrower shape (22 blocks vs 12 at equal FLOPs) and small per-channel matrix multiplies at 100 tokens. The abstract, Sec. 6 and App. F now report these numbers and state that all accuracy results were produced with the original operator.

  7. Mixed single-seed and three-seed numbers. Fixed. Every headline number is a three-seed mean; the remaining single-seed entries are marked. We also corrected a ViT-Small parameter count (21.41M, not 22.01M) and disclosed a learning-rate schedule issue that affects every arm identically.

  8. Jargon; Eq. 4 has no figure. We removed the informal phrases you flagged (e.g., "funding the mechanism"); "slot-aware head" is kept but defined where it is introduced. A HAP diagram did not fit within the page limit in this revision; we will add it to the appendix in a final version.

We would be grateful if you would reconsider your assessment in light of these revisions.

Thank you to the authors for the rebuttal. The FastOp rewrite shows improvement over the original operator. Beyond that, my major concerns are not addressed and still exists. DepthViT2 now trails at matched budget comparisons. Table 3 shows that the private per-channel control does not drive accuracy which is the main point of distinction from other methods like DaViT and ChannelViT. The ECAL benchmark is sparse which makes it effectively single-channel and no ECAL-zeroed ablation was provided. So, I maintain my original rating.

Reviewer_1x47OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DepthViT2, a vision transformer for inputs with physically distinct sensor channels. It replaces spatial token attention with channel-axis attention and restores spatial communication through hierarchical attention pooling (HAP) blocks. At matched parameter and FLOP budgets on an LHC jet benchmark, DepthViT2 underperforms a ViT at small scales but outperforms it at larger scales. Their same-scaffold controls suggest that channel asymmetry is the cheapest configuration rather than the most accurate one. Additional experiments validate generalization performance to natural images, variable input resolutions, and runtime cost.

Critical strengths
  1. Rigorous matched-budget evaluation.
  • Parameters and FLOPs are closely matched across four capacity tiers, providing a more informative comparison.
  • A closed-form parameter model exactly reproduces the first three tiers.
  1. Detailed analysis of architectural costs.
  • The paper quantifies how parameters shift across components as the model scales.
  • The paper diagnoses the gap between theoretical FLOPs and actual runtime, and also demonstrates a fused-kernel remedy on the isolated operator.
Critical weaknesses
  1. The contribution of channel-axis attention is not isolated.
  • Same-scaffold controls: Removing channel asymmetry does not seem to change accuracy much at 22M. The authors acknowledge this, but it is still unclear whether channel-axis attention helps beat ViT. A baseline with token attention and the same HAP blocks and head would answer this, but it is missing.
  • CIFAR-100 results: The large accuracy gap (62.76% vs. 52.92%) does not seem to come from the proposed operator. It more likely comes from the HAP blocks and head, or an undertuned ViT baseline.
  • Motivation: A standard patch embedding already uses separate learned weights for each input channel, so it does not seem to lose sensor identity. The channel-summing control also performs about as well as the full model. Therefore, it is not yet clear that keeping channels separate brings a real benefit.
  1. The crossover explanation is confounded.
  • As k grows, the share of channel attention, the share of HAP, and the residual width all grow together. DepthViT2’s residual width becomes close to the ViT’s right around the crossover. Therefore, the loss at small budgets seems to come from a narrow residual width rather than from too few parameters in channel attention. Other scaling rules, such as growing the FFN width with k, are not tested.
Questions for authors
  1. How does a token-attention model with the same HAP blocks and head compare with DepthViT2 at matched parameter and FLOP budgets?
  2. Can the parameter-allocation hypothesis be distinguished from the residual-width hypothesis, for example through width-matched comparisons or alternative FFN scaling rules?
  3. Were the ViT baselines trained with the same SGD settings as DepthViT2? If so, do the crossover and the CIFAR-100 advantage still hold when the ViT is trained with settings tuned for it?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1x47: operator-swap control, allocation test, tuned ViT baselines

Thank you for the review. We attempted to address all three of your concerns.

  1. Channel attention is not isolated from HAP and the head. We kept the private embedding, HAP blocks, head and protocol fixed and replaced only the channel-attention operator with standard multi-head token self-attention, re-solving k to match parameters (within 0.4%). Three seeds each.

    • At 5.4M params, Channel attention: 75.12 ± 0.13 vs. Token attention with the same scaffold: 74.35 ± 0.11
    • At 22M params, Channel attention: 75.34 ± 0.20 vs. Token attention with the same scaffold: 74.83 ± 0.27

    The uncertainty bands don't overlap. With the chansum/symmix control, this places the accuracy in mixing along the channel axis (asymmetric or symmetric), not in the scaffold alone (new Sec. 3.2, App. G).

  2. The crossover is confounded by residual width. We tested allocation directly at a fixed budget and fixed residual width. At 1M (k=23), we widen only channel attention's inner dimension by a factor e and pay for it with a narrower feed-forward block (768 → 644 / 479). This raises the mechanism's share from 4.6% to about 18% / 37%. Accuracy goes from 72.37 ± 0.41 to 74.15 ± 0.10 (e=4) and 73.88 ± 0.53 (e=8). Moving parameters into the mechanism therefore helps with width unchanged. At 164K the same move does not help (67.59 ± 1.88 vs 69.15 ± 2.00, within seed spread), and we say so (Sec. 4).

  3. The ViT baseline may be under-tuned. This was a legitimate concern that turned out to be the case. We changed to AdamW and did a learning rate scan for ViT. Unfortunately, due to lack of time and limited compute resources we weren't able to carry out the same detailed scan with DepthViT2 so we defaulted to using the optimal settings for ViT which could potentially underreport the DepthViT2 results. With AdamW (wd 0.05, lr in {5e-4, 1e-3, 2e-3}, three seeds per rate), the tuned ViT beats DepthViT2 at every jets budget:

    TierDepthViT2 (best recipe)ViT, tuned
    164K69.15 ± 2.0075.21 ± 0.06
    1M72.37 ± 0.4175.39 ± 0.24
    5.4M75.12 ± 0.1375.57 ± 0.08
    22M75.34 ± 0.2075.66 ± 0.04

    DepthViT2 trained with AdamW at ViT's best rate does worse than with SGD (it peaks at epoch 10–12, then overfits), so each model is reported at its best recipe. This supports that using the ViT settings for DepthViT2 might underreport its accuracy. We have removed the claim that DepthViT2 wins at 5.4M and 22M. The paper now reports both protocols and claims a deficit that closes from 6.06 to 0.32 points as channel attention's share grows. On CIFAR-100 we were able to tune both ViT and DepthViT2, DepthViT2 leads 63.91 ± 0.38 vs 56.75 ± 0.34.

  4. The patch embedding already keeps per-channel weights. Agreed. We softened the motivation: a strided convolution applies per-channel filters before summing, and chansum shows the early sum costs little here.

  5. Additional corrections. We found some other additional corrections while running these tests and have added them to the paper. The 22M ViT-Small has 21.41M parameters (not 22.01M), so DepthViT2 is 3.0% larger there. The 5.4M SGD ViT-Tiny entry is a single seed. Our cosine schedule cycles because its horizon was computed with too few steps, identically for every arm. All three are disclosed in the revision.

We hope that the other features of our results justify acceptance. Those being comparable performance on Jets, better performance on CIFAR where we also had time to fully tune the DepthViT2 model as well as the ViT, and flexibility of one trained model to accept images of different shapes and aspect ratios (any patch-aligned size up to 480x480) without cropping, during both training and inference. The revised, narrower claims are supported by the evidence to the best of what we could do under limited time and would be grateful if you would reconsider your assessment in light of them.

Thank you for the detailed response and the additional experiments. However, with the tuned ViT, DepthViT2 no longer outperforms ViT at any jets budget, which removes the main empirical claim. Therefore, I keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.