- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper extends DepthViT, which applies attention across sensor channels, with spatial communication between image regions. It compares the resulting model with ViTs at matched parameter budgets, finding that its relative performance improves from a disadvantage at small budgets to an advantage at larger budgets. It also investigates channel asymmetry, variable image sizes, and runtime optimization. Validation includes jet classification, natural-image experiments, component tests, and an isolated fused-kernel benchmark.
Critical strengths
-
The matched-budget evaluation provides useful empirical evidence. Comparing several model sizes under the same training protocol is more informative than comparing a small proposed model against a much larger standard ViT. Repeated runs support the advantages at the larger budgets and show that the relative performance depends on model size.
-
The same-scaffold controls help qualify the role of channel asymmetry. At 22M parameters, replacing the asymmetric operator with a symmetric alternative yields similar accuracy. This is an informative negative result, and the authors appropriately acknowledge that asymmetry itself does not explain the accuracy advantage at this budget.
Critical weaknesses
-
The experiments do not sufficiently establish the contribution of the new components or explain the gains. Spatial mixing and the classification head are tested separately on ImageNet, but their contributions are not isolated on the main jet-classification task. In the reported spatial-mixing experiment, the advantage decreases from 3.3 percentage points at epoch 30 to 0.1 at epoch 40. The head's reported 5.3-point improvement is accompanied by limited experimental details. Furthermore, increasing model width changes several components simultaneously, so the parameter-allocation analysis does not establish which changes cause the performance crossover. These gaps limit the evidence for the architectural contributions and their proposed explanation.
-
The evidence for the claimed practical advantages is incomplete. The full model trains approximately four times slower than ViT, while the reported 71x acceleration concerns an isolated forward operator that has not been integrated into training. The variable-resolution experiment also changes both architecture and preprocessing: the proposed model receives uncropped, bucketed images, while ViT uses fixed crops and different augmentation. Consequently, the experiments establish neither an end-to-end speed advantage nor an architectural advantage under comparable image-processing conditions.
-
One thing that confuses me is that the authors claims without HAP blocks, the new architecture achieves 49.0% and 56.3% for ImageNet-1K sweep. How come only channel-wise attention without spatial information exchange could perform well for image classification. Or do I missing something in the paper?
Questions for authors
- For the classification-head comparison, what encoder configuration, parameter counts, training settings, and number of runs were used? Were all other components held fixed? Are component-wise results available for the main jet-classification task?
- Is parameter allocation intended as a possible explanation or an experimentally established mechanism? What evidence distinguishes its effect from the simultaneous changes in spatial mixing and representation width?
- Are full-model training measurements available with the fused operator integrated? If not, can the efficiency conclusions be explicitly limited to FLOPs and isolated-operator performance?
- Are results available for a ViT using the same image buckets and augmentation pipeline? Without this comparison, what evidence attributes the variable-resolution gain to the architecture rather than the different inputs?
2 comments
Thanks for addressing my raised concerns. We still don't understand why the fully-depth wise model with no HAP blocks could catch up with the pattern (DDD*H)*4 on ImageNet at epoch 40. It seems like the inserted HAP blocks has no impact, which is against my intuition. I will keep the original score.
Response to Reviewer kjkg: isolation control, allocation vs width, efficiency now measured end to end
Thank you for the review. We address each point and mark where we could not run the requested experiment in time.
HAP and head versus channel attention (isolation). New experiment: same embedding, HAP blocks and head, with only the operator swapped for parameter-matched token attention. Channel attention wins at both tiers: 75.12 ± 0.13 vs 74.35 ± 0.11 (5.4M) and 75.34 ± 0.20 vs 74.83 ± 0.27 (22M), three seeds. The scaffold alone does not account for the accuracy (Sec. 3.2).
Allocation versus width. At a fixed 1M budget and fixed residual width, widening only channel attention's inner dimension (paid for by the feed-forward block) raises accuracy from 72.37 ± 0.41 to 74.15 ± 0.10. At 164K it does not help, within seed spread (Sec. 4).
Imagewoof arms differ in preprocessing and augmentation. We agree, and did not have time to rerun with swapped pipelines. The revision now states that the ViT arm was not tuned and keeps the augmentation caveat. Tuning would likely change performance of both. On jets and CIFAR-100, tuning improved ViT results. On CIFAR-100 where we had time to tune DepthViT2 as well, it improved performance. We restate the result as a capability, one trained model accepting five input shapes without crop or positional-table surgery, rather than as an accuracy margin.
HAP and the head are only ablated on ImageNet; HAP gives no gain by epoch 40. Agreed on epoch 40. The paper's claim is faster early learning at an unchanged ceiling, not a higher ceiling, and the paper states this in Sec. 2. Jets ablations of HAP and the head did not fit in the discussion period. The operator swap above partly addresses the related question of whether the scaffold carries the result. We will add full details of the 5.3-point head result (encoder configuration, parameter counts of both heads, epochs, seeds and what was held fixed) to the appendix.
How does a model without HAP reach 49.0 / 56.3% on ImageNet? Without HAP, tokens do not exchange information inside the encoder, but the model is still a bag of local features: each p-by-p patch is embedded, processed by per-token layers, and the head attention-pools over all tokens, so global aggregation happens at readout. Bag-of-local-features models are known to reach this accuracy range on ImageNet (BagNet; Brendel & Bethge, 2019). We will add this explanation to the paper.
Efficiency claims are under-evidenced. We agree the kernel claim was overstated: the original Triton kernel was 71x faster only on the isolated forward operator and never ran in training, so we now measure end-to-end training instead. For C=2 the channel softmax is a sigmoid, so we rewrote the operator in closed form (DepthViT2 Fast Op), which torch.compile fuses into Triton kernels for the forward and backward passes. It matches the original to ≤7e-7 in fp32 and gives matching loss curves on real data. Through our training pipeline on one A100 (300 batches of 128, data loading included), DepthViT2-22M takes 47 s with Fast Op vs 181 s with the original operator and 36 s for ViT-Small: 3.9x faster, and 1.3x the cost of ViT-Small instead of 5.0x. Per GPU step, with both models compiled whole, the gap is 1.7x, narrowing to 1.59x at 1,600 tokens. A second FLOP count (PyTorch FlopCounterMode) gives 4.28 vs 4.30 GFLOPs, matching Table 2. Details are in Sec. 6 and App. F; all accuracy results use the original operator.
Corrections. ViT-Small has 21.41M parameters (DepthViT2 is 3.0% larger at 22M), the 5.4M SGD ViT-Tiny entry is a single seed, and the learning-rate schedule cycles identically for every arm. All three are disclosed in the revision.
We would be grateful if you would reconsider your assessment in light of the new controls and the narrower claims.