- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RCViT, a Vision Transformer that encodes spatial structure through deterministic sparse row-column attention masks. Directional heads attend to local and dilated neighbors along alternating row and column axes, while the remaining heads retain global attention. The construction adds no trainable parameters and can be combined with absolute, relative, conditional, and rotary positional encodings. Experiments on small-image datasets, ImageNet-100, and ImageNet-1K study the method with and without positional encoding, together with head-allocation, attention-pattern, and resolution/layout evaluations.
Critical strengths
- The proposed method is simple, parameter-free, and evaluated under carefully matched settings.
- The empirical analysis covers several complementary aspects of the proposed spatial bias.
Critical weaknesses
- The methodological novelty and empirical advantage over closely related structured-attention methods are not sufficiently established. The core novelty is closely related to axial attention, CSWin, and particularly LookHere, which already uses fixed directional attention patterns to encode spatial structure in ViTs. RCViT introduces sparse local/dilated row-column neighborhoods and layer-wise axis alternation, but the paper provides no direct comparison with these closely related methods. It is therefore difficult to assess whether the proposed construction offers a meaningful advantage beyond a different instantiation of an established directional-attention paradigm.
- The current experiments do not fully isolate the benefit of row-column geometry. The large improvements without positional embeddings are only compared against unrestricted global attention. Since any fixed non-permutation-symmetric attention mask can potentially provide token-position information, it would be important to compare RCViT with random or local masks under the same no-positional-encoding setting.
Questions for authors
- Why does RCViT reduce ImageNet-1K accuracy for both ViT-T configurations with nD-RoPE? Would a full directional/global head-allocation study on ImageNet-1K show that this is mainly caused by transferring from ImageNet-100?
- For the resolution and layout experiments, can the authors further clarify whether RCViT improves robustness relative to its clean accuracy, rather than mainly benefiting from a higher clean baseline, particularly for CPVT-CPE and nD-RoPE?
2 comments
Thanks for your response. Some of my concerns have been addressed. However, my main concern regarding the methodological novelty and empirical advantages over LookHere remains. The results do not demonstrate a clear and consistent advantage of RCViT over LookHere, particularly given its modest gains under nD-RoPE and inferior performance without additional positional encoding and at higher resolutions. Therefore, I maintain my original rating.
rebuttal
Thanks for your review and valuable suggestions.
W1: RCViT composes sparse local and dilated axial neighborhoods across layers, retaining a configurable number of global heads in a plain ViT. Its connection range and interaction pattern differ from complete axial attention, CSWin stripes, and LookHere’s angular fields of view and distance biases. The added experiments test this construction directly. With the same six-head backbone, training recipe, and nD-RoPE, RCViT achieves 86.41 ± 0.07%, versus 86.09 ± 0.10% for global attention and 86.07 ± 0.33% for LookHere. The paired gain is +0.32 ± 0.03 percentage points (pp) over global attention and +0.34 ± 0.40 pp over LookHere. These direct comparisons support the ability of row–column connections to complement positional encoding.
W2: The new results show that breaking arbitrary permutation symmetry with a fixed mask is insufficient to explain RCViT’s performance. We compare global, RCViT, fixed random, and local attention on CIFAR-10/100 and SVHN without positional embeddings. The three masked methods constrain all four heads, matching key counts and self-connections per head and query, including boundary-dependent counts. They share initial weights, a fixed 10% training holdout, and best-EMA evaluation. RCViT obtains 89.53 ± 0.01% / 66.23 ± 0.16% / 95.70 ± 0.04%; local attention obtains 86.63 ± 0.81% / 63.97 ± 0.01% / 95.22 ± 0.04%; random obtains 78.57 ± 0.27% / 52.63 ± 0.49% / 84.15 ± 5.05% (Table 20). The paired gains are +2.90 ± 0.79, +2.26 ± 0.14, and +0.48 ± 0.00 pp over local, and +10.96 ± 0.25, +13.60 ± 0.65, and +11.55 ± 5.01 pp over random.
Q1: These comparisons test transfer of k=1 from ImageNet-100 to ImageNet-1K. Three- and four-head ViT-T with nD-RoPE give paired changes of −0.471 ± 0.230 and −0.112 ± 0.014 pp. With APE, ViT-T at the same k=1 improves by +0.684 ± 0.016 pp. The full ImageNet-100 allocation study already shows different preferred global-head counts: two for APE/relative bias, three for CPVT-CPE/nD-RoPE. Therefore, a value of k=1 for transfer may not be suitable for all dataset and settings.
Q2: For CPVT-CPE, the benefit is mainly higher mean transformed accuracy; for nD-RoPE, it also depends on the transformation. Define Δd=(clean−transformed)RCViT−(clean−transformed)global. Across resolution/aspect-ratio/canvas groups, CPVT-CPE gives +0.19/+0.16/+0.18 pp and nD-RoPE gives −0.17/+0.34/−0.09; negative means a smaller mean drop. This does not explain all generalization gains as a higher starting point. APE gives −0.95/−0.30/−1.16 pp and relative bias −0.58/−0.45/−0.34: both reduce mean drops in all three groups. The revised text states the findings by encoding. The new LookHere comparison uses the same metric: with common nD-RoPE, RCViT’s mean accuracy exceeds global attention at all five sizes.