acceptodds
Under review as a conference paper at ICLR 2027

RCViT: Encoding Spatial Structure through Alternating Row–Column Attention

Abstract

Positional encodings provide Vision Transformers with information about the spatial arrangement of image patches. Spatial relationships can also be specified through the attention pattern itself: row and column connections represent horizontal and vertical alignment, and alternating these connections across layers composes dependencies along both image axes. We investigate this observation through \method, a Vision Transformer with sparse row–column attention. Deterministic masks combine local and dilated neighbors, alternate the active axis across layers, and assign directional roles to attention heads. Global heads preserve unrestricted interactions, controlling the strength of the spatial inductive bias. The construction adds no trainable parameters and integrates with absolute, relative, conditional, and rotary positional encodings. Classification experiments demonstrate the utility of the attention pattern without positional embeddings and its complementarity with positional encoding in mixed directional–global models. Ablations and resolution and layout evaluations identify the importance of axis alternation and the dependence on head allocation and model configuration. The results support alternating row–column attention as a spatial inductive bias complementary to global attention and positional encoding.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Transformer, positional embeddings, spatial information

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Eow5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RCViT, a Vision Transformer that encodes spatial structure through deterministic sparse row-column attention masks. Directional heads attend to local and dilated neighbors along alternating row and column axes, while the remaining heads retain global attention. The construction adds no trainable parameters and can be combined with absolute, relative, conditional, and rotary positional encodings. Experiments on small-image datasets, ImageNet-100, and ImageNet-1K study the method with and without positional encoding, together with head-allocation, attention-pattern, and resolution/layout evaluations.

Critical strengths
  • The proposed method is simple, parameter-free, and evaluated under carefully matched settings.
  • The empirical analysis covers several complementary aspects of the proposed spatial bias.
Critical weaknesses
  1. The methodological novelty and empirical advantage over closely related structured-attention methods are not sufficiently established. The core novelty is closely related to axial attention, CSWin, and particularly LookHere, which already uses fixed directional attention patterns to encode spatial structure in ViTs. RCViT introduces sparse local/dilated row-column neighborhoods and layer-wise axis alternation, but the paper provides no direct comparison with these closely related methods. It is therefore difficult to assess whether the proposed construction offers a meaningful advantage beyond a different instantiation of an established directional-attention paradigm.
  2. The current experiments do not fully isolate the benefit of row-column geometry. The large improvements without positional embeddings are only compared against unrestricted global attention. Since any fixed non-permutation-symmetric attention mask can potentially provide token-position information, it would be important to compare RCViT with random or local masks under the same no-positional-encoding setting.
Questions for authors
  1. Why does RCViT reduce ImageNet-1K accuracy for both ViT-T configurations with nD-RoPE? Would a full directional/global head-allocation study on ImageNet-1K show that this is mainly caused by transferring k=1k=1 from ImageNet-100?
  2. For the resolution and layout experiments, can the authors further clarify whether RCViT improves robustness relative to its clean accuracy, rather than mainly benefiting from a higher clean baseline, particularly for CPVT-CPE and nD-RoPE?
2 comments

Thanks for your review and valuable suggestions.

W1: RCViT composes sparse local and dilated axial neighborhoods across layers, retaining a configurable number of global heads in a plain ViT. Its connection range and interaction pattern differ from complete axial attention, CSWin stripes, and LookHere’s angular fields of view and distance biases. The added experiments test this construction directly. With the same six-head backbone, training recipe, and nD-RoPE, RCViT achieves 86.41 ± 0.07%, versus 86.09 ± 0.10% for global attention and 86.07 ± 0.33% for LookHere. The paired gain is +0.32 ± 0.03 percentage points (pp) over global attention and +0.34 ± 0.40 pp over LookHere. These direct comparisons support the ability of row–column connections to complement positional encoding.

W2: The new results show that breaking arbitrary permutation symmetry with a fixed mask is insufficient to explain RCViT’s performance. We compare global, RCViT, fixed random, and local attention on CIFAR-10/100 and SVHN without positional embeddings. The three masked methods constrain all four heads, matching key counts and self-connections per head and query, including boundary-dependent counts. They share initial weights, a fixed 10% training holdout, and best-EMA evaluation. RCViT obtains 89.53 ± 0.01% / 66.23 ± 0.16% / 95.70 ± 0.04%; local attention obtains 86.63 ± 0.81% / 63.97 ± 0.01% / 95.22 ± 0.04%; random obtains 78.57 ± 0.27% / 52.63 ± 0.49% / 84.15 ± 5.05% (Table 20). The paired gains are +2.90 ± 0.79, +2.26 ± 0.14, and +0.48 ± 0.00 pp over local, and +10.96 ± 0.25, +13.60 ± 0.65, and +11.55 ± 5.01 pp over random.

Q1: These comparisons test transfer of k=1 from ImageNet-100 to ImageNet-1K. Three- and four-head ViT-T with nD-RoPE give paired changes of −0.471 ± 0.230 and −0.112 ± 0.014 pp. With APE, ViT-T at the same k=1 improves by +0.684 ± 0.016 pp. The full ImageNet-100 allocation study already shows different preferred global-head counts: two for APE/relative bias, three for CPVT-CPE/nD-RoPE. Therefore, a value of k=1 for transfer may not be suitable for all dataset and settings.

Q2: For CPVT-CPE, the benefit is mainly higher mean transformed accuracy; for nD-RoPE, it also depends on the transformation. Define Δd=(clean−transformed)RCViT−(clean−transformed)global. Across resolution/aspect-ratio/canvas groups, CPVT-CPE gives +0.19/+0.16/+0.18 pp and nD-RoPE gives −0.17/+0.34/−0.09; negative means a smaller mean drop. This does not explain all generalization gains as a higher starting point. APE gives −0.95/−0.30/−1.16 pp and relative bias −0.58/−0.45/−0.34: both reduce mean drops in all three groups. The revised text states the findings by encoding. The new LookHere comparison uses the same metric: with common nD-RoPE, RCViT’s mean accuracy exceeds global attention at all five sizes.

Thanks for your response. Some of my concerns have been addressed. However, my main concern regarding the methodological novelty and empirical advantages over LookHere remains. The results do not demonstrate a clear and consistent advantage of RCViT over LookHere, particularly given its modest gains under nD-RoPE and inferior performance without additional positional encoding and at higher resolutions. Therefore, I maintain my original rating.

Reviewer_k5fgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper asks whether a ViT can get spatial information from the structure of its attention pattern instead of coming from positional encodings. RCViT uses fixed attention masks so that some heads, referred to as directional heads, attend only to a few nearby patches along one image axis, with the axis alternating between rows and columns across layers. The remaining heads stay global, so the number of directional heads controls how strong the spatial constraint is. The masks add no parameters and work on top of existing positional encodings. Without positional embeddings, directional attention greatly improves accuracy on CIFAR and SVHN. With positional encodings on ImageNet-100, making all heads directional hurts, but keeping a single directional head alongside global heads gives small gains over every positional baseline, and alternating the axis does slightly better than row-only, column-only, random, or local masks. On ImageNet-1K the gains are mixed, helping in some backbone and encoding combinations and hurting in others. The model also scores higher when tested at other resolutions and layouts, though not under transpose or rotation. The authors conclude that alternating row–column connectivity is a spatial inductive bias that complements global attention and positional encoding, with the right strength depending on the dataset, backbone, and encoding.

Critical strengths
  1. The no-positional-embedding experiment is a clean test. With position embedding removed, the mask is the only spatial signal, and directional attention still improves CIFAR and SVHN by a large margin.
  2. The controls in Table 5 are nice. Random and local masks are matched on the number of keys per query, and row-only, column-only, and fixed-head variants are included.
Critical weaknesses
  1. The gain from axis alternation falls within the error bars. In Table 5, alternating masks beat a plain local mask by 0.2 points with ±0.2 std over three runs, and the fixed-head variant does better than the proposed cyclic one.
  2. ImageNet-1K is mixed. RCViT hurts in two of the four configurations, and the gains in the other two are under a point, so it is unclear when the constraint actually helps.
  3. Most of the improvement comes from settings that favor a strong spatial prior. With positional encodings and ImageNet-scale data the gain is small and sometimes negative. It would help to be clearer about which regime the method is meant for, since the current results suggest the benefit shrinks as the baseline gets stronger.
  4. The writing is dense and hard to follow.
Questions for authors
  1. Could you compare against LookHere under the same recipe, at least on ImageNet-100 and the resolution tests? It also mixes directed and global heads and targets resolution extrapolation, so it is the natural baseline.
  2. Section 4.7 shows RCViT scoring higher than the baseline at every resolution and layout, which reads as if it is more robust to these changes. But Table 11 shows its drop from clean accuracy is not consistently smaller, and in some cases is slightly larger. So RCViT mostly starts higher and stays higher, rather than losing less. Could you state clearly whether the claim is robustness or just a higher baseline that carries over?
2 comments

Thank you for the review and for recognizing the controls and experiments without positional embeddings.

W1: Axis alternation and head assignment are different variables. The fixed-head model still alternates axes. The direct single-axis controls give paired gains of +0.30 ± 0.02 percentage points (pp) over row-only and +0.61 ± 0.21 pp over column-only. Individual-model SDs are not SDs of paired differences; overlapping error bars do not negate these comparisons. The original paired difference from the local control is +0.20 ± 0.41 pp. New experiments without positional embeddings constrain all four heads: RCViT has higher mean accuracy than local and random attention on three datasets, with paired gains over local of 2.90 ± 0.79, 2.26 ± 0.14, and 0.48 ± 0.00 pp (Table 20).

W2: APE improves by +0.684 ± 0.016 pp; three- and four-head ViT-T with nD-RoPE give −0.471 ± 0.230 and −0.112 ± 0.014 pp; ViT-S gives +0.157 ± 0.573 pp (Table 4). This tests transfer of k=1 from ImageNet-100, not allocation selected separately for each model. Together with the allocation study below, it supports choosing constraint strength for the encoding and backbone, while establishing an ImageNet-1K benefit with APE.

W3: The benefit extends beyond weak baselines or absent positional encoding. On ImageNet-100, one directional head improves all four encodings, including +0.43 ± 0.21 pp with nD-RoPE. The added six-head nD-RoPE comparison also improves by +0.32 ± 0.03 pp. The allocation study gives practical guidance: with four total heads, APE and relative bias peak at two global heads, while CPVT-CPE and nD-RoPE peak at three.

W4: We rewrote the abstract, introduction, and experimental discussion. The method first explains k=1: most heads stay global; one uses self, adjacent, and distant axial connections; the next layer switches axes. We then define connections and head assignment separately. Detailed settings and summary statistics are in the appendix.

Q1: We compare the methods with the same width-192, six-head, 12-layer backbone, global average pooling, initial weights, augmentation, optimizer, effective batch 512, 400 epochs, and final EMA. LookHere uses the authors’ supported LH-180 configuration: four directional and two unrestricted heads, retaining its distance bias. With common nD-RoPE, global/RCViT/LookHere achieve 86.09 ± 0.10% / 86.41 ± 0.07% / 86.07 ± 0.33%. The paired gain is +0.32 ± 0.03 pp over global attention and +0.34 ± 0.40 pp over LookHere. Without additional positional encoding, the results are 78.74 ± 0.08% / 83.50 ± 0.14% / 84.28 ± 0.08% (Table 3).

Q2: Starting higher does not explain every encoding. Define Δd=(clean−transformed)RCViT−(clean−transformed)global; negative means a smaller drop. For resolution/aspect-ratio/canvas changes, mean Δd is −0.95/−0.30/−1.16 pp with APE and −0.58/−0.45/−0.34 with relative bias. Both improve transformed accuracy and reduce the mean drop in all three groups. CPVT-CPE gives +0.19/+0.16/+0.18 pp, and nD-RoPE gives −0.17/+0.34/−0.09. Thus, CPVT-CPE mainly retains its accuracy advantage, while nD-RoPE depends on the transformation. The main text now states both findings: higher mean transformed accuracy for all four encodings, and smaller mean drops in all three groups for APE and relative bias.

Thank you for the added analyses and clarifications. I am raising soundness to 3 and informativeness to 2. I still retain my original overall rating because of the mixed ImageNet-1K results and limited advantage over LookHere.

Reviewer_m6SWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether spatial structure can be carried by the attention pattern itself. The proposed method apply deterministic sparse masks to a subset of heads in a standard ViT, admitting only local and dilated neighbours along one image axis, alternating the active axis across layers, and cycling four directional roles over heads; in the mixed configuration the remaining heads stay global. The construction adds no trainable parameters and is compatible to absolute, relative, conditional and rotary positional encodings. Without positional embeddings, row–column attention improves clearly over global attention on CIFAR-10/100 and SVHN; with positional encoding, a single directional head yield small gains on all four ImageNet-100 baselines, while ImageNet-1K results varies with backbone and positional mechanism.

Critical strengths
  1. Research on fundmental model architecture is always a important problem.
  2. The setup is clean, no extra parameters, no change to the backbone, and it drops into all four positional mechanisms.
  3. The appendix is generous, including allocation curves, training curves, 29 resolution and layout settings, step time and memory.
Critical weaknesses
  1. My main concerns is whether alternation does anything. In Table 5, alternating (86.03) is basically tied with fixed-head (86.13), local (85.83) and row-only (85.73) over three seeds; what is actually robust is that one restricted head beats global (+0.43 to +2.38). So the row–column alternation in the title reads more like a plausible guess than a tested conclusion.
  2. It is confusing that in the ImageNet-1K table, two of the four comparisons are negative (+0.68 to −0.47), and although the caption promises mean±std, the table has no ± and no per-run paired differences
  3. The paper doesn't really ask why it works. Does the directional head actually look inside its mask, do global heads compensate, is the fixed mask just acting as a hard-coded positional code? None of this is measured. And compared to axial attention, CSWin(Dong et al., 2022)、NAT(Hassani et al., 2023)、LookHere(Fuller et al., 2024), the mechanism feel like a recombination rather than something new.
Questions for authors

How much of the directional head's attention mass lands inside the mask? What happens if its projections are frozen or randomly fixed?

1 comment

Thanks for your review and feedback.

W1: The fixed-head control retains row–column alternation; it only fixes the directional head index. Thus, 86.13% versus 86.03% tests head assignment, not whether alternation helps. Both models use the proposed alternating connections. We relabel the control “alternating axes, fixed head index.” Row-only and column-only are the direct controls for axis alternation. RCViT has paired gains of +0.30 ± 0.02 percentage points (pp) over row-only and +0.61 ± 0.21 pp over column-only.Local attention tests neighborhood shape. The original paired difference is +0.20 ± 0.41 pp. We additionally tested all four heads without positional embeddings on three datasets, matching key counts and self-connections. RCViT has higher mean accuracy than local and random attention, with paired gains over local of 2.90 ± 0.79, 2.26 ± 0.14, and 0.48 ± 0.00 pp (Table 20).

W2: We report ImageNet-1K results as means and sample SDs using epoch-300 EMA models (Table 4). ViT-T with APE improves by +0.684 ± 0.016 pp, showing that the connections can add value when positional encoding is present. The changes are −0.471 ± 0.230 and −0.112 ± 0.014 pp for three- and four-head ViT-T with nD-RoPE, and +0.157 ± 0.573 pp for ViT-S. This variation concerns how a fixed head allocation interacts with each configuration; it does not refute the spatial information supplied by row–column connections.

W3: The contribution has a concrete construction and controlled evidence. Axial attention aggregates along axes, CSWin uses stripes, NAT uses local neighborhoods, and LookHere uses angular fields of view and distance biases. RCViT composes sparse local and dilated axial connections across layers, retaining a configurable number of global heads in a plain ViT without extra parameters. Beyond the 2.90 ± 0.79, 2.26 ± 0.14, and 0.48 ± 0.00 pp gains over local attention without positional embeddings, the same-backbone comparison with common nD-RoPE gives 86.41% for RCViT, 86.09% for global attention, and 86.07% for LookHere (Table 3). Without additional positional encoding, LookHere obtains 84.28% versus RCViT’s 83.50%; it is also higher at larger resolutions. These comparisons reveal different interactions with positional encoding.

Q1: We directly test neighborhood use and projection learning. Masked-out weights are zero, so mass inside the allowed support is one by construction. Non-self mass and perturbations reveal how that support is used. On all 50,000 ImageNet-1K validation images, one trained ViT-S assigns 36.49% of directional attention to non-self positions. Uniform weights over the same keys reduce accuracy from 79.946% to 79.192%; zeroing directional outputs in every layer gives 68.802%, and zeroing one global head per layer gives 72.054% (Table 19). Both branches contribute, and learned directional weights improve on uniform aggregation. On ImageNet-100, uniform weights cost 1.71 ± 0.17 pp, versus 0.19 ± 0.24 pp for removing only distant keys, separating the contribution of learned weights from the effect of distant keys. We also fixed directional Q/K/V weights and biases at their initial random values throughout 400 training epochs, while learning the output projection and other parameters. Accuracy decreases by 0.42 ± 0.00 pp, from 86.41 ± 0.07% to 85.99 ± 0.07% (Table 18), directly confirming a contribution from learning these projections.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.