- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RCPA, a row–column attention that adapts FloydNet’s Pivotal Attention to image grids, and builds the ViF backbone on it. The idea of bringing explicit relation modeling into visual backbones is interesting. However, ViF uses more layers and parameters than the ViT baselines, it is never compared against closely related axial or criss-cross attention, and its efficiency advantage does not appear at standard resolutions.
Critical strengths
- Taking the “update through a middle point” idea from Pivotal Attention and building it into a vision backbone is a nice angle. Most backbones just mix tokens, so trying to model relations directly is interesting.
- RCPA is a drop-in replacement for self-attention. The rest of the block stays the same. The authors test it on classification, segmentation, detection, and scene graphs, and the ablations show the joint softmax works better than the other variants they tried.
Critical weaknesses
- ViF Small and Base use 16 blocks, while the ViT baselines use 12, so ViF has a lot more parameters and compute. ViF-S has about 50% more parameters than ViT-S and still only ties it on ImageNet. Most of the gains could just come from the bigger model.The author mentioned "the Small and Base ViF classifiers have 16 blocks, compared with 12 in the corresponding DeiT references; their block-stack MAC ratios at T = 196 are approximately 1.45 and 1.50."
- There are two similar work: [1] CCNet: Criss-cross attention for semantic segmentation. [2] Axial attention in multidimensional transformers,. RCPA only lets each patch look at its own row and column. That’s the same connectivity used by CCNet and axial attention. Although these two paper are cited in the appendix, they are not compared experimentally.
Questions for authors
- Can the authors compare ViF and ViT at the same depth and parameter count? Right now ViF has 16 blocks and ViT has 12, so it’s hard to tell where the gains come from.
- How does RCPA do against axial attention and CCNet under the same setup? They use the same row-column connections, so this is the key test of whether the joint pairing actually helps.
- Is there a geometric reason for pairing the j-th patch in a row with the j-th patch in a column? And how does RCPA handle non-square inputs, which are common in detection and segmentation?
- Do you have results at higher resolution or smaller patch size, where RCPA is supposed to be faster? That would back up the efficiency claim.
3 comments
Response to Reviewer Q9LN (Part 2/2)
We appreciate the opportunity to clarify the same-index prior, rectangular inputs, and the scope of the efficiency claim.
Q3. Why use the same pivot index, and how are rectangular inputs handled?
Is there a geometric reason to pair the j-th row patch with the j-th column patch, and how does RCPA handle non-square inputs used in detection and segmentation?
Response: The shared index is a correspondence prior. For target (i, k), the two sources (i, j) and (j, k) refer to the same intermediate index j, and the retained row and column coordinates meet at (i, k). Using independent indices allows the update to combine sources that do not share an intermediate.
The shared pivot is better in all six scale/split comparisons:
| Scale | Pivot setting | IID MAE | Style OOD | Angle OOD |
|---|---|---|---|---|
| Tiny | Shared | 2.23 | 29.55 | 32.21 |
| Tiny | Independent | 4.03 | 36.20 | 37.01 |
| Small | Shared | 1.96 | 26.50 | 23.04 |
| Small | Independent | 2.84 | 29.31 | 23.95 |
For a square grid, RCPA applies one joint softmax over the shared pivot. For a rectangular H x W grid, it applies one softmax per axis and sums the outputs. This preserves sparse row-column support, but it is not the same joint-normalization operator. We therefore limit the joint-softmax and O(T^(3/2)) statements to square grids.
Q4. Does the efficiency advantage appear in the high-token regime?
Are there higher-resolution or smaller-patch results where RCPA is expected to be faster?
Response: We added a higher-resolution ImageNet experiment and an attention-core scaling study. We did not add a new smaller-patch training experiment.
| Model | Resolution | Top-1 raw |
|---|---|---|
| ViT-Tiny | 384 | 78.438 |
| ViT-CC Tiny | 384 | 77.898 |
| ViF-Tiny | 384 | 79.105 |
At 384x384, ViF improves raw Top-1 by 0.667 points over ViT and 1.207 points over ViT-CC.
| Grid | Equivalent p16 input | Self-attention latency / memory | Matched axial latency / memory | RCPA latency / memory |
|---|---|---|---|---|
| 14x14 | 224x224 | 0.439 ms / 15.1 MiB | 2.320 ms / 15.4 MiB | 1.802 ms / 17.1 MiB |
| 56x56 | 896x896 | 2.385 ms / 372.7 MiB | 2.268 ms / 62.4 MiB | 1.739 ms / 73.0 MiB |
| 84x84 | 1344x1344 | 10.045 ms / 1773.4 MiB | 3.043 ms / 134.1 MiB | 2.463 ms / 157.6 MiB |
| 112x112 | 1792x1792 | 30.768 ms / 5506.3 MiB | 4.890 ms / 248.4 MiB | 4.080 ms / 293.4 MiB |
RCPA is slower at 14x14, but it is 1.37x/4.08x/7.54x faster than explicit self-attention at 56x56/84x84/112x112. At 112x112, peak allocated memory decreases from 5506.3 to 293.4 MiB. RCPA is also faster than the matched axial implementation at the three larger grids, while axial uses less memory.
These measurements use one attention module, random inputs and weights, FP16 autocast, batch size one, and one V100. They do not measure a complete backbone or an end-to-end high-resolution application. We therefore use them to support a high-token scaling claim, not a general runtime-superiority claim.
Follow-up on our response
Dear Reviewer Q9LN,
We would like to follow up on our responses to your comments. We would greatly appreciate it if you could let us know whether we have addressed your concerns or if any further clarification would be helpful. Thank you for your time and consideration.
Response to Reviewer Q9LN (Part 1/2)
We thank the reviewer for the concrete requests on capacity matching and related row-column operators.
Q1. What happens at the same depth and parameter count?
Can ViF and ViT be compared at the same depth and parameter count?
Response: We now compare ViF and ViT at the same 12-layer depth and much closer capacity. The match is not exact, but ViF-S/B use only about 16% more parameters and 8–12% more FLOPs, rather than the roughly 50% parameter gap in the original submission.
The same-depth Small/Base results are:
The near-capacity Tiny control gives the same relation-task pattern:
The consistent result across matched depth and near-matched capacity supports RCPA rather than extra depth as the source of the gain.
Q2. How does RCPA compare with axial attention and CCNet?
Under the same setup, does joint pairing improve over axial attention and CCNet?
Response: We implemented both baselines in the same Tiny framework. ViT-Axial follows the original axial-attention design. ViT-CC replaces the self-attention kernel in each ViT block with CCNet's criss-cross attention kernel. This directly tests whether row-column support alone is sufficient.
Relative to the Tiny criss-cross model, RCPA lowers IID, Style OOD, and Angle OOD MAE by 34.4%, 38.2%, and 36.0%. ImageNet Top-1 differs by -0.03 points.
RCPA gives the best VG and COCO values in this Tiny comparison. These results show that row-column connectivity alone does not reproduce the benefit of shared-pivot joint selection.