acceptodds
Withdrawn from ICLR 2027

FLOYD VISION TRANSFORMER: RELATION-CENTRIC VISUAL BACKBONES WITH PIVOTAL ATTENTION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As vision models are increasingly applied to complex real-world tasks, visual understanding must extend beyond recognizing individual objects to modeling their relationships, interactions, and scene dynamics. Mainstream visual backbones typically operate on patches, leaving relations between patches implicit in the resulting token representations. Pivotal Attention (PA) makes relations explicit through ordered-pair features and their updates, but its all-pairs computation is costly for images. We introduce Row Column Pivotal Attention (RCPA), which exploits the organized image grid to jointly select and aggregate row and column features. This structure preserves global two-hop coverage through sparse connections, as characterized by the rook's graph, while reducing attention interactions from PA's to for patches on a square grid. Building on RCPA, we develop Vision Transformer Floyd (ViF), a general-purpose backbone that brings PA-inspired relation modeling to visual recognition and dense prediction. On ClockBench relative-angle prediction, PA and RCPA Small reduce IID angular error by 83.6% and 67.3% relative to ViT Small. On real images, ViF Base improves VG150 SGDet R@100 by 4.6% relative to ViT Base and transfers effectively to ImageNet-1K, ADE20K, and COCO. These results establish a practical route from explicit relation modeling to visual backbone design and motivate its extension to richer relational and multimodal tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Transformers, Visual Relation Modeling, Pivotal Attention, Sparse Attention, Scene Graph Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q9LNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RCPA, a row–column attention that adapts FloydNet’s Pivotal Attention to image grids, and builds the ViF backbone on it. The idea of bringing explicit relation modeling into visual backbones is interesting. However, ViF uses more layers and parameters than the ViT baselines, it is never compared against closely related axial or criss-cross attention, and its efficiency advantage does not appear at standard resolutions.

Critical strengths
  • Taking the “update through a middle point” idea from Pivotal Attention and building it into a vision backbone is a nice angle. Most backbones just mix tokens, so trying to model relations directly is interesting.
  • RCPA is a drop-in replacement for self-attention. The rest of the block stays the same. The authors test it on classification, segmentation, detection, and scene graphs, and the ablations show the joint softmax works better than the other variants they tried.
Critical weaknesses
  • ViF Small and Base use 16 blocks, while the ViT baselines use 12, so ViF has a lot more parameters and compute. ViF-S has about 50% more parameters than ViT-S and still only ties it on ImageNet. Most of the gains could just come from the bigger model.The author mentioned "the Small and Base ViF classifiers have 16 blocks, compared with 12 in the corresponding DeiT references; their block-stack MAC ratios at T = 196 are approximately 1.45 and 1.50."
  • There are two similar work: [1] CCNet: Criss-cross attention for semantic segmentation. [2] Axial attention in multidimensional transformers,. RCPA only lets each patch look at its own row and column. That’s the same connectivity used by CCNet and axial attention. Although these two paper are cited in the appendix, they are not compared experimentally.
Questions for authors
  • Can the authors compare ViF and ViT at the same depth and parameter count? Right now ViF has 16 blocks and ViT has 12, so it’s hard to tell where the gains come from.
  • How does RCPA do against axial attention and CCNet under the same setup? They use the same row-column connections, so this is the key test of whether the joint pairing actually helps.
  • Is there a geometric reason for pairing the j-th patch in a row with the j-th patch in a column? And how does RCPA handle non-square inputs, which are common in detection and segmentation?
  • Do you have results at higher resolution or smaller patch size, where RCPA is supposed to be faster? That would back up the efficiency claim.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Q9LN (Part 1/2)

We thank the reviewer for the concrete requests on capacity matching and related row-column operators.

Q1. What happens at the same depth and parameter count?

Can ViF and ViT be compared at the same depth and parameter count?

Response: We now compare ViF and ViT at the same 12-layer depth and much closer capacity. The match is not exact, but ViF-S/B use only about 16% more parameters and 8–12% more FLOPs, rather than the roughly 50% parameter gap in the original submission.

The same-depth Small/Base results are:

ModelDepthParams (M)GFLOPsIN1K Top-1VG R@100 / mR@100ADE20K mIoU
ViT-S1222.059.2080.5234.50 / 7.8244.13
ViF-S1225.529.9380.3435.30 / 8.0146.16
ViT-B1286.5735.1381.5235.80 / 8.2045.41
ViF-B12100.5939.2882.2236.40 / 8.4747.15

The near-capacity Tiny control gives the same relation-task pattern:

ModelParams (M)IID MAEStyle OODAngle OOD
ViT5.727.6969.8336.80
ViT-Axial6.3810.6657.4245.69
ViT-CC5.533.4047.7850.33
ViF-RCPA6.572.2329.5532.21

The consistent result across matched depth and near-matched capacity supports RCPA rather than extra depth as the source of the gain.

Q2. How does RCPA compare with axial attention and CCNet?

Under the same setup, does joint pairing improve over axial attention and CCNet?

Response: We implemented both baselines in the same Tiny framework. ViT-Axial follows the original axial-attention design. ViT-CC replaces the self-attention kernel in each ViT block with CCNet's criss-cross attention kernel. This directly tests whether row-column support alone is sufficient.

ModelAttentionParams (M)IID MAEStyle OODAngle OODIN1K Top-1
ViTSelf-attention5.727.6969.8336.8074.25
ViT-AxialAxial attention6.3810.6657.4245.6973.48
ViT-CCCriss-cross attention5.533.4047.7850.3375.23
ViFRCPA6.572.2329.5532.2175.20

Relative to the Tiny criss-cross model, RCPA lowers IID, Style OOD, and Angle OOD MAE by 34.4%, 38.2%, and 36.0%. ImageNet Top-1 differs by -0.03 points.

ModelVG R@100VG mR@100COCO Box APCOCO Mask AP
ViT32.667.2343.839.0
ViT-CC32.807.5144.038.9
ViF33.267.5844.139.3

RCPA gives the best VG and COCO values in this Tiny comparison. These results show that row-column connectivity alone does not reproduce the benefit of shared-pivot joint selection.

AuthorsOpenReview ↗

Response to Reviewer Q9LN (Part 2/2)

We appreciate the opportunity to clarify the same-index prior, rectangular inputs, and the scope of the efficiency claim.

Q3. Why use the same pivot index, and how are rectangular inputs handled?

Is there a geometric reason to pair the j-th row patch with the j-th column patch, and how does RCPA handle non-square inputs used in detection and segmentation?

Response: The shared index is a correspondence prior. For target (i, k), the two sources (i, j) and (j, k) refer to the same intermediate index j, and the retained row and column coordinates meet at (i, k). Using independent indices allows the update to combine sources that do not share an intermediate.

The shared pivot is better in all six scale/split comparisons:

ScalePivot settingIID MAEStyle OODAngle OOD
TinyShared2.2329.5532.21
TinyIndependent4.0336.2037.01
SmallShared1.9626.5023.04
SmallIndependent2.8429.3123.95

For a square grid, RCPA applies one joint softmax over the shared pivot. For a rectangular H x W grid, it applies one softmax per axis and sums the outputs. This preserves sparse row-column support, but it is not the same joint-normalization operator. We therefore limit the joint-softmax and O(T^(3/2)) statements to square grids.

Q4. Does the efficiency advantage appear in the high-token regime?

Are there higher-resolution or smaller-patch results where RCPA is expected to be faster?

Response: We added a higher-resolution ImageNet experiment and an attention-core scaling study. We did not add a new smaller-patch training experiment.

ModelResolutionTop-1 raw
ViT-Tiny38478.438
ViT-CC Tiny38477.898
ViF-Tiny38479.105

At 384x384, ViF improves raw Top-1 by 0.667 points over ViT and 1.207 points over ViT-CC.

GridEquivalent p16 inputSelf-attention latency / memoryMatched axial latency / memoryRCPA latency / memory
14x14224x2240.439 ms / 15.1 MiB2.320 ms / 15.4 MiB1.802 ms / 17.1 MiB
56x56896x8962.385 ms / 372.7 MiB2.268 ms / 62.4 MiB1.739 ms / 73.0 MiB
84x841344x134410.045 ms / 1773.4 MiB3.043 ms / 134.1 MiB2.463 ms / 157.6 MiB
112x1121792x179230.768 ms / 5506.3 MiB4.890 ms / 248.4 MiB4.080 ms / 293.4 MiB

RCPA is slower at 14x14, but it is 1.37x/4.08x/7.54x faster than explicit self-attention at 56x56/84x84/112x112. At 112x112, peak allocated memory decreases from 5506.3 to 293.4 MiB. RCPA is also faster than the matched axial implementation at the three larger grids, while axial uses less memory.

These measurements use one attention module, random inputs and weights, FP16 autocast, batch size one, and one V100. They do not measure a complete backbone or an end-to-end high-resolution application. We therefore use them to support a high-token scaling claim, not a general runtime-superiority claim.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer Q9LN,

We would like to follow up on our responses to your comments. We would greatly appreciate it if you could let us know whether we have addressed your concerns or if any further clarification would be helpful. Thank you for your time and consideration.

Reviewer_U7rYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Row Column Pivotal Attention (RCPA), a PA-inspired attention operator for visual backbones. For each target patch, RCPA pairs row and column features through a shared coordinate index, scores them jointly, and aggregates both sources using shared attention weights. The resulting Vision Transformer Floyd (ViF) is evaluated on ClockBench, VG150, ImageNet, ADE20K, and COCO. The results are promising, although the independent contribution of the proposed mechanism remains insufficiently established.

Critical strengths
  1. A clear and practical operator with informative computational analysis.
  2. Positive evidence from both controlled experiments and natural-image tasks.
Critical weaknesses
  1. The main comparisons maynot sufficiently isolate the effects of RCPA. Small/Base ViF use 16 blocks vs. 12 for the ViT references. In Table 4, FLOPs increase by 43–49%, while Small accuracy is unchanged and Base improves by only 0.92 points. Although disclosed, these differences leave RCPA’s independent contribution unclear. The matched Tiny results are encouraging but do not resolve this issue at larger scales.
  2. Joint selection lacks sufficiently controlled validation. Figure 7 changes both normalization and scaling, so it does not isolate joint normalization. Consequently, the contribution of joint pairing to relational-task improvements remains unclear.
Questions for authors
  1. Please clarify the configurations and training recipes for each ViT/ViF pair. Do the gains persist under matched depth and training settings?
  2. Does joint selection retain its advantage with controlled scaling and against a closely related row–column attention baseline, particularly on a relational task?
2 comments
AuthorsOpenReview ↗

Response to Reviewer U7rY

We thank the reviewer for focusing on the depth confound and the interpretation of joint normalization.

Q1. Do the gains remain with matched depth and training settings?

Please give the configurations and recipes for each pair and test whether the gains remain with matched depth and training.

Response: We re-pretrained ViF-S/B with 12 blocks and transferred the resulting backbones to VG150 and ADE20K. Within each task, the ViT/ViF pair uses the same evaluation protocol: ImageNet uses 300 optimization epochs plus 20 warm-up epochs with Muon; VG150 uses 60k detector-pretraining and 40k relation-training iterations with the detector frozen during relation training; and ADE20K uses UPerNet for 80k iterations with AdamW.

ModelDepthParams (M)GFLOPsIN1K Top-1VG R@100VG mR@100ADE20K mIoU
ViT-S1222.059.2080.5234.507.8244.13
ViF-S1225.529.9380.3435.308.0146.16
ViT-B1286.5735.1381.5235.808.2045.41
ViF-B12100.5939.2882.2236.408.4747.15

At Small scale, ViF improves VG R@100/mR@100 by +0.80/+0.19 and ADE20K mIoU by +2.03. At Base scale, the corresponding differences are +0.70 ImageNet Top-1, +0.60/+0.27 VG R@100/mR@100, and +1.74 ADE20K mIoU. These results show that extra depth is not the source of the relation and dense-prediction gains. The corrected counts reduce the remaining gap to about 16% in parameters and 8–12% in FLOPs.

Q2. Does joint selection help after controlling score scale?

Does joint selection still help with matched scaling and against a closely related row-column baseline on a relation task?

Response: The original Figure 7 changed normalization and score scale together. We added a dual-softmax control that changes normalization while preserving the score scale.

VariantIID MAE change from RCPA
Default RCPA0.00
Matched-scale dual softmax+1.17
Dynamic pivot shift+2.02
Matched axial attention+5.90

The matched-scale dual-softmax variant is worse by 1.17 degrees, the dynamic pivot shift by 2.02, and matched axial attention by 5.90. Independent pivots are also worse in all six Tiny/Small and IID/Style OOD/Angle OOD comparisons. On ImageNet-1K, independent row/column normalization lowers Top-1 from 72.694% to 71.346%. The gain therefore comes from joint selection and shared alignment, not a favorable logit scale.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer U7rY,

We would like to follow up on our responses to your comments. We would greatly appreciate it if you could let us know whether we have addressed your concerns or if any further clarification would be helpful. Thank you for your time and consideration.

Reviewer_HGYkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Row-Column Pivotal Attention (RCPA) and builds upon it to construct a visual backbone named Vision Transformer Floyd (ViF). While standard Pivotal Attention (PA) explicitly represents relations via ordered node-pair features, it requires maintaining O(T<sup>2</sup>C) feature states and performing O(T<sup>3</sup>) attention interactions. In contrast, RCPA operates directly on patch features arranged on a regular image grid: for a target spatial location (i, k), it pairs row features (i, j) and column features (j, k) sharing a common pivot index j as candidate pairs, selects candidates via joint scoring and a shared Softmax distribution, and subsequently sums the two value branches. This formulation reduces the maintained feature state complexity to O(TC) and lowers spatial interaction complexity on a square grid to O(T<sup>3/2</sup>), while preserving two-hop reachability between arbitrary patches via a Rook’s Graph topology.

Empirically, the authors introduce ClockBench, a synthetic benchmark for relative angle prediction, and evaluate ViF across VG150 scene graph generation, ImageNet-1K classification, ADE20K semantic segmentation, and COCO object detection and instance segmentation, alongside module ablations and computational cost analyses. The paper reports notable gains of RCPA over ViT on ClockBench, consistent improvements of ViF across model scales on VG150, and competitive results on general vision tasks. Based on these findings, the authors argue that RCPA offers a viable pathway toward relation-centric general-purpose visual backbones.

Critical strengths

1. A clean and scalable PA-inspired spatial operator

The computational formulation of RCPA is well-defined. Rather than naively concatenating two independent row and column attention branches, it jointly leverages evidence from both row and column interactions for candidate selection prior to normalization. Reducing PA’s O(T<sup>3</sup>) interaction complexity and O(T<sup>2</sup>C) pairwise state storage to O(T<sup>3/2</sup>) interactions and O(TC) patch-level state storage addresses a practically meaningful bottleneck. The graph-theoretic analysis of two-hop coverage, as well as the discussion in Appendix D regarding the trade-off between projection overhead and spatial interaction costs at finite resolutions, also helps clarify the operating regime of the proposed operator.

2. Broad experimental coverage with promising results on selected tasks

The paper evaluates a unified backbone across synthetic geometric relation reasoning, scene graph generation, image classification, semantic segmentation, and object detection. ViF consistently outperforms the authors’ implemented ViT baseline across three model scales on VG150 and achieves gains on several spatial predicates as well as on ADE20K (Small/Base settings). Ablation studies—such as those validating joint normalization—also provide empirical justification for specific design choices within RCPA. These results suggest that the direction warrants further investigation.

Critical weaknesses

1. Fundamental architectural limitation: RCPA does not actually maintain "relations" as explicit representation objects

The core theoretical motivation of the paper is to explicitly model visual relations. However, the defining mechanism of PA is that it maintains and iteratively updates an independent state RikR_{ik} for each ordered pair across layers. RCPA merely transplants this pairwise computational form onto unary patch features; its inputs and outputs remain point-wise patch representations xikx_{ik} and yiky_{ik}. What the paper refers to as a "relation" is only transiently formed by combining a row patch and a column patch within a single attention pass, without being preserved or updated across layers as an independent representation. Moreover, two-hop reachability on a Rook’s Graph only guarantees short information propagation paths between patches; it does not imply that the model learns semantic relations. Consequently, RCPA functions more as a structured spatial attention mechanism that jointly selects row and column evidence, rather than the explicit relational modeling architecture claimed by the paper.

2. Flaws in experimental design: Current evaluations cannot attribute performance gains to relational modeling or to RCPA itself

ClockBench neither distinguishes relational learning from independent orientation estimation nor employs controlled training comparisons: Relative angles between clock hands can be trivially derived by independently estimating the absolute orientation of each hand and subtracting them; thus, a low angular error alone does not prove that the model has learned relational representations. Meanwhile, different models are evaluated under inconsistent batch sizes, training budgets, and checkpoint selection criteria—notably, the IID result for PA Tiny is taken from the best checkpoint prior to an early termination around epoch 55. Consequently, the performance differences in Table 1 cannot be cleanly attributed to the relational inductive bias of RCPA.

The evaluation pipeline on VG150 confounds relational modeling, entity representation quality, and detector capability: The current setup merely compares a "ViT/ViF backbone + relation prediction head" pipeline, and the authors explicitly acknowledge that stronger entity detection contributes to the end-to-end gains. In the absence of evaluations under the standard Predicate Classification (PredCls) protocol with fixed ground-truth entity inputs, or controlled experiments that solely swap the backbone within an established scene graph generation framework, it is impossible to determine whether the improvements stem from superior relational modeling, stronger general visual features, or simply better object detection.

General vision and efficiency comparisons fail to control for model capacity and computational budget: ViF-Small has approximately 53% more parameters and 43% more GFLOPs than ViT-Small, yet achieves a slightly lower ImageNet Top-1 accuracy; ViF-Base uses roughly 54% more parameters and 49% more GFLOPs for a modest gain of only 0.92 percentage points. Furthermore, ViF-Small/Base utilize 16 blocks compared to 12 blocks in the corresponding DeiT reference models. Figure 5 further reveals that the full RCPA model achieves an inference throughput of only 43.9 images/s, which is substantially slower than ViT at 176.2 images/s. Therefore, the existing results neither demonstrate that RCPA provides a genuine performance advantage under iso-parameter/iso-FLOPs budgets, nor support the claim that it is practically more efficient than ViT.

3. Other

Limited OOD evidence: The validation range of the Angle OOD split still overlaps with 5% of the training sampling distribution rather than representing a strictly unseen angular regime; moreover, RCPA results are entirely missing from the geometric deformation tests, rendering the evidence insufficient to support broad claims of relational generalization.

Lack of comparisons against established structured attention mechanisms: Although the paper discusses Axial Attention, CCNet, CSWin, and MaxViT, it omits controlled empirical comparisons against them on real-image benchmarks, leaving the incremental contribution of RCPA over existing row-column/criss-cross attention variants unclear.

Inconsistent gains across scales on general vision tasks: On COCO, ViF-Small achieves 47.0 box AP and 41.6 mask AP, trailing ViT-Small by 0.3 AP on both metrics. This indicates that downstream gains remain scale-dependent, and claims of ViF serving as a "consistently superior general backbone" warrant more cautious phrasing.

Questions for authors
  1. In what precise sense does RCPA achieve the “explicit relational modeling” claimed in the paper? Standard PA explicitly maintains and updates pairwise states R<sub>ik</sub>, whereas RCPA’s inputs and outputs remain patch features x<sub>ik</sub> and y<sub>ik</sub>. Could the authors explicitly clarify which tensor or representation in the network corresponds to a “relation,” how relational information is independently preserved and updated across layers, and how RCPA fundamentally differs in theoretical objective—not merely in computational form—from general row-column spatial feature mixing?

  2. Does ClockBench genuinely validate relational modeling rather than standard orientation estimation? Because this task can be solved by estimating the orientation of each of the three hands independently and subtracting them, low angular error alone does not prove that the model has learned relational representations. Under strictly unified training budgets and convergence criteria, does RCPA still maintain its advantage over ViT and a direct “orientation-estimation-then-subtraction” baseline? Furthermore, how do the authors account for the impact of PA’s non-unified training schedule and the omission of RCPA from the geometric deformation tests on their core conclusions?

  3. Do the improvements on VG150 truly stem from enhanced relational modeling capabilities? The current experiments primarily attach a relation prediction head to a ViF/ViT feature backbone, and the authors acknowledge that improved entity detection contributes to the end-to-end gains. If evaluated under protocols with fixed ground-truth entity inputs (such as PredCls), or by solely swapping the backbone within an identical specialized scene graph model, does ViF’s advantage in relation prediction persist? Can the authors rule out increased model capacity and stronger entity features to directly support their relational modeling claim?

  4. Have the core advantages of RCPA over existing structured attention mechanisms and ViT been rigorously validated? When strictly matched in parameter count, network depth, and training recipes, does RCPA still hold a consistent advantage over ViT? Especially given that RCPA’s full-model throughput in Figure 5 is roughly 4× lower than ViT’s, what is the primary practical contribution the authors wish to emphasize: relational modeling capability, general visual representation performance, or computational efficiency? Which specific controlled experiments in the current manuscript directly substantiate this primary contribution?

3 comments
AuthorsOpenReview ↗

Response to Reviewer HGYk (Part 1/2)

We thank the reviewer for the careful assessment and for identifying the main conceptual and experimental confounds.

Q1. In what precise sense does RCPA model relations?

Which representation is the relation, how is it preserved and updated across layers, and how does RCPA differ from general row-column mixing?

Response: The relation in RCPA is an update computed from patch-indexed states, rather than a persistent pair tensor. For target (i, k), each candidate j combines the row source (i, j) and column source (j, k). The two sources produce one combined score, one softmax weight, and one joint value update. The next layer recomputes this composition from the updated patch states.

Full PA explicitly maintains pair states, whereas RCPA retains an H x W x C feature grid and uses shared-pivot composition as an image-scale approximation. On a square grid, the update has O(T^(3/2)) interactions. Positional encoding exposes the row and column identities, and their intersection identifies the target (i, k). We agree that two-hop coverage is a connectivity result, not evidence of semantic relation learning; ClockBench and VG150 provide the task-level evidence.

The pivot controls give the following MAE:

ScalePivot settingIIDStyle OODAngle OOD
TinyShared2.2329.5532.21
TinyIndependent4.0336.2037.01
SmallShared1.9626.5023.04
SmallIndependent2.8429.3123.95

Closely related row-column controls also increase IID MAE:

ControlIID MAE change from default RCPA
Matched-scale dual softmax+1.17
Matched axial attention+5.90

These results support shared-pivot joint selection beyond row-column connectivity; they do not imply that RCPA preserves PA's full pairwise state.

Q2. Does ClockBench validate relation learning?

Does RCPA still win under a unified budget and against an orientation-then-subtraction baseline, and how do the schedule and missing deformation results affect the claim?

Response: ClockBench supports relation-task ability and appearance-stable relative geometry, but it does not identify a unique solution mechanism. The target can be reduced to estimating two hand orientations and subtracting them, so we have narrowed the claim accordingly.

The final Small results and matched Tiny learning-speed comparison are:

ComparisonModelIID MAEStyle OODAngle OOD
TinyViT-T45.2082.20—
TinyRCPA-T6.5652.85—
SmallViT-S5.9958.8532.87
SmallRCPA-S1.9626.5023.04

The matched Tiny models use the same dataset, seed, optimizer, batch size, 100-epoch budget, and checkpoint rule.

Under a color-only intervention with fixed hand geometry, RCPA retains a circular angle structure with r=0.997 and changes its prediction by 25.3 degrees on average, while ViT obtains r=0.739 and changes by 73.4 degrees. This indicates that RCPA's relative-angle representation is more stable under this appearance intervention.

We do not report a separately supervised orientation-then-subtraction baseline. The historical PA rows also use model-specific budgets, and PA Tiny stopped after its best checkpoint rather than completing the full schedule. Angle OOD has 5% training exposure for every model and is therefore low-support rather than strictly unseen. Because the deformation diagnostic did not include RCPA, we removed it from the revision and make no deformation claim.

AuthorsOpenReview ↗

Response to Reviewer HGYk (Part 2/2)

We appreciate the opportunity to clarify the real-image evidence, the remaining capacity difference, and the practical role of RCPA.

Q3. Do the Visual Genome gains come from relation prediction or object detection?

Does the gain remain with fixed ground-truth entities or an otherwise identical scene-graph model, and can capacity and stronger entity features be ruled out?

Response: PredCls fixes the boxes and object labels. The remaining gain therefore shows that ViF's backbone embeddings improve relation capture.

Over three seeds, the results are:

ModelPredCls R@100PredCls mR@100
ViT-S66.94 +/- 0.0217.18 +/- 0.03
ViF-S67.33 +/- 0.0217.72 +/- 0.06

SGDet gives the same direction: ViF-S improves R@100/mR@100 from 34.50/7.82 to 35.30/8.01, and ViF-B improves them from 35.80/8.20 to 36.40/8.47. Both models use the same Motif head and training pipeline, with the detector frozen during relation training.

The predicate analysis localizes the gain. ViF improves all four reported spatial predicates: in front of +2.62, behind +3.38, at +1.70, and on +2.85 mR@100 points. Their detection-conditioned recall also rises. Interaction predicates are mixed, so our claim is specific to the observed spatial-relation gain. Better detection contributes to aggregate SGDet, while the fixed-entity PredCls result shows that it is not the only source.

Q4. Are the gains caused by depth and compute, and what is the practical contribution?

Under matched depth, parameters, and recipes, does RCPA still help, and is the main contribution relation modeling, general representation quality, or efficiency?

Response: The gains are not explained by depth alone.

  • We re-pretrained 12-layer ViF-S/B models and evaluated them with the corresponding task protocols.
ModelDepthParams (M)GFLOPsIN1K Top-1VG R@100 / mR@100ADE20K mIoU
ViT-S1222.059.2080.5234.50 / 7.8244.13
ViF-S1225.529.9380.3435.30 / 8.0146.16
ViT-B1286.5735.1381.5235.80 / 8.2045.41
ViF-B12100.5939.2882.2236.40 / 8.4747.15
  • The paired differences and remaining capacity gaps are:
ScaleParams ΔGFLOPs ΔIN1K ΔVG R@100 / mR@100 ΔADE20K Δ
Small+15.7%+7.9%-0.18+0.80 / +0.19+2.03
Base+16.2%+11.8%+0.70+0.60 / +0.27+1.74
  • The corrected counts reduce the parameter gap from roughly 50% to about 16%. A tighter Tiny comparison gives the same relation-task pattern:
ModelParams (M)IID MAEStyle OODAngle OOD
ViT5.727.6969.8336.80
ViT-Axial6.3810.6657.4245.69
ViT-CC5.533.4047.7850.33
ViF-RCPA6.572.2329.5532.21

RCPA therefore retains the relation and dense-prediction gains after matching depth and substantially narrowing the capacity gap.

  • The efficiency comparisons are:
MeasurementFull PARCPAViT / self-attention
Small-model GFLOPs2203.679.93—
Small-model peak memory (MiB)1465.3159.4—
224px throughput (img/s)5.943.9176.2
112x112-grid core latency (ms)—4.08030.768
112x112-grid core memory (MiB)—293.45506.3

RCPA makes PA practical and exhibits the expected high-token crossover, although the 112x112 results are attention-core measurements rather than end-to-end deployment speed.

We therefore claim competitive general transfer with a task- and scale-dependent boundary.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer HGYk,

We would like to follow up on our responses to your comments. We would greatly appreciate it if you could let us know whether we have addressed your concerns or if any further clarification would be helpful. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.