- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HRoPE, a homography-based relative positional encoding for multi-view Transformers. It introduces per-patch plane-induced transformations into attention to incorporate local surface geometry. Experiments on several novel view synthesis benchmarks demonstrate improvements over existing geometric positional encodings, with additional evaluations on generalization and cross-architecture transfer.
Critical strengths
- The paper is clearly written and addresses a meaningful problem in geometric positional encoding for multi-view Transformers. Incorporating local surface information into attention is well motivated.
- The experiments cover multiple datasets and architectures, with encouraging improvements and reasonably comprehensive ablations.
Critical weaknesses
-
Limited methodological novelty. The proposed formulation largely extends existing relative geometric attention mechanisms, particularly GTA and PRoPE, by replacing camera-level transformations with per-patch homography-structured matrices. Since plane-induced homographies are well established, the fundamental novelty beyond this combination remains insufficiently demonstrated.
-
Insufficient validation of the geometric claims. The per-patch rotation, translation, and plane parameters are predicted independently without explicit physical consistency constraints. Consequently, it is unclear whether the improvements arise from meaningful surface geometry or simply from more expressive learned feature transformations. The current ablations do not fully disentangle these factors.
Questions for authors
-
Could the authors compare HRoPE against a capacity-matched, unconstrained 3×3 transformation? The existing 4×4 comparison does not fully isolate the benefit of the homography structure.
-
How can the authors justify the physical interpretation of the learned homographies when camera rotation, translation, and plane parameters are independently predicted for each patch? Quantitative cross-view geometric consistency or correspondence evaluations would be helpful.
-
To what extent do the improvements depend on pretrained geometric features such as VGGT? A more controlled comparison under matched feature sources and computational budgets would strengthen the conclusions.
6 comments
Official Response by Authors (2/4)
W2 / Q1 — Comparison with an unconstrained 3×3 transformation
Thanks for your suggestion. Following your suggestion, we add an unconstrained 3×3 baseline on both ASE and RealEstate10K. Both variants use VGGT features and 3×3 transformations, addressing the matrix-size difference in the original 4×4 comparison.
| Dataset | Transformation | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| ASE | Unconstrained 3×3 | 28.198 | 0.803 | 0.219 |
| ASE | HRoPE | 28.555 | 0.809 | 0.198 |
| RealEstate10K | Unconstrained 3×3 | 27.103 | 0.839 | 0.093 |
| RealEstate10K | HRoPE | 27.410 | 0.848 | 0.091 |
HRoPE improves PSNR by 0.357 dB on ASE and 0.307 dB on RealEstate10K, with improvements in SSIM and LPIPS on both datasets. Since both methods operate on 3×3 transformations and use the same pretrained feature source, this comparison reduces the confounding effects of transformation size and feature availability. The results support the effectiveness of HRoPE's specific geometric parameterization over directly learning an unrestricted matrix.
Official Response by Authors (3/4)
W2 / Q2 — Physical interpretation and cross-view geometric consistency
Our central contribution is to introduce a homography-inspired geometric inductive bias into relative attention. This factorization provides a geometric inductive bias that guides learning under photometric supervision. We therefore expect the learned parameters to remain closely related to the underlying scene geometry (evidented in figure 5 in original paper), even though exact recovery of physical surface normals is not guaranteed.
Following your suggestion, we distinguish the physical interpretation of calibrated homographies from the geometric inductive bias provided by their learned parameterization. We quantify cross-view normal consistency on ASE. We establish patch correspondences using GT depth and cameras, with visibility and reprojection checks. All conditions use the same GT-based correspondences and weights. We report oriented angular disagreement between corresponding normals in a common reference frame. As a control, we keep the source normals fixed and randomly permute target normals among the co-visible patches, preserving the target normal distribution while disrupting spatial correspondence.
| Normal source | Mean angle (°) ↓ | Angle <10° (%) ↑ | Angle <30° (%) ↑ |
|---|---|---|---|
| HRoPE (VGGT) | 15.41 | 60.56 | 86.89 |
| HRoPE (VGGT), spatially shuffled | 55.36 | 21.97 | 35.33 |
| HRoPE (ASE GT supervision) | 10.40 | 72.47 | 90.87 |
| HRoPE (ASE GT supervision), spatially shuffled | 76.46 | 18.23 | 25.17 |
| GT planes fitted independently in each view | 7.54 | 81.21 | 91.02 |
The predicted normals agree more closely at GT-matched locations than under spatial shuffling. This supports correspondence-related organization of the learned normal directions beyond their marginal distribution. The ASE-supervised predictions achieve still lower angular disagreement. Together with the transformation and orientation ablations, these results support local surface geometry as a useful inductive bias for attention.
Correspondence evaluation. We additionally investigate whether HRoPE improves the geometric alignment of cross-view attention itself (Appendix A.8). We compare HRoPE (VGGT), HRoPE (learned, without geometric supervision), PRoPE, and LVSM at the third LVSM layer, using 9,579 geometrically valid queries across 22 ASE scenes and both directions between the two input views. GT correspondences are obtained by depth-based reprojection with visibility and reprojection checks. For each query, we average the full-sequence softmax probabilities over all 16 heads, then normalize the weights within the other input view. Metrics are averaged within each scene and then equally across scenes.
GT mass@14 measures the conditional attention probability assigned to key-patch centers within 14 pixels of the GT correspondence. PCK@14 measures how often the highest-attention patch center lies within that radius. Uniform enrichment@14 divides the GT-neighborhood mass by the probability that a uniform distribution over the 576 key patches would assign to the same neighborhood, computed per query before aggregation.
| Method | GT mass@14 ↑ | PCK@14 ↑ | Uniform enrichment@14 ↑ |
|---|---|---|---|
| HRoPE (VGGT) | 32.03% | 68.91% | 60.72× |
| HRoPE (learned, without geometric supervision) | 20.57% | 60.76% | 37.03× |
| PRoPE | 18.39% | 50.03% | 34.66× |
| LVSM | 10.00% | 39.48% | 18.73× |
Compared with PRoPE, HRoPE (VGGT) improves GT mass@14 by 13.64 percentage points and PCK@14 by 18.88 points. Its attention is therefore more concentrated around geometrically corresponding regions, rather than being distributed across unrelated patches. Learned HRoPE without geometric supervision also exceeds PRoPE on all three metrics, improving GT mass@14 and PCK@14 by 2.18 and 10.73 percentage points, respectively, and uniform enrichment@14 from 34.66× to 37.03×. We have supplemented these quantitative results with attention heatmaps using identical queries and a shared visualization protocol, marking the GT correspondence and attention peak (Fig. 8).
Together, the normal-consistency and correspondence experiments provide complementary evidence that HRoPE's learned geometric modulation is associated with improved cross-view organization and attention alignment. These observations support its geometric inductive bias without requiring exact recovery of physical plane parameters.
Official Response by Authors (1/4)
We thank the reviewer for the constructive feedback and for recognizing the motivation of incorporating local surface information into attention. We appreciate the concerns regarding methodological novelty, geometric interpretation, and the contribution of pretrained features, and address them below.
W1 — Contribution: a surface-aware geometric inductive bias for attention
Our focus isn't on introducing a new plane-induced homography, but introduce local surface-awareness into token-level relative attention in multi-view transformer.
Existing camera-level encodings such as GTA and PRoPE encode camera relationships, with a camera-level geometric factor shared across patches in a view. Although point- and ray-based approaches incorporate finer geometric information, they do not explicitly parameterize the local surface orientation that determines neighborhood deformation across views. For example, two surfaces passing through the same 3D point can have different orientations and therefore induce different perspective deformations, despite sharing the same center position and depth. HRoPE incorporates this additional local geometric variable into attention.
The key contribution of HRoPE is a structured, surface-inspired parameterization of relative attention, rather than an unconstrained feature-dependent transformation. The rotation-plus-rank-one factorization, normalized directions, and bounded corrections introduce an optimization bias motivated by classical plane-induced geometry, while retaining an efficient closed-form inverse. Importantly, this geometric bottleneck constrains how features modulate attention, without restricting the high-dimensional visual representations themselves.
We empirically validate its contribution through controlled comparisons. The new unconstrained 3×3 comparison below evaluates the transformation structure while retaining the matrix size and VGGT feature source. We also compare learned normals with fixed fronto-parallel normals on RealEstate10K, using the same VGGT features, backbone, and training protocol. Learning surface orientation improves PSNR from 26.946 to 27.410, SSIM from 0.837 to 0.848, and LPIPS from 0.096 to 0.091. These results support the benefit of explicitly parameterizing local surface orientation in attention.