acceptodds
Under review as a conference paper at ICLR 2027

HROPE: HOMOGRAPHY RELATIVE POSITIONAL ENCODING

20 views

Abstract

Geometric Relative Positional Encoding has become the primary mechanism for injecting spatial inductive biases into multi-view transformers. Existing designs condition attention on camera-level frustums or ray-marched depth, associating cross-view tokens through the sensor geometry. Such sensor-centric formulations, however, describe only where a token is observed from, leaving the rich surface geometry of the scene that the token depicts unmodeled. In this paper, we advocate a shift toward surface-aware positional encoding that explicitly grounds patch tokens on the physical surfaces. We introduce HRoPE, which represents the local geometry of each patch as a slanted 3D plane, following the slanted support-window assumption of classical multi-view stereo. We inject token-level, plane-induced homographies into attention as relative transformations. This explicitly enables the transformer to rectify perspective distortions and align patch features across viewpoints rather than having to learn this alignment implicitly from data. By construction, this design preserves the global frame invariance of prior relative encodings, while explicitly enabling the transformer to rectify perspective distortions and align patch features across viewpoints rather than having to learn this alignment implicitly from data. Our encoding requires no ground-truth geometry: the plane parameters can be predicted end-to-end under a view-synthesis objective alone, or derived from foundation models. Extensive experiments on RealEstate10K, CO3D, and DL3DV show that HRoPE consistently outperforms the evaluated geometric positional encodings for novel view synthesis. Extensive ablations and analysis shows the effectiveness of the proposed encoding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Homography, Positional Encoding, Novel View Synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 5 reviews, above 85% of the others.
Reviewer_Nm6WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HRoPE, a homography-based relative positional encoding for multi-view Transformers. It introduces per-patch plane-induced transformations into attention to incorporate local surface geometry. Experiments on several novel view synthesis benchmarks demonstrate improvements over existing geometric positional encodings, with additional evaluations on generalization and cross-architecture transfer.

Critical strengths
  1. The paper is clearly written and addresses a meaningful problem in geometric positional encoding for multi-view Transformers. Incorporating local surface information into attention is well motivated.
  2. The experiments cover multiple datasets and architectures, with encouraging improvements and reasonably comprehensive ablations.
Critical weaknesses
  1. Limited methodological novelty. The proposed formulation largely extends existing relative geometric attention mechanisms, particularly GTA and PRoPE, by replacing camera-level transformations with per-patch homography-structured matrices. Since plane-induced homographies are well established, the fundamental novelty beyond this combination remains insufficiently demonstrated.

  2. Insufficient validation of the geometric claims. The per-patch rotation, translation, and plane parameters are predicted independently without explicit physical consistency constraints. Consequently, it is unclear whether the improvements arise from meaningful surface geometry or simply from more expressive learned feature transformations. The current ablations do not fully disentangle these factors.

Questions for authors
  1. Could the authors compare HRoPE against a capacity-matched, unconstrained 3×3 transformation? The existing 4×4 comparison does not fully isolate the benefit of the homography structure.

  2. How can the authors justify the physical interpretation of the learned homographies when camera rotation, translation, and plane parameters are independently predicted for each patch? Quantitative cross-view geometric consistency or correspondence evaluations would be helpful.

  3. To what extent do the improvements depend on pretrained geometric features such as VGGT? A more controlled comparison under matched feature sources and computational budgets would strengthen the conclusions.

6 comments
AuthorsOpenReview ↗

Official Response by Authors (1/4)

We thank the reviewer for the constructive feedback and for recognizing the motivation of incorporating local surface information into attention. We appreciate the concerns regarding methodological novelty, geometric interpretation, and the contribution of pretrained features, and address them below.

W1 — Contribution: a surface-aware geometric inductive bias for attention

Our focus isn't on introducing a new plane-induced homography, but introduce local surface-awareness into token-level relative attention in multi-view transformer.

Existing camera-level encodings such as GTA and PRoPE encode camera relationships, with a camera-level geometric factor shared across patches in a view. Although point- and ray-based approaches incorporate finer geometric information, they do not explicitly parameterize the local surface orientation that determines neighborhood deformation across views. For example, two surfaces passing through the same 3D point can have different orientations and therefore induce different perspective deformations, despite sharing the same center position and depth. HRoPE incorporates this additional local geometric variable into attention.

The key contribution of HRoPE is a structured, surface-inspired parameterization of relative attention, rather than an unconstrained feature-dependent transformation. The rotation-plus-rank-one factorization, normalized directions, and bounded corrections introduce an optimization bias motivated by classical plane-induced geometry, while retaining an efficient closed-form inverse. Importantly, this geometric bottleneck constrains how features modulate attention, without restricting the high-dimensional visual representations themselves.

We empirically validate its contribution through controlled comparisons. The new unconstrained 3×3 comparison below evaluates the transformation structure while retaining the matrix size and VGGT feature source. We also compare learned normals with fixed fronto-parallel normals on RealEstate10K, using the same VGGT features, backbone, and training protocol. Learning surface orientation improves PSNR from 26.946 to 27.410, SSIM from 0.837 to 0.848, and LPIPS from 0.096 to 0.091. These results support the benefit of explicitly parameterizing local surface orientation in attention.

AuthorsOpenReview ↗

Official Response by Authors (2/4)

W2 / Q1 — Comparison with an unconstrained 3×3 transformation

Thanks for your suggestion. Following your suggestion, we add an unconstrained 3×3 baseline on both ASE and RealEstate10K. Both variants use VGGT features and 3×3 transformations, addressing the matrix-size difference in the original 4×4 comparison.

DatasetTransformationPSNR ↑SSIM ↑LPIPS ↓
ASEUnconstrained 3×328.1980.8030.219
ASEHRoPE28.5550.8090.198
RealEstate10KUnconstrained 3×327.1030.8390.093
RealEstate10KHRoPE27.4100.8480.091

HRoPE improves PSNR by 0.357 dB on ASE and 0.307 dB on RealEstate10K, with improvements in SSIM and LPIPS on both datasets. Since both methods operate on 3×3 transformations and use the same pretrained feature source, this comparison reduces the confounding effects of transformation size and feature availability. The results support the effectiveness of HRoPE's specific geometric parameterization over directly learning an unrestricted matrix.

AuthorsOpenReview ↗

Official Response by Authors (3/4)

W2 / Q2 — Physical interpretation and cross-view geometric consistency

Our central contribution is to introduce a homography-inspired geometric inductive bias into relative attention. This factorization provides a geometric inductive bias that guides learning under photometric supervision. We therefore expect the learned parameters to remain closely related to the underlying scene geometry (evidented in figure 5 in original paper), even though exact recovery of physical surface normals is not guaranteed.

Following your suggestion, we distinguish the physical interpretation of calibrated homographies from the geometric inductive bias provided by their learned parameterization. We quantify cross-view normal consistency on ASE. We establish patch correspondences using GT depth and cameras, with visibility and reprojection checks. All conditions use the same GT-based correspondences and weights. We report oriented angular disagreement between corresponding normals in a common reference frame. As a control, we keep the source normals fixed and randomly permute target normals among the co-visible patches, preserving the target normal distribution while disrupting spatial correspondence.

Normal sourceMean angle (°) ↓Angle <10° (%) ↑Angle <30° (%) ↑
HRoPE (VGGT)15.4160.5686.89
HRoPE (VGGT), spatially shuffled55.3621.9735.33
HRoPE (ASE GT supervision)10.4072.4790.87
HRoPE (ASE GT supervision), spatially shuffled76.4618.2325.17
GT planes fitted independently in each view7.5481.2191.02

The predicted normals agree more closely at GT-matched locations than under spatial shuffling. This supports correspondence-related organization of the learned normal directions beyond their marginal distribution. The ASE-supervised predictions achieve still lower angular disagreement. Together with the transformation and orientation ablations, these results support local surface geometry as a useful inductive bias for attention.

Correspondence evaluation. We additionally investigate whether HRoPE improves the geometric alignment of cross-view attention itself (Appendix A.8). We compare HRoPE (VGGT), HRoPE (learned, without geometric supervision), PRoPE, and LVSM at the third LVSM layer, using 9,579 geometrically valid queries across 22 ASE scenes and both directions between the two input views. GT correspondences are obtained by depth-based reprojection with visibility and reprojection checks. For each query, we average the full-sequence softmax probabilities over all 16 heads, then normalize the weights within the other input view. Metrics are averaged within each scene and then equally across scenes.

GT mass@14 measures the conditional attention probability assigned to key-patch centers within 14 pixels of the GT correspondence. PCK@14 measures how often the highest-attention patch center lies within that radius. Uniform enrichment@14 divides the GT-neighborhood mass by the probability that a uniform distribution over the 576 key patches would assign to the same neighborhood, computed per query before aggregation.

MethodGT mass@14 ↑PCK@14 ↑Uniform enrichment@14 ↑
HRoPE (VGGT)32.03%68.91%60.72×
HRoPE (learned, without geometric supervision)20.57%60.76%37.03×
PRoPE18.39%50.03%34.66×
LVSM10.00%39.48%18.73×

Compared with PRoPE, HRoPE (VGGT) improves GT mass@14 by 13.64 percentage points and PCK@14 by 18.88 points. Its attention is therefore more concentrated around geometrically corresponding regions, rather than being distributed across unrelated patches. Learned HRoPE without geometric supervision also exceeds PRoPE on all three metrics, improving GT mass@14 and PCK@14 by 2.18 and 10.73 percentage points, respectively, and uniform enrichment@14 from 34.66× to 37.03×. We have supplemented these quantitative results with attention heatmaps using identical queries and a shared visualization protocol, marking the GT correspondence and attention peak (Fig. 8).

Together, the normal-consistency and correspondence experiments provide complementary evidence that HRoPE's learned geometric modulation is associated with improved cross-view organization and attention alignment. These observations support its geometric inductive bias without requiring exact recovery of physical plane parameters.

Reviewer_4MFoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HRoPE (homography relative positional encoding), which is a geometric positional encoding mechanism for multi-view transformers. Different from existing camera-based or ray-based relative positional encoding, HRoPE incorporates surface orientation by modeling local patch regions as slanted 3D planes. It predicts parameters of plane-induced homographies into attention to take patch-level image warping into consideration. The method is evaluated primarily on novel view synthesis and it outperforms existing position encoding methods. The additional validation proves that the local geometry information is leading to the performance improvement.

Critical strengths
  1. It proposed a patch-level positional encoding method based on local surface information, which is novel and different from existing point-based methods. The intuition of taking multi-view geometry into consideration is also theoretically sound.

  2. The proposed method outperforms existing PE methods on the novel view synthesis task on the benchmarking datasets. There are also thorough ablation experiments showing the effectiveness of the homography inductive bias instead of coming from pure feature capacity.

Critical weaknesses
  1. HRoPE benefits most from the pre-trained VGGT features which is a large foundation model. Table 4 indicates that using VGGT more than doubles the training cost (76 GPU-hours vs. 33 GPU-hours for baseline LVSM or the scratch-trained variant). The paper does not provide inference runtime latency, peak GPU memory overhead or FLOPs comparisons for computing and multiplying the per-patch 3x3 matrices.
Questions for authors
  1. What is the inference latency of HRoPE compared to 2D RoPE or PRoPE?

  2. If an image patch contains a non-planar region, will this slanted plane assumption hurt the performance? Do you have an example and analysis?

4 comments
AuthorsOpenReview ↗

Official Response by Authors (1/2)

We thank the reviewer for the constructive feedback and for recognizing the motivation behind surface-aware positional encoding and the supporting ablations. We address your comments below.

W1 / Q1 — Inference latency and memory cost

Thanks for the suggestion, we measure inference latency and peak GPU memory for LVSM, 2D RoPE, PRoPE, and HRoPE under the same input setting: a batch size of 4, two reference views and one target view at 336 × 336 resolution, on a single RTX 4090(48G) with FP16 autocast. The HRoPE results use our VGGT- and DA3-based variants, including feature extraction. We use CUDA Graphs in all experiments to reduce scheduling overhead. Latency is averaged per batch, excluding loading, warm-up, and graph capture.

MethodInference latency (ms/batch)Peak GPU memory (GiB)
LVSM22.910.381
2D RoPE24.940.389
PRoPE25.750.400
HRoPE (DA3)42.191.719
HRoPE (VGGT)78.156.170

The VGGT-based variant has higher latency and memory usage, while achieving the highest RealEstate10K PSNR in the parameter-source ablation (Sec. 4.2.1, Fig. 4(b)). This ablation fixes the HRoPE parameterization and compares a predictor trained from scratch with pretrained feature sources, including DINOv2, DA3, and VGGT. Table 4 reports the corresponding training costs, showing that the choice of feature source provides an accuracy–cost trade-off within the same encoding formulation. DA3 therefore offers a good balance between reconstruction quality and inference and training costs.

We would also like to clarify the scope of our work. Our primary goal is to investigate a generalizable geometric positional encoding mechanism, rather than to develop a computationally optimized NVS system. We adopt novel view synthesis as a representative testbed because it provides a direct and measurable way to evaluate the effectiveness of geometric inductive biases in multi-view attention. While inference efficiency remains an important practical consideration, our emphasis is on validating the proposed surface-aware encoding mechanism. The additional computational cost of our strongest configuration comes partly from the pretrained VGGT feature extractor, rather than the homography modulation alone. As demonstrated by our parameter-source ablations, HRoPE also supports lightweight predictors trained from scratch, providing flexibility in balancing geometric modeling capability and computational cost across different applications.

AuthorsOpenReview ↗

Official Response by Authors (2/2)

Q2 — Non-planar regions and the local plane assumption

Thank you for raising this important question. The local plane assumption is an approximation rather than a requirement that the underlying scene be globally planar. Each image token covers a local image region, whose underlying surface can often be approximated by a plane, following the local planar assumption used in classical multi-view stereo. For smooth curved surfaces, this provides a first-order approximation, while independently predicted per-patch transformations allow the local orientation to vary across the surface.

Importantly, the default learned HRoPE does not impose a hard planar reconstruction constraint or perform explicit pixel warping. In HRoPE, this approximation provides geometric guidance for attention. The plane parameters determine how feature interactions are modulated, while the tokens retain their high-dimensional visual representations. The backbone can therefore continue to encode texture, contours, and variation within a patch; the encoding does not require the reconstructed content to be planar. This separates the local geometric prior from the model's capacity to represent fine visual detail.

We therefore view local planarity as a useful geometric inductive bias for approximately smooth surface regions, rather than a universally accurate geometric representation.

Reviewer_4MFoOpenReview ↗

Reply to rebuttal

I want to thank the authors for the detailed response and for providing the inference latency and memory benchmarks across feature backbones. The numbers clearly illustrate the accuracy/ efficiency trade-off, and the clarification regarding the local plane acting as a soft attention prior rather than a hard warping constraint addresses my question on non-planar regions. Please make sure to include the inference benchmark table in the revised appendix. My concerns are resolved

Thank you for your constructive feedback. We are glad that our responses have addressed your concerns, and we will include the inference benchmark table in the appendix of the final version of the paper.

Reviewer_aTWyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes HRoPE, a geometric relative positional encoding for multi-view transformers. Existing methods such as PRoPE and RayRoPE incorporate camera intrinsic and extrinsic geometry and estimated 3D locations of patches into attention, but they do not explicitly model the orientation of those patches. The authors propose representing each patch using a local tangent plane, motivated by the "slanted support-window formulation" in classical multi-view stereo. A plane provides a compact representation of how the neighborhood of the image patch undergoes perspective deformation across different viewpoints, as opposed to simply describing the position of its centre. This provides the model with a richer inductive-bias representation that it can learn to exploit.

The proposed method constructs a homography-structured transformation for each patch using neural network-predicted rotation, translation, surface normal, and inverse plane distance parameters. The relative composition of these transformations is incorporated into the transformer's attention scores (and also later into value vectors), analogous to how RoPE encodes relative positional information. The parameters are learned through view-synthesis supervision without requiring ground-truth surface geometry.

The method is evaluated primarily on novel view synthesis, using LVSM across RealEstate10K, CO3Dv2, Objaverse, and DL3DV. The paper also includes ablations of the transformation parameterization, a calibrated-camera variant, and experiments transferring the method to a diffusion-based view-synthesis model and a multimodal LLM.

Critical strengths
  1. The core contribution is moderately novel and interesting. Incorporating local planar orientation into attention provides a richer geometric inductive bias than simply representing the 3D position of patches. The paper introduces a meaningful extension by replacing camera-level transformations with per-patch transformations, allowing attention to capture differences in local surface geometry, even between patches of the same image. The proposed parameterization, along with its efficient closed-form inverse is a technically sound contribution. I believe this is a useful insight that could motivate further work on geometric positional encodings.

  2. The method is empirically well supported, with consistent improvements across multiple datasets and architectures. The ablations are comprehensive.

Critical weaknesses
  1. The paper's central contribution is incorporating local surface orientation into attention, but the ablations do not directly isolate the benefits of this orientation information. A controlled comparison between 1) flexible and learned surface normals; and 2) fixed fronto-parallel normals within the same HRoPE formulation would help establish the specific contribution of learning surface orientation.

  2. The strongest improvements appear to rely considerably on pretrained geometric features. In Table 1, HRoPE trained from scratch achieves 26.419 PSNR on RealEstate10K compared with 26.836 for URoPE, whereas HRoPE with VGGT reaches 27.410. The paper does not directly compare HRoPE and URoPE with access to the same pretrained geometric features. Such a comparison would help establish how much of the improvement comes specifically from the proposed positional encoding rather than the additional geometric information provided by VGGT.

Questions for authors
  1. How does the target pose enter the model when target tokens use A = I? Are Plücker rays kept for all methods (which LVSM uses)? What exactly does “without ground-truth camera poses” mean?
5 comments
AuthorsOpenReview ↗

Official Response by Authors (1/3)

We thank the reviewer for the constructive feedback and for recognizing the contribution of incorporating local surface geometry into positional encoding. We address each concern below.

W1 — Contribution of surface orientation

Thanks for raising the important issue. Following your suggestion, we conduct a controlled ablation comparing learned per-patch normals against fixed fronto-parallel normals within the same HRoPE formulation on RealEstate10K. In the fixed-normal variant, the normal direction is held constant throughout training and inference, while rotation, translation, and inverse plane distance remain learnable. Both variants use identical VGGT features, LVSM backbones, and training protocols, isolating the effect of allowing the normal direction to adapt to each patch.

MethodPSNR ↑SSIM ↑LPIPS ↓
HRoPE (fixed fronto-parallel normals)26.9460.8370.096
HRoPE (learned normals)27.4100.8480.091

Learning the per-patch normals improves PSNR by 0.464 dB, with higher SSIM and lower LPIPS. With the feature source held fixed, these results support the benefit of adaptive surface orientation within HRoPE.

AuthorsOpenReview ↗

Official Response by Authors (2/3)

W2 — Pretrained features and the geometric inductive bias

We agree that pretrained geometric features contribute substantially to our best values. However, HRoPE does not depend on VGGT to be effective, the key aspect of our design is the geometric inductive bias that parameterized in homography-structure , which constrains how the features modulate token interactions and allows the same encoding to use features learned from scratch, DINOv2, VGGT, or other sources.

First, HRoPE remains effective without pretrained 3D features. As shown in Table 1, on RealEstate10K, HRoPE with a predictor trained from scratch achieves 26.419 dB, outperforming PRoPE (26.142 dB) by 0.277 dB. Using DINOv2 features further improves performance to 26.701 dB (+0.559 dB over PRoPE), without requiring a pretrained 3D geometry backbone. These results demonstrate that HRoPE's benefits are not exclusive to VGGT-based features.

Second, our existing ablations isolate the effect of transformation structure under a fixed feature source. In Fig. 4(a), we use the same frozen VGGT features to compare different transformation parameterizations and an additive positional encoding. HRoPE achieves the best PSNR among these variants, showing that the way pretrained features modulate attention matters, rather than the performance gains arising solely from access to VGGT.

Regarding a VGGT-conditioned URoPE baseline, URoPE is originally a parameter-free encoding based on predefined depth anchors and calibrated camera geometry. Unlike HRoPE, it does not directly consume image features to predict geometric transformations. Incorporating VGGT would therefore require an additional feature-integration design, such as using VGGT-derived depth estimates or injecting its features into the token representations. This would constitute a modified URoPE variant rather than its original formulation. Such a comparison would vary both the feature-integration design and the positional encoding, so matching the feature source alone would not isolate the effect of the encoding. We acknowledge that such a comparison could provide further insight. Our current evidence instead consists of complementary controls: evaluations without pretrained 3D features and matched-VGGT comparisons across different encoding structures, including the new fixed-normal ablation above.

Together, these results support our central claim that the homography-inspired geometric inductive bias contributes to performance, while stronger pretrained geometric features provide additional benefits.

AuthorsOpenReview ↗

Official Response by Authors (3/3)

Q1 — Target-pose conditioning and Plücker-ray inputs

HRoPE retains LVSM’s Plücker-ray inputs for both reference and target views. Target-view tokens are initialized from rays computed using the target camera pose, so they already carry pose information when the target-side transform in Algorithm 1 is applied. In contrast, the GTA, PRoPE, RayRoPE, and URoPE baselines in Table 1 use camera-frame rays (CamRay) as token inputs, with camera extrinsics incorporated through their respective geometric positional encodings.

We agree that the phrase "without ground-truth cameras" in Sec. 1 may cause misunderstanding. Our intended claim concerns the predicted HRoPE transformation parameters, which are inferred from image features without requiring ground-truth camera extrinsics, rather than the entire NVS framework. Camera intrinsics are still used in constructing the encoding, and the LVSM backbone retains its camera-conditioned ray inputs. We have revised Sec. 1 (L.103-104), Sec. 3.3 (L.284-288), and Sec. 4.1 (L.317-318) to clarify this distinction.

Importantly, HRoPE is not intrinsically tied to Plücker-ray conditioning. In our MLLM experiment (Appendix A.7), HRoPE is applied directly to the visual tokens of LLaVA-Video, which are mapped to per-patch transformations and incorporated into the language model's visual-token attention. No explicit camera poses or Plücker-ray inputs are required in this setting. This demonstrates that HRoPE can also be integrated into architectures without LVSM-style camera-conditioned input tokens.

Reviewer_JzFaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed technique, HRoPE, extends PRoPE. While PRoPE uses one camera matrix per view, HRoPE uses one matrix per patch: H=K(R+κ tn⊤)H = K(R + \kappa\, t n^\top), shaped like a plane-induced homography. The parameters come from patch features (learned from scratch, DINOv2, or frozen VGGT). Attention depends on Hi,uHj,v−1H_{i,u} H_{j,v}^{-1}. Then calibration-based version combines PRoPE's camera matrix with a per-patch plane (P⋅FP \cdot F). With VGGT features, HRoPE beats PRoPE by 0.53, 1.27 and 1.03 dB PSNR on CO3Dv2, RealEstate10K and Objaverse. Furthermore, the results on EscherNet and a video MLLM are reported in the appendix.

Critical strengths
  1. Earlier encodings only give information about where a patch is. But HRoPE also tells 'how the patch is tilted'. This comes from PatchMatch Stereo. None of the compared encodings models tilt.

  2. The inverse has an exact closed form.

  3. Ablation coverage is good. The rank-one term adds about 0.5 dB over a pure SO(3)SO(3) transform. SO(3)SO(3) is perfectly conditioned, so the gain is not only from stable numbers.

  4. VGGT never trained on RealEstate10K, and HRoPE still improves there, even without VGGT (Learnable: +0.28 dB).

Critical weaknesses

1. The main version has no target-camera information: Input tokens use H=KAH = K A, where AA is predicted from image features with no camera pose. Target tokens use only H=KH = K (Algorithm 1). So the target-source factor is KtAj,v−1Kj−1K_t A_{j,v}^{-1} K_j^{-1}. It contains no target pose. VGGT sees only the input images, so AA cannot know the target camera. PRoPE does encode this relation (PtPj−1P_t P_j^{-1}). HRoPE only encodes geometry 'between input views'. The target viewpoint must come from LVSM's Plucker-ray tokens. RayRoPE and URoPE keep the real cameras and add depth. HRoPE removes the cameras from the encoding completely. The paper says HRoPE beats PRoPE "without ground-truth camera poses." If the Plucker rays still use ground-truth poses, this is true only for the encoding, not the whole model.

2. The "plane" meaning is not guaranteed : Attention only sees A=R+κ tn⊤A = R + \kappa\, t n^\top. Since RR and tt are free for each patch, κ tn⊤\kappa\, t n^\top can be any small rank-one matrix. Nothing forces it to describe a real plane. Splitting AA into (R,t,n,κ)(R, t, n, \kappa) is not unique. Homography decomposition has a known ambiguity, and the scale between κ\kappa and ∥t∥\|t\| is free. So the predicted normals may not be real surface normals.

3. The gains may come from VGGT, not from geometry. : VGGT was trained with 3D labels on Co3Dv2, DL3DV, ScanNet, ASE and Objaverse-like data. This overlaps with most test datasets. Without VGGT, the gain over PRoPE is small: +0.12 dB on CO3Dv2 and +0.28 dB on RealEstate10K (Learnable). With VGGT, the additive encoding gets within about 0.2–0.3 dB of HRoPE. The introduction says additive injection "sharply degrades performance," but this holds only without VGGT. Ground-truth geometry on ASE adds only 0.20 dB. But VGGT trained on ASE, so this test is not informative. The main model has only ~25M parameters. It is not shown that the benefit stays at larger scale.

4. Some claims are too strong:

  • "Rectifies perspective distortions": HH multiplies 3-dim feature chunks, no pixels are warped. What could be shown is that attention looks at the locations the homography predicts.
  • "Strictly generalizes point- and ray-based PEs": with a fronto-parallel normal, HRoPE becomes a depth-only homography, not RayRoPE's multi-frequency encoding.
  • "Existing PEs assume fronto-parallel surfaces": true for RayRoPE (its three corner rays share one depth). Not true for PRoPE, which has no depth, so it assumes nothing about tilt.

5. The evidence is weak for such small gains: Many differences are only a few tenths of a dB, from single runs with no variance. Baseline rankings disagree with other papers. The URoPE paper ranks RayRoPE below PRoPE on RealEstate10K, but this paper ranks it above. RayRoPE reports the best LPIPS on CO3D, but here it has the worst among the encodings. Also, it is not said which HRoPE version is shown in the out-of-distribution plots (Fig. 7).

Questions for authors
  1. Do you keep the Plucker-ray tokens for input and target views? Does the Learnable predictor use hidden states from each layer? Does adding PRoPE's target–source factor PtPj−1P_t P_j^{-1} help?

  2. How accurate are the predicted normals compared with ASE ground truth? What happens if you perturb nn at test time? Does a simple "rotation + rank-one" version, without plane meaning, work equally well?

  3. Can you show that attention looks where the predicted homography says it should?

  4. What are the exact numbers for the additive VGGT encoding in Fig. 4a? How does PRoPE do with the same VGGT features, on all datasets?

  5. Please add VLM-3R to Table 6, and train your own control with the same backbone, epochs, frames and VGGT branch.

  6. How many seeds did you run? How did you configure the baselines? Which HRoPE version is in Fig. 7?

6 comments
AuthorsOpenReview ↗

Official Response by Authors (1/6)

We thank the reviewer for the detailed feedback and for recognizing the value of modeling local surface orientation, the closed-form inverse, and the ablation studies. We address the camera-conditioning, the role of the geometric parameterization, and the attribution of the gains below.

W1 / Q1 — Camera conditioning and the Learnable predictor

We retain LVSM's Plücker-ray inputs for both reference and target views, hence the target tokens carry the information specifying the requested viewpoint. We agree that "without ground-truth cameras" in Sec. 1 may cause confusion. Our intended claim is HRoPE’s per-patch transformation parameters are inferred from hidden states (learnable HRoPE setting) or pretraining features (VGGT and DINO setttings) without requiring ground-truth camera extrinsics as inputs to the encoding. We have revised Sec. 1 (L.103-104), Sec. 3.3 (L.284-288), and Sec. 4.1 (L.317-318) to make this distinction explicit.

For the Learnable predictor, the HRoPE parameters indeed are predicted from reference-view hidden states at each transformer layer.

On adding PRoPE's target–source factor: We also evaluate a variant that incorporates explicit camera-pose conditioning through PRoP through replacing the complementary 2D RoPE component with PRoPE and uses CamRay token initialization. PRoPE supplies the relative camera factor PtPj−1P_tP_j^{-1}, allowing this configuration to use CamRay in place of Plücker-ray initialization. Both configurations below use VGGT features for HRoPE.

Encoding on RealEstate10KToken initializationPSNR ↑SSIM ↑LPIPS ↓
HRoPE + PRoPECamRay27.1870.8430.096
HRoPE + 2D RoPEPlücker rays27.4100.8480.091

The HRoPE + PRoPE + CamRay configuration achieves 27.187 dB, demonstrating that HRoPE can operate with camera poses supplied through the positional encoding. Our default configuration achieves 0.223 dB higher PSNR, with better SSIM and LPIPS. This comparison evaluates two camera-conditioning configurations, jointly changing the complementary encoding and token initialization.

The calibrated P⋅FP\cdot F instance in Sec. 3.3 provides complementary evidence: it retains PRoPE's camera projection and adds a local surface frame in a 4×4 construction. On RealEstate10K, it improves PSNR from 26.142 for PRoPE to 27.115 (Table 3), supporting the usefulness of local surface geometry alongside calibrated cameras.

AuthorsOpenReview ↗

Official Response by Authors (2/6)

W2 / Q2 — Plane grounding, normal consistency, and perturbation

Our central contribution is to introduce a homography-inspired geometric inductive bias into relative attention.

The transformation A=R+κtn⊤A=R+\kappa tn^\top follows the algebraic structure of a plane-induced homography, a classical geometric model of cross-view correspondence. This factorization provides a geometric inductive bias that guides learning under photometric supervision. We therefore expect the learned parameters to remain closely related to the underlying scene geometry (please see figure 5 in original paper), even though exact recovery of physical surface normals is not guaranteed.

To confirm HRoPE's parameters meaning and demonstrate the resemblance to physical surface parameters, we conduct three complementary analyses.

1. Cross-view normal consistency. On ASE, we use GT depth and cameras to establish cross-view correspondences, with visibility and round-trip consistency checks. We then evaluate the angular disagreement between corresponding predicted directions after expressing them in a common evaluation frame. As a control, we spatially shuffle the predicted directions among co-visible patches, preserving their marginal distribution while disrupting spatial correspondence.

Normal sourceCross-view mean angle (°) ↓Angle <10° (%) ↑Angle <30° (%) ↑
HRoPE (VGGT)15.4160.5686.89
HRoPE (VGGT), spatially shuffled55.3621.9735.33
HRoPE (ASE GT supervision)10.4072.4790.87
HRoPE (ASE GT supervision), spatially shuffled76.4618.2325.17
GT planes fitted independently in each view7.5481.2191.02

The learned directions show substantially stronger cross-view consistency than their spatially shuffled counterparts. This indicates that the predictions exhibit meaningful correspondence-related spatial organization. Importantly, this is a cross-view consistency diagnostic rather than a measurement of predicted-versus-GT normal accuracy, and does not require a uniquely identifiable physical decomposition.

2. Test-time direction perturbation. We perturb normals as n′=normalize⁡(n+σϵ)n'=\operatorname{normalize}(n+\sigma\epsilon), with ϵ∼N(0,I3)\epsilon\sim\mathcal{N}(0,I_3), and recompute the transformation and its inverse while holding the other parameters fixed. The angle column reports the mean perturbation relative to the unperturbed normal.

Parameter sourceNoise σ\sigmaMean perturbation (°)PSNR ↑SSIM ↑LPIPS ↓
HRoPE (VGGT)0028.5550.8090.198
HRoPE (VGGT)0.053.5928.5160.8070.195
HRoPE (VGGT)0.2517.9527.4530.7510.372
HRoPE (VGGT)0.5035.1426.0840.6840.541
HRoPE (GT)0028.7510.8200.192
HRoPE (GT)0.053.5928.6030.8180.198
HRoPE (GT)0.2517.9527.9580.8090.242
HRoPE (GT)0.5035.1427.2940.8000.281

For the VGGT version, a small 3.59° perturbation changes PSNR by only −0.039 dB, whereas 17.95° and 35.14° perturbations reduce it by 1.102 and 2.471 dB. The GT instance also degrades as the perturbation grows. These results show that the directional factors actively influence reconstruction, although sensitivity alone does not establish physical normal accuracy.

3. Rotation plus free rank-one control. Following the suggestion, we evaluate A=R+uv⊤A=R+uv^\top using VGGT features, with freely learned u,vu,v and the same rotation construction. The control bounds the rank-one update by the same worst-case spectral-norm limit as HRoPE and uses the same closed-form inverse.

Parameterization on ASEPSNR ↑SSIM ↑LPIPS ↓
Rotation + free rank-one (VGGT)28.0920.7990.227
HRoPE (VGGT)28.5550.8090.198

HRoPE improves PSNR by 0.463 dB. The two formulations have comparable algebraic expressive power, since the factor (\kappa t) can be absorbed into a free vector. Therefore, we do not interpret this result as proving a strictly smaller transformation family. Instead, it supports the effectiveness of HRoPE's specific factorization, normalization, and optimization constraints.

Together, these experiments support the usefulness and cross-view organization of HRoPE's learned transformations, while acknowledging that the individual latent factors are not guaranteed to represent unique physical quantities.

AuthorsOpenReview ↗

Official Response by Authors (3/6)

W3 / Q4 — Gain comes from pretrained feature or our design

Indeed, pretrained geometric features contribute substantially to the best-performing HRoPE configuration. However, the results in Table 1 of the original paper also provide evidence that HRoPE’s geometric inductive bias offers benefits beyond those features:

1. Mapping from DINOv2 features. Using standard DINOv2 features without backbone fine-tuning or 3D-supervised pretraining, HRoPE improves over PRoPE by 0.237, 0.559, and 0.777 dB on CO3Dv2, RealEstate10K, and Objaverse, respectively (Table 1). It also outperforms URoPE by 0.475 dB on Objaverse.

2. Learnable HRoPE. With a predictor trained from scratch and no pretrained feature backbone, HRoPE improves over PRoPE by 0.117, 0.277, and 0.291 dB on CO3Dv2, RealEstate10K, and Objaverse, respectively (Table 1). It also exceeds RayRoPE by 0.664 dB on CO3Dv2 and achieves comparable PSNR to URoPE on Objaverse.

We further evaluate predictors trained from scratch on ASE, with and without explicit GT geometric supervision:

MethodPSNR ↑SSIM ↑LPIPS ↓
PRoPE27.8290.8020.264
HRoPE (learned, without geometric supervision)28.1630.8030.225
HRoPE (learned, with ASE GT supervision)28.4870.8120.195
HRoPE (VGGT)28.5550.8090.198

The oracle geometry experiment in Table 3 serves a separate purpose: replacing predicted parameters with GT geometry improves ASE PSNR from 28.555 to 28.751 dB, indicating that the formulation can benefit from more accurate geometric inputs. These results show that pretrained features are not required for HRoPE to improve reconstruction quality, while geometric supervision provides an alternative source of guidance for learning the same parameterization.

3. Matched feature source comparisons. We report the exact additive VGGT result from Fig. 4(a) alongside HRoPE on RealEstate10K.

Method on RealEstate10KPSNR ↑SSIM ↑LPIPS ↓
LVSM + 768-dimensional additive VGGT encoding27.1590.8380.099
HRoPE27.4100.8480.091

The VGGT APE baseline adds a 768-dimensional representation of VGGT features to LVSM tokens; HRoPE uses VGGT features to predict attention transformations without direct feature fusion. The purpose of this ablation is to test whether directly retaining high-dimensional VGGT features is sufficient to outperform HRoPE's structured geometric modulation; HRoPE remains 0.251 dB ahead.

PRoPE + VGGT APE would not constitute a fair, controlled comparison of positional encodings alone: relative to HRoPE, it changes both the direct feature-fusion pathway and the positional encoding. We therefore treat direct feature fusion as a separate design choice and evaluate the transformation parameterization through the controls below. Extending the PRoPE + VGGT APE comparison to all datasets also exceeds our available time and computational resources.

We have removed “sharply” from the statement in Sec. 1 (L.105-106) to better reflect the observed performance difference.

4. Transformation-structure controls. With VGGT features held fixed, Fig. 4(a) compares several geometric parameterizations. We additionally evaluate an unconstrained 3×3 transformation, directly addressing the reviewer's capacity-control suggestion.

ParameterizationRealEstate10K PSNR ↑ASE PSNR ↑
Unconstrained 3×327.10328.198
HRoPE27.41028.555

HRoPE improves PSNR by 0.307 dB on RealEstate10K and 0.357 dB on ASE. Together with the free rank-one control above, these comparisons suggest that the specific structured parameterization matters beyond access to pretrained features alone.

5. Larger backbone. Table 2 additionally reports results using a larger 53M-parameter LVSM on DL3DV. HRoPE improves over PRoPE by 0.710, 0.416, and 0.287 dB with 2, 4, and 8 input views, respectively. Appendix A.7 also reports transfer to the substantially larger EscherNet architecture.

Reviewer_adoSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HRoPE, a relative positional encoding for multi-view transformers. Following the slanted support-window assumption of PatchMatch-style stereo, each patch is modeled as a local 3D plane. The plane-induced homography H = K(R + κtnᵀ) is injected into attention as a relative transformation, so that the encoding captures local surface orientation in addition to camera geometry. Plane parameters are predicted per patch from image features (learned from scratch, DINOv2, or VGGT), and the closed-form inverse follows from the Sherman–Morrison formula. Experiments on RealEstate10K, CO3Dv2, Objaverse, and DL3DV show consistent gains over existing geometric positional encodings for novel view synthesis.

Critical strengths
  • The idea is well motivated. Bringing the PatchMatch slanted-plane prior into relative positional encoding is a natural and interesting step from point/ray-level to surface-level geometry.
  • The design is clean: the rotation-plus-rank-one structure gives a closed-form inverse.
  • The experiments are broad, covering several NVS datasets, ablations on the transformation parameterization and feature sources, and transfer to EscherNet.
Critical weaknesses
  • In the predicted mode, R and t are predicted per patch rather than taken from camera extrinsics (Sec. 3.2), and target-view tokens use A = I, so their encoding contains only the intrinsics K_t (Alg. 1, line 3; App. A.3). The geometric factor between input and target tokens therefore contains no target extrinsics. The paper does not explain how the target camera pose enters the network. This matters for interpreting the method as a geometric encoding and for the claim that HRoPE outperforms PRoPE "without ground-truth camera poses" (Sec. 1).
Questions for authors
  1. In the predicted mode, how does the target camera pose enter the network? Are the Plücker ray inputs of LVSM retained for HRoPE and for all baselines? If so, how should "without ground-truth camera poses" in Sec. 1 be understood?
1 comment
AuthorsOpenReview ↗

Official Response by Authors

We thank the reviewer for the constructive comments. We have clarified the corresponding descriptions in the revised manuscript and address each point below.

W1 / Q1 — Target-pose conditioning and scope of the camera claim

Thank you for pointing this out. We adapt the LVSM model, and hence retain LVSM’s Plücker-ray inputs for both reference and target views. The target-view tokens are initialized from Plücker-ray of the target camera, while HRoPE introduces geometric inductive biases through its learned per-patch transformations. We admit that the phrase``"without ground-truth cameras'' in Sec. 1 may cause confusion. Our intended claim is HRoPE’s per-patch transformation parameters are inferred from image features without requiring ground-truth camera extrinsics as inputs to the encoding. We have revised Sec. 1 (L.103-104), Sec. 3.3 (L.284-288), and Sec. 4.1 (L.317-318) to make this distinction explicit.

Importantly, we want to respectfully emphasize that HRoPE itself does not require known camera extrinsics. For example, in the MLLM experiment (Appendix A.7), HRoPE is applied directly to visual tokens without introducing camera-pose inputs, further demonstrating its applicability beyond camera-conditioned NVS architectures.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (2/2)

Summary of Revisions

We summarize our manuscript revisions and additional responses below.

1. Experimental Validation

  • Surface Orientation Ablation: Comparison of learned and fixed fronto-parallel normals (R3W1, R2Q4, R5W1).
  • Transformation Controls: Comparison with unconstrained 3×3 and rotation-plus-free-rank-one transformations (R2Q2, R2Q4, R5Q1).
  • Normal Consistency and Perturbation: Cross-view consistency against shuffled controls and sensitivity to normal perturbations (R2Q2, R5Q2).
  • Attention Correspondence: Quantitative evaluation and heatmaps against GT cross-view correspondences (R2Q3, R5Q2; Appendix A.8, Tab. 7, Fig. 8).
  • Learning from Scratch: ASE experiments with and without GT geometric supervision (R2Q4, R5Q3).
  • Camera-conditioning Variants: Comparison of PRoPE + CamRay and 2D RoPE + Plücker-ray conditioning within HRoPE (R2Q1).
  • Training-seed Variability: Five-run mean and standard deviation for PRoPE and HRoPE (VGGT) on RealEstate10K (R2Q6).

2. Methodological Clarifications

  • Camera Conditioning: We clarify target-pose conditioning, the camera claim, and baseline ray-input conventions (R1Q1, R2Q1, R3Q1; Sec. 1, L.103-104; Sec. 3.3, L.284-288; Sec. 4.1, L.317-318).
  • Geometric Interpretation: We distinguish the learned geometric prior from exact physical parameter recovery (R2Q2, R5W1, R5Q2).
  • Local Plane Assumption: We clarify local planarity as an attention prior without a hard planar reconstruction constraint (R4Q2).
  • Claim Refinement: We soften the distortion, generalization, and additive-feature claims (R2W3, R2W4; L.021-023, L.095-096, L.105-106).
  • OOD Configuration: We specify that the OOD results use VGGT-based HRoPE (R2Q6; Fig. 7, L.981-984).

3. Efficiency and Extended Discussions

  • Inference Efficiency: Latency and peak memory comparisons for VGGT- and DA3-based HRoPE and baseline methods (R4Q1).
  • Pretrained Features: We clarify their contribution using existing DINOv2 results and matched-VGGT ablations (R2Q4, R3W2, R5Q3; Tab. 1, Fig. 4(a)).
  • MLLM Evaluation: We discuss published VLM-3R results and the scope of the MLLM experiment (R2Q5).

Below, we address the reviewers’ questions and concerns in detail.

AuthorsOpenReview ↗

General Response (1/2)

We thank R1 (adoS), R2 (JzFa), R3 (aTWy), R4 (4MFo), and R5 (Nm6W) for their constructive feedback. We appreciate the reviewers’ recognition of HRoPE’s motivation, structured design, and empirical results. The strengths highlighted in the reviews include:

  1. Surface-Aware Positional Encoding: R1 describes incorporating the slanted-plane prior into relative positional encoding as a “natural and interesting step” from point/ray-level to surface-level geometry. R3 recognizes the “richer geometric inductive bias” provided by local surface orientation, while R4 considers the method “novel and different from existing point-based methods.” R5 also acknowledges that incorporating local surface information into attention is “well motivated.”
  2. Clean and Technically Sound Design: R1 highlights that “the design is clean,” with the rotation-plus-rank-one structure enabling a closed-form inverse. R2 likewise recognizes the exact closed-form inverse, and R3 considers the proposed parameterization and efficient inverse a “technically sound contribution.”
  3. Broad Evaluation and Encouraging Results: R1 recognizes the breadth of the experiments, including multiple NVS datasets, parameterization and feature-source ablations, and transfer to EscherNet. R3 highlights “consistent improvements across multiple datasets and architectures” and “comprehensive” ablations. R2 similarly notes that “ablation coverage is good,” while R5 acknowledges “encouraging improvements” across the evaluations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.