acceptodds
Under review as a conference paper at ICLR 2027

WorldRoPE: Probabilistic World-Centric Rotary Embedding for Multi-View Attention

20 views

Abstract

Rotary Positional Embedding (RoPE) is widely used to encode relative positions in Transformers. In multi-view settings, however, patch coordinates are defined independently in each image: the same 3D point can therefore have different coordinates across views, making them insufficient for establishing cross-view correspondences. We introduce WorldRoPE, a probabilistic world-centric RoPE for multi-view attention. At each layer, WorldRoPE back-projects each patch token into a set of volumetric Gaussian primitives in a shared world frame. These primitives represent multiple plausible 3D supports for a token, jointly capturing alternative depth hypotheses, depth uncertainty, and the finite spatial extent of its patch frustum. When a key token’s support is projected into a query view, it induces a distribution over where that key may appear in the query image. WorldRoPE integrates the rotary operator over this distribution in closed form, combining rotations at the projected primitive centers according to their mixture weights with covariance-dependent attenuation. This makes attention less sensitive to uncertain key locations in the query view, without rasterizing or warping token features. The formulation reduces to deterministic RoPE when the support collapses to a point. Across layers, the predicted geometry guides cross-view attention, while the updated features refine the geometry used at the next layer. Experiments on novel view synthesis, depth estimation, and 3D geometry reconstruction demonstrate the effectiveness of WorldRoPE for multi-view Transformers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Probabilistic, RoPE, Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_Z1LwOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes WorldRoPE, a probabilistic world-centric rotary positional encoding for multi-view attention. It represents each image token using a mixture of volumetric Gaussian primitives to capture multiple depth hypotheses, depth uncertainty, and patch extent. By deriving a closed-form expectation of the rotary operator over the projected Gaussian distributions, WorldRoPE incorporates probabilistic 3D geometry into cross-view attention. Experiments across multiple tasks and architectures demonstrate its effectiveness.

Critical strengths
  1. Elegant and principled formulation. The paper presents an elegant mathematical derivation that provides a solid theoretical foundation for the proposed design. The resulting method is well motivated and interpretable.
  2. Strong empirical results. WorldRoPE achieves consistent improvements across multiple architectures, tasks, and benchmarks, demonstrating its practical effectiveness and broad applicability.
  3. Meaningful learned geometry. The visualizations in Figures 7, 10, and 11 suggest that WorldRoPE can learn plausible 3D scene structures without explicit depth supervision, providing further qualitative support for the proposed geometric modeling.
Critical weaknesses
  1. Computational overhead. Projecting Gaussian primitives across views introduces additional geometric computation, which is an important factor for application. The efficiency analysis in Figure 13 deserves a place in the main paper, and exact runtime measurements should be reported rather than only plots.
  2. Potential information loss from hypothesis aggregation. WorldRoPE combines multiple Gaussian hypotheses into a single expected rotary operator per frequency block. However, distinct multimodal distributions may produce identical expected rotations, potentially discarding geometric information relevant to cross-view matching. It would be interesting to compare the current approach with an alternative that computes attention distributions separately for each Gaussian hypothesis and aggregates them after softmax. Such a comparison could help assess whether the current aggregation strategy loses useful information.
Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to reviewer Z1Lw

Thank you for your thoughtful and thorough review of our work.

[W1] “Computational overhead”

Thank you for your feedback, which improved our paper. WorldRoPE introduces modest additional runtime over RayRoPE and URoPE. As suggested, we added exact runtime and memory comparisons to the main paper (Table 7), with full results in Appendix A.10 of the revised PDF. Below, we compare LVSM-25M at 256 × 256 resolution on one NVIDIA A6000 GPU using torch.compile.

Reference viewsMethodTraining (ms/step)Inference (ms/scene)Training memory (GiB)Inference memory (GiB)
2RayRoPE100.35.442.590.49
URoPE98.25.372.570.50
WorldRoPE102.45.742.610.53
4RayRoPE171.511.423.740.78
URoPE170.011.373.660.81
WorldRoPE175.211.953.910.85
8RayRoPE404.629.637.311.29
URoPE402.729.807.121.42
WorldRoPE413.330.687.681.43
16RayRoPE1223.292.6513.852.60
URoPE1222.193.6213.303.14
WorldRoPE1241.395.2014.762.91
  • Runtime. At sixteen reference views, WorldRoPE adds only 1.5–1.6% training time and 1.7–2.8% inference time over RayRoPE and URoPE. The relative runtime overhead decreases as more views are added. We interpret this trend as the cost of attention itself increasingly dominating the additional RoPE computation.

  • Memory. At sixteen views, training memory is 6.6–11.0% higher, while inference memory lies between the two baselines. Results for LVSM-150M are included in the same appendix.

To further reduce overhead, a future direction is to perform attention directly in a shared 3D world frame, where our Gaussian rotary expectation also has a closed form. This could preserve probabilistic geometric modeling while avoiding repeated query-view projections.

[W2] “Potential information loss from hypothesis aggregation”

We thank the reviewer for this insightful question. We agree that aggregating hypotheses before softmax can, in principle, discard information. To test whether this matters in practice, we implemented your suggested alternative along with two additional variants. The suggested alternative achieves nearly identical performance to our original aggregation, which uses only a quarter of the rotated key tokens.

For a query, let sims_i^m be its attention score to key token i∈1,…,Ni \in \\{1, \dots, N\\} under hypothesis m∈1,…,Mm \in \\{1, \dots, M\\}, and let αim\alpha_i^m be the corresponding mixture weight. Our method averages the hypothesis scores, ∑mαimsim\sum_m \alpha_i^m s_i^m, before softmax, which requires only one rotated key per token because the score is linear in the rotary operator. The alternatives instead compute a separate score for each hypothesis, increasing the number of rotated key tokens from NN to NMNM and requiring MM times as many attention scores per query, while the value and output transformations remain unchanged:

  • Mixture of softmax (reviewer-suggested). We compute a separate attention distribution for each hypothesis and combine them with the mixture weights, renormalizing because the weights vary across key tokens:

    Aimix=∑m=1MαimBim∑j=1N∑n=1MαjnBjn,Bim=exp⁡(sim)∑j=1Nexp⁡(sjm).A_i^{\mathrm{mix}} = \frac{\sum_{m=1}^{M} \alpha_i^m B_i^m}{\sum_{j=1}^{N} \sum_{n=1}^{M} \alpha_j^n B_j^n}, \qquad B_i^m = \frac{\exp(s_i^m)}{\sum_{j=1}^{N} \exp(s_j^m)}.

  • Score scaling (additional variant). We multiply each hypothesis score by its mixture weight before a joint softmax over all N×MN \times M token–hypothesis pairs, using unnormalized weights exp⁡(αimsim)\exp(\alpha_i^m s_i^m), and then sum over hypotheses.

  • Prior weighting (additional variant). We instead multiply each exponentiated score by its mixture weight, using unnormalized weights αimexp⁡(sim)\alpha_i^m \exp(s_i^m), before the same joint normalization and summation.

Full definitions are provided in Appendix A.13 of the revised PDF. The results below (Table 32) use M=4M = 4 hypotheses.

Method (aggregation rule)# of rotated key tokensPSNR ↑SSIM ↑LPIPS ↓
WorldRoPE (default)×126.750.8460.073
WorldRoPE with mixture of softmax (suggested)×M26.760.8450.073
WorldRoPE with score scaling×M26.620.8430.074
WorldRoPE with prior weighting×M26.720.8450.073

The suggested mixture of softmax improves PSNR by only 0.01 dB, with slightly lower SSIM and the same reported LPIPS, while the other two variants lower PSNR. These results support our aggregation as an effective accuracy–efficiency tradeoff: comparable performance using only 1/M1/M of the rotated key tokens in this comparison.

We sincerely look forward to engaging in further discussions.

Dear Reviewer Z1Lw,

Thank you again for your insightful review and constructive comments. Following your suggestions, we added runtime and memory comparisons to the main paper and compared our aggregation with your suggested alternative. Your feedback helped us provide stronger evidence of WorldRoPE's practicality and demonstrate that its default design offers an effective accuracy–cost tradeoff.

We truly appreciate your positive assessment of our work. Please let us know if anything else is unclear; we would be happy to clarify further.

Note: We have also uploaded an interactive 3D viewer as supplementary material, allowing you to explore the Gaussian primitives predicted at each attention layer. Please open the 3d_viewer.html file in Supplementary Material to use it.

Best regards,

Authors

Dear Reviewer Z1Lw,

Follow-up on [W2]: Additional ablation on Objaverse

To check whether the result in our original response holds beyond RE10K, we additionally ran the same ablation on Objaverse. The table below compares three WorldRoPE variants that differ only in hypothesis aggregation: one suggested by the reviewer and two additional variants we explored (see Appendix A.13 for the detailed equations). We set M = 4 in this comparison. We hope these results further address your question.

RE10KObjaverse
Method# of rotated key tokensPSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
WorldRoPE (default)×126.750.8460.07322.520.8960.113
WorldRoPE with mixture of softmax (reviewer suggested)×M26.760.8450.07322.520.8950.113
WorldRoPE with score scaling (additional variant)×M26.620.8430.07422.310.8930.117
WorldRoPE with prior weighting (additional variant)×M26.720.8450.07322.620.8970.110
  • WorldRoPE with reviewer's suggested mixture of softmax again performs on par with the default, whereas WorldRoPE with score scaling degrades all metrics. Interestingly, WorldRoPE with prior weighting further improves all three metrics on Objaverse (+0.10 dB PSNR, +0.001 SSIM, −0.003 LPIPS vs. the default), although, like the other variants, it requires four times as many rotated key tokens. (M=4)
  • Notably, all four variants still outperform RayRoPE and URoPE on both datasets, suggesting that WorldRoPE's improvement is robust to the choice of aggregation rule.
  • These results further support the default aggregation as an effective accuracy–cost tradeoff: comparable performance using only 1/M1/M of the rotated key tokens.

Additional remark

We sincerely appreciate your comments, which have been very helpful in improving our work. If there are any other questions or areas where you would like more information, please feel free to reach out, and we will gladly provide further details.

Best regards,

Authors

Reviewer_yo2nOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WorldRoPE, a probabilistic, world-centric RoPE for multi-view attention. Each key token is represented by multiple volumetric Gaussians in world space. These Gaussians encode depth hypotheses with their uncertainty, as well as the extent of the image patch. They are projected into the query view, and RoPE is integrated over the resulting distribution in closed form; when the position is certain, this reduces to standard RoPE. Experiments on novel view synthesis, depth estimation, and feed-forward 3D reconstruction show gains over existing camera-aware positional encodings.

Critical strengths

1. A principled and elegant formulation

  • Taking the expectation of RoPE over uncertain token positions gives a simple, interpretable closed form. It unifies depth uncertainty, multiple depth hypotheses, and patch extent in one formulation. Since it reduces to standard RoPE when the position is certain, it is a natural generalization of RoPE.

2. Consistent gains as a plug-and-play component

  • The method improves results across three tasks and backbones.
  • The ablations in the main paper support the design: removing probabilistic modeling causes the largest drop, and multiple depth hypotheses help (Tables 3–4).
  • This suggests that it can serve as a general component for multi-view transformers.
Critical weaknesses

1. The contribution relative to RayRoPE could be articulated more clearly

  • WorldRoPE builds directly on RayRoPE, which also models uncertain token positions along camera rays (Table 1).
  • The extensions are clearly presented. Still, a more explicit discussion of what is conceptually new relative to RayRoPE, and why it matters, would help assess the significance of the contribution.

2. Some gains are modest, and the practical cost could be quantified further

  • On Objaverse, the improvement over the strongest baseline is 0.11 dB (Table 2), and the gains on VGGT are also small (Table 7). Reporting the mean and variance over multiple random seeds would help assess whether these gains are significant.
  • The authors note that the geometric computation grows as O(N²) with the number of views (App. A.8). Concrete runtime and memory numbers would help assess the practical cost, especially for larger models and more views.
Questions for authors
  1. Relation to RayRoPE. Could you further clarify the main conceptual differences from RayRoPE, and in which settings they matter most?
  2. Stability of results. If feasible, could you report the mean and variance over multiple random seeds for the main results, particularly on Objaverse?
  3. Computational cost. Could you provide concrete training time, inference time, and memory usage for different numbers of views? Ideally, this would include the closest baselines, such as RayRoPE and URoPE.
  4. Learned geometry. The depth hypotheses and their uncertainties are learned without direct supervision (App. C.1). Do they correspond to the actual scene geometry? A brief quantitative check, for example comparing them with ground-truth depth where available, would strengthen the probabilistic interpretation beyond the qualitative visualizations (Fig. 4).
5 comments
AuthorsOpenReview ↗

Response to reviewer yo2n [1/2]

[W1 & Q1] “Relation to RayRoPE”

We thank the reviewer for this insightful question. 3D Gaussian Splatting offers a powerful representation of the 3D world. WorldRoPE connects this volumetric representation to 2D image Transformers directly within attention, without rasterization or changing the overall 2D Transformer architecture. This gives 2D Transformers explicit 3D awareness.

  • In terms of uncertainty modeling, RayRoPE uses a single 1D hat-shaped distribution along a ray. WorldRoPE uses multiple learned 3D Gaussian hypotheses to model the viewing cone of a finite image patch, more faithfully capturing world-space geometry.

  • Why volumetric modeling matters Depth uncertainty alone cannot represent the finite spatial extent of an image patch. This distinction is particularly clear at small camera baselines, where a 1D ray distribution projects to almost a single point in the query view, whereas our 3D primitives retain a finite footprint. In this setting, adding spatial extent modeling on top of depth uncertainty improves PSNR by 0.17 dB (Appendix A.9, Table 14 of the revised PDF). This also motivates future extensions to video Transformers, where neighboring frames often have small camera baselines.

  • Why Gaussian modeling matters. Integrating rotary encodings over a Gaussian distribution yields positive, non-oscillatory attenuation of rotary frequencies. Consistent with Gaussian scale-space smoothing, greater projected uncertainty smoothly strengthens this attenuation, suppressing rotary components that are sensitive to positional uncertainty.

  • Benefits across rotary operators. WorldRoPE improves performance with both standard RoPE and MixRoPE, showing that the gains from volumetric spatial modeling hold across different rotary operators (Figure 5 and Table 5).

[W2 & Q2] “Stability across random seeds on LVSM & VGGT”

Thank you for raising this point. Following your suggestion, we repeated the main comparisons across five training seeds and reported them in Revised PDF - Appendix A.11. WorldRoPE retains the best mean performance on all reported metrics, supporting the consistency of the main results across training runs.

  1. LVSM on the RE10K and Objaverse datasets

    WorldRoPE achieves the best mean PSNR, SSIM, and LPIPS on both the RE10K and Objaverse datasets, with PSNR standard deviations of 0.06 dB and 0.04 dB, respectively. These results support consistent benefits across repeated runs.

    RE10KObjaverse
    MethodPSNR ↑SSIM ↑LPIPS ↓PSNR ↑SSIM ↑LPIPS ↓
    PRoPE25.47 ± 0.080.812 ± 0.0020.093 ± 0.00222.41 ± 0.020.894 ± 0.0000.118 ± 0.001
    RayRoPE26.11 ± 0.050.831 ± 0.0010.082 ± 0.00122.10 ± 0.160.891 ± 0.0020.125 ± 0.004
    URoPE26.31 ± 0.030.835 ± 0.0010.078 ± 0.00122.03 ± 0.020.890 ± 0.0000.126 ± 0.001
    WorldRoPE26.69 ± 0.060.845 ± 0.0010.073 ± 0.00122.57 ± 0.040.896 ± 0.0010.111 ± 0.001
  2. VGGT on the ScanNet++ datasets

    WorldRoPE achieves the lowest mean errors across all four metrics on the ScanNet++ dataset. In particular, the rotation-error improvement is accompanied by a smaller standard deviation, supporting both accuracy and consistency across runs.

    MethodDepth AbsRel ↓Norm. RMSE ↓Cam.-pts AbsRel ↓Rotation error (°) ↓
    RayRoPE0.0456 ± 0.00690.0632 ± 0.00040.0465 ± 0.00621.211 ± 0.148
    URoPE0.0466 ± 0.00640.0638 ± 0.00070.0476 ± 0.00571.080 ± 0.038
    WorldRoPE0.0454 ± 0.00630.0630 ± 0.00060.0460 ± 0.00591.001 ± 0.019

We further evaluate LVSM and VGGT on additional datasets to examine whether the gains extend to other settings in the Revised PDF - Appendix A.14.

  • LVSM on CO3D. WorldRoPE outperforms the compared baselines across all three metrics, improving PSNR by 0.28 dB over RayRoPE and 0.58 dB over PRoPE. These results extend the observed benefits beyond the RE10K and Objaverse datasets.

  • VGGT on outdoor scenes from DL3DV3K. WorldRoPE improves all four metrics, including about 13% reduction in camera rotation error relative to URoPE, extending the evaluation beyond indoor scenes.

AuthorsOpenReview ↗

Response to reviewer yo2n [2/2]

[Q3] “Computational cost”

Thank you for raising this point. WorldRoPE introduces modest additional runtime over RayRoPE and URoPE across the evaluated view counts. We expanded the runtime analysis in Original PDF - Appendix A.8, Figure 13 with exact training and inference times and GPU memory usage. The full results are provided in Revised PDF - Appendix A.10, Tables 15–18.

Below, we compare LVSM-25M at 256 × 256 resolution on one NVIDIA A6000 GPU using torch.compile.

Reference viewsMethodTraining (ms/step)Inference (ms/scene)Training memory (GiB)Inference memory (GiB)
2RayRoPE100.35.442.590.49
URoPE98.25.372.570.50
WorldRoPE102.45.742.610.53
4RayRoPE171.511.423.740.78
URoPE170.011.373.660.81
WorldRoPE175.211.953.910.85
8RayRoPE404.629.637.311.29
URoPE402.729.807.121.42
WorldRoPE413.330.687.681.43
16RayRoPE1223.292.6513.852.60
URoPE1222.193.6213.303.14
WorldRoPE1241.395.2014.762.91
  • Runtime. At sixteen reference views, WorldRoPE adds only 1.5–1.6% training time and 1.7–2.8% inference time over RayRoPE and URoPE. The relative runtime overhead decreases as more views are added. We interpret this trend as the cost of attention itself increasingly dominating the additional RoPE computation.

  • Memory. At sixteen views, training memory is 6.6–11.0% higher, while inference memory lies between the two baselines. Results for LVSM-150M are included in the same appendix.

To further reduce overhead, a future direction is to perform attention directly in a shared 3D world frame, where our Gaussian rotary expectation also has a closed form. This could preserve probabilistic geometric modeling while avoiding repeated query-view projections.

[Q4] “Quantitative evaluation of learned geometry”

We thank the reviewer for this insightful question, which helped us strengthen our analysis. We evaluate predicted depth and uncertainty against aligned UniDepthV2 reference depths on the RE10K dataset. The results suggest that successive feature updates improve depth accuracy while making the predicted uncertainty more informative about depth errors.

  1. Depth accuracy. Depth MSE generally decreases across layers, from 15.37 at Layer 1 to 0.86 at Layer 5, followed by a slight increase to 1.03 at Layer 6. This indicates that the geometry predicted from updated token features better agrees with the reference depths, although the improvement is not strictly monotonic.

  2. Uncertainty quality. To evaluate the reliability of predicted uncertainty, we adopt the Area Under the Sparsification Error curve (AUSE), which measures how well uncertainty ranks patches by depth error; lower is better. The advantage over random ordering becomes more pronounced in later layers, with AUSE values of 0.419 and 0.760, respectively, at Layer 6. This indicates that the learned uncertainty helps identify less reliable depth predictions.

Please see Revised PDF - Appendix A.5 for details of both evaluations.

We sincerely look forward to engaging in further discussions.

Dear Reviewer yo2n,

Thank you again for your insightful questions and valuable feedback. Following your suggestions, we clarified the relation to RayRoPE, extended our evaluation to multiple seeds and datasets, and analyzed the computational cost as well as the predicted depth and uncertainty. These efforts strengthened the evidence for the consistency and generality of WorldRoPE and clarified the benefits of its volumetric modeling and the quality of its learned geometry.

We truly appreciate your positive assessment of our work. If there are any other questions or areas where you would like more information, please feel free to reach out, and we will gladly provide further details.

Note: We have also uploaded an interactive 3D viewer as supplementary material, allowing you to explore the Gaussian primitives predicted at each attention layer. Please open the 3d_viewer.html file in Supplementary Material to use it. We hope it also helps with your question on the learned geometry.

Best regards,

Authors

Reviewer_CZTeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RoPE applied independently per view fails cross-view correspondence, since the same 3D point projects to different 2D coordinates in each view. WorldRoPE predicts a depth distribution per layer and token, builds volumetric Gaussian primitives in world space, and integrates RoPE in closed form over the projected distribution, yielding a scaled rotation block with covariance-dependent attenuation and token-adaptive temperature (deterministic RoPE as the limiting case). It is validated across novel-view synthesis, stereo depth estimation, and feed-forward 3D reconstruction, with component ablations, robustness sweeps, and negligible overhead.

Critical strengths
  1. Clean mathematical contribution: a closed-form expected rotary operator over probabilistic depth positions, extended to Lie-group RoPE, with a sensitivity bound and deterministic RoPE as the limiting case, which directly explains why per-view image-space RoPE fails.
  2. Broad, plug-and-play empirical validation across three tasks (novel-view synthesis, depth estimation, feed-forward 3D reconstruction), including unseen-set generalization, comprehensive component ablations, and robustness sweeps, all at negligible overhead.
Critical weaknesses
  1. Sometimes the improvement seems to be a bit marginal (e.g., Table 3 and 4), I wonder how the model would perform with different training runs (e.g., how large the performance fluctuates) to better understand the significance of the performance gains.
  2. The projection to the query view relies on a first-order local affine approximation, and per-view geometric cost grows with the number of reference views. I wonder how the proposed method scales to more reference views and higher-resolution images.
  3. The paper does not include an explicit limitation section.
Questions for authors
  1. Could the authors share seed-wise variance and paired significance tests?
  2. How does the local-affine approximation error behave for large depth hypotheses or wide baselines?
  3. What might the runtime overhead look like at larger numbers of reference views, and how does it scale to longer sequences?
3 comments
AuthorsOpenReview ↗

Response to reviewer CZTe [1/2]

Thank you for your thoughtful and thorough review of our work.

[W1 & Q1] “Stability across random seeds”

Thank you for your feedback, which improved our paper. Following your suggestion, we repeated the ablations in Tables 3 and 4 across five training seeds and report them in Revised PDF - Appendix A.11 (Tables 27–29). We report PSNR as mean ± sample standard deviation, with p-values from two-sided paired t-tests. Performance varies little across seeds, and all key gains have p < 0.05.

  1. Component ablations (Table 27, extending Table 3)

    All configurations use four hypotheses. The ablation baseline (w/o probabilistic modeling) disables both depth uncertainty and spatial extent, and the full model enables both.

    ConfigurationDepth uncertaintySpatial extentPSNR ↑p-value (vs. full)p-value (vs. baseline)
    w/o probabilistic modeling26.22 ± 0.08<0.001—
    w/o depth uncertainty✓26.45 ± 0.05<0.0010.001
    w/o spatial extent✓26.60 ± 0.040.022<0.001
    Full model✓✓26.69 ± 0.06—<0.001

    Note: The w/o multi-hypothesis setting in Table 3 of the original paper is omitted from the updated table, as it corresponds to the single-hypothesis configuration in the number-of-hypotheses ablation (Tables 4 and 28).

    Depth uncertainty and spatial extent make complementary contributions. Compared with the ablation baseline, spatial extent alone improves PSNR by 0.23 dB (p = 0.001), and depth uncertainty alone by 0.38 dB (p < 0.001). Combining them achieves the highest PSNR, outperforming either component alone (p < 0.05 in both cases). The benefit of spatial extent is more pronounced for small reference distances and trajectory deviations, where it adds 0.17 dB on top of depth uncertainty, compared with 0.07 dB overall in the same single-seed evaluation (Revised PDF - Appendix A.9, Table 14).

  2. Number of hypotheses (Table 28, extending Table 4)

    Both depth uncertainty and spatial extent are enabled; only the number of hypotheses varies.

    HypothesesPSNR ↑p-value (vs. four hypotheses)p-value (vs. one hypothesis)
    126.52 ± 0.020.006—
    226.64 ± 0.030.140<0.001
    4 (default)26.69 ± 0.06—0.006
    826.71 ± 0.040.487<0.001

    Every tested multi-hypothesis configuration outperforms the single-hypothesis setting (p < 0.01 in all cases), supporting the benefit of representing multiple plausible depths. The gains then level off beyond two hypotheses (+0.05 dB from two to four, p = 0.140; +0.02 dB from four to eight, p = 0.487), suggesting that a few hypotheses already capture most of the benefit. We therefore use four hypotheses as a practical balance between accuracy and computation, achieving mean PSNR within 0.02 dB of eight.

  3. Additional evaluations

    Beyond the requested ablations, we also evaluated LVSM on the RE10K and Objaverse datasets and VGGT on the ScanNet++ dataset across five seeds. WorldRoPE retains the best mean performance on all reported metrics. Please refer to Appendix A.11, Tables 23–26 for the means, standard deviations, and paired t-test results.

AuthorsOpenReview ↗

Response to reviewer CZTe [2/2]

[W2 & Q2] “Local-affine approximation under large depths and wide baselines”

We thank the reviewer for this interesting question, which helped us strengthen our analysis. With absolute depth uncertainty fixed, approximation error decreases with mean depth and increases with lateral baseline, i.e., it grows with the primitive's projected support in the query view. We verified these trends against a quasi-Monte Carlo reference and also evaluated a second-order Taylor correction (Revised PDF-Appendix A.12). Both tables report projection RMSE in pixels.

  1. Depth: Increasing depth reduces error when the baseline, absolute depth uncertainty, and source patch footprint are fixed. As mean depth increases, the primitive typically lies farther from both cameras and appears smaller in the query view, leading to a smaller approximation error.

    Mean depth45681015
    First-order affine (ours)9.002.741.390.640.410.23
    Second-order Taylor7.341.550.560.170.080.03
  2. Baseline: Under aligned cameras, increasing the lateral baseline while keeping mean depth and uncertainty fixed makes the token's primitive occupy a larger region in the query view and increases the approximation error.

    Baseline0.0340.0550.1390.3620.5250.7121.2551.633
    First-order affine (ours)0.630.640.711.061.391.803.043.92
    Second-order Taylor0.200.210.240.410.560.731.251.62
  3. Additional experiment (beyond the reviewer's request): We further tested whether the second-order Taylor correction's higher projection accuracy benefits model performance. In a single-seed pilot evaluation, replacing the first-order approximation with this correction did not improve PSNR (26.75 → 26.69 dB). We therefore retain the first-order approximation, though the accuracy gain warrants further investigation. Results are in Appendix A.12.

[W2 & Q3] “Computational cost and scalability”

Thank you for raising this point. WorldRoPE adds modest runtime over RayRoPE and URoPE across the evaluated view counts and resolutions. We expanded the runtime analysis in Original PDF-Appendix A.8 with exact training/inference times and GPU memory. Full results are in Revised PDF-Appendix A.10. Below, we compare LVSM-25M at 256 × 256 on one NVIDIA A6000 GPU with torch.compile.

ViewsMethodTraining (ms/step)Inference (ms/scene)Training memory (GiB)Inference memory (GiB)
2RayRoPE100.35.442.590.49
URoPE98.25.372.570.50
WorldRoPE102.45.742.610.53
4RayRoPE171.511.423.740.78
URoPE170.011.373.660.81
WorldRoPE175.211.953.910.85
8RayRoPE404.629.637.311.29
URoPE402.729.807.121.42
WorldRoPE413.330.687.681.43
16RayRoPE1223.292.6513.852.60
URoPE1222.193.6213.303.14
WorldRoPE1241.395.2014.762.91
  1. Longer sequences (table above): More reference views increase attention and view-pair projection costs for all three methods (RayRoPE, URoPE, and WorldRoPE). WorldRoPE's relative runtime overhead generally shrinks with more views, as attention itself increasingly dominates the extra RoPE computation. At 16 views, WorldRoPE adds only 1.5–1.6% training time and 1.7–2.8% inference time over RayRoPE and URoPE; training memory is 6.6–11.0% higher, while inference memory lies between the two baseline methods.
  2. Higher-resolution images (table in Revised PDF-Appendix A.10): With the view count fixed, the relative gap narrows as resolution increases: at 1024 × 1024 with eight views, inference time is within 1% of 2D RoPE, RayRoPE, and URoPE.
  3. How to scale it?: To further reduce overhead, we could perform attention in a shared 3D world frame, or project onto the global xy/yz/zx planes instead of each query view, which would keep the per-view geometric cost from growing with more views. Our Gaussian rotary expectation also admits a closed form in both, so our probabilistic geometric modeling could still apply. We will explore this direction and thank the reviewer for the intuition behind it.

[W3] “Explicit discussion of limitations”

Thank you for this helpful suggestion. We have added an explicit discussion of limitations and future work to the main paper. Since WorldRoPE learns 3D geometry in attention layers without depth supervision, geometric ambiguity naturally increases under small baselines; using available depth to guide attention could help.

We sincerely look forward to further discussions.

Dear Reviewer CZTe,

We appreciate your careful feedback and valuable comments. Following your suggestions, we conducted multi-seed evaluations and further analyses of the local-affine approximation and computational cost, along with a discussion of limitations. These additions helped us provide stronger evidence for the stability and scalability of WorldRoPE and clarified its approximation behavior and future directions.

With the discussion ending soon, we would be grateful if you could share your thoughts on our responses. Please let us know if any concerns remain or if there is anything we could clarify further. If your concerns have been addressed, we would appreciate it if you could take our responses into account in your final evaluation.

Note: We have also uploaded an interactive 3D viewer (3d_viewer.html in the supplementary material). Please feel free to use it to explore the Gaussian primitives predicted at each attention layer.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary

Thank you for managing the review process. Below is our rebuttal summary.


Strengths

WorldRoPE:

  • introduces a principled mathematical formulation that generalizes RoPE to incorporate world-space 3D geometry into 2D Transformer attention. (mentioned by all reviewers)

    • “clean mathematical contribution” (CZTe); “principled and elegant formulation” (yo2n); “solid theoretical foundation” (Z1Lw)
  • achieves consistent gains as a plug-and-play component across three tasks. (mentioned by all reviewers)

    • “plug-and-play empirical validation” (CZTe); “can serve as a general component” (yo2n); “strong empirical results” (Z1Lw)
  • learns meaningful geometry without explicit depth supervision. (mentioned by Z1Lw)


Discussion

The discussion strengthened our work as follows:

DiscussionsR-CZTeR-yo2nR-Z1LwOur Response
Reporting computational overhead✔✔✔1. / Main paper & Appx. A.10
Stability across random seeds✔✔2. / Appx. A.11
Comparison with RayRoPE✔3. / Appx. A.9
Local-affine approximation error✔4. / Appx. A.12
Quantitative evaluation of learned geometry✔5. / Main paper & Appx. A.5
Ablation on hypothesis aggregation strategies✔6. / Main paper & Appx. A.13
Discussion of limitations✔7. / Main paper
Initial Score344
Response Acknowledgement✔As of Nov 18th (end of reviewer–author discussion)
  1. Computational overhead: We extend original analysis (Appx. A.8 of the original paper) with exact measurements and additional settings; ours adds under 3% runtime at 16 views over RayRoPE and URoPE, with a moderate increase in training memory. Relative overhead decreases with more views and higher resolution.
  2. Stability across seeds: Five-seed runs with means, standard deviations, and paired t-tests support consistent improvements in the main results and ablations (p-value < 0.05 for component ablations).
  3. Comparison with RayRoPE: WorldRoPE models geometry more faithfully by jointly capturing multiple plausible depths and volumetric patch extent, beyond RayRoPE’s 1D hat-shaped distribution along a ray; WorldRoPE outperforms RayRoPE across both axial and mixed rotary operators. Within WorldRoPE, volumetric extent improves PSNR by 0.17 dB at small camera baselines versus 0.07 dB overall. Gaussian modeling also yields non-oscillatory attenuation.
  4. Local-affine approximation: The requested analysis shows that error decreases with depth and increases with baseline. We additionally implemented a second-order Taylor correction, which reduces projection error but does not improve model performance in our evaluation, supporting the first-order design.
  5. Learned geometry: We quantitatively evaluated depth accuracy and uncertainty quality using depth MSE and AUSE, respectively. Depth MSE decreases from 15.37 to as low as 0.86 in deeper layers, while uncertainty achieves an AUSE of 0.419 versus 0.760 for random ordering (lower is better), supporting meaningful geometry learning without explicit depth supervision.
  6. Hypothesis aggregation: Combining hypotheses in the rotary operator keeps the number of rotated key tokens unchanged. Our default matches the suggested mixture of softmax variant, while the alternatives require 4× as many rotated key tokens. Notably, all these WorldRoPE variants outperform RayRoPE and URoPE.
  7. Limitations and future work: The main paper explicitly discusses limitations and future directions, including using available depth to further guide attention, particularly under small camera baselines where geometric ambiguity increases.

Contributions

We believe WorldRoPE opens a general path to making 2D Transformers 3D-aware: a plug-and-play positional encoding that requires neither architectural modification nor explicit depth supervision, with no rasterization or feature warping. It brings the volumetric representation behind 3D Gaussian Splatting, a powerful model of the 3D world, directly into attention.

Concretely, each 2D token is lifted into a probabilistic 3D support in world space: a mixture of volumetric Gaussians that captures multiple plausible depths, depth uncertainty, and finite patch extent. Integrating the rotary operator over its projection into the query view yields a closed-form, uncertainty-aware generalization of RoPE, which recovers deterministic RoPE when positional uncertainty vanishes.

Beyond the three tasks evaluated here, we believe WorldRoPE offers a promising direction for bringing 3D awareness to 2D Transformers, with the potential to advance both perception and generation across research and industry. We also envision future extensions to egocentric, multi-view, and camera-controlled world models, where reliable cross-view correspondence is essential.

Best regards,

Authors

AuthorsOpenReview ↗

Final Remarks

Dear Reviewers, Area Chair, Senior Area Chair, and Program Chairs,

We sincerely thank you for your time and effort throughout the review process. The reviewers’ thoughtful feedback helped us strengthen both the analysis and experimental evidence in our paper. We believe our rebuttal addresses the concerns raised by all three reviewers, with all additional experiments and analyses included in the revised PDF.

We would also be grateful if you could take a look at the supplementary interactive 3D viewer (3d_viewer.html), which illustrates the predicted 3D Gaussian supports underlying alignment between key and query views.

WorldRoPE brings 3D awareness to 2D Transformers by incorporating probabilistic 3D geometry into attention, without architectural changes or additional depth supervision for the positional encoding. We believe this offers a promising direction for future research in spatial intelligence.

Thank you again for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.