acceptodds
Under review as a conference paper at ICLR 2027

GaussRoPE: Rethinking 3D Rotary Encoding When Geometry Is Uncertain

10 views

Abstract

3D rotary position embeddings (RoPE) use camera poses and scene depths that can be uncertain. Position errors can distort the relative phases on which attention relies, while geometry refined inside the network cannot inform subsequent encodings. We introduce GaussRoPE, which connects rotary encoding, uncertainty propagation, and layerwise refinement through a unified structured Gaussian geometry state. Propagation maps pose and depth uncertainty into rotary phases, while cross-token shared camera errors are implemented through token-wise rotations. This shared-error sampling preserves the modeled phase correlations and remains compatible with standard attention, without requiring explicit pairwise uncertainty tensors. Visual features update the geometry and its uncertainty, enabling refined states to inform subsequent encodings. We evaluate GaussRoPE across five host models spanning novel-view synthesis, Gaussian reconstruction, 3D object detection, and depth estimation. Relative to the strongest competing encodings in our evaluation, the full GaussRoPE framework improves PETR mAP by approximately 28% under both original and perturbed inputs. Under geometric perturbations, it also improves MVSplat PSNR by 0.48 dB and reduces UniMatch AbsRel by 7.4% relative to competing methods under the same equally weighted pose/depth/joint protocol.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D vision, geometry-aware attention, rotary positional embeddings, uncertainty modeling, multi-view reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dQXgOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GaussRoPE, a 3D RoPE with uncertain-aware camera poses and depths for multiple attention-based visual or geometry foundation models. GaussRoPE keeps a single structured Gaussian state over camera poses and depths in which the ray and point descriptors are built. Both RoPE phase for fusing mean geometry and error propagation of shared and local geometry are analytical. The layerwise geometry refinement further updates the geometry and its uncertainty for the following RoPEs. The experiments are conducted on 5 host models where the authors aim to demonstrate the advantages of GaussRoPE.

Critical strengths
  1. This paper incorporate the pose uncertainty into the 3D RoPE which is not discussed in the previous 3D RoPEs. To support the standard Flash attention interface, the authors propose the shared-error sampling strategy, which avoid explicit pairwise uncertainty tensors.
  2. This papers test the model on multiple visual and geometry foundation models, and within the matched training setup including initialization, inputs, and training budgets, the proposed method successfully improves the metrics. And the ablation studies also shows the importance of the layerwise geometry refinement.
Critical weaknesses
  1. The unconverged models' performance can not validate the effectiveness of the proposed method. Although we understand the training budgets are limited for authors, the training regime (including training step and training dataset) is far from the one to train the vanilla versions of the hosts so nearly all metrics have clear gap with the original hosts. Even if we consider the reported results as the beginning stage of the full training, we can not predict the full training version of the proposed method is better than the baselines. Also, a single seed result can not validate the proposed method is always better in the early training stage.
  2. The hyperparameter selection is not discussed in detail. For layerwise updates, the hyperparameter of the layer gap for updating is not discussed. For example, why we have G_0 to G_5 for Pi3X at specific blocks and what is the effect of different number of updates. For the ray and point channels, what is the best ratio of them? Is the RoPE model sensitive to the variation of the ratio on different hosts?
Questions for authors
  1. Is it possible to report the fully-training results of GaussRoPE and other baselines at least on one host?
  2. Could you report the mean and variance on multiple seeds on each host to validate the current results are not a lucky run?
  3. Do analysis or ablation studies on the hyperparameters as mentioned above.
  4. The description of static and full version is not clear. Could you explain more on that?
7 comments
AuthorsOpenReview ↗

Response to Reviewer dQXg (1/3)

We thank the reviewer for the detailed review and for noting the pose-uncertainty modeling, the FlashAttention-compatible shared-error sampling, and the matched training setup. Revisions in the updated PDF are marked in blue; page numbers refer to the revised PDF.


Q4. Static, Unc, and Full

We thank the reviewer for reminding us to clarify the definitions of the variants. The revised paper now gives all three side by side at the start of Sec. 4.1 (p. 6). The supervision and per-host details are in Sec. 3.5 (p. 6) and Table 10(b) (p. 19):

VariantRay–point 3D RoPEUncertainty propagation (sampled shared-camera and depth phase errors)Inter-layer geometry updatesSupervision on PETR
GaussRoPE (static), "Static"✓— (mean phase)— (fixed G0G_0)task loss only
GaussRoPE (unc), PETR only✓✓— (fixed G0G_0)task loss + σ NLL
GaussRoPE (full), "Full"✓✓✓ (G0G_0 → G5G_5)task loss + pose/depth residual + σ NLL

In short, Static is the plain 3D RoPE baseline on every host. It uses mean phases and a fixed G0G_0, and one checkpoint is evaluated at all perturbation levels. Unc is a new intermediate variant for the step-by-step PETR ablation (Table 2, p. 8). Note that the Pi3X No state feedback control (Table 4(a)) differs from Static: it keeps and supervises all intermediate geometry heads so that the training losses match exactly, but it does not pass the updated state to later encoding layers.

AuthorsOpenReview ↗

Response to Reviewer dQXg (2/3)

Q1. The gap to the original hosts and the 24-epoch PETR retraining

We strongly agree with the reviewer that validating a method on hosts trained to a competitive level is essential for judging its real gains. We would first like to briefly clarify two points about the design of the original submission and the level of its hosts:

  1. Some hosts already perform at the publicly reported level. Not all hosts in the original submission show an under-convergence gap. For example, Pi3X is fine-tuned directly from the official fully pretrained Pi3X weights (an extension of π3\pi^3). Without added perturbation, Full reaches a depth AbsRel of 0.0307 (Table 4(c), p. 10), which is already a well-converged level.
  2. Strictly controlled matched comparison. The core design of the original submission is a matched comparison: within each host, all arms start from the same baseline and are trained with exactly the same data and budget. The comparison therefore measures relative differences under the same conditions, rather than matching externally tuned absolute scores. PETR and UniMatch used shortened schedules mainly because of the compute cost of the full method × perturbation grid.

Building on this, we fully agree with the reviewer that short, single-seed early results are not enough to confirm that these relative advantages still hold after full training. We therefore followed the reviewer's suggestion and retrained all nine methods with the standard StreamPETR PETRv1-R50 24-epoch recipe (ImageNet-pretrained ResNet-50, 704×256, global batch 8, learning rate 2×10−42\times 10^{-4}, 23,736 updates; about 28% of the nuScenes training split and 2,279 validation frames). Each method uses the same three training seeds (27 runs in total). The compared 3D RoPE methods have no nuScenes detection checkpoints, so for a fair comparison all methods start from the same ImageNet backbone. The new results replace Table 1 (p. 7) and Table 2 (p. 8); all pose levels are in Table 19 (p. 31), and the original 3,000-step results are kept in Appendix D.3.5 (Table 20, p. 32).

Mean±SD over three training seeds (×100); Noisy = mean of P1D0–P3D0.

MethodOriginal mAPNoisy mAPP3D0 mAPOriginal NDSNoisy NDSP3D0 NDS
Native PETR17.55±1.4517.2816.98±1.4921.67±1.9221.5021.23±1.92
PRoPE17.54±1.9117.4217.23±1.8321.54±2.1721.4521.32±2.07
URoPE15.00±0.9614.8714.68±0.8920.00±1.6919.8019.58±1.59
RoRE18.76±1.3818.4118.04±1.1622.37±2.0322.1421.88±1.81
RayRoPE18.70±1.4918.4618.14±1.4222.82±1.3922.6022.38±1.30
GTA18.79±2.5118.5618.24±2.4822.36±2.7022.1621.90±2.61
GaussRoPE (static)18.87±1.9318.4717.99±1.8822.64±2.0122.3122.02±1.85
GaussRoPE (unc)19.37±1.8819.1818.92±1.8722.94±2.1022.7722.57±2.05
GaussRoPE (full)19.93±1.9619.6019.33±1.7423.65±2.2123.5123.39±1.96

With the full 24-epoch schedule, Full has the highest three-seed mean mAP and NDS at every pose level. On Original / Noisy inputs, its mAP is +2.38 / +2.32 higher than native PETR and +1.14 / +1.04 higher than GTA, the best competing encoding in mAP. Its NDS is +0.83 / +0.91 higher than RayRoPE, the best competing encoding in NDS. Compared with Static, Full adds +1.06 / +1.13 / +1.34 mAP on P0D0 / Noisy / P3D0, and +1.01 to +1.37 NDS across P0–P3. We are also continuing to train PETR on the full nuScenes training split, and to train UniMatch with a longer schedule. We will report both results in the final version.

AuthorsOpenReview ↗

Response to Reviewer dQXg (3/3)

Q2. Multiple seeds and repeatability

We would like to point out that, together with the new three-seed PETR runs, the paper now provides multiple training seeds or paired statistical tests on four hosts:

HostEvidenceWhere (revised PDF)
LVSMthree training seeds for the ray / ray–point / segment-bound representations, sampling off and on; sampling improves PSNR and SSIM AUC in all threeTable 16, p. 29 (original Table 13)
MVSplatshared-camera vs. sharing-removed: +0.319 dB perturbed PSNR, 95% paired-scene interval [+0.058, +0.596]Table 3, p. 9; Appendix E, Table 22
Pi3XFull vs. retrained controls, paired bootstrap over 24 spaces; vs. No state feedback all three intervals are below zeroTable 4(a), p. 10; Table 9, p. 17
PETR (new)nine methods × three training seeds, 24 epochsTables 1, 2, 19

The remaining main-table entries use strictly matched comparisons with a single training seed. Extending multiple seeds and larger-scale training to the other hosts is future work.


W2 / Q3. Update placement and number of updates

The layer gap is not a hand-tuned hyperparameter. An update is placed between every two consecutive geometry-aware attention modules, so the total number of updates is set entirely by each host's architecture (Sec. 3.4, pp. 5–6; Table 10(a), p. 19):

  • LVSM, MVSplat, PETR, StreamPETR, and UniMatch use the geometry encoding in six consecutive blocks, so they update at five inter-layer boundaries (G0G_0 → G5G_5).
  • Pi3X injects the geometry encoding at five sparse decoder slots (blocks 1, 9, 17, 25, and 33). It therefore updates only at the four boundaries between these slots (reading G0G_0–G4G_4 in turn) and reads out G5G_5 at the end. An update at an intermediate block without the geometry encoding would produce a state that no encoding layer uses.

For the effect of the number of updates, we summarize on Pi3X the fixed-checkpoint interventions (Table 4(b)), an additional one-update evaluation, and the same-loss retrained controls (Table 4(a); all paired intervals are below zero):

Pi3X settingAbsRel ↓Rotation (°) ↓Center (mm) ↓
Fixed checkpoint, 0 updates (always read G0G_0)0.04220.95334.64
Fixed checkpoint, first update only (read G1G_1; additional evaluation)0.04030.85231.04
Fixed checkpoint, all 4 inter-layer updates (read G0G_0–G4G_4)0.04030.84530.66
Same-loss retraining, No state feedback (same heads and losses)0.04280.85132.09
Same-loss retraining, Full0.04030.79528.13

As shown above, the first update gives the largest error reduction, and the later updates further lower the rotation and camera-center errors.


W2 / Q3. Ray–point ratio

By default, every host splits the rotary frequency pairs in each head 1:1 between ray and point channels, without per-host tuning (Sec. 3.3, p. 4). Appendix D.3.4 of the original submission (Table 16, p. 29; original Table 13) includes a related three-seed study of representation allocation on a six-layer, 16-head LVSM. It allocates whole heads: 16 ray heads vs. 8 ray + 8 point heads vs. segment-bound.

LVSM / RE10K, pose-noise PSNR AUC (dB)Sampling offSampling on
Ray only21.868±0.02322.074±0.052
Ray–point (8 + 8 heads)21.782±0.10121.984±0.182
Segment bounds21.683±0.05021.872±0.121

With sampling both off and on, ray-only and 1:1 ray–point differ by only about 0.09 dB, and shared-error sampling consistently improves PSNR and SSIM AUC in all three allocations. As discussed in Appendix D.3.4, rays and image features already give strong constraints in novel view synthesis, whereas point phases also depend on estimated depths. We did not grid-search the within-head channel ratio on every host, and the paper states conclusions only for the evaluated configurations.


We hope these results address the reviewer’s concern, and if so, we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.

Reviewer_JftFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

GaussRoPE introduces a geometry-aware rotary positional encoding framework for multi-view transformers with uncertain camera poses and depths. It models camera-shared pose errors and token-local depth errors using a structured Gaussian state, propagates these uncertainties into ray–point rotary phases, and uses shared-error sampling to preserve modeled cross-token correlations within each phase. The full framework also updates geometry and uncertainty from intermediate visual features for subsequent encoding stages. Evaluation covers host-specific integrations for novel-view synthesis, Gaussian reconstruction, 3D object detection, and depth estimation. The full method improves all five primary noisy-input metrics over its static encoding variant, although comparisons with competing methods vary by task and perturbation protocol.

Critical strengths
  1. Well-motivated uncertainty modeling. Distinguishing camera-shared pose errors from token-local depth errors captures dependencies that independent per-token uncertainty models miss. This is consequential because relative rotary phases depend on error correlations, and shared errors can cancel when their effects on two tokens agree.

  2. Compatibility with standard attention. Shared-error sampling incorporates correlated phase uncertainty through token-wise rotations without explicit pairwise uncertainty tensors. This preserves standard attention interfaces and supports fused attention kernels, making the approach more practical to integrate into existing transformers.

  3. Direct supporting evidence. The matched MVSplat ablation supports shared-camera error modeling, improving perturbed PSNR by 0.319 dB.

  4. Broad applicability. Experiments span view synthesis, Gaussian reconstruction, 3D detection, and depth estimation. Full improves all five primary noisy-input metrics over Static under their respective protocols, suggesting that the combined framework has utility across multiple tasks, although the magnitude of improvement varies.

Critical weaknesses
  1. Difficult to follow model-dependent presentation. The description moves between the common formulation, Pi3X-specific details, and adaptations to other models. This makes it difficult to reconstruct each integration’s forward pass and distinguish shared components from model-specific modifications.

  2. Unclear depth initialization. The architecture and initialization of LVSM’s depth prediction pathway are insufficiently specified. For MVSplat, it is unclear how depth becomes available before the first depth-dependent attention operation. These omissions hinder reproducibility and assessment of additional computation and supervision.

  3. Uneven incremental benefits and limited component attribution. Full jointly introduces uncertainty propagation and geometry updates, so its comparison with Static does not isolate their individual contributions. Although some improvements are substantial, others are small or absent: PETR gains only about 0.03 percentage points in mAP, while LVSM’s original-input PSNR slightly decreases, despite improvements in SSIM and LPIPS. Predominantly single-seed evaluations leave the reliability of the smaller gains unresolved, making the benefit of the added complexity difficult to assess.

Questions for authors
  1. LVSM depth initialization: Appendix C identifies a host-specific depth readout, but its implementation in LVSM remains unclear. What features produce the initial encoding depth, what is the readout architecture, and how is it initialized and supervised during source training?

  2. Practical benefit of Full over Static: Full provides substantial improvements in some settings but only small gains in others, such as PETR. What evidence establishes that the smaller gains are repeatable and practically meaningful? How much additional computation does Full require relative to Static under matched execution conditions?

  3. MVSplat execution order: Table 8 lists native RGB depth prediction as the geometry source. Since the original MVSplat predicts depth after Transformer feature extraction, how is depth available for the first depth-dependent GaussRoPE operation? Does initialization require a separate branch or an additional forward pass?

Overall, I found the paper difficult to follow because the common framework and model-specific implementations are interleaved, with important details spread across the main text and appendix. A consolidated explanation of the computation flow, together with answers to the questions above, would help resolve these uncertainties and provide a clearer basis for assessing the contribution and its practical benefits.

4 comments
AuthorsOpenReview ↗

Response to Reviewer JftF (1/2)

We thank the reviewer for the careful reading and for recognizing the shared-camera motivation and the matched MVSplat ablation. We answer each point below.


W1 — Presentation of shared and host-specific components

We agree that the shared framework and the host-specific details should be presented more clearly. In the revision, Sec. 3.1 (p. 3) now states the shared pipeline explicitly. The host-specific choices are collected in Table 10 in the appendix (p. 19), and Sec. 3 refers the reader to it. We summarize both below.

Shared pipeline. At each geometry-attention stage ℓ\ell, every host runs four steps:

  1. Initialize. Before the first geometry-aware attention layer, build G0G_0 from the given poses and observed depth; where depth is missing, use a pre-attention depth readout F0F_0.
  2. Sample. Linearize GℓG_\ell into ray–point phase sensitivities and draw one set of shared-error token phases.
  3. Attend. Apply rotary attention with these phases inside the host’s own attention block.
  4. Update. At each feedback boundary, an updater MLP proposes geometry and scale updates, which Eq. 5 fuses with G0G_0 into Gℓ+1G_{\ell+1}. The host’s native task head decodes the final outputs.

Host-specific choices. Hosts differ mainly in three choices: the query roles, whether rotation is applied to Q/K or to Q/K/V/O, and which updated states receive auxiliary supervision. Table 10 lists these differences and the remaining details, one row per host.


W2 / Q1 — LVSM depth initialization

In LVSM, F0F_0 comes from each token's 256-d image feature. A three-layer MLP (256→256→256→1, GELU) reads it out before the first geometry-aware encoder block. No external depth is used.

  • Initialization. The last layer has zero weights and a bias of log 5, so every token starts at a depth of 5 m. In DeMoN source training, context tokens with valid RGB-D depth use that depth. All other tokens, including target queries, use F0F_0. Target-view depth is only a training label, never an input.
  • Supervision. No auxiliary loss acts on G0G_0 (or F0F_0) directly; its gradients come through fusion and rendering. The auxiliary losses act only on the updated states G1G_1–G5G_5 (pose and depth residuals, uncertainty scales), together with the rendering loss.
  • Transfer. In RE10K transfer (RGB only), F0F_0 initializes all tokens. The depth MLP and the update MLPs are frozen.

Appendix C (p. 18) and the LVSM row of Table 10 give the full architecture, loss weights, and initial scales.


W2 / Q3 — MVSplat execution order

MVSplat's own depth is decoded only after the Transformer and the cost volume, so it comes too late for F0F_0. We therefore add a small depth readout on the CNN backbone features. It sits after the 2D positional encoding and before the first Transformer block. Everything runs in one forward pass:

  1. The CNN backbone extracts features for each view.
  2. The depth readout predicts F0F_0 and builds G0G_0.
  3. Six Transformer blocks run cross-view attention, with five updates (G0G_0 → G5G_5).
  4. The refined features and updated poses go into MVSplat's own cost volume and Gaussian head.

So initialization adds only a small branch, and the backbone and the Transformer each run once. Table 10(a) (original Table 8) now states that F0F_0 comes from this readout, and Appendix C (p. 18) describes the flow.


GaussRoPE (static) uses ray–point 3D RoPE with mean phases and the fixed initial geometry G0G_0. Unc adds uncertainty propagation, which samples shared camera and depth phase errors. Full further adds the five inter-layer geometry updates. Unc is a new intermediate variant for PETR (Sec. 4.1, p. 6).

AuthorsOpenReview ↗

Response to Reviewer JftF (2/2)

W3 / Q2 — PETR gain and repeatability of smaller gains

We thank the reviewer for raising this important point. To test the gain of Full over Static more reliably, we retrained all nine methods with the StreamPETR PETRv1-R50 24-epoch recipe, using the same three training seeds. The setup is: ImageNet-pretrained ResNet-50, 704×256 input, global batch 8, about 28% of nuScenes train, and 2,279 validation frames. These results now replace Table 1 (p. 7). Table 2 (p. 8) shows the step-by-step variants, and Table 19 (p. 31) shows all levels. The original results stay in Appendix D.3.5 (Table 20, p. 32).

PETR, mean±SD over 3 seeds (×100)P0D0 mAPP3D0 mAPNoisy mAPP0D0 NDSP3D0 NDSNoisy NDSΔNDS (P3−P0)
GaussRoPE (static)18.87±1.9317.99±1.8818.4722.64±2.0122.02±1.8522.31−0.62
GaussRoPE (unc)19.37±1.8818.92±1.8719.1822.94±2.1022.57±2.0522.77−0.37
GaussRoPE (full)19.93±1.9619.33±1.7419.6023.65±2.2123.39±1.9623.51−0.26

With 24-epoch training, Full beats Static by +1.06 / +1.34 / +1.13 mAP (P0D0 / P3D0 / Noisy) and by +1.01 to +1.37 NDS across P0–P3.

And the original submission also checked the smaller gains on three other hosts, with multiple seeds or paired tests:

  • LVSM (three seeds; Table 16, p. 29; original Table 13): shared-error sampling improves PSNR and SSIM AUC for all three geometric encodings.
  • MVSplat (Table 3): +0.319 dB, with a 95% interval of [+0.058, +0.596].
  • Pi3X (Table 9): all paired intervals for state feedback exclude zero.

Rerunning multiple seeds on all five hosts and all perturbation levels needs much more compute and time. We will include a larger-scale multi-seed evaluation in the final version.


W3 — Component attribution: propagation vs. geometry updates

To measure each component on its own, we use the Pi3X controls from the original submission (Appendix B.3, original Table 7). Each variant removes one component. All variants keep the same intermediate heads, auxiliary losses, and loss weights as Full. They are trained for 500 updates with one seed, and we report paired bootstrap intervals over 24 spaces.

Pi3X, joint perturbationAbsRel ↓Rotation (°) ↓Center (mm) ↓
No state feedback (later layers read G0G_0)0.04280.85132.09
Mean-phase (removes propagation)0.04140.83128.64
Full0.04030.79528.13
  • Geometry updates. Without state feedback, all three errors go up, and all paired 95% intervals exclude zero (Table 9, p. 17).
  • Propagation. Without it (Mean-phase), rotation error goes up (Mean-phase − Full: [0.0079°, 0.0550°]). On perturbed pixels, the AbsRel and center intervals include zero. Over all valid observed depths, AbsRel goes up at 0×, 1×, and 2× joint perturbation, and all intervals exclude zero (Table 4(c)).

So the gain of Full over Static does not come from one component alone. These controls are now in the main text as Table 4(a) and 4(c) (p. 10).


W3 — LVSM original-input PSNR

We agree that LVSM's Original PSNR drops slightly, and Sec. 4.2 (p. 8) now states this trade-off. On original inputs, PSNR goes from 23.48 to 23.45 dB, while SSIM rises from 0.647 to 0.690 and LPIPS falls from 0.385 to 0.288 (Table 13(a), p. 27). Under noisy inputs, PSNR rises by 0.83 dB (22.06 → 22.89 dB, Table 1). Under pose-only noise, it rises by 0.77 dB (21.84 → 22.61 dB, Table 13(b)).


Q2 — Additional computation under matched execution

We measured all methods on the same hardware and settings: 4× A100 for training (batch 2 per GPU), 1× A100 for inference (batch 1), and FP16. The full results are now in Table 5 (p. 10) and Appendix C.4, Table 11 (p. 23).

PETRTrain step math / flash (s)Train peak mem math / flash (GiB)Inference math / flash (ms)
Native PETR0.429 / 0.4262.18 / 2.1837.96 / 37.02
RayRoPE1.372 / 1.3677.29 / 6.02268.72 / 265.22
GaussRoPE (static)0.427 / 0.3997.21 / 4.9390.32 / 81.39
GaussRoPE (unc)1.087 / 1.0539.04 / 5.76228.83 / 232.57
GaussRoPE (full)1.153 / 1.1319.21 / 6.11245.17 / 243.14

With FlashAttention, Full takes 2.99× the inference time and 2.83× the training step time of Static. Most of the extra cost comes from uncertainty propagation and phase preparation (Static → Unc). The five geometry updates (Unc → Full) add only 10.57 ms (+4.55%) at inference and 0.077 s (+7.34%) per training step. Full is also faster than RayRoPE in both training and inference.


We hope these results address the reviewer’s concern, and if so, we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.

Dear Reviewer JftF,

Thank you again for your constructive feedback and for the time and effort you have devoted to reviewing our paper.

As the author–reviewer discussion period is coming to an end in the next few hours, we would be very grateful if you could take a look at our previous rebuttal and let us know whether it has addressed your concerns. In particular, we have:

  • Reorganized the presentation of the shared pipeline vs. host-specific designs, explicitly detailing the unified four-step flow (Initialize → Sample → Attend → Update) in Sec. 3.1 and consolidating all host-specific choices in Table 10 (W1).

  • Clarified the depth initialization and single-pass execution flow in LVSM and MVSplat, specifying the pre-attention depth readout architecture, initialization, supervision, and forward-pass order in Appendix C and Table 10 (W2, Q1 & Q3).

  • Verified the repeatability of gains and isolated individual component contributions, retraining all nine methods on PETR for 24 epochs across three seeds (Full beats Static by +1.06 to +1.34 mAP and +1.01 to +1.37 NDS) and foregrounding the loss-matched ablations separating uncertainty propagation from geometry updates in Table 4(a) (W3 & Q2). Furthermore, we also provided detailed information about Pi3X in the dQXg response.

  • Reported the end-to-end training/inference runtime and memory breakdown across Static → Unc → Full and baselines, showing that the five geometry updates add minimal overhead (+4.55% inference, +7.34% training) and that Full is faster than RayRoPE (Q2).

We believe that our rebuttal and additional experiments have substantively addressed the concerns raised in your review and would be very grateful if you could consider increasing your rating in light of these responses.

If any questions or concerns remain, we would be more than happy to discuss them further before the end of the discussion period. Thank you again for your time and consideration!

Thank you for the detailed clarifications and additional experiments. The explanations of depth initialization and execution order solved my implementation concerns, and the new PETR results provide evidence of improvement. However, I remain uncertain whether full’s gains over static justify its nearly three times inference cost. Could you clarify which practical settings this trade-off would make full preferable to Static? While I appreciate the detailed response, this concern leads me to maintain my current rating.

Reviewer_tL57OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper focuses on 3D rotary position embeddings for multi-view transformers when the camera poses and depths behind those embeddings are noisy. Instead of building pairwise uncertainty terms, it samples the errors once and reuses each camera’s sample across all of that camera’s tokens. The method is tested on top of five base reconstruction models. The paper reports gains under perturbed geometry.

Critical strengths
  1. The paper is well-written, with clear motivation, description of the method and experimental settings. The opening robot example and Fig. 1 make the problem easy to grasp: pose errors move a camera’s rays together, while depth errors move individual points along their rays. The experimental protocols are documented in unusual detail, which makes the setup transparent.

  2. The evaluation considered many reconstruction tasks and transformer-based models. The shared-camera ablation (Table 2/18) is a controlled experiment with paired bootstrap intervals. The method is compared against six positional encodings under both clean and perturbed geometry. Perturbations cover pose-only, depth-only, and combined noise at several strengths.

Critical weaknesses
  1. The hosts are far below their normal performance, which makes the main gains hard to interpret. For example, PETR reaches about 5% mAP on nuScenes, where a normally trained PETR gets around 30–40%. UniMatch’s AbsRel is around 1.0, where values near 0.1 are typical. In this setting, differences in inductive bias can dominate in ways that vanish at scale. At least one base model trained to a competitive level would make the setting more convincing.

  2. Overall cost isn’t reported. The cost study only measures factor preparation with fixed phases. The full cost at every feedback stage is not reported. I couldn’t find end-to-end training or inference time and memory for like Full vs. Static vs. baselines.

Questions for authors
  1. Can you show GaussRoPE on at least one host trained to near-standard performance, such as PETR at its usual nuScenes schedule?

  2. How much do predictions vary across phase samples at inference? Does averaging K samples, or using the analytic amplitude, do better than one sample?

  3. Can you separate uncertainty propagation from geometry updates on the main base models, using the same auxiliary losses?

4 comments
AuthorsOpenReview ↗

Response to Reviewer tL57 (1/3)

We thank the reviewer for the careful reading and for recognizing our motivation, the documented protocols, and the controlled shared-camera ablation.


W1 / Q1: host performance and a 24-epoch, three-seed PETR evaluation

We strongly agree with the reviewer that validating a method on hosts trained to a competitive level is essential for judging its real gains. We would first like to briefly clarify two points about the design of the original submission and the level of its hosts:

  1. Some hosts already perform strongly. Not all hosts in the original submission are at an early stage of convergence. For example, Pi3X is fine-tuned directly from the official fully pretrained Pi3X weights (an extension of π3\pi^3). Without added perturbation, Full reaches a depth AbsRel of 0.0307 (Table 4(c), p. 10).
  2. Strictly controlled matched comparison. The core design of the original submission is a matched comparison: within each host, all arms share the same starting point and are trained with exactly the same data and budget. The comparison therefore measures relative differences under the same conditions, rather than matching externally tuned absolute scores. PETR and UniMatch used shortened schedules mainly because of the compute cost of the full method × perturbation grid.

Building on this, we fully agree with the reviewer that testing whether these relative advantages hold under longer, higher-level training would establish the effectiveness and robustness of the method more convincingly. We therefore followed the reviewer's suggestion and retrained all nine methods on PETR with the standard StreamPETR PETRv1-R50 24-epoch recipe. Each method uses the same three training seeds (27 runs in total). The setup is:

  • ImageNet-pretrained ResNet-50 backbone, 704×256 input, global batch 8, learning rate 2×10−42\times 10^{-4}, and 23,736 updates;
  • about 28% of the nuScenes training split, and 2,279 validation frames.

The compared 3D RoPE methods have no nuScenes detection checkpoints. For a fair comparison, all methods start from the same ImageNet backbone. The new results replace Table 1 (p. 7), Table 2 (p. 8), and Table 19 (p. 31). The original 3,000-step screening results are kept in Appendix D.3.5 (Table 20, p. 32).

Mean±SD over three training seeds (×100). Noisy = mean of P1D0–P3D0; Δ = P3D0 − P0D0. Static uses ray–point 3D RoPE with mean phases and the fixed initial geometry G0G_0. Unc adds uncertainty propagation, which samples shared camera and depth phase errors. Full further adds the five inter-layer geometry updates. Unc is a new intermediate variant for PETR (Sec. 4.1, p. 6).

MethodP0D0 mAPP3D0 mAPNoisy mAPΔmAPP0D0 NDSP3D0 NDSNoisy NDSΔNDS
Native PETR17.55±1.4516.98±1.4917.28−0.5721.67±1.9221.23±1.9221.50−0.44
PRoPE17.54±1.9117.23±1.8317.42−0.3121.54±2.1721.32±2.0721.45−0.22
URoPE15.00±0.9614.68±0.8914.87−0.3220.00±1.6919.58±1.5919.80−0.42
RoRE18.76±1.3818.04±1.1618.41−0.7222.37±2.0321.88±1.8122.14−0.49
RayRoPE18.70±1.4918.14±1.4218.46−0.5622.82±1.3922.38±1.3022.60−0.44
GTA18.79±2.5118.24±2.4818.56−0.5522.36±2.7021.90±2.6122.16−0.46
GaussRoPE (static)18.87±1.9317.99±1.8818.47−0.8822.64±2.0122.02±1.8522.31−0.62
GaussRoPE (unc)19.37±1.8818.92±1.8719.18−0.4522.94±2.1022.57±2.0522.77−0.37
GaussRoPE (full)19.93±1.9619.33±1.7419.60−0.6023.65±2.2123.39±1.9623.51−0.26

P1D0 and P2D0 for all methods are in Table 19.

With the full 24-epoch schedule, Full has the highest mean mAP and NDS at every pose level P0–P3. Specifically:

  • On Original / Noisy inputs, Full's mAP is +2.38 / +2.32 higher than native PETR. It is +1.14 / +1.04 higher than GTA, the best competing encoding in mAP. Its NDS is +0.83 / +0.91 higher than RayRoPE, the best competing encoding in NDS.
  • Compared with its own Static variant, Full adds +1.06 / +1.13 / +1.34 mAP on P0D0 / Noisy / P3D0, and +1.01 to +1.37 NDS across P0–P3.

We are also continuing to train PETR on the full nuScenes training split, and to train UniMatch with a longer schedule. We will report both results in the final version.

AuthorsOpenReview ↗

Response to Reviewer tL57 (2/3)

W2: end-to-end cost, including every feedback stage

We thank the reviewer for suggesting a runtime cost analysis. We measured the end-to-end PETR cost of all nine methods under the same settings (training: 4× A100, batch 2 per GPU; inference: 1× A100, batch 1; FP16; Table 5, p. 10; Appendix C.4, Table 11, p. 23). Overall, most of the extra cost of Full over Static comes from uncertainty propagation and phase preparation. The five inter-layer geometry updates add very little (+4.55% at inference, +7.34% in training). Full is also faster than the competing method RayRoPE in both end-to-end training and inference.

MethodTrain step math / flash (s)Train peak mem math / flash (GiB)Inference math / flash (ms)Head mem increment math / flash (MiB)
Native PETR0.429 / 0.4262.18 / 2.1837.96 / 37.02156.8 / 74.3
PRoPE0.466 / 0.4676.26 / 4.7168.49 / 68.47183.3 / 91.3
RoRE0.458 / 0.4576.40 / 4.8660.80 / 62.27179.0 / 91.3
URoPE0.504 / 0.4996.37 / 4.8480.08 / 78.80186.9 / 91.3
GTA0.515 / 0.5056.33 / 4.7970.79 / 71.41183.7 / 91.3
RayRoPE1.372 / 1.3677.29 / 6.02268.72 / 265.22103.1 / 91.3
GaussRoPE (static)0.427 / 0.3997.21 / 4.9390.32 / 81.3990.1 / 75.7
GaussRoPE (unc)1.087 / 1.0539.04 / 5.76228.83 / 232.5797.3 / 90.0
GaussRoPE (full)1.153 / 1.1319.21 / 6.11245.17 / 243.1499.8 / 92.7
  • Total cost and per-stage breakdown (Static → Unc → Full). With FlashAttention, Full takes 2.99× the inference time of Static (243.14 vs. 81.39 ms) and 2.83× its training step time (1.131 vs. 0.399 s). Most of this comes from Static → Unc. The five geometry updates (Unc → Full) add only 10.57 ms at inference (about 2.1 ms per update), 0.077 s per training step, and 2.7 MiB of head memory. We updated Sec. 4.4 (p. 9) based on these numbers, and it now lists the costs of all three variants.
  • Comparison with RayRoPE. Full is faster than RayRoPE in training (1.131 vs. 1.367 s) and inference (243.14 vs. 265.22 ms). Its FlashAttention training memory is about the same (6.11 vs. 6.02 GiB).
  • Note on inference head memory. The last column is the peak memory added by the six-layer decoder at inference. Memory before the head is 387.0 MiB for the baselines and 421.0 MiB for GaussRoPE. Native PETR has the largest head memory with the math backend (156.8 MiB). Its eager attention builds the full 900 × 4,224 × 8 score tensor at once. GaussRoPE instead splits the queries into blocks by reference camera. Softmax is computed per query, so the result is mathematically the same (Appendix C.4, p. 23).
AuthorsOpenReview ↗

Response to Reviewer tL57 (3/3)

Q2. Variation across inference phase samples; K-sample averaging; analytic amplitude

We thank the reviewer for this insightful question. With a fixed Full checkpoint, we tested the variation across inference seeds and compared inference operators. Overall, the default single-sample inference (K=1K=1) gives very stable dataset-level metrics (seed spread ≤ 0.05). It is also clearly better than switching, at inference time only, to multi-sample averaging (K=2,4K=2, 4), the mean phase, or the analytic attenuation operator.

Very small variation across inference seeds. Over four inference seeds, pooled NDS on the evaluation set ranges from 23.3506 to 23.3972 (spread only 0.047), and pooled mAP ranges from 19.5244% to 19.5666% (spread only 0.042 pp).

Inference seedNDS (×100)mAP (%)
123.392919.5244
223.395319.5504
323.397219.5666
423.350619.5484

K-sample averaging and analytic amplitude. On P3D0 (2,279 frames), averaging K sampled outputs inside cross-attention (K=2,4K=2, 4), the mean phase (Mean-phase), and the analytic relative-variance attenuation (pairwise-analytic) are all worse than K=1K=1, which is used in training (NDS −0.79, −0.81, −0.79, −2.69; mAP −0.59, −0.60, −0.58, −3.04). These drops are much larger than the seed spread of K=1K=1. So with K=1K=1 training, keeping the same single-sample estimate at inference works better than changing the inference operator afterwards.

Inference modeNDS (×100)mAP (%)ΔNDSΔmAP
K=1K=1 (default)23.4019.56——
K=2K=222.6118.97−0.79−0.59
K=4K=422.5918.96−0.81−0.60
Mean-phase22.6118.98−0.79−0.58
Pairwise-analytic20.7116.53−2.69−3.04

Q3. Separating uncertainty propagation from geometry updates under the same auxiliary losses

We thank the reviewer for pointing out that the contributions of the two core mechanisms should be separated from changes in the auxiliary losses. We would like to clarify that the original submission already included controlled experiments that remove uncertainty propagation (Mean-phase) and inter-layer state feedback (No state feedback) separately, with exactly the same prediction heads and auxiliary losses (Appendix B.3, original Table 7). To make them easier to find for the reviewer and readers, we moved them into the main text as Table 4(a) in Sec. 4.3 (p. 10). The full paired bootstrap intervals are in Appendix B.3, Table 9 (p. 17). On Pi3X, both retrained control variants keep all intermediate prediction heads and use the same auxiliary losses and weights as Full.

Pi3X, joint perturbationAbsRel ↓Rotation (°) ↓Center (mm) ↓
No state feedback (always reads G0G_0)0.04280.85132.09
Mean-phase (no sampled phase)0.04140.83128.64
Full0.04030.79528.13
  • Removing uncertainty propagation (Full vs. Mean-phase, same auxiliary losses). Phase propagation significantly lowers the rotation error (95% interval [−0.0550°, −0.0079°]). Over all valid observed depths, Full also has significantly lower depth AbsRel at 0×, 1×, and 2× joint perturbation, and all three intervals exclude zero (Table 4(c), p. 10).
  • Removing inter-layer geometry updates (Full vs. No state feedback, same auxiliary losses). State feedback significantly lowers all three geometry errors, and all 95% intervals exclude zero: AbsRel [−0.0045, −0.0005], rotation [−0.1100°, −0.0022°], and center [−5.25, −2.56] mm. This matches the fixed-checkpoint test in Table 4(b) (no feedback: 0.953°, 34.64 mm; normal feedback: 0.845°, 30.66 mm).

On PETR, Table 2 (p. 8) adds the components one at a time. Adding propagation (Static → Unc) raises P3D0 mAP from 17.99 to 18.92. It also halves the P0→P3 mAP drop (−0.88 → −0.45). Adding the five updates (Unc → Full) further raises P0D0 / P3D0 mAP to 19.93 / 19.33. It also shrinks the NDS drop from −0.37 to −0.26. On PETR, the auxiliary losses of each variant grow with the enabled prediction heads (Sec. 3.5, p. 6). The text next to Table 2 states this setting and points readers to the loss-matched ablation in Table 4(a) above.


We hope these results address the reviewer’s concern, and if so, we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.

Dear Reviewer tL57,

Thank you again for your constructive feedback and for the time and effort you have devoted to reviewing our paper.

As the author–reviewer discussion period is coming to an end in the next few hours, we would be very grateful if you could take a look at our previous rebuttal and let us know whether it has addressed your concerns. In particular, we have:

  • Retrained all nine methods on PETR with the standard 24-epoch recipe across three seeds (27 runs), showing that GaussRoPE (full) consistently achieves the best mAP and NDS across all clean and noisy settings (W1 & Q1). Furthermore, we also provided detailed information about Pi3X in the dQXg response.

  • Reported end-to-end training/inference time and memory across all feedback stages (Static → Unc → Full) and baselines, confirming that the five geometry updates add minimal overhead (+4.55% inference, +7.34% training) and that Full is faster than RayRoPE (W2).

  • Evaluated inference variance across seeds and compared K-sample averaging and analytic amplitude, showing that K=1K=1 is highly stable (spread ≤ 0.05) and outperforms switching inference operators at test time (Q2).

  • Foregrounded the loss-matched ablations in the main text (Table 4(a)), isolating uncertainty propagation and inter-layer geometry updates under identical auxiliary losses with significant paired bootstrap gains (Q3).

We believe that our rebuttal and additional experiments have substantively addressed the concerns raised in your review and would be very grateful if you could consider increasing your rating in light of these responses.

If any questions or concerns remain, we would be more than happy to discuss them further before the end of the discussion period. Thank you again for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.