- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes GaussRoPE, a 3D RoPE with uncertain-aware camera poses and depths for multiple attention-based visual or geometry foundation models. GaussRoPE keeps a single structured Gaussian state over camera poses and depths in which the ray and point descriptors are built. Both RoPE phase for fusing mean geometry and error propagation of shared and local geometry are analytical. The layerwise geometry refinement further updates the geometry and its uncertainty for the following RoPEs. The experiments are conducted on 5 host models where the authors aim to demonstrate the advantages of GaussRoPE.
Critical strengths
- This paper incorporate the pose uncertainty into the 3D RoPE which is not discussed in the previous 3D RoPEs. To support the standard Flash attention interface, the authors propose the shared-error sampling strategy, which avoid explicit pairwise uncertainty tensors.
- This papers test the model on multiple visual and geometry foundation models, and within the matched training setup including initialization, inputs, and training budgets, the proposed method successfully improves the metrics. And the ablation studies also shows the importance of the layerwise geometry refinement.
Critical weaknesses
- The unconverged models' performance can not validate the effectiveness of the proposed method. Although we understand the training budgets are limited for authors, the training regime (including training step and training dataset) is far from the one to train the vanilla versions of the hosts so nearly all metrics have clear gap with the original hosts. Even if we consider the reported results as the beginning stage of the full training, we can not predict the full training version of the proposed method is better than the baselines. Also, a single seed result can not validate the proposed method is always better in the early training stage.
- The hyperparameter selection is not discussed in detail. For layerwise updates, the hyperparameter of the layer gap for updating is not discussed. For example, why we have G_0 to G_5 for Pi3X at specific blocks and what is the effect of different number of updates. For the ray and point channels, what is the best ratio of them? Is the RoPE model sensitive to the variation of the ratio on different hosts?
Questions for authors
- Is it possible to report the fully-training results of GaussRoPE and other baselines at least on one host?
- Could you report the mean and variance on multiple seeds on each host to validate the current results are not a lucky run?
- Do analysis or ablation studies on the hyperparameters as mentioned above.
- The description of static and full version is not clear. Could you explain more on that?
7 comments
Response to Reviewer dQXg (2/3)
Q1. The gap to the original hosts and the 24-epoch PETR retraining
We strongly agree with the reviewer that validating a method on hosts trained to a competitive level is essential for judging its real gains. We would first like to briefly clarify two points about the design of the original submission and the level of its hosts:
- Some hosts already perform at the publicly reported level. Not all hosts in the original submission show an under-convergence gap. For example, Pi3X is fine-tuned directly from the official fully pretrained Pi3X weights (an extension of ). Without added perturbation, Full reaches a depth AbsRel of 0.0307 (Table 4(c), p. 10), which is already a well-converged level.
- Strictly controlled matched comparison. The core design of the original submission is a matched comparison: within each host, all arms start from the same baseline and are trained with exactly the same data and budget. The comparison therefore measures relative differences under the same conditions, rather than matching externally tuned absolute scores. PETR and UniMatch used shortened schedules mainly because of the compute cost of the full method × perturbation grid.
Building on this, we fully agree with the reviewer that short, single-seed early results are not enough to confirm that these relative advantages still hold after full training. We therefore followed the reviewer's suggestion and retrained all nine methods with the standard StreamPETR PETRv1-R50 24-epoch recipe (ImageNet-pretrained ResNet-50, 704×256, global batch 8, learning rate , 23,736 updates; about 28% of the nuScenes training split and 2,279 validation frames). Each method uses the same three training seeds (27 runs in total). The compared 3D RoPE methods have no nuScenes detection checkpoints, so for a fair comparison all methods start from the same ImageNet backbone. The new results replace Table 1 (p. 7) and Table 2 (p. 8); all pose levels are in Table 19 (p. 31), and the original 3,000-step results are kept in Appendix D.3.5 (Table 20, p. 32).
Mean±SD over three training seeds (×100); Noisy = mean of P1D0–P3D0.
| Method | Original mAP | Noisy mAP | P3D0 mAP | Original NDS | Noisy NDS | P3D0 NDS |
|---|---|---|---|---|---|---|
| Native PETR | 17.55±1.45 | 17.28 | 16.98±1.49 | 21.67±1.92 | 21.50 | 21.23±1.92 |
| PRoPE | 17.54±1.91 | 17.42 | 17.23±1.83 | 21.54±2.17 | 21.45 | 21.32±2.07 |
| URoPE | 15.00±0.96 | 14.87 | 14.68±0.89 | 20.00±1.69 | 19.80 | 19.58±1.59 |
| RoRE | 18.76±1.38 | 18.41 | 18.04±1.16 | 22.37±2.03 | 22.14 | 21.88±1.81 |
| RayRoPE | 18.70±1.49 | 18.46 | 18.14±1.42 | 22.82±1.39 | 22.60 | 22.38±1.30 |
| GTA | 18.79±2.51 | 18.56 | 18.24±2.48 | 22.36±2.70 | 22.16 | 21.90±2.61 |
| GaussRoPE (static) | 18.87±1.93 | 18.47 | 17.99±1.88 | 22.64±2.01 | 22.31 | 22.02±1.85 |
| GaussRoPE (unc) | 19.37±1.88 | 19.18 | 18.92±1.87 | 22.94±2.10 | 22.77 | 22.57±2.05 |
| GaussRoPE (full) | 19.93±1.96 | 19.60 | 19.33±1.74 | 23.65±2.21 | 23.51 | 23.39±1.96 |
With the full 24-epoch schedule, Full has the highest three-seed mean mAP and NDS at every pose level. On Original / Noisy inputs, its mAP is +2.38 / +2.32 higher than native PETR and +1.14 / +1.04 higher than GTA, the best competing encoding in mAP. Its NDS is +0.83 / +0.91 higher than RayRoPE, the best competing encoding in NDS. Compared with Static, Full adds +1.06 / +1.13 / +1.34 mAP on P0D0 / Noisy / P3D0, and +1.01 to +1.37 NDS across P0–P3. We are also continuing to train PETR on the full nuScenes training split, and to train UniMatch with a longer schedule. We will report both results in the final version.
Response to Reviewer dQXg (3/3)
Q2. Multiple seeds and repeatability
We would like to point out that, together with the new three-seed PETR runs, the paper now provides multiple training seeds or paired statistical tests on four hosts:
| Host | Evidence | Where (revised PDF) |
|---|---|---|
| LVSM | three training seeds for the ray / ray–point / segment-bound representations, sampling off and on; sampling improves PSNR and SSIM AUC in all three | Table 16, p. 29 (original Table 13) |
| MVSplat | shared-camera vs. sharing-removed: +0.319 dB perturbed PSNR, 95% paired-scene interval [+0.058, +0.596] | Table 3, p. 9; Appendix E, Table 22 |
| Pi3X | Full vs. retrained controls, paired bootstrap over 24 spaces; vs. No state feedback all three intervals are below zero | Table 4(a), p. 10; Table 9, p. 17 |
| PETR (new) | nine methods × three training seeds, 24 epochs | Tables 1, 2, 19 |
The remaining main-table entries use strictly matched comparisons with a single training seed. Extending multiple seeds and larger-scale training to the other hosts is future work.
W2 / Q3. Update placement and number of updates
The layer gap is not a hand-tuned hyperparameter. An update is placed between every two consecutive geometry-aware attention modules, so the total number of updates is set entirely by each host's architecture (Sec. 3.4, pp. 5–6; Table 10(a), p. 19):
- LVSM, MVSplat, PETR, StreamPETR, and UniMatch use the geometry encoding in six consecutive blocks, so they update at five inter-layer boundaries ( → ).
- Pi3X injects the geometry encoding at five sparse decoder slots (blocks 1, 9, 17, 25, and 33). It therefore updates only at the four boundaries between these slots (reading – in turn) and reads out at the end. An update at an intermediate block without the geometry encoding would produce a state that no encoding layer uses.
For the effect of the number of updates, we summarize on Pi3X the fixed-checkpoint interventions (Table 4(b)), an additional one-update evaluation, and the same-loss retrained controls (Table 4(a); all paired intervals are below zero):
| Pi3X setting | AbsRel ↓ | Rotation (°) ↓ | Center (mm) ↓ |
|---|---|---|---|
| Fixed checkpoint, 0 updates (always read ) | 0.0422 | 0.953 | 34.64 |
| Fixed checkpoint, first update only (read ; additional evaluation) | 0.0403 | 0.852 | 31.04 |
| Fixed checkpoint, all 4 inter-layer updates (read –) | 0.0403 | 0.845 | 30.66 |
| Same-loss retraining, No state feedback (same heads and losses) | 0.0428 | 0.851 | 32.09 |
| Same-loss retraining, Full | 0.0403 | 0.795 | 28.13 |
As shown above, the first update gives the largest error reduction, and the later updates further lower the rotation and camera-center errors.
W2 / Q3. Ray–point ratio
By default, every host splits the rotary frequency pairs in each head 1:1 between ray and point channels, without per-host tuning (Sec. 3.3, p. 4). Appendix D.3.4 of the original submission (Table 16, p. 29; original Table 13) includes a related three-seed study of representation allocation on a six-layer, 16-head LVSM. It allocates whole heads: 16 ray heads vs. 8 ray + 8 point heads vs. segment-bound.
| LVSM / RE10K, pose-noise PSNR AUC (dB) | Sampling off | Sampling on |
|---|---|---|
| Ray only | 21.868±0.023 | 22.074±0.052 |
| Ray–point (8 + 8 heads) | 21.782±0.101 | 21.984±0.182 |
| Segment bounds | 21.683±0.050 | 21.872±0.121 |
With sampling both off and on, ray-only and 1:1 ray–point differ by only about 0.09 dB, and shared-error sampling consistently improves PSNR and SSIM AUC in all three allocations. As discussed in Appendix D.3.4, rays and image features already give strong constraints in novel view synthesis, whereas point phases also depend on estimated depths. We did not grid-search the within-head channel ratio on every host, and the paper states conclusions only for the evaluated configurations.
We hope these results address the reviewer’s concern, and if so, we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.
Response to Reviewer dQXg (1/3)
We thank the reviewer for the detailed review and for noting the pose-uncertainty modeling, the FlashAttention-compatible shared-error sampling, and the matched training setup. Revisions in the updated PDF are marked in blue; page numbers refer to the revised PDF.
Q4. Static, Unc, and Full
We thank the reviewer for reminding us to clarify the definitions of the variants. The revised paper now gives all three side by side at the start of Sec. 4.1 (p. 6). The supervision and per-host details are in Sec. 3.5 (p. 6) and Table 10(b) (p. 19):
In short, Static is the plain 3D RoPE baseline on every host. It uses mean phases and a fixed G0, and one checkpoint is evaluated at all perturbation levels. Unc is a new intermediate variant for the step-by-step PETR ablation (Table 2, p. 8). Note that the Pi3X No state feedback control (Table 4(a)) differs from Static: it keeps and supervises all intermediate geometry heads so that the training losses match exactly, but it does not pass the updated state to later encoding layers.