acceptodds
Under review as a conference paper at ICLR 2027

SGA3R: Spherical Geometry-Aware 3D Reconstruction from Panoramas

Abstract

Panoramic cameras capture the surrounding scene in a single exposure and eliminate errors caused by external multi-view registration. Nevertheless, a critical limitation remains when adapting perspective-pretrained 3D reconstruction models to panoramic imagery: conventional planar projection and positional encodings do not adequately reflect spherical geometry. We present SGA3R, a framework that incorporates spherical geometry into view projection and attention for 3D reconstruction from single or multiple panoramas. To represent each panorama with a compact set of informative perspective views, we formulate view projection as information-guided spherical-cap optimization under explicit coverage and overlap constraints. To encode spherical relationships within the pretrained backbone, we introduce Dual-Path Harmonic Encoding (DPHE) that integrates Additive Harmonic Encoding and Rotary Harmonic Encoding, where the latter lifts RoPE from SO(2) to SO(3) relative rotation between patch directions via irreducible subspaces of multiple orders. We further extend SGA3R to multiple panoramas using alternating attention across view, frame, and global domains, together with frame-conditioned pose refinement, while preserving the single-frame architecture. Experiments on indoor datasets Matterport3D, Stanford 2D-3D-S, as well as the outdoor dataset Mapillary Metropolis show that SGA3R achieves the lowest absolute relative depth error (AbsRel) among the evaluated panoramic methods, and the full model reduces AbsRel on Matterport3D by 11.9% relative to the strongest evaluated baseline, while using only six views per panorama and no cross-view layering. Multi-frame experiments on Holo360D further demonstrate improved performance in point cloud reconstruction. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Vision, Panoramic 3D Reconstruction, Feed-Forward 3D Reconstruction, Rotary Position Encoding, Geometric Vision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qsBROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SGA3R, a feed-forward 3D reconstruction framework designed for single-view and multi-view panoramic images. It addresses the key issue that existing 3D foundation models, primarily trained on perspective camera images, cannot be directly transferred to panoramic imagery. The paper first investigates perspective image cropping methods, selecting appropriate sequences of perspective views from panoramic images based on information density. Subsequently, it proposes Dual-Path Harmonic Encoding (DPHE)—comprising AHE (for absolute position encoding) and RHE (for relative relationship encoding)—and formulates the encoding similarly to RoPE, thereby enabling the transfer of encoding capabilities from SO(2) to SO(3). Finally, the framework is extended to handle multiple frames by decomposing attention into view, frame, and global components and incorporating frame-conditioned pose refinement to achieve multi-view reconstruction. The authors evaluated single-view reconstruction on Matterport3D, Stanford 2D-3D-S, and Mapillary Metropolis, and multi-view reconstruction on Holo360D, achieving state-of-the-art (SOTA) performance.

Critical strengths
  1. Proposed DPHE, enabling the attention mechanism to simultaneously perceive both absolute and relative geometric relationships on the sphere.
  2. Conducted extensive quantitative experiments, evaluating the method on multiple indoor and outdoor datasets.
Critical weaknesses
  1. The distinctions and boundaries of innovation regarding the proposed method versus existing work have not been sufficiently articulated: the backbone design of SGA3R draws upon VGGT-Omega (as already noted in the paper); the information-based panoramic image projection is similar to that of CAPDepth; and the frame-conditioned pose refinement resembles that of V-DPM. The core innovation lies in the DPHE module.
  2. The ablation study is inadequate: in Table 4, the performance gain of the full model over the VO+RF baseline is marginal, indicating that the DPHE—the method's core innovation—does not yield significant results in practice; furthermore, adding RHE or AHE in isolation actually resulted in a drop in accuracy.
Questions for authors
  1. AAA employs a three-layer model architecture, whereas the perspective-image-based VFM (VGGT-Omega) utilizes a two-layer design. Given this structural difference, how are the VGGT-Omega weights loaded, and is any additional initialization required?
1 comment

W1. We agree with the reviewer's division of the components: SGA3R uses VGGT-Ω\Omega directly as its backbone, and our contribution is a spherical adaptation on top of it rather than a redesign of the trunk. For view projection, information-driven panoramic projection is indeed in the same family as CAPDepth and VGGT-360. The difference is that CAPDepth uses a content-aware Pannini projection, adjusting the projection parameters from distortion cues, and serves CNN depth estimation on a single panorama, whereas our view optimization is spherical-cap optimization with explicit coverage and overlap constraints, with the view budget and the FOV adjustable, and is designed for a 3D foundation model rather than a per-patch depth network. For the pose head, we do follow the frame-conditioned decoder of V-DPM, as stated in the paper ("following V-DPM"). Our modification connects that head to view/frame/global alternating attention and couples it with the pose prior produced during view projection.V-DPM itself has no such spherical multi-frame structure.

W2. As the reviewer notes, DPHE is one of the paper's novel components. its purpose is to inject spherical geometric priors into a 3D foundation model trunk and to anchor attention to position. AHE and RHE provide an absolute geometric anchor and a relative frame for attention, respectively, and the two act in synergy. By contrast, PanoVGGT's positional encoding provides a single spherical absolute position, and adding it to SGA3R lowers accuracy. What matters is therefore the synergy of AHE and RHE rather than either positional encoding alone.

To facilitate the comparison, we report a Matterport3D ablation under the same training protocol (a 15k-step arm, mean / median) together with attention visualisations.

ConfigurationAbsRel ↓\downarrowδ1\delta_1 ↑\uparrowRMSE ↓\downarrow
Mean / MedMean / MedMean / Med
VO+RF5.90 / 4.8196.15 / 98.040.96 / 0.72
Full model4.49 / 3.6397.20 / 98.300.87 / 0.68

Q1. The three nested attention domains that AAA places side by side inside the trunk keep the same depth as VGGT-Ω\Omega, so the pretrained weights transfer module by module: the view domain and the frame domain are initialised directly from VGGT-Ω\Omega's intra-frame and cross-frame attention, and the remaining shared parameters are loaded by matching tensor names. The newly added global domain has no counterpart in the pretrained model and is responsible for a different representation from the frame domain at a different level, so it is initialised to identity. The three-layer model therefore reduces exactly to the pretrained two-layer behaviour at initialisation, and the new global domain is then activated gradually. This both inherits the perspective prior and avoids the disturbance that random initialisation or shared weights would cause to the existing representation.

Reviewer_MEsZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SGA3R, a spherical geometry-aware framework for 3D reconstruction from panoramic images. The method combines adaptive spherical-cap view optimization, Dual-Path Harmonic Encoding (DPHE), and confidence-weighted spherical ray fusion with a pretrained VGGT-Ω backbone. It further extends the framework to multiple panoramas through alternating attention across view, frame, and global domains. Experiments conducted on various indoor and outdoor datasets demonstrate that this approach achieves performance improvements in both panoramic depth estimation and multi-frame reconstruction.

Critical strengths
  • The paper addresses a practical problem of adapting perspective-pretrained 3D foundation models to panoramic reconstruction.

  • The proposed framework combines several geometric modeling strategies into a coherent reconstruction pipeline and reports promising improvements on multiple benchmarks.

Critical weaknesses
  1. Limited technical novelty. Several key components closely resemble existing methods. Adaptive view projection has been explored in VGGT-360, while spherical positional encoding has been investigated in UniK3D, PanoVGGT, and GTA.

  2. Limited effectiveness of the proposed modules. Table 4 shows that RHE alone slightly worsens AbsRel (12.92 → 12.97), while AHE substantially degrades it (12.92 → 18.28). Combining both provides only a marginal improvement (12.92 → 12.75), raising questions about their effectiveness.

  3. Unclear fairness of multi-frame comparisons. The proposed model is fine-tuned on Holo360D, but it is unclear whether PanoVGGT receives equivalent fine-tuning. This raises concerns about whether the improvements stem from the proposed architecture or additional training.

  4. Insufficient evidence of outdoor generalization. On Mapillary Metropolis, SGA3R achieves only a marginal AbsRel improvement over VGGT-360 (12.65 vs. 12.68), while exhibiting substantially worse RMSE (8.84 vs. 5.23). This raises concerns about its robustness to domain shifts, particularly compared with training-free approaches.

Questions for authors
  1. Can the authors clarify the key technical novelty of SGA3R compared with existing methods such as VGGT-360, UniK3D, and PanoVGGT?

  2. Can the authors explain why AHE and RHE provide only marginal improvements, or even degrade performance, in Table 4?

  3. Was PanoVGGT fine-tuned on Holo360D under the same training settings as SGA3R? If not, can the authors provide a fair comparison?

  4. Can the authors explain the substantially worse RMSE on Metropolis and provide additional cross-dataset experiments to demonstrate generalization to unseen outdoor scenes?

4 comments

We begin with one change to the evaluation protocol that affects Table 2. The original protocol truncated depth at 50 m, a setting inherited from prior work for comparability, but this truncation excludes far-field error from the metrics. With every other setting identical, we re-evaluated every method, ours and all baselines, on Metropolis without truncation.The corrected numbers form the fourth column group of Table 2. The top of the ranking is unchanged, with SGA3R still ahead of VGGT-360, while some baselines reorder below the first two places. Without the cap SGA3R is the best on all three metrics (mean AbsRel 13.62 against 18.97, mean δ1\delta_1 86.45 against 67.53, mean RMSE 18.47 against 19.27), and its mean-AbsRel lead over the runner-up widens to 28.2%.Under the cap it remains the best on AbsRel and δ1\delta_1. The main conclusion therefore holds, and is stronger under a physically reasonable protocol. The revised manuscript and the merged table have been updated in the corresponding paragraph.

Q1. On the reviewer's concern about novelty.

  1. VGGT-360's adaptive projection places KK additional views adaptively within a fixed total budget (NB=8N_B=8 base views plus K=2K=2 neighbours). Its own ablation shows that simply increasing the number of views has marginal benefit at high cost. Extending this to multiple frames requires the view budget to be consistent across frames, otherwise cross-frame attention is undefined. SGA3R makes the budget an explicit adjustable quantity (Table 1, V=4/5/6V{=}4/5/6) and uses a fixed VV per frame to support cross-frame attention, which offers a workable solution to the view-redundancy cost of the projection-based panoramic reconstruction route.

  2. PanoVGGT is designed for the ERP-native route, and spherical continuity is difficult to genuinely enforce through positional encoding, so its depth prediction breaks down at the two ends of the panorama (Figure 4). Our approach is another attempt within the projection-based route, encoding cross-view relations, spherical continuity and spherical relative position. GTA encodes token geometry as a query--key relative transformation in a multi-view perspective transformer (camera encoding plus SO(2)\mathrm{SO}(2) image positional encoding) and depends on known camera extrinsics, whereas SGA3R operates in a pose-free setting without external camera extrinsics.

Q2. AHE and RHE provide an absolute geometric anchor and a relative frame for attention, respectively. From two different directions they jointly unify a set of views on one continuous sphere at the attention level. This means the model processes not only pixel signals but also prior geometric signals. Moreover, just as PanoVGGT's positional encoding provides a spherical absolute position, adding it to SGA3R does not improve the metrics. For AHE, the l=7l=7 SH basis is more expressive but harder to optimise, so, as the ablation shows, the synergy of AHE and RHE yields the accuracy gain rather than either positional encoding alone.

We have added an ablation on Matterport3D under the same training protocol (a 15k-step ablation arm, reported as mean / median). The conclusion is consistent. In addition, we have added attention visualisations to Appendix B, illustrating how DPHE anchors and distributes attention.

ConfigurationAbsRel ↓\downarrowδ1\delta_1 ↑\uparrowRMSE ↓\downarrow
Mean / MedMean / MedMean / Med
VO+RF5.90 / 4.8196.15 / 98.040.96 / 0.72
Full model4.49 / 3.6397.20 / 98.300.87 / 0.68

Q3. We thank the reviewer for pointing out the issue with the multi-frame comparison. In the paper's table we evaluated PanoVGGT with its released weights and did not fine-tune it on Holo360D. As a supplementary experiment, we fine-tuned PanoVGGT on Holo360D with the same data and optimiser as SGA3R for 10k steps.The results are in the table below. SGA3R retains its advantage in point-cloud Comp and Overall, and remains ahead in the pose metric AUC@30 (computed with max⁡(R,T)\max(R,T)) and the translation error. In addition, the cross-frame module is a module extension compatible with the original VGGT-Ω\Omega architecture.It yields a consistent improvement over SGA3R's own single-frame + ICP baseline and produces consistent multi-frame reconstruction. We do not claim state-of-the-art multi-frame performance. This module yields consistent gains over the single-frame + ICP baseline.

ModelTTAcc ↓\downarrowComp ↓\downarrowOverall ↓\downarrowAUC@30 ↑\uparrowRR ↓\downarrowTT ↓\downarrow
SGA3R (Multi-frame)20.1468 / 0.10270.0758 / 0.04800.1113 / 0.07470.1476 / 0.035013.0692 / 9.882956.0680 / 59.6594
SGA3R (Multi-frame)30.1638 / 0.11480.0728 / 0.04180.1183 / 0.07950.1451 / 0.023816.2073 / 12.362054.3613 / 55.1449
PanoVGGT (Refine)20.1558 / 0.12530.1049 / 0.09190.1303 / 0.10950.0860 / 0.00006.6583 / 4.956962.2193 / 63.9340
PanoVGGT (Refine)30.1615 / 0.13230.0914 / 0.07540.1264 / 0.10380.0966 / 0.00009.1907 / 6.561060.1982 / 60.4726

Q4. We thank the reviewer for the question about evaluating SGA3R on the outdoor dataset. We have added to the paper an evaluation setting that applies no depth truncation at all. Under it SGA3R becomes the best on all three metrics, with a mean-AbsRel improvement of 28.2% over the runner-up. In addition, we have completed a zero-shot experiment on Deep360: SGA3R leads on δ1\delta_1 and RMSE, with mean δ1\delta_1 and mean RMSE ahead of the runner-up by 9.2% and 32.7%, respectively.

MethodAbsRel ↓\downarrowδ1\delta_1 ↑\uparrowRMSE ↓\downarrow
Mean / MedMean / MedMean / Med
UniK3D60.69 / 55.9842.17 / 42.5566.04 / 61.71
DAP47.19 / 44.6548.31 / 48.4157.72 / 58.24
VGGT-36040.71 / 38.2645.40 / 44.2653.77 / 51.62
PanoVGGT27.05 / 25.3061.86 / 63.6052.53 / 49.39
Ours32.54 / 33.1867.55 / 68.7235.36 / 32.29

Dear Reviewer MEsZ,

Thank you for your thoughtful follow-up. If there are any remaining questions or points requiring further clarification, we would be pleased to engage in further discussion.

Best regards, The Authors

Reviewer_MEsZOpenReview ↗

Comments

I appreciate the authors providing additional experiments and clarification. These new results help address some of the concerns I raised. However, my main concern regarding methodological novelty remains. While I acknowledge the practical improvements over prior work, I find that the proposed designs offer limited new methodological insights. Therefore, I will maintain my original rating.

Reviewer_hgqrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SGA3R adapts a perspective-pretrained 3D reconstruction backbone to panoramic images. It projects each panorama into six perspective views using a non-learned, information-guided procedure that adjusts viewing directions and fields of view subject to coverage and overlap constraints. The reconstruction network incorporates Dual-Path Harmonic Encoding, combining additive spherical-harmonic features with rotary encoding based on SO(3) representations. Predictions are mapped back to the sphere and combined through confidence-weighted ray fusion. The multi-frame extension introduces attention across view, frame, and global domains, together with frame-conditioned pose refinement.

Experiments evaluate single-frame reconstruction on Matterport3D, Stanford 2D-3D-S, and Mapillary Metropolis, and multi-frame reconstruction on Holo360D. The method reports an 11.9% relative reduction in mean AbsRel over PanoVGGT on Matterport3D.

Critical strengths
  1. The paper addresses a relevant problem: bringing strong perspective-pretrained reconstruction models to panoramic imagery. Incorporating spherical directions into attention and reconciling predictions along shared rays gives the framework a coherent geometric motivation. The resulting system is potentially useful where panoramic training data are limited.

  2. The complete system achieves competitive results across indoor and outdoor panoramic datasets, including the lowest reported mean AbsRel among the evaluated methods in Table 2. The multi-frame results also suggest practical value beyond single-panorama depth estimation. These results are encouraging, although the method does not lead on every metric.

Critical weaknesses
  1. The evaluation leaves uncertainty about component attribution and comparability.

In Table 4, adding VO+RF simultaneously changes view selection, fusion, and the supervision/evaluation representation from per-view depth to fused ERP depth. The reported 9.5% AbsRel improvement therefore supports the combined configuration, but does not isolate the contribution of adaptive view selection. I recognize that Table 1 provides useful additional evidence for adaptive projection; however, its fixed and adaptive configurations also use different initial FOVs, so the effects of direction selection and FOV cannot be read separately from that comparison.

Section 4.1 evaluates some perspective baselines per view and SGA3R on a fused ERP grid. This does not establish that the comparison is unfair, but the different ray sampling, overlap handling, and aggregation warrant clarification before interpreting the margins. The existing timing results in Tables 1, 4, and 5 are helpful. Clearer reporting of input resolutions and which pipeline stages the timings include would make the accuracy–cost trade-off easier to assess. I do not regard non-learned preprocessing as a flaw by itself, nor do I require ordinary perspective-dataset experiments for a paper whose stated scope is panoramic reconstruction.

  1. Several definitions and component claims need more precise support.

The most consequential issue is AUC@30. Section 4.3.2 describes the minimum of rotation and translation errors, whereas the cited VGGT evaluation code uses their maximum (https://github.com/facebookresearch/vggt/blob/evaluation/evaluation/test_co3d.py). These criteria have different meanings: the minimum can conceal a large error in one component. It is unclear whether this is a wording error or the criterion actually used in Table 3. If the implementation follows VGGT, correcting the description would resolve this concern; otherwise, the score should be distinguished from the referenced protocol and the implications for the comparison explained.

The view-selection objective integrates a fixed information-density field over the union of covered regions. With at least 99% coverage, its ability to distinguish layouts may be limited, and it does not directly describe how observations are allocated within the already-covered region. The paper explicitly presents a heuristic and reports information gains in Table 5, which I appreciate. What remains unclear is how those gains relate to the stated objective and to the centroid attraction, repulsion, FOV expansion, and first-feasible stopping rule. A clearer interpretation would suffice; a proof of optimality is not necessary.

The multi-frame quantitative comparison is limited to PanoVGGT. Figure 5 includes useful single-frame-plus-ICP qualitative evidence, but the separate effects of AAA and frame-conditioned pose refinement are not quantified. Finally, Appendix B's argument about dependence only on relative rotation needs qualification when polar truncation is activated by each token's absolute latitude. The latitude-dependent projections should remain explicit in that argument. This concern does not by itself contradict yaw invariance, since yaw preserves latitude.

Overall, I appreciate the effort to build a coherent panoramic reconstruction framework, and the reported results suggest practical value. My current recommendation is a weak rejection, primarily because the evaluation does not yet make the component-specific interpretations sufficiently clear. The AUC implementation and the interpretation of VO+RF are the two points that would most affect my assessment. I do not expect a comprehensive new benchmark during rebuttal: clear explanations and relevant results already available to the authors may address several concerns. I would be happy to reconsider my recommendation in light of that clarification.

Questions for authors

The following are clarification requests that could affect my assessment, rather than a request for a comprehensive set of new experiments.

  1. Does Table 3 use the minimum or maximum of rotation and translation errors for AUC@30, and how are the pairwise errors aggregated? If the code follows VGGT, please correct the wording. If it uses the minimum, please distinguish this score from the referenced protocol and, if feasible, provide the comparison under the maximum-based criterion.

  2. How should the 9.5% improvement from VO+RF be interpreted given the simultaneous changes to fusion, supervision, and evaluation? Table 1 already offers useful evidence: do existing results help separate direction adaptation, FOV changes, and fusion? If not, please qualify the attribution to the combined configuration.

  3. For the per-view versus fused-ERP comparisons, please clarify ground-truth ray sampling, masks, overlap handling, alignment, and metric aggregation. Is there existing evidence that these choices do not materially alter the reported comparison?

  4. Please clarify the input resolutions of the main baselines and whether reported inference times include view optimization, projection, and fusion. Any available runtime or peak-memory measurements, accompanied by their hardware and measurement conditions, would help assess the practical trade-off. I am not asking for every baseline to be rerun on identical hardware.

  5. How does Table 5's information-gain measure relate to the union-integral objective once coverage is nearly complete? In Appendix B, under what conditions does the relative-rotation-only statement hold when latitude-dependent polar truncation is active? Keeping these projections explicit would help distinguish that statement from the yaw-invariance result.

  6. Are there existing quantitative comparisons or ablations for single-frame reconstruction plus ICP, AAA, or frame-conditioned pose refinement? If these are unavailable, please clarify which claims are supported by the full-system comparison and which remain unisolated.

3 comments

Q1. We thank the reviewer for pointing out the issue with the AUC@30 protocol. The AUC@30 reported in the paper is computed from the minimum of the rotation and translation errors.Our attribution of this protocol to VGGT was inaccurate, and we thank the reviewer for catching it. The original benchmark computes, for each unordered pair of frames, the geodesic rotation error and the scale-invariant translation-direction error, combines the two as min⁡(R,T)\min(R,T), and averages the normalised CDF over 00--30∘30^\circ. To remove the ambiguity, we re-evaluated all methods under VGGT's maximum criterion.SGA3R and PanoVGGT obtain 0.1476/0.14510.1476/0.1451 (T=2/3T{=}2/3) and 0.0022/0.00080.0022/0.0008 (T=2/3T{=}2/3), respectively. The ranking and the conclusions are unchanged. In the paper we remove the attribution to VGGT, state the minimum criterion explicitly in the table caption, and report the maximum criterion alongside it, so that the protocol actually used is not conflated with the reference protocol.

Q2. Regarding the reviewer's concern about the VO+RF ablation, we re-ran the experiment. With view optimization only and every other setting unchanged, the mean AbsRel and mean δ1\delta_1 are 13.08 and 88.89. This is slightly weaker than the mean AbsRel / mean δ1\delta_1 of VO+RF (12.92 / 91.50).The gap is 0.16 / 2.61. The gain from ray fusion should be attributed both to the removal of point-cloud layering in overlap regions and to its regularising effect on depth errors there, together with the difference between the per-view and fused-ERP protocols themselves (detailed in Q3). Concretely, the per-view protocol computes the error of each repeated observation in an overlap separately and then averages them.By Jensen's inequality, the error of the averaged depth is no larger than the average of the per-view errors.

This also demonstrates that view optimization improves the representation of the model, in particular by giving low-texture regions more context to reason from. The complete ablation is Table 4, with its uncapped counterpart in Appendix D, and before/after visualisations of view optimization on outdoor and indoor scenes have been added to Appendix C.The optimised view distribution drives views away from weakly informative regions such as the ceiling and the sky.

Q3. On the per-view and fused-ERP protocols: the former computes the metric for each view and averages the per-view metrics as the metric of that ERP panorama, whereas the latter computes the metric directly on the ERP grid. Both use the same ERP ground truth and the same metric implementation. The per-view baselines sample along their own viewing rays on the native ERP image without interpolation. SGA3R is scored directly on the ERP grid after fusion. The validity mask is projected with the same spherical mapping as the RGB views. All methods use the same alignment protocol, with a single IRLS-Cauchy scale solved per panorama.

The difference concentrates in four places.

First, the ray set. The per-view protocol samples inside six 5122512^2 views at their own angular spacing (about 0.22∘0.22^\circ per pixel at a 115∘115^\circ field of view), so the same surface in an overlap band is sampled once by each of two views. The fused protocol keeps a single value per ray on the native ERP grid (1024×20481024\times2048 for MP3D and Metropolis, about 0.18∘0.18^\circ per pixel), and scores only the cells that receive a depth prediction.

Second, the overlap region. The per-view protocol computes the error of each view separately and then averages. The fused protocol first merges the observations into one depth and then scores it. The difference between the two is the inter-view inconsistency at that point, which is exactly what fusion is meant to resolve.

Third, the number of valid pixels. When views have different numbers of valid pixels, the equal-weight average over views and the pixel average differ by a weighting by valid-pixel count.

Fourth, RMSE is systematically smaller under the per-view protocol. Perspective-view depth is the camera-frame zz coordinate, whereas ERP depth is radial range (r=z⋅normr=z\cdot\mathrm{norm}). This unit change increases RMSE under the fused-ERP protocol.

We quantified the difference between the two protocols sample by sample on Metropolis. The two protocols differ little in AbsRel and δ1\delta_1, whereas RMSE under the ERP protocol is on average 6.97 larger than under the per-view protocol. In the table, Δ\Delta denotes MetricERP−Metricper-view\text{Metric}_\text{ERP} - \text{Metric}_\text{per-view}.

MetricΔ\Delta meanΔ\Delta medΔ\Delta stdΔ\Delta p10Δ\Delta p90
AbsRel (%)−0.058-0.0580.094.403−1.573-1.573+2.060+2.060
RMSE (m)+6.970+6.970+6.798+6.798+2.605+2.605+3.995+3.995+10.148+10.148
δ1\delta_1 (%)+0.087+0.087+0.354+0.3543.400−2.282-2.282+2.743+2.743

Q4. On the reviewer's question about resolution: all methods are scored against the native ERP ground truth of each dataset (1024×20481024\times2048 for MP3D and Metropolis, 2048×40962048\times4096 for Stanford 2D-3D-S). ERP-native baselines run at their released input resolution and are scored on their native grid. Methods that take perspective views share six 512×512512\times512 crops.π3\pi^3 and VGGT-Ω\Omega are scored view by view, and SGA3R uses the same six crops and is scored on the fused ERP grid.

On inference and training efficiency: the reported inference time is the end-to-end time per panorama (view optimization, projection, forward pass and fusion). Peak inference memory is below 7 GB. For full-model training with a frozen patch embedding, peak memory is 47.1 GB per GPU on an RTX Pro 5000.

Q5. On the reviewer's question about view optimization: coverage and information density are defined on different spaces, so the information-density gain remains valid once the coverage constraint is met. Coverage is the fraction of the sphere covered by the union, counting each spherical point once, whereas global information integrates the information density carried by each patch, including patches in overlap regions. Concretely, the coverage repulsion and the FOV growth provide the coverage guarantee, while the information attraction pulls views towards denser regions according to per-patch information density. In practice we stop the optimisation the first time both the minimum angular overlap and the coverage constraint are satisfied, but the procedure also supports further refinement once the constraints are met. The comparison figure added to Appendix C shows the view distribution of the same panorama before and after optimisation under the coverage constraint.The optimised view centres concentrate on building and structural texture.

On the reviewer's question about the RHE module: the retained l≤1l\le1 subspace of the latitude truncation is itself an SO(3)\mathrm{SO}(3) representation, and the projection commutes with the Wigner-D matrices, so after truncation the score still depends only on the relative rotation Rp−1Rp′R_p^{-1}R_{p'}. This holds both when both ends are truncated and when one end is truncated and the other is complete. Concretely, the truncation is a per-token latitude gating projection. With φ(p)=arcsin⁡(p⋅ez)\varphi(p)=\arcsin(p\cdot e_z), a pole latitude threshold φpole\varphi_{\rm pole}, and m(p)=1[∣φ(p)∣≤φpole]m(p)=\mathbf 1[|\varphi(p)|\le\varphi_{\rm pole}], define

P(p)=diag(I1, I3, m(p)I5, …, m(p)I15).P(p)=\mathrm{diag}\big(I_1,\ I_3,\ m(p)I_5,\ \dots,\ m(p)I_{15}\big).

The l≤1l\le1 blocks are always retained and mm gates only the l≥2l\ge2 blocks. P(p)P(p) is block diagonal with coefficients 0 or 1 in each block, hence commutes with any Wigner-D matrix. Truncating both ends gives P,D(Rp−1Rp′),PP,D(R_p^{-1}R_{p'}),P, and truncating one end gives D(Rp−1Rp′),PD(R_p^{-1}R_{p'}),P.Both depend only on the relative rotation. Yaw is fixed to eze_z, so P(Sp)=P(p)P(Sp)=P(p) and RSp=SRpR_{Sp}=SR_p, and neither the relative rotation nor the projection changes.

Q6. We have a quantitative comparison for single-frame + ICP, and we state explicitly what the full-system comparison does and does not support.

  1. Single-frame + ICP. The revised Table 3 reports the per-frame ICP-aligned single-frame pipeline together with the other baselines. it is weaker than the multi-frame pipeline in Acc, Comp and Overall. This supports the conclusion that cross-frame modelling is preferable to per-frame reconstruction followed by ICP.

  2. AAA. AAA and the frame-conditioned pose refinement are building blocks of the multi-frame extension, and we do not have a complete independent ablation for them. For reference, on an earlier path we applied a DynamicVGGT-style variant that attaches full global attention and cross-frame register attention at the output of the VGGT backbone as a multi-frame extension. Under the same data and 40k steps, that variant obtained mean Acc / Comp / Overall = 0.57 / 1.43 / 1.00 and AUC@30 / R / T = 0.0017 / 26.91 / 55.70, which is why we moved to the final view/frame/global alternating design. The frame-conditioned pose refinement follows the time-conditioned decoder of V-DPM.

We hope the clarifications above address the reviewer's questions about the AUC@30 protocol, the attribution of gains to VO+RF, and the remaining points. We would be grateful if the reviewer would reconsider the recommendation.

Dear Reviewer hgqr,

Thank you for your time and feedback. We sincerely appreciate you taking the time to review our paper and offer your valuable support.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.