- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SGA3R, a feed-forward 3D reconstruction framework designed for single-view and multi-view panoramic images. It addresses the key issue that existing 3D foundation models, primarily trained on perspective camera images, cannot be directly transferred to panoramic imagery. The paper first investigates perspective image cropping methods, selecting appropriate sequences of perspective views from panoramic images based on information density. Subsequently, it proposes Dual-Path Harmonic Encoding (DPHE)—comprising AHE (for absolute position encoding) and RHE (for relative relationship encoding)—and formulates the encoding similarly to RoPE, thereby enabling the transfer of encoding capabilities from SO(2) to SO(3). Finally, the framework is extended to handle multiple frames by decomposing attention into view, frame, and global components and incorporating frame-conditioned pose refinement to achieve multi-view reconstruction. The authors evaluated single-view reconstruction on Matterport3D, Stanford 2D-3D-S, and Mapillary Metropolis, and multi-view reconstruction on Holo360D, achieving state-of-the-art (SOTA) performance.
Critical strengths
- Proposed DPHE, enabling the attention mechanism to simultaneously perceive both absolute and relative geometric relationships on the sphere.
- Conducted extensive quantitative experiments, evaluating the method on multiple indoor and outdoor datasets.
Critical weaknesses
- The distinctions and boundaries of innovation regarding the proposed method versus existing work have not been sufficiently articulated: the backbone design of SGA3R draws upon VGGT-Omega (as already noted in the paper); the information-based panoramic image projection is similar to that of CAPDepth; and the frame-conditioned pose refinement resembles that of V-DPM. The core innovation lies in the DPHE module.
- The ablation study is inadequate: in Table 4, the performance gain of the full model over the VO+RF baseline is marginal, indicating that the DPHE—the method's core innovation—does not yield significant results in practice; furthermore, adding RHE or AHE in isolation actually resulted in a drop in accuracy.
Questions for authors
- AAA employs a three-layer model architecture, whereas the perspective-image-based VFM (VGGT-Omega) utilizes a two-layer design. Given this structural difference, how are the VGGT-Omega weights loaded, and is any additional initialization required?
W1. We agree with the reviewer's division of the components: SGA3R uses VGGT-Ω directly as its backbone, and our contribution is a spherical adaptation on top of it rather than a redesign of the trunk. For view projection, information-driven panoramic projection is indeed in the same family as CAPDepth and VGGT-360. The difference is that CAPDepth uses a content-aware Pannini projection, adjusting the projection parameters from distortion cues, and serves CNN depth estimation on a single panorama, whereas our view optimization is spherical-cap optimization with explicit coverage and overlap constraints, with the view budget and the FOV adjustable, and is designed for a 3D foundation model rather than a per-patch depth network. For the pose head, we do follow the frame-conditioned decoder of V-DPM, as stated in the paper ("following V-DPM"). Our modification connects that head to view/frame/global alternating attention and couples it with the pose prior produced during view projection.V-DPM itself has no such spherical multi-frame structure.
W2. As the reviewer notes, DPHE is one of the paper's novel components. its purpose is to inject spherical geometric priors into a 3D foundation model trunk and to anchor attention to position. AHE and RHE provide an absolute geometric anchor and a relative frame for attention, respectively, and the two act in synergy. By contrast, PanoVGGT's positional encoding provides a single spherical absolute position, and adding it to SGA3R lowers accuracy. What matters is therefore the synergy of AHE and RHE rather than either positional encoding alone.
To facilitate the comparison, we report a Matterport3D ablation under the same training protocol (a 15k-step arm, mean / median) together with attention visualisations.
Q1. The three nested attention domains that AAA places side by side inside the trunk keep the same depth as VGGT-Ω, so the pretrained weights transfer module by module: the view domain and the frame domain are initialised directly from VGGT-Ω's intra-frame and cross-frame attention, and the remaining shared parameters are loaded by matching tensor names. The newly added global domain has no counterpart in the pretrained model and is responsible for a different representation from the frame domain at a different level, so it is initialised to identity. The three-layer model therefore reduces exactly to the pretrained two-layer behaviour at initialisation, and the new global domain is then activated gradually. This both inherits the perspective prior and avoids the disturbance that random initialisation or shared weights would cause to the existing representation.