- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents LaRI3R, a feed-forward two-view framework for reconstructing visible and occluded 3D geometry. The method introduces a Bidirectional Ray Intersection representation that organizes layered geometry using both forward and backward ray orders, together with geometry-aware feature aggregation that uses intermediate 3D point maps to exchange information across views. The approach is evaluated on object- and scene-level reconstruction benchmarks and further applied to relative camera pose estimation. The reported results show improvements over representative generative, iterative, layered, and feed-forward reconstruction methods, with ablations supporting the contributions of the proposed representation and geometry-aware aggregation.
Critical strengths
-
The paper addresses an important and well-motivated reconstruction setting with a simple but meaningful representation design. The bidirectional layered representation directly targets the instability of forward-only layer assignment under severe occlusion, while retaining the efficiency of a feed-forward model.
-
The experimental evaluation is broad and generally supports the proposed method. The paper evaluates object-level and scene-level reconstruction, provides component and layer-number ablations, analyzes different view-overlap and visibility regions, and further demonstrates potential benefits for downstream camera pose estimation.
Critical weaknesses
-
The downstream relative-pose evaluation is not sufficiently reliable or internally consistent. The paper states that AUC@5 and AUC@20 are computed from the minimum of the rotation and translation angular errors. If interpreted literally, a prediction with, for example, rotation error but translation-direction error would still be assigned a joint error of only , allowing a severely incorrect pose to be counted as accurate. In addition, Eq. (8) formulates pose recovery as a 3D-to-3D similarity alignment over rotation, translation, and scale, whereas the experimental description states that the reported results are obtained using PnP within RANSAC. These are different estimation problems and require different correspondences. Since Table 3 is used to support the downstream benefit of occluded geometry, both the metric definition and the actual pose-estimation protocol need to be unambiguous for this conclusion to be trustworthy.
-
The core BRI formulation and the evidence isolating its contribution are not sufficiently precise. Algorithm 1 does not fully specify how forward and backward selections are handled when they overlap; for example, if only one valid intersection remains on a ray, the forward-most and backward-most points are identical and could be assigned to two different layers. The relationship between the layers in the point-map formulation and the layers produced by BRI is also left implicit, while Eq. (5) omits the layer index and does not explicitly define the scale normalizers. Moreover, the geometry-aware branch simultaneously introduces a PTv3 point transformer with Sonata pretraining, so the improvement from CD 0.0206 / [email protected] 0.685 to 0.0200 / 0.700 does not isolate whether the gain comes from the proposed 3D aggregation mechanism, additional model capacity, or pretrained geometric features. These issues make both the exact formulation and the empirical attribution of the main contributions difficult to assess.
Questions for authors
-
The relative-pose evaluation metric appears logically problematic. The paper states that AUC@5 and AUC@20 are computed based on the minimum of the rotation and translation angular errors. If this definition is literal, a prediction with, for example, a rotation error of but a translation-direction error of would still have a joint error of only . Therefore, a pose can be severely incorrect in one component while still being counted as accurate. Since Table 3 is used to support the downstream benefit of the reconstructed occluded geometry, this metric definition directly affects the validity of that conclusion.
-
The pose-estimation protocol is internally inconsistent between the mathematical formulation and the experimental description. Eq. (8) defines pose recovery as a 3D-to-3D similarity alignment that jointly estimates rotation, translation, and scale, whereas the following paragraph states that the reported results are obtained using PnP within RANSAC. These are different estimation problems. For example, Eq. (8) requires 3D-to-3D correspondences, while PnP conventionally requires 3D-to-2D correspondences and camera intrinsics. The manuscript does not explain which solver actually produces the results in Table 3.
-
The Bidirectional Ray Intersection construction contains unresolved ambiguities when the forward and backward selections overlap. Algorithm 1 extracts and from the same remaining point set, assigns them to two different layers, and only then removes their union. For example, if a ray contains only one remaining valid surface intersection, the forward-most and backward-most points are necessarily identical, so the same point could be assigned to both layers under the stated procedure. The paper does not specify how overlapping selections, odd numbers of intersections, missing intersections, or depth ties are handled.
-
The ablation does not fully isolate the claimed benefit of geometry-aware 3D aggregation from additional model capacity and pretraining. The geometry-aware branch introduces a PTv3-based point transformer initialized with Sonata self-supervised weights, whereas the BRI-only model does not contain this additional pretrained geometric branch. Table 4 improves from CD 0.0206 / [email protected] 0.685 with BRI to 0.0200 / 0.700 after adding geometry-aware aggregation. However, this comparison does not isolate whether the gain comes specifically from explicit 3D positional aggregation, additional network capacity, or the Sonata initialization.
-
Some headline improvements are not based on input-matched comparisons. The paper highlights a 28% reduction in CD and a 15% improvement in [email protected] over LaRI on GSO, with CD improving from 0.025 to 0.018. However, LaRI is a single-view layered reconstruction method, whereas LaRI3R uses two input views. Therefore, this improvement cannot be attributed solely to the proposed bidirectional representation and geometry-aware aggregation, since the observation budget also changes.
-
The scale-invariant regression objective is under-specified for the proposed multi-layer representation. The prediction is defined as , whereas Eq. (5) writes the regression loss using only and sums over , without specifying how the layer index is handled. In addition, the scale normalizers and are only described as being computed from the valid point set without an explicit definition. For example, it is unclear whether normalization is performed jointly across all layers or independently for each layer, which would result in different supervision.
4 comments
Some substantive uncertainties remain regarding the pose-estimation protocol and the scope of the downstream claims. In particular, the paper expresses both predicted point maps in the first camera’s coordinate frame, while the rebuttal states that each view’s focal length is estimated from its first-layer point map. It remains unclear how the second view’s point map is obtained in its own camera coordinate frame for this estimation. Moreover, the pose-estimation gains from including occluded geometry are substantial on GSO but much smaller on Omni3D, leaving limited evidence for a consistently strong downstream benefit across datasets. Nevertheless, the explanations of the BRI edge cases and multi-layer normalization, the additional controls for Sonata initialization and cross-view aggregation, and the input-matched comparison have alleviated several of my original concerns. Taking these improvements and the remaining uncertainties into account, I will increase my rating to 2 . I hope the authors will incorporate the corrected metric definition, clarified training and evaluation procedures, and additional experiments into the revised manuscript, as the PDF currently available to me still retains the original, unrevised descriptions.
We thank the reviewer for the follow-up comments and for raising the score.
Q1 Obtaining the focal length for the second camera view.
We follow the practice of DUSt3R [1] by constructing symmetric input pairs. Specifically, the focal length of view 1 is computed by inputting the view pair in the order , while the focal length of view 2 is computed by inputting the view pair , such that each view's focal length is correctly computed under its own coordinate system. We will add further details and refer readers to the original paper.
[1] DUSt3R: Geometric 3D Vision Made Easy. CVPR 2024.
Q2. Downstream benefit on Omni3D.
-
Omni3D combines images and objects collected from several different source datasets, most of which fall outside our training distribution, so its scenes are more diverse and complex than GSO. This is a challenge for layered reconstruction methods such as LaRI and LaRI3R, which are trained on much less data than visible-surface models such as DUSt3R. As a relatively new task, occluded-geometry reconstruction still lacks large-scale training data. We plan to scale up the training data in future work.
-
Occluded geometry still helps under this generalization challenge. Even on such out-of-domain data, adding the occluded layers does not hurt pose estimation and still improves it (AUC@5 0.209 → 0.210, AUC@20 0.688 → 0.694). LaRI3R also remains the best among all feedforward methods on Omni3D (AUC@5 0.210 vs. 0.141 for VGGT and 0.123 for Reloc3r).
On the revised manuscript.
We sincerely thank the reviewer for the reminder. We had planned to revise the manuscript after collecting all reviewers' feedback, so that every suggestion could be addressed together. Unfortunately, given the time limit, we may not be able to upload the revised version in time.
However, we promise that all of the points raised by the reviewer will be corrected or added in the revised manuscript, including the corrected metric definition, the clarified training and evaluation procedures, and the additional experiments. The same applies to all points raised by the other reviewers. We sincerely appreciate the reviewer's constructive suggestions and will make sure that each of them is fully reflected in the final version.
Response to Reviewer gRkN
We sincerely thank the reviewer for the constructive feedback. We address each concern below and will update the final paper accordingly.
Q1. Definition of the pose AUC
Thank you very much for pointing this out. This is a writing typo instead of a technical misalignment. We strictly follow the evaluation codebase of Reloc3r [1], where the AUC in Tab. 3 is computed on the maximum, rather than the minimum, of the rotation and translation-direction errors of each pair. The experiments align with the previous methods, and the gains are evaluated under the same previous protocol. We will correct the typos in the revision.
Q2. Pose estimation protocol
Thank you for pointing this out. Our original description was not sufficiently clear. We intended to illustrate that the relative pose can be solved by the 3D-3D alignment in Eq. (8), but since this alignment is sensitive to noise and outliers, PnP within RANSAC is a better alternative, so we use it in the evaluation.
Following DUSt3R, all results in Tab. 3 are obtained with PnP within RANSAC. Specifically, for each view, the focal length is estimated from its predicted first-layer point map with the Weiszfeld solver as in DUSt3R, and the principal point is set to the image center. The 2D-3D correspondences are formed by the pixels of one view and its point map in the frame of the other view. Since the occluded points lie along the same camera rays, they share these pixels and serve as additional correspondences.
We will revise the description around Eq. (8) and add the PnP formulation in the revision to avoid misunderstanding.
Q3. Edge cases in Algorithm 1
In each iteration of Algorithm 1, the forward-most and backward-most points on each ray are assigned to the two corresponding layers and then removed from the remaining set. When only one point remains, as on a ray with an odd number of intersections, this point is duplicated into both layers. When no points remain, the corresponding layers are marked as invalid in a validity mask and are not supervised in the loss. We will clarify this in the revision.
Q4. Multi-layer regression objective
Our loss follows DUSt3R and extends it to multiple layers. The N layers in the point maps correspond exactly to the 2K layers produced by BRI, with K=2 and N=4 in all experiments. In Eq. (5), the summation runs over all valid pixels and layers (u,v,n)∈D. As in DUSt3R, z and zgt are the average distances of the valid points to the origin, computed jointly over all layers of both views for the prediction and the ground truth, respectively. We will make Eq. (5) explicit in the revision.
Q5. Isolating the effect of geometry-aware aggregation
To separate the contribution of 3D aggregation from pretraining and model capacity, we add the following experiments:
We conduct an additional experiment by removing the Sonata initialization of the point transformer. As shown in Tab. R8 (d), the Sonata initialization brings no performance gain compared with (c), possibly due to the gap between its pretraining data and our training data.
We keep the same PTv3 architecture but remove cross-view aggregation in 3D (e). This variant leads to a performance drop compared to our proposed approach (c), indicating that the gain comes from cross-view aggregation in 3D rather than additional capacity.
Tab. R8. Ablation on geometry-aware aggregation on GSO.
Q6. Input-matched comparison
LaRI3R addresses a new setting between single-view occluded-geometry reconstruction (e.g., LaRI) and two-view visible-surface reconstruction (e.g., DUSt3R). The improvements are therefore meaningful relative to both settings. With the same two-view input, our method still outperforms baseline (a) in Tab. R8, which uses LaRI’s unidirectional layered representation for occluded scene reconstruction, reducing CD by 11.5% and improving [email protected] by 6.9%.
In the revision, we will clarify the comparison with LaRI to make clear that the performance gains come from two sources, including the additional input view and our proposed design.
[1] Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization. https://github.com/ffrivera0/reloc3r