acceptodds
Under review as a conference paper at ICLR 2027

LaRI3R: Bidirectional Ray Intersections for Occluded 3D Scene Reconstruction from Two Views

Abstract

Recovering complete 3D scene geometry from limited observations remains challenging due to severe occlusions and restricted viewpoints. Although observing a scene from many viewpoints can progressively improve reconstruction completeness, dense multi-view reconstruction often requires costly exploration and computation. Recent single-view feedforward methods address the trade-off between reconstruction quality and efficiency through layered representations, but they still suffer from strong ambiguity in heavily occluded regions. Extending layered reconstruction to two views is also non-trivial because occlusion ordering becomes view-dependent, and visually corresponding evidence can be weak or absent across views. We present LaRI3R, a new feedforward two-view framework for complete 3D scene reconstruction under severe occlusions. Our approach is built on two core designs: 1) We introduce a bidirectional layered representation that predicts geometry in both forward and backward ray orders, which reduces geometric inconsistencies in heavily occluded regions. 2) Instead of relying purely on appearance-based matching, we perform geometry-aware feature aggregation using explicit 3D positional cues. This design enables geometrically related but visually dissimilar regions to communicate across views. Experiments on both object- and scene-level benchmarks show that LaRI3R outperforms representative generative, iterative, and feedforward methods while preserving efficient inference. We further show that the recovered occluded geometry serves as a strong structural cue for downstream 3D vision tasks such as camera pose estimation with notable improvements over existing large geometric models.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
3D reconstruction, occluded geometry reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gRkNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LaRI3R, a feed-forward two-view framework for reconstructing visible and occluded 3D geometry. The method introduces a Bidirectional Ray Intersection representation that organizes layered geometry using both forward and backward ray orders, together with geometry-aware feature aggregation that uses intermediate 3D point maps to exchange information across views. The approach is evaluated on object- and scene-level reconstruction benchmarks and further applied to relative camera pose estimation. The reported results show improvements over representative generative, iterative, layered, and feed-forward reconstruction methods, with ablations supporting the contributions of the proposed representation and geometry-aware aggregation.

Critical strengths
  1. The paper addresses an important and well-motivated reconstruction setting with a simple but meaningful representation design. The bidirectional layered representation directly targets the instability of forward-only layer assignment under severe occlusion, while retaining the efficiency of a feed-forward model.

  2. The experimental evaluation is broad and generally supports the proposed method. The paper evaluates object-level and scene-level reconstruction, provides component and layer-number ablations, analyzes different view-overlap and visibility regions, and further demonstrates potential benefits for downstream camera pose estimation.

Critical weaknesses
  1. The downstream relative-pose evaluation is not sufficiently reliable or internally consistent. The paper states that AUC@5 and AUC@20 are computed from the minimum of the rotation and translation angular errors. If interpreted literally, a prediction with, for example, 2∘2^\circ rotation error but 60∘60^\circ translation-direction error would still be assigned a joint error of only 2∘2^\circ, allowing a severely incorrect pose to be counted as accurate. In addition, Eq. (8) formulates pose recovery as a 3D-to-3D similarity alignment over rotation, translation, and scale, whereas the experimental description states that the reported results are obtained using PnP within RANSAC. These are different estimation problems and require different correspondences. Since Table 3 is used to support the downstream benefit of occluded geometry, both the metric definition and the actual pose-estimation protocol need to be unambiguous for this conclusion to be trustworthy.

  2. The core BRI formulation and the evidence isolating its contribution are not sufficiently precise. Algorithm 1 does not fully specify how forward and backward selections are handled when they overlap; for example, if only one valid intersection remains on a ray, the forward-most and backward-most points are identical and could be assigned to two different layers. The relationship between the NN layers in the point-map formulation and the 2K2K layers produced by BRI is also left implicit, while Eq. (5) omits the layer index and does not explicitly define the scale normalizers. Moreover, the geometry-aware branch simultaneously introduces a PTv3 point transformer with Sonata pretraining, so the improvement from CD 0.0206 / [email protected] 0.685 to 0.0200 / 0.700 does not isolate whether the gain comes from the proposed 3D aggregation mechanism, additional model capacity, or pretrained geometric features. These issues make both the exact formulation and the empirical attribution of the main contributions difficult to assess.

Questions for authors
  1. The relative-pose evaluation metric appears logically problematic. The paper states that AUC@5 and AUC@20 are computed based on the minimum of the rotation and translation angular errors. If this definition is literal, a prediction with, for example, a rotation error of 2∘2^\circ but a translation-direction error of 60∘60^\circ would still have a joint error of only 2∘2^\circ. Therefore, a pose can be severely incorrect in one component while still being counted as accurate. Since Table 3 is used to support the downstream benefit of the reconstructed occluded geometry, this metric definition directly affects the validity of that conclusion.

  2. The pose-estimation protocol is internally inconsistent between the mathematical formulation and the experimental description. Eq. (8) defines pose recovery as a 3D-to-3D similarity alignment that jointly estimates rotation, translation, and scale, whereas the following paragraph states that the reported results are obtained using PnP within RANSAC. These are different estimation problems. For example, Eq. (8) requires 3D-to-3D correspondences, while PnP conventionally requires 3D-to-2D correspondences and camera intrinsics. The manuscript does not explain which solver actually produces the results in Table 3.

  3. The Bidirectional Ray Intersection construction contains unresolved ambiguities when the forward and backward selections overlap. Algorithm 1 extracts QkfwdQ_k^{\mathrm{fwd}} and QkbwdQ_k^{\mathrm{bwd}} from the same remaining point set, assigns them to two different layers, and only then removes their union. For example, if a ray contains only one remaining valid surface intersection, the forward-most and backward-most points are necessarily identical, so the same point could be assigned to both layers under the stated procedure. The paper does not specify how overlapping selections, odd numbers of intersections, missing intersections, or depth ties are handled.

  4. The ablation does not fully isolate the claimed benefit of geometry-aware 3D aggregation from additional model capacity and pretraining. The geometry-aware branch introduces a PTv3-based point transformer initialized with Sonata self-supervised weights, whereas the BRI-only model does not contain this additional pretrained geometric branch. Table 4 improves from CD 0.0206 / [email protected] 0.685 with BRI to 0.0200 / 0.700 after adding geometry-aware aggregation. However, this comparison does not isolate whether the gain comes specifically from explicit 3D positional aggregation, additional network capacity, or the Sonata initialization.

  5. Some headline improvements are not based on input-matched comparisons. The paper highlights a 28% reduction in CD and a 15% improvement in [email protected] over LaRI on GSO, with CD improving from 0.025 to 0.018. However, LaRI is a single-view layered reconstruction method, whereas LaRI3R uses two input views. Therefore, this improvement cannot be attributed solely to the proposed bidirectional representation and geometry-aware aggregation, since the observation budget also changes.

  6. The scale-invariant regression objective is under-specified for the proposed multi-layer representation. The prediction is defined as Pi∈RH×W×N×3P_i \in \mathbb{R}^{H \times W \times N \times 3}, whereas Eq. (5) writes the regression loss using only P(u,v)P(u,v) and sums over (u,v)∈D(u,v) \in D, without specifying how the layer index is handled. In addition, the scale normalizers zz and zgtz_{\mathrm{gt}} are only described as being computed from the valid point set without an explicit definition. For example, it is unclear whether normalization is performed jointly across all layers or independently for each layer, which would result in different supervision.

4 comments
AuthorsOpenReview ↗

Response to Reviewer gRkN

We sincerely thank the reviewer for the constructive feedback. We address each concern below and will update the final paper accordingly.

Q1. Definition of the pose AUC

Thank you very much for pointing this out. This is a writing typo instead of a technical misalignment. We strictly follow the evaluation codebase of Reloc3r [1], where the AUC in Tab. 3 is computed on the maximum, rather than the minimum, of the rotation and translation-direction errors of each pair. The experiments align with the previous methods, and the gains are evaluated under the same previous protocol. We will correct the typos in the revision.

Q2. Pose estimation protocol

Thank you for pointing this out. Our original description was not sufficiently clear. We intended to illustrate that the relative pose can be solved by the 3D-3D alignment in Eq. (8), but since this alignment is sensitive to noise and outliers, PnP within RANSAC is a better alternative, so we use it in the evaluation.

Following DUSt3R, all results in Tab. 3 are obtained with PnP within RANSAC. Specifically, for each view, the focal length is estimated from its predicted first-layer point map with the Weiszfeld solver as in DUSt3R, and the principal point is set to the image center. The 2D-3D correspondences are formed by the pixels of one view and its point map in the frame of the other view. Since the occluded points lie along the same camera rays, they share these pixels and serve as additional correspondences.

We will revise the description around Eq. (8) and add the PnP formulation in the revision to avoid misunderstanding.

Q3. Edge cases in Algorithm 1

In each iteration of Algorithm 1, the forward-most and backward-most points on each ray are assigned to the two corresponding layers and then removed from the remaining set. When only one point remains, as on a ray with an odd number of intersections, this point is duplicated into both layers. When no points remain, the corresponding layers are marked as invalid in a validity mask and are not supervised in the loss. We will clarify this in the revision.

Q4. Multi-layer regression objective

Our loss follows DUSt3R and extends it to multiple layers. The NN layers in the point maps correspond exactly to the 2K2K layers produced by BRI, with K=2K = 2 and N=4N = 4 in all experiments. In Eq. (5), the summation runs over all valid pixels and layers (u,v,n)∈D(u, v, n) \in \mathcal{D}. As in DUSt3R, zz and zgtz_{gt} are the average distances of the valid points to the origin, computed jointly over all layers of both views for the prediction and the ground truth, respectively. We will make Eq. (5) explicit in the revision.

Q5. Isolating the effect of geometry-aware aggregation

To separate the contribution of 3D aggregation from pretraining and model capacity, we add the following experiments:

  1. We conduct an additional experiment by removing the Sonata initialization of the point transformer. As shown in Tab. R8 (d), the Sonata initialization brings no performance gain compared with (c), possibly due to the gap between its pretraining data and our training data.

  2. We keep the same PTv3 architecture but remove cross-view aggregation in 3D (e). This variant leads to a performance drop compared to our proposed approach (c), indicating that the gain comes from cross-view aggregation in 3D rather than additional capacity.

Tab. R8. Ablation on geometry-aware aggregation on GSO.

MethodCD ↓[email protected] ↑[email protected] ↑
(a) Baseline0.02260.9090.655
(b) Baseline + BRI0.02060.9290.685
(c) Baseline + BRI + Geo. Agg. (Ours)0.02000.9360.700
(d) Ours w/o Sonata initialization0.02000.9350.700
(e) Ours with separate PTv30.02060.9300.688

Q6. Input-matched comparison

LaRI3R addresses a new setting between single-view occluded-geometry reconstruction (e.g., LaRI) and two-view visible-surface reconstruction (e.g., DUSt3R). The improvements are therefore meaningful relative to both settings. With the same two-view input, our method still outperforms baseline (a) in Tab. R8, which uses LaRI’s unidirectional layered representation for occluded scene reconstruction, reducing CD by 11.5% and improving [email protected] by 6.9%.

In the revision, we will clarify the comparison with LaRI to make clear that the performance gains come from two sources, including the additional input view and our proposed design.

[1] Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization. https://github.com/ffrivera0/reloc3r

Some substantive uncertainties remain regarding the pose-estimation protocol and the scope of the downstream claims. In particular, the paper expresses both predicted point maps in the first camera’s coordinate frame, while the rebuttal states that each view’s focal length is estimated from its first-layer point map. It remains unclear how the second view’s point map is obtained in its own camera coordinate frame for this estimation. Moreover, the pose-estimation gains from including occluded geometry are substantial on GSO but much smaller on Omni3D, leaving limited evidence for a consistently strong downstream benefit across datasets. Nevertheless, the explanations of the BRI edge cases and multi-layer normalization, the additional controls for Sonata initialization and cross-view aggregation, and the input-matched comparison have alleviated several of my original concerns. Taking these improvements and the remaining uncertainties into account, I will increase my rating to 2 . I hope the authors will incorporate the corrected metric definition, clarified training and evaluation procedures, and additional experiments into the revised manuscript, as the PDF currently available to me still retains the original, unrevised descriptions.

We thank the reviewer for the follow-up comments and for raising the score.

Q1 Obtaining the focal length for the second camera view.

We follow the practice of DUSt3R [1] by constructing symmetric input pairs. Specifically, the focal length of view 1 (I0)(I_{0}) is computed by inputting the view pair in the order (I0,I1)(I_{0},I_{1}), while the focal length of view 2 (I1)(I_{1}) is computed by inputting the view pair (I1,I0)(I_{1},I_{0}), such that each view's focal length is correctly computed under its own coordinate system. We will add further details and refer readers to the original paper.

[1] DUSt3R: Geometric 3D Vision Made Easy. CVPR 2024.

Q2. Downstream benefit on Omni3D.

  1. Omni3D combines images and objects collected from several different source datasets, most of which fall outside our training distribution, so its scenes are more diverse and complex than GSO. This is a challenge for layered reconstruction methods such as LaRI and LaRI3R, which are trained on much less data than visible-surface models such as DUSt3R. As a relatively new task, occluded-geometry reconstruction still lacks large-scale training data. We plan to scale up the training data in future work.

  2. Occluded geometry still helps under this generalization challenge. Even on such out-of-domain data, adding the occluded layers does not hurt pose estimation and still improves it (AUC@5 0.209 → 0.210, AUC@20 0.688 → 0.694). LaRI3R also remains the best among all feedforward methods on Omni3D (AUC@5 0.210 vs. 0.141 for VGGT and 0.123 for Reloc3r).

On the revised manuscript.

We sincerely thank the reviewer for the reminder. We had planned to revise the manuscript after collecting all reviewers' feedback, so that every suggestion could be addressed together. Unfortunately, given the time limit, we may not be able to upload the revised version in time.

However, we promise that all of the points raised by the reviewer will be corrected or added in the revised manuscript, including the corrected metric definition, the clarified training and evaluation procedures, and the additional experiments. The same applies to all points raised by the other reviewers. We sincerely appreciate the reviewer's constructive suggestions and will make sure that each of them is fully reflected in the final version.

Reviewer_Uqb8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method for two-view point cloud reconstruction, including for occluded surfaces along the input rays. Analogously to DuSt3R, the approach directly predicts point maps in parallel streams, except that here the predictions are layered point maps and the supervision is a bidirectional front-back data structure. A point transformer than refines the predictions, allowing for geometric information to be shared even if it is not between visible surface points. This is a significant problem that lies on the boundary between regression vs generation and single-view vs multi-view geometry. The contributions are (1) a two view layered point map predictor; (2) a bidirectional layered representation that has favorable surface fragmentation properties; and (3) demonstrating utility on a downstream task (two-view camera pose estimation). The approach is evaluated on the GSO, Omni3D, 3D-FRONT and SCRREAM datasets, demonstrating compelling results.

Critical strengths
  1. The motivation and main arguments made by the paper are sound. I am in whole-hearted agreement with the arguments about the trade-off between exploration vs reconstruction completeness and accuracy, and the importance of thinking carefully about the representation to reduce fragmentation and thereby make the network's life easier.

  2. The bidirectional representation. While there are some empirical issues with the N>2 layer version of this representation (see below), the idea itself is elegant, theoretically sound, and a great improvement over the most immediate related work (LaRI).

  3. Approach. The approach (directly predict layered point maps with bidirectional depth supervision, analogous to Dust3R, then refine with a point transformer) is appropriate for the task, is simple, and performant.

  4. Downstream application. The two-view pose experiments provide an excellent additional motivation for the proposed task, and lead to surprisingly strong results. I found this quite compelling.

  5. Clarity. The paper is well-written and well-presented, making it very easy to follow the idea, approach and arguments. The diagrams (Figs 2 and 3) are particularly helpful at conveying the main ideas.

Critical weaknesses
  1. Representation contribution. The bidirectional ray intersection representation is the core contribution of the method. However, Table 7 shows that the N=2 layers version of this representation (first and final surface) is practically indistinguishable in performance from N>2. The N=2 version is well-studied in the literature [A, B, C, D], which undercuts the central contribution. For example, [A, B] predicts the depth of the first and last surface per ray for human-shaped objects, [C] does so for general objects, and [D] does the same for scenes, rather than objects. Shin et al. 2019 (cited in the manuscript) also predict, among other things, the first and last depths. Moreover, the rationale for predicting first and last surfaces in these works is largely the same as the motivation provided in this manuscript: so that neighbouring pixels look at coherent surfaces per-layer. The remaining difference between the proposed representation and those used in the literature is therefore the N>2 case. I find this case elegantly handled, conceptually, by the proposed method. However, the performance boost attributable to this is -0.0004 CD and +0.005 [email protected] (Table 7). This is unlikely to be statistically significant.

  2. Claims.

    1. Several claims are made about the new representation, including "better preserves object contours, reduces surface splitting, and improves consistency across layered point maps" [L151 and similar on L456]. While this certainly seems plausible, there does not seem to be supporting evidence presented for these properties. One idea for supporting this claim would be to propose a coherence measure that, e.g., counts the number of discontinuities between adjacent pixels in a single layer.
    2. The claim "acquiring dense observations requires...substantial computation" [L062] is, I think, not intentional. If it is a mistake, remove this clause; if not, provide some evidence for this.
  3. Missing task/methodological information. The approach is somewhat under-specified.

    1. Task definition. The scene-level task is not well-defined. What, specifically/technically, is meant by the final surface in a scene? At face-value, this would be the final surface before outer-space. I suspect you mean to define it with respect to the dataset instead. In which case, how do the datasets (implicitly) define it? Is it consistent? If not, your network is being asked to make conflicting predictions.
    2. In algorithm 1 (data construction), what happens if there are no points remaining in Q_k - do the remaining layers get assigned a null point? If so, how is this represented? Does it effect the evaluation? What happens if Q_k^fwd and Q_k^bwd are the same point - does the representation duplicate the point?
    3. Similarly, for the prediction itself, what is predicted if there are no internal points/layers? Is there supervision for this? If not, are the depth predictions inside the object arbitrary?
    4. K (number of representation construction iterations) and N (number of layers) are not specified in the paper. It's alluded to in the appendix that N=4 is used, but not stated explicitly there either.
  4. Missing evaluation information.

    1. Chamfer distance: is this bidirectional or uni-directional (from which direction)? This isn't specified, but it is critical for interpreting the results. For example, if it is uni-directional from the ground-truth point cloud to the prediction, then it can be optimised by predicting densely along the ray.
    2. In general, the evaluation isn't entirely clear. It is unclear what "final" surface means in the context of scenes where, e.g., the wall has a back surface, and there might even be a room behind that (see task definition weakness above), and so it is rather unclear if the network has failed or not. The paper states that the ground-truth is "pre-processed" [L320]: please summarise how, since it directly impacts whether the evaluation is meaningful.
    3. How is "unseen" defined? Presumably it is unseen by both views, but this needs to be made explicit.
  5. Evaluation.

    1. Compare one or two of the front-back depth methods listed below [A,B,C,D], even if just on a human subset, since these approaches appear to be more direct competitors.
    2. It is somewhat unexpected to see evaluation on VGGT and not VGGT-Omega and MoGe instead of MoGe-v2 (or v3).
  6. Design choices. Given that a point transformer is introduced, it seems somewhat counter-productive to revert to two layered point maps (eq 4), rather than predicting an unstructured point cloud directly. This would have the benefit, for example, of handling grazing edges better (listed as a limitation), since the point transformer could better allocate the 3D points across the surface.

  7. Minor.

    1. Typo: "extracted points are assigned...before removal from the remaining point set" [L220] - as written, this implies that the points are removed twice, impossibly.
    2. Trivia. There is no Monsieur Chamfer, so there is no need for the capital in "Chamfer distance".

[A] Gabeur et al., "Moulding Humans", ICCV 2019.

[B] Smith et al., "FACSIMILE", ICCV 2019.

[C] Xian et al., "Any-Shot GIN", 3DV 2022.

[D] Bahrami et al., "Room Envelopes", AJCAI 2025.

Questions for authors
  1. How does the proposed bidirectional representation demonstrate improvements over existing first-last representations? I'd recommend an evaluation where you remove the first and last surfaces from the ground-truth and predictions, and evaluate on the remaining points. I expect this would demonstrate a clear contribution for your representation, if it is working as intended.

  2. Can you provide some supporting evidence for the claim that the representation "better preserves object contours, reduces surface splitting, and improves consistency across layered point maps"? Such as via a coherence measure.

  3. What is the scene-level task definition?

  4. What is the model and algorithm's behavior when there are no internal points?

  5. Is the chamfer distance bidirectional or uni-directional (from which direction)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Uqb8 (Part 1).

We sincerely thank the reviewer for the deep, detailed, and constructive feedback. We address each question below and will update the results in the final paper.

Q1. Contribution of the bidirectional representation beyond first and last surfaces

The improvement in interior areas is not fully reflected in Table 7, as they account for only a small portion (15% in GSO) of all points compared to the first and last layers.

Following the suggestions of the reviewer, we additionally evaluate the model by removing the first and last layers. Since the baseline uses the unidirectional ray intersection representation, its layers cannot be directly compared with those of BRI. We therefore reorganize the baseline predictions into BRI layers following Algorithm 1 for evaluation.

As shown in Tab. R4, despite the relatively inferior performance of interior areas reasonably led by the problem setting, the improvement ratio in chamfer distance of the interior areas is higher than that of the first and last layers. This demonstrates that the proposed approach works for both outermost and interior areas, and we will update the experiments in the revised paper.

We truly appreciate the reviewer's comment on related papers, and we will add a detailed discussion of [A, B, C, D] in the revision.

Tab. R4. Reconstruction results on different surface regions of GSO.

MethodFirst & Last ↓Interior ↓
Baseline (Unidirectional)0.02340.0660
Ours (Bidirectional)0.02040.0537

Q2. Evidence for improved layer coherence

We thank the reviewer for this constructive suggestion. We measure the coherence of each layer directly on the ground-truth layered point maps. Specifically, for each valid pixel, we compute the average 3D distance to its valid 4-neighbors within the same layer. A larger value indicates more discontinuities, i.e., more surface splitting within the layer. As shown in Tab. R5, for the occluded layers, BRI consistently yields smaller distances on both datasets, indicating that it reduces surface splitting.

Tab. R5. Average 4-neighbor distance within each layer of the ground-truth layered point maps (lower is better).

DatasetMethodFirstInteriorLastAll
GSOUnidirectional0.01150.01930.03760.0161
GSOBidirectional0.01150.01710.01360.0132
3D-FRONTUnidirectional0.01160.02330.03250.0196
3D-FRONTBidirectional0.01160.01690.01250.0136

Q3. Statement in L062

Thank you for pointing this out. We will remove the original clause that is misleading.

Q4. Definition of the scene-level task

We aim to confine scene-level prediction within a single room, so that the reconstruction coverage does not go beyond the semantic cues of an image pair. We achieve this by data preprocessing across datasets:

  1. For 3D-FRONT, we crop the scene geometry using the room bounding box provided by the dataset, so that geometry outside the current room is excluded.

  2. For ScanNet++, since some scans contain multiple rooms, we follow LaRI's practice and truncate the ray intersections at 1.2 times the maximum depth of the visible surfaces.

We will clarify this in the revision.

Q5. Clarification of Algorithm 1

When QkfwdQ_k^{\mathrm{fwd}} and QkbwdQ_k^{\mathrm{bwd}} coincide, for example when only one intersection remains on a ray, the point is duplicated into both layers. When no points remain in QkQ_k, the remaining layers are marked as invalid and are not supervised in the loss. Following LaRI, only the valid layers are evaluated. We use K=2K = 2 and N=4N = 4 for both object-level and scene-level models. We will clarify these details in the revision.

Q6. Evaluation details

  1. As described in Q4, the scene-level evaluation is also defined within a single room.
  2. Following LaRI, we use the bidirectional chamfer distance, which considers both the prediction-to-ground-truth and ground-truth-to-prediction directions.
  3. In addition, in Tab. 2, the visible region refers to the first layer, and the unseen region refers to the subsequent layers.

We will clarify this in the revision.

AuthorsOpenReview ↗

Response to Reviewer Uqb8 (Part 2).

Q7. Comparison with [D]

As [A, B, C] have no public code, we compare with [D] on 3D-FRONT. Since [D] predicts only the visible surface and the room envelope, corresponding to the first and last surfaces of BRI, we conduct the comparison on these two surfaces. As shown in Tab. R6, our method outperforms [D] on both surfaces.

Tab. R6. Comparison with [D].

MethodFirst CD ↓First [email protected] ↑Last CD ↓Last [email protected] ↑
[D]0.0400.7860.1780.283
Ours0.0350.8850.0730.744

Q8. Newer versions of baselines

Thank you for the suggestion. We additionally evaluate the newer VGGT-Ω and MoGe-3 on 3D-FRONT and SCRREAM, as shown in Tab. R7, and will include them in the revised paper. Benefiting from large-scale and diverse training data, these methods achieve better visible-surface accuracy, while our method achieves the lowest overall CD on both datasets by recovering occluded geometry. In the future, we plan to train LaRI3R on larger and more diverse datasets to further improve its generalization.

Tab. R7. Results of newer baselines on 3D-FRONT and SCRREAM.

3D-FRONTSCRREAM
MethodVisible CD ↓Overall CD ↓Visible CD ↓Overall CD ↓
MoGe-30.0300.0760.0220.054
VGGT-Ω0.0270.0740.0180.052
Ours0.0350.0410.0430.049

Q9. Layered point maps instead over an unstructured point cloud

Thank you for this thoughtful suggestion. In this paper, we aim to design pixel-aligned layered point maps following previous feedforward approaches. This keeps an explicit correspondence between each 3D point and its image pixel, allowing a relatively smoother adaptation to image-based downstream tasks such as relative pose estimation.

We also agree that an unstructured point cloud could better allocate points on grazing surfaces. Taking our layered point maps as input and combining them with generative models to directly output unstructured point clouds is a promising direction for future work.

Q10. Behavior when there are no internal surfaces

For scene-level tasks, the multiple objects in a scene and their mutual occlusion naturally produce multiple surfaces along each ray, so empty interior layers rarely occur.

For object-level tasks, when an object has no internal structures, we observe that the model tends to predict an inner-shell-like surface between the first and last surfaces. This surface resembles the inner wall of a hollow shell and roughly reflects the thickness of the object surface. We believe this prior is transferred from training objects that contain internal structures. In addition, the model also produces some noisy points inside the object.

Q11. Minor issues

Thank you for the careful reading. We will follow the reviewer's comments to revise the following issues:

  • revise L220 to clarify that the extracted points are removed from the remaining point set only once
  • use the lowercase “chamfer distance” throughout the paper.

Moreover, we will carefully examine the remaining text to correct any potential typos.

Reviewer_MDUAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes, LaRI3R, a two-view model for 3D scene reconstruction under severe occlusions. The key innovations are the bidirectional layered representation, predicting geometry in both forward and backward ray orders, helping for occluded regions and geometry-aware feature aggregation using 3D positional cues. The work is a continuation / improvement of LaRI (ICML 2026), extending it to two-views while incorporating bidirectional layered representations and geometry-aware feature aggregation. The downstream results are strong, consistently outperforming LaRI along other methods.

Critical strengths
  • The paper is well-written and clearly motivates its core design
  • The method is conceptually quite nice and achieves strong performance on standard benchmarks
  • Both scene-level and object-level performance seems solid, even showing usefulness in relative pose estimation
Critical weaknesses
  • Seems not obvious to me why stopping at 2 views in particular is the right choice.
  • Performance does not seem "revolutionary", it is a nice improvement, but not a huge step-change.
  • The Geometry-aware Aggregation does not seem to give much improvement in Tab 4.
  • Novelty is fine I think, it is not some completely new insight or direction but instead taking reasonable premises and combining them into a nice paper.

Take my criticism with a grain of salt, I am not very knowledgeable on this paper's exact related work and general find the paper a quite nice read.

Questions for authors

I will prefice by saying I am not an expert in this exact topic. Although, I am decently well-versed in feed-forward reconstruction methods, like VGGT.

  • Why are these multi-level pointmaps needed? Why can't one just use HxWx3 outputs?
  • How does it compare to DUSt3R on reconstructing pairs without occlusion, like maybe CO3D or something?
  • What happens if there is no covisibility at all between the pair? I guess DUSt3R worked fine for some of these
3 comments
AuthorsOpenReview ↗

Response to Reviewer MDUA (Part 1).

We sincerely thank the reviewer for the insightful comments and constructive questions. We address each question below and will revise the final paper accordingly.

Q1. Why two views?

LaRI3R provides a reference in the cross-view design for multi-view settings, and can be easily adapted to multi-view inputs.

  1. We follow the trajectory along which 3D reconstruction has evolved, from monocular to two-view models (e.g., DUSt3R) and then to multi-view settings, so that effective designs can be developed in simpler settings and reused in more complex ones. As a step along this path, LaRI3R addresses the cross-view occlusion reconstruction problem left open by LaRI, offering a viable design choice for future multi-view work.

  2. Similar to DUSt3R, LaRI3R can also be extended to multiple input views through multi-view global alignment. We evaluate this on 3D-FRONT with N=3,5,10N = 3, 5, 10 views. As shown in Tab. R1, LaRI3R consistently outperforms DUSt3R under all settings.

Moreover, we fully agree that extending LaRI3R to natively process multiple views in a single forward pass is promising, and we will explore this direction in future work.

Tab. R1. Multi-view reconstruction on 3D-FRONT with global alignment.

MethodNNCD ↓[email protected] ↑
DUSt3R30.1700.648
LaRI3R (Ours)30.1250.843
DUSt3R50.2170.658
LaRI3R (Ours)50.1570.855
DUSt3R100.2210.704
LaRI3R (Ours)100.1880.844

Q2. Highlight the performance gain and novelty.

LaRI3R does not claim revolutionary novelty or dramatic performance gains. It is arguably among the first approaches to address occluded geometry reconstruction across views in feedforward approaches, and it delivers consistent gains across multiple applications:

  1. Cross-view effectiveness. The proposed bidirectional ray intersections provide a more coherent layered representation that reduces surface splitting under occlusion. This makes the layered geometry easier to learn and leads to more accurate occluded-surface reconstruction for feedforward approaches. Building on this, the geometry-aware aggregation fuses cross-view cues in 3D space, allowing information aggregation that is agnostic to the appearance variance. Compared to previous feedforward 3D reconstruction approaches, the proposed method serves different goals (visible vs. occluded), and therefore leads to different design choices.

  2. Application scenarios. LaRI3R achieves the best occluded geometry reconstruction on both object- and scene-level benchmarks. It also shows promise in downstream tasks such as relative pose estimation, revealing that occluded (stereo) geometry reconstruction contributes to a broader scope of applications with potentially promising performance gains.

We also hope that combining these designs and extending them into multi-view using a scaled dataset can lead to a further performance boost. And we will leave it to future work.

Q3. Why are multi-layer point maps needed?

Compared with visible-layer-only point maps, multi-layer point maps aim to achieve similar scene coverage from limited observations. This is useful in several scenarios.

  1. In online reconstruction, e.g., for autonomous driving, occlusion reasoning is important, but future observations are not yet available. Visible-layer-only point maps cannot represent occluded regions in this case, whereas multi-layer point maps allow the model to infer the geometry behind visible surfaces from the current observation.

  2. For images collected from the Internet, only one or a few views of a scene are available, and visible-layer-only point maps inevitably lead to incomplete reconstructions.

  3. Moreover, multi-layer point maps do not contradict current multi-view visible-layer-only methods (e.g., VGGT) but are a good complement to them. Extending the multi-view models with native capability of occlusion modeling from unobserved scenes attains both strong 3D priors and scene reasoning capability.

AuthorsOpenReview ↗

Response to Reviewer MDUA (Part 2).

Q4. Comparison with DUSt3R on CO3D

Thanks for the constructive comments. We additionally evaluate visible-surface reconstruction on the development set of CO3Dv2, where we select the top 20 sequences with the highest ground-truth quality scores and randomly sample 400 images from them to construct image pairs.

Tab. R2. Visible-surface reconstruction on CO3Dv2.

Method[email protected] ↑[email protected] ↑[email protected] ↑
DUSt3R0.8540.7250.502
LaRI3R (Ours)0.8880.7400.453

As shown in Tab. R2, LaRI3R outperforms DUSt3R at [email protected] and [email protected], while it falls behind at the strict threshold [email protected]. DUSt3R is trained on more diverse data, including CO3Dv2, which is in-domain for the evaluation set. Our method still performs competitively with DUSt3R without being trained on CO3Dv2. In the future, we plan to train LaRI3R on larger and more diverse datasets to further improve its generalization.

Q5. Pairs without covisibility

We construct GSO image pairs without covisibility and compare them with the standard test pairs used in Tab. 1. As shown in Tab. R3, both methods degrade in this setting. Without covisibility, the task becomes more challenging, as the relative placement of the two views can only be inferred from shape priors. Nevertheless, our method still substantially outperforms DUSt3R without covisibility. Including such challenging pairs in training could help the model learn stronger shape priors for this setting.

Tab. R3. Reconstruction results on GSO pairs with and without covisibility.

SettingMethodCD ↓[email protected] ↑[email protected] ↑
With covisibilityDUSt3R0.05900.6400.382
With covisibilityOurs0.02000.9360.700
Without covisibilityDUSt3R0.08480.5310.288
Without covisibilityOurs0.03040.8420.568
Reviewer_MDUAOpenReview ↗

Response

Dear authors,

Thank you for your detailed response and answer to my questions. I don't have any outstanding critical concerns. However, I recommend the authors to integrate the above discussion into the manuscript as to improve the paper.

I will keep my score of 3 (WA). I will not increase it to a 4 as I believe the novelty / impact is not sufficient.

As for my expertise here, I will re-iterate that I am mostly familiar with the feed-forward reconstruction lineage and thus find this work intriguing and a valuable contribution, but I cannot attest strongly to if these type of evaluations are commonplace for this lineage of work nor will i give a high certainty score to my rating.

Overall, I think the comparison to DUSt3R looks interesting and I appreciate the addition on non-overlapping views and more standard benchmarks (from my point of view, looking at the DUSt3R line of research).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their constructive feedback. We are encouraged that the reviewers find the problem setting important and well motivated (Uqb8, gRkN), the bidirectional representation elegant and meaningful (Uqb8, gRkN), the pose estimation results compelling (Uqb8, MDUA), the evaluation broad (gRkN), and the paper clear and easy to follow (MDUA, Uqb8).

In response to the comments, we have conducted the following additional experiments.

  1. Extending LaRI3R to the multi-view input setting via global alignment (Tab. R1, MDUA).
  2. Visible-surface comparison with DUSt3R on CO3Dv2 (Tab. R2, MDUA).
  3. Evaluation on image pairs without covisibility (Tab. R3, MDUA).
  4. Separate evaluation of the outermost and interior surfaces (Tab. R4, Uqb8).
  5. Layer coherence analysis on the ground-truth layered point maps (Tab. R5, Uqb8).
  6. Comparison with the front-back depth method Room Envelopes (Tab. R6, Uqb8).
  7. Comparison with the newer baselines VGGT-Ω and MoGe-3 (Tab. R7, Uqb8).
  8. Ablation on the Sonata initialization and cross-view aggregation in the point transformer (Tab. R8, gRkN).

We have also clarified the pose evaluation metric and protocol, the edge cases of Algorithm 1, the multi-layer regression objective, the definition of the scene-level task, and the input-matched comparison with LaRI. All these results and clarifications will be incorporated into the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.