acceptodds
Under review as a conference paper at ICLR 2027

HOME: Multi-Room Geometry Estimation via Top-Down Guidance

Abstract

Feed-forward 3D foundation models struggle in multi-room environments where cameras are segregated across rooms divided by opaque walls, breaking visual overlap and spatial continuity. We propose HOME, a method that adapts 3D foundation models to incorporate an unposed top-down RGB render alongside perspective captures for multi-room geometry and relative camera pose estimation. To support this, we construct TD-HOMES, a dataset suite spanning real-world scans, designer-crafted spaces, and agentic procedural environments. While providing global top-down context largely resolves catastrophic pose and geometry failures across disjoint rooms, naively sharing parameters induces representational interference that compromises pre-trained metric precision on standard overlapping scenes. To alleviate this issue, HOME introduces a view-type decoupling design that selectively decouples the projection weights and feed-forward networks within global cross-frame attention layers between top-down and perspective views. Experiments show that HOME reliably recovers coherent multi-room geometry across non-overlapping views while reducing performance drops on standard overlapping scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Extreme-View Geometry Estimation, 3D Foundation Model, Indoor Scene Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_gJZfOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses geometry and camera pose estimation for multi-room indoor scenes, where perspective views in different rooms share little or no overlap. The proposed HOME3 feeds an unposed top-down render of the whole floor into a pre-trained 3D foundation model (π3 or VGGT-Ω) as an extra view, and uses view-type-specific QKV projections and FFNs in the global attention layers to reduce interference between top-down and perspective tokens. The authors also build TD-HOMES, pairing perspective images with top-down renders from HM3D, Structured3D and Code-as-Room scenes. Experiments show clear gains over perspective-only baselines in the multi-room setting, with a smaller performance drop on standard benchmarks than naive joint fine-tuning.

Critical strengths
  1. Clear presentation and a simple, easy-to-follow method.
  2. Clear quantitative and qualitative improvements within the proposed setting.
Critical weaknesses

The setting is weakly motivated. Existing geometry foundation models fail in multi-room scenes because the rooms share almost no visual overlap, so their relative placement cannot be inferred from the perspective images. The paper addresses this by injecting a top-down view of the whole floor as an extra observation, which enables joint reconstruction of otherwise independent rooms. However, this scenario is rare in real applications, and top-down images of indoor scenes are generally hard to obtain. In fact, every top-down image in TD-HOMES is rendered from the complete 3D scene itself; having such an image presupposes that the floor has already been scanned or modelled, which largely removes the need for the task. The paper calls this "a considerably more practical setting", but provides no experiment with realistic top-down sources such as real floor plans or sketches. This weakness largely drives my recommendation: the main finding, that giving the network a global map of the scene helps it place cameras within that scene, is expected and of limited significance.

Questions for authors

All top-down images used in the paper are rendered from complete 3D scenes (reconstructed meshes, annotated layouts, or generated scenes). Could the authors clarify how such top-down images would be obtained for a large number of in-the-wild real-world scenes, and demonstrate the practical value of the method on a broad set of real scenes where the top-down input comes from a realistically available source rather than from the ground-truth scene itself?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We respectfully disagree with the view that our problem setup lacks practical relevance or benchmark validity. Both Reviewers 5BML and fuAD recognized the value of this setting and the design of TD-HOMES: Reviewer 5BML noted that the task is "challenging yet meaningful" and that the TD-HOMES dataset is "meaningful to the community under this line of research" due to its three complementary sources; similarly, Reviewer fuAD noted that disjoint multi-room reconstruction represents "a well-defined and consequential failure regime" requiring exocentric layout cues, and described TD-HOMES as "a useful dataset and reasonably controlled comparisons". Below, we further clarify why our paired top-down formulation reflects realistic deployment scenarios where end-users lack access to the underlying 3D model, and show that HOME³ directly extends to non-rendered 2D floor plans:

  • Practical motivation. While our training set renders top-down views from 3D scenes for supervision, having a 2D top-down image at inference time does not mean the end-user has access to the 3D model. In many real-world scenarios, such as real-estate platforms, shopping malls, and museums, venue owners publish static 2D top-down dollhouse renders or overview maps for human viewing, while withholding raw 3D meshes due to privacy, intellectual property, and bandwidth constraints. When a user or agent captures a few sparse, non-overlapping perspective photos on-site, the 2D overview alone lacks 3D structure and camera poses. Recovering globally consistent 3D geometry and relative camera poses from these sparse inputs is needed for downstream tasks such as multi-room 3D scene reconstruction, snapshot-based camera localization, and cross-room spatial layout estimation.

  • Extending to floorplans. To further evaluate our framework on realistically available, non-rendered top-down sources, we extend our experiments by incorporating 2D architectural floorplans into the training of HOME³$ (denoted as HOME³† in Table R1). The floorplans are either derived from existing annotations or synthesized through an agentic procedural generation pipeline. When evaluated strictly with 2D floor plans at inference time, HOME³† outperforms the perspective-only baseline across most metrics under both in-domain and unseen zero-shot settings. For instance, improving RRA@30 from 69.75 to 84.26, RTA@30 from 68.09 to 83.55, and reducing point map accuracy error from 0.497 to 0.285 on TD-HOMES; although slight performance drop occurs on RRA@30 in TD-HM3D zero-shot. Table R1 has been added as Table 4 in the updated manuscript.

    Table R1. Floorplan-guided Camera Pose and Point Map Estimation.

    DatasetMethod / guideRRA@30 ↑RTA@30 ↑AUC@30 ↑Acc. ↓Comp. ↓N.C. ↑
    TD-HOMESπ³ (Perspective-FT; no guide)69.7568.0944.650.4970.6610.749
    HOME³† — RGB render89.5788.8072.820.2110.2370.905
    HOME³† — abstract floorplan84.2683.5565.800.2850.3620.865
    Zero-shotπ³ (Perspective-FT; no guide)97.3284.6574.170.2360.3490.852
    HOME³† — RGB render98.6089.6880.510.1510.1600.933
    HOME³† — abstract floorplan96.5787.1375.920.1950.2230.893
  • Clarification. When referring to "a considerably more practical setting" in Section 1, we specifically contrast treating the global prior as an unposed, uncalibrated 2D visual guide against requiring known camera extrinsics and exact metric scale alignment relative to the perspective observations. In real-world deployments, external top-down maps or floor plans rarely provide calibrated camera poses or metric scaling, so avoiding this assumption makes our formulation significantly easier to apply than pose-aligned map localization methods. We have revised the phrasing in Section 1 (Line48-49) to make this distinction explicit and avoid any ambiguity.

AuthorsOpenReview ↗

Official Comment by Authors

We would like to briefly follow up on our response above, particularly the additional experiments using 2D floor plans as guidance (Table R1, now Table 4 in the revised manuscript) and our clarification of the intended deployment scenarios.

When you have a chance, we would appreciate your thoughts on whether these additions help address your concerns about the practical relevance of our setting. Please also let us know if any concerns remain or if there are other points you would like us to clarify. We would be happy to provide further details. Thank you for your consideration.

After reading the response of authors, my concerns are resolved, so I raised my score.

Reviewer_5BMLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to tackle the multi-view, multi-floor indoor scene reconstruction problem under a sparse input setting. Although remarkable progress has been made by feed-forward 3D foundation models, 3D recovery quality under sparse views & top-down views and scalable environment still remains challenging. The paper first constructs a large-scale benchmark dataset named TD-HOMES from 3 types of sources: (1) read-world scans (2) synthetic design-oriented crafting pipelines (3) an agentic synthesis pipeline. Since most feed-forward foundation models are pretrained on perspective data only, authors find it hard to generalize well onto the top-down & sparse view inputs on both reconstruction and pose recovery. Motivated by the distinctive representation from the two types of views, authors propose to decoupled, separate attention weights (Q,K,V) matrixes to process the tokens from the two types of views on global cross attention. Extensive experiments are performed on TD-HOMES as well as well-established datasets. Authors also verified the design is agnostic to the selection of the 3D foundation model. The method shows notable improvements on both point map reconstruction and pose estimation.

Critical strengths
  1. The problem that the paper aims to tackle is challenging yet meaningful. Sparse view reconstruction associated with a top-down view could facilitate efficient reconstruction on real-life application, such as CAD design, floorplan reconstruction, etc. Most of the previous methods focus on dense and perspective views.
  2. The motivation is clear and strightforward and the method is simple but effective. It is flexible to be plug-and-play for any 3D foundation models without adding new modules or changing the foundational design.
  3. The curated dataset is meaningful to the community under this line of research. The three sources of data can be complementary with each other to ensure the soundness for benchmarking new methods.
  4. Experimental studies are comprehensive and well designed. It is important to see the performance does not have significant degrade on the perspective setups.
Critical weaknesses
  1. The effectiveness compared with a competing baseline: The improvement of the proposed decoupled pipeline is not significant compared with Joint-FT on both point map estimation and camera pose recovery. Given that the new design still introduces an amount of new parameters on both training and inference, I suspect the necessity on revising the design w.r.t performing joint FT, unless there are some important scenarios that the joint-FT always fail to generalize. In other words, if we train both pipelines in a more scaled-up benchmark, can we further mitigate the gap between the two pipelines?

  2. The authors did not mention the dataset will be released to the public, which raise some potential concerns on the value of the research for community, as well as the reproducibility.

  3. I would like to see the transferability of the method on some wild data such as outdoor datasets. However, given the challenge to sample the top-down view of an outdoor environment, it is understandable to skip this study. If some existing datasets can satisfy the same setting, I would encourage the authors to extend the study.

Questions for authors

(Minor): There is some typesetting issue for table 23 on the table size. Please double check before the camera ready version.

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for recognizing the value of our work. Below, we address the questions point by point. (W1 briefly denotes Weakness 1, and Q1 briefly denotes Question 1.)

  • (W1) Effectiveness of the decoupled model compared with Joint-FT

    To investigate whether additional training data could eliminate the performance gap, we augmented our training set with 824 real-world multi-room scenes from RealSee3D [1]. For each scene, we first unprojected the panoramas using the provided camera matrices and depth maps to obtain a scene-level point cloud. We then rendered a top-down orthographic view and generated 100 perspective images with corresponding depth maps. Finally, we trained both HOME3 (Joint-FT) and HOME3 (Decouple) on the expanded dataset, with the same training steps, and evaluated them on the TD-HOMES benchmark.

    Table R1. Camera Pose and Point Map Estimation with Scaled-Up Training Data.

    MethodTD-HOMESTD-HOMES (zero-shot)
    RRA@30↑RTA@30↑AUC@30↑Acc.↓Comp.↓N.C.↑RRA@30↑RTA@30AUC@30↑Acc.Comp.↓N.C.↑
    HOME³ (Joint-FT)89.0188.0871.090.2230.2630.89798.8790.0082.210.1420.1570.938
    HOME³ (Decouple)89.4688.6572.960.2070.2440.90199.1090.0084.900.1340.1460.954

    As shown in Table R1, HOME³ (Decouple) outperforms Joint-FT on all reported metrics across both benchmarks, except zero-shot RTA@30, where the two methods tie. We agree that further scaling up paired multi-room and top-down datasets is a valuable direction, and we will build on this study to explore larger-scale data construction and further analyze its scaling behavior in future work. We have updated Table R1 in Table 5 (Section 6).

  • (W2) Dataset release plan

    We sincerely thank the reviewer for pointing this out. Upon acceptance, we will publicly release our curated dataset TD-HOMES, along with the source code to support reproducibility and future research.

  • (W3) Transferability of the model on in-the-wild data

    We agree that extending our approach to in-the-wild outdoor scenarios is valuable for broadening its applications and assessing and improving cross-domain generalization. However, constructing suitable datasets is challenging, and outdoor ground-level observations differ substantially in distribution from indoor views spanning multiple rooms. Addressing these challenges extends beyond our current focus on indoor reconstruction, but we hope to explore this direction in future work (Section 7), as we have stated in Line 530.

  • (Q1) Typesetting issue for table 23 on the table size

    We appreciate the reviewer’s careful reading and for bringing the sizing issue in Table 23 (which corresponds to Table 29 in the updated paper) to our attention. We acknowledge that this formatting detail was overlooked during manuscript preparation. We have corrected the table’s size and layout in the updated manuscript to ensure a clear and consistent presentation.

[1] Li, L., Wu, Y., Li, X., Wang, L., Rao, T., Zhou, J., Pan, C., & Hui, X. (2026). Realsee3D: A Large-Scale Multi-View RGB-D Dataset of Indoor Scenes (Version v1.1.0) [Computer software]. Zenodo.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Official Comment by Authors

We appreciate the reviewer’s positive assessment of our work and the detailed comments on the manuscript. Below, we address each of your questions and suggestions in turn. (W1 briefly denotes Weakness 1, and Q1 briefly denotes Question 1.)

  • (W4, W1; Q1) Sensitivity to imperfect guidance and the extension to independently sourced guidance.
    • (1) Sensitivity to imperfect top-down guidance. (W4; Q1)

      HOME3 already possesses a degree of robustness to imperfect top-down guidance because we apply photometric and image-quality augmentations, specifically color jitter, JPEG compression, and downsampling–upsampling degradation to top-down renders during training (while geometric augmentations such as random-scale cropping and in-plane rotation are not applied to the top-down view). We additionally evaluate HOME3 under four kinds of perturbations applied to the top-down guide: Gaussian noise, removal of 25% of the guide canvas, spatial rescaling by factors within [0.7, 1.3], and arbitrary in-plane rotation within [-180°, 180°].

      Table R1a. Camera pose estimation on TD-HOMES under perturbed top-down guidance.

      Top-down input / methodRRA@30 ↑RTA@30 ↑AUC@30 ↑
      π³ (Perspective-FT; no guide)69.7568.0944.65
      HOME³ — clean render88.7087.9671.46
      Gaussian noise, σ = 0.0587.1886.6169.62
      Gaussian noise, σ = 0.1085.9885.7867.62
      Missing 25% of canvas79.1178.0757.28
      Rotation, [−180°, 180°]79.0675.3850.57
      Scale, [0.7, 1.3]87.9687.1370.47

      Table R1b. Camera pose estimation on TD-HOMES (zero-shot) under perturbed top-down guidance.

      Top-down input / methodRRA@30 ↑RTA@30 ↑AUC@30 ↑
      π³ (Perspective-FT; no guide)97.3284.6574.17
      HOME³ — clean render99.0589.8383.28
      Gaussian noise, σ = 0.0598.9089.6582.29
      Gaussian noise, σ = 0.1098.5389.7981.80
      Missing 25% of canvas97.7588.1080.03
      Rotation, [−180°, 180°]98.7689.6182.08
      Scale, [0.7, 1.3]99.1089.8183.34

      Table R2a. Point-map reconstruction on TD-HOMES under perturbed top-down guidance. All metrics report medians.

      Top-down input / methodAcc. (Median) ↓Comp. (Median) ↓N.C. (Median) ↑
      π³ (Perspective-FT; no guide)0.4970.6610.749
      HOME³ — clean render0.2230.2520.897
      Gaussian noise, σ = 0.050.2310.2800.886
      Gaussian noise, σ = 0.100.2510.3030.876
      Missing 25% of canvas0.3790.5060.804
      Rotation, [−180°, 180°]0.4050.5320.793
      Scale, [0.7, 1.3]0.2290.2710.894

      Table R2b. Point-map reconstruction on TD-HOMES (zero-shot) under perturbed top-down guidance. All metrics report medians.

      Top-down input / methodAcc. (Median) ↓Comp. (Median) ↓N.C. (Median) ↑
      π³ (Perspective-FT; no guide)0.2360.3490.852
      HOME³ — clean render0.1320.1480.948
      Gaussian noise, σ = 0.050.1400.1560.946
      Gaussian noise, σ = 0.100.1450.1700.940
      Missing 25% of canvas0.1820.1990.917
      Rotation, [−180°, 180°]0.1260.1350.947
      Scale, [0.7, 1.3]0.1320.1420.95

      As shown in Tables R1a–R2b, Gaussian noise and spatial rescaling cause only minor degradation across both benchmarks. In contrast, removing 25% of the guide or applying arbitrary rotations leads to more noticeable drops. This is expected, as masking out regions directly removes structural priors for the occluded areas, while arbitrary orientations were unseen during training.
      Crucially, under every tested perturbation, HOME3 still consistently outperforms the perspective-only baseline (π³ Perspective-FT) across all metrics on both benchmarks. This confirms that imperfect top-down guidance remains practically beneficial, while pointing to occlusion and rotation augmentation as straightforward future improvements.


Unfinished, please keep reading the comments.

AuthorsOpenReview ↗

Official Comment by Authors

  • (W4, W1; Q1) Sensitivity to imperfect guidance and the extension to independently sourced guidance.
    • (2) Extension to abstract floorplans. (W1; Q1)

      To examine a different guide representation, we additionally evaluate the HOME³† variant with decoupled design under the guidance of abstract floorplans and top-down RGB renders, respectively. We report this experiment separately because it uses a distinct model variant and tests the applicability of the framework to floorplan guidance.

      Table R3. Evaluation with abstract floorplans and RGB renders. Point-map metrics report medians.

      Evaluation splitMethod / guideRRA@30 ↑RTA@30 ↑AUC@30 ↑Acc. ↓Comp. ↓N.C. ↑
      TD-HOMESπ³ (Perspective-FT; no guide)69.7568.0944.650.4970.6610.749
      HOME³† — RGB render89.5788.8072.820.2110.2370.905
      HOME³† — abstract floorplan84.2683.5565.800.2850.3620.865
      Zero-shotπ³ (Perspective-FT; no guide)97.3284.6574.170.2360.3490.852
      HOME³† — RGB render98.6089.6880.510.1510.1600.933
      HOME³† — abstract floorplan96.5787.1375.920.1950.2230.893

      As shown in Table R3, compared with RGB guidance, floorplan guidance reduces AUC@30 from 72.82 to 65.80 in-domain and from 80.51 to 75.92 zero-shot. However, it still exceeds the perspective-only baseline in AUC@30 and all three reported point-map metrics on both splits. This supports the usefulness of abstract layout guidance, although the richer RGB renders yield better results in this comparison.


Unfinished, please keep reading the comments.

AuthorsOpenReview ↗

Official Comment by Authors

  • (W2; Q2) The benefit specifically attributable to view-type decoupling remains partly confounded.

    To clarify the contribution of view-type decoupling, we compare three variants: (a) TD-Adapt, which adds a TD-specific feature adapter after the shared image encoder while retaining shared downstream QKV/FFN transformations; (b) HOME³ (TD branch only), which retains the decoupled architecture but trains only TD-specific QKV/FFN parameters, freezing all original pretrained parameters; and (c) HOME³ (Joint-FT) (Separate-Encoder), which stems from HOME³ (Joint-FT) with a separate top-down encoder, where both encoders are initialized from pretrained DINO weights; the perspective encoder remains frozen, while the TD encoder is fine-tuned alongside the shared downstream QKV projections and FFNs.

    Tables R4 & R5 provide complementary evidence. TD-Adapt performs the worst on both evaluation sets, indicating that the tested input-level adaptation does not substitute effectively for specialization within the fusion layers. HOME³ (TD branch only) improves over HOME³ (Joint-FT) in zero-shot pose AUC and reconstruction quality, yet performs substantially worse in-domain, while HOME³ (Decouple) outperforms it on both sets. Under the same training protocol and with identical top-down inputs, HOME³ (Joint-FT) (Separate-Encoder) substantially degrades pose estimation performance relative to Joint-FT on both in-domain and zero-shot evaluation, despite introducing additional encoder capacity.

    Table R4. Architecture Comparison for Camera Pose Estimation.

    DatasetMethodRRA@30 ↑RTA@30 ↑AUC@30 ↑
    TD-HOMESπ³ (TD-Adapt)69.5568.2544.83
    HOME³ (TD branch only)78.7479.2456.72
    HOME³ (Joint-FT) (Separate-Encoder)67.2666.5742.36
    HOME³ (Joint-FT)87.5787.0669.14
    HOME³ (Decouple)88.7087.9671.46
    TD-HOMES (zero-shot)π³ (TD-Adapt)97.4184.3674.83
    HOME³ (TD branch only)97.5088.3281.98
    HOME³ (Joint-FT) (Separate-Encoder)97.2384.4975.85
    HOME³ (Joint-FT)98.2089.2580.06
    HOME³ (Decouple)99.0589.8383.28

    Table R5. Architecture Comparison for Point Map Estimation.

    DatasetMethodAcc. Mean ↓Acc. Med. ↓Comp. Mean ↓Comp. Med. ↓N.C. Mean ↑N.C. Med. ↑
    TD-HOMESπ³ (TD-Adapt)0.5650.4990.7910.6490.6630.748
    HOME³ (TD branch only)0.4020.3350.5210.4070.7210.838
    HOME³ (Joint-FT) (Separate-Encoder)0.5820.5150.8300.6880.6540.735
    HOME³ (Joint-FT)0.2950.2400.3620.2770.7650.889
    HOME³ (Decouple)0.2780.2230.3330.2520.7730.897
    TD-HOMES (zero-shot)π³ (TD-Adapt)0.2780.2280.4230.3440.7410.857
    HOME³ (TD branch only)0.1870.1560.2020.1640.8190.934
    HOME³ (Joint-FT) (Separate-Encoder)0.2880.2440.4490.3710.7440.857
    HOME³ (Joint-FT)0.1940.1560.2180.1730.8010.928
    HOME³ (Decouple)0.1610.1320.1780.1480.8260.948

    The local/global ablation (Appendix A.3.2, Tables 8–9 of the updated manuscript) further shows that adding more specialized trainable parameters does not consistently improve performance. Decoupling both local and global layers introduces more parameters than global-only decoupling, yet yields no consistent reconstruction gains and performs worse in zero-shot evaluation.


Unfinished. Please keep reading the comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.