acceptodds
Under review as a conference paper at ICLR 2027

Building Rome from a Single Image

Abstract

Single-image scene generation aims to produce a complete 3D scene mesh from a single image, including surfaces the camera did not observe. While pretrained 3D object generators encode a strong shape prior, they are mainly designed for isolated objects in a fixed canonical volume and focus mostly on indoor scenes, since diverse 3D data for outdoor scenes are quite limited. In this work, we present a method that redesigns such an object-centric generator, , Trellis 2, to work on both indoor and outdoor scenes while retaining its prior. We accomplish this by (a) partitioning the scene into adaptive chunks that scale relative to the distance to the camera; nearby chunks have a smaller size to keep the finer detail, while distant structures, , buildings, are covered by large chunks; (b) making the generator capture explicit 2D-3D correspondence by lifting image features and making the model aware of the free space, observed surface, and unobserved region; (c) synthesizing around 4,000 outdoor scenes to broaden the training data, as existing scene datasets are largely indoor. Experiments on Tanks and Temples, ScanNet++, and in-the-wild images show that our method outperforms all baselines in geometric accuracy and perceptual quality across both indoor and outdoor scenes.

Primary area:
generative models
Keywords:
3D generation, 3D reconstruction

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_6aCgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts TRELLIS.2 for generating 3D scene meshes from a single image. It combines adaptive chunking, depth and visibility conditioning, and approximately 4,000 synthetic outdoor training scenes to reconstruct observed surfaces and complete unobserved geometry.

Critical strengths

The paper addresses a challenging task across indoor and large outdoor scenes. Adaptive chunking provides a promising balance between geometric quality and efficiency. Experiments include ablations examining chunking, conditioning, and the contribution of additional training data.

Critical weaknesses
  1. The evaluation does not sufficiently distinguish reconstruction from generation. Generating a complete scene mesh from a single image involves several cases: reconstructing observed objects, completing partially observed objects, and generating objects with little or no direct visual evidence. Examples include completing the partially visible pavilion in the bridge image in Figure 1 and generating additional background buildings in the supplementary Notre-Dame example. However, CD and F1 measure alignment between predicted mesh/points and ground truth mesh/points. Those two metrics are okay for the observed objects, but do not fully capture the plausibility of alternative completions in unobserved regions. Although the paper also reports perceptual metrics and a user study, the evaluation does not clearly separate observed object reconstruction from unobserved region generation. The generation extent, evaluation region, and ICP settings also need clarification, particularly when different methods generate different amounts of additional geometry.

  2. The mesh level effects of adaptive chunking are insufficiently demonstrated. Figure A1 visualizes chunk allocation over the MoGE-3 point map, and Table 2 provides quantitative comparisons of chunking strategies. However, the resulting mesh geometry associated with individual chunks is difficult to identify in the visualizations. For the bridge, car, and Notre-Dame examples, I would like to see the final meshes colored by chunk, with the physical chunk sizes and close-ups around boundaries. Smaller nearby chunks should provide finer physical resolution, while larger distant chunks provide broader coverage at coarser resolution. The current visualizations do not clearly demonstrate how this allocation affects local detail or objects spanning multiple chunks.

  3. Cross scale latent assembly and mesh decoding are not sufficiently explained. Section 4.3 states that latents from previously generated chunks are reused in overlapping regions and trilinearly resampled when chunk scales differ. However, it remains unclear how these latents, associated with different chunk sizes, are assembled and decoded into the final mesh. Are they first resampled onto a common grid and decoded jointly, or are chunks decoded separately and their meshes subsequently merged? Since TRELLIS.2 uses O-Voxel geometry, how are edges generated across different voxels in different chunk sizes? GenRecon’s uniform chunk scale avoids this particular cross resolution issue. The paper should explain how its adaptive representation handles these interfaces and demonstrate whether cracks, duplicate surfaces, or discontinuities occur.

Questions for authors
  1. Could you provide more visualizations and statistics for the proposed outdoor dataset? Figures 4 and A5 only illustrate the same building scene. It would be helpful if the author could show representative samples across the stated 20 categories, including non-building scenes, and report their distribution. I would like to see the visual differences between the newly collected dataset and the current in-the-wild visualizations.

  2. Could you include the original pretrained TRELLIS.2 as a baseline? Table 4 evaluates training with and without the additional outdoor data, but neither setting represents TRELLIS.2 without additional scene fine-tuning. GenRecon includes this baseline, but training-free approaches such as SynCity demonstrate that TRELLIS based scene generation can achieve promising results without additional finetuning. Qualitative and quantitative comparisons with the original pretrained TRELLIS.2 would therefore help clarify the benefits of scene specific adaptation in this task.

  3. Does generating an object across multiple chunks affect its geometry quality? In the supplementary bridge, Notre-Dame, Jeep, and Taj Mahal examples, I observed holes and missing geometry. I did one test by myself, I send the same reference images into TRELLIS.2 to run a single object mesh generation. Amazingly, it sometimes produced meshes with fewer holes and better looking local geometry. Then I raise a question that could splitting an object across multiple chunks lead to some kind of perfomrance degradation? A comparison of the same object generated within one chunk, across multiple same-scale chunks, and across different-scale chunks would help clarify this question.

  4. Could you clarify the training and inference procedures for sequential chunk generation? This world "autoregressive" might be a little bit misleading. This paper copies overlapping latents from previously generated chunks into the current chunk, which sounds like conditional chunk generation, such as NuiScene. It is unclear whether the model is trained to condition on previous chunks, how much historical context it receives, or whether this dependency is introduced only during inference. Does the denoiser use overlap masks and known latent regions during training? At each denoising timestep of the current chunk, are the inherited latents kept fully denoised, or are they first converted to a noisy state corresponding to the current timestep?

9 comments

We thank the reviewer for the thoughtful and constructive feedback, and for recognizing that adaptive chunking offers a promising approach. We address each weakness and question below.

[W1] Evaluation does not separate reconstruction from generation

We agree that generating a scene from a single image involves three cases:

  1. reconstructing observed surfaces
  2. completing partially observed objects
  3. generating objects with little or no visual evidence

CD and F1 in Table 1 are computed against the complete scan inside the evaluation region, so they cover the three cases together.

Why CD and F1 are meaningful beyond the observed surfaces. For a partially observed object, the visible part constrains the hidden part. For example, the visible half of a car or of the pavilion largely determines the other half. The scan is a valid target for the hidden part too, and the distance to it measures the quality of the completion. These are the common metrics for completion from partial observations, used for example in ActionMesh [1], Helix4D [2], and Fast4DMesh [3], which report CD and F1 against the complete ground-truth geometry.

Separating visible and occluded geometry in Table 1. To make the distinction clear, we split the scores of Table 1 into a visible part and an occluded part. In Tanks and Temples, the scans are not aligned with the pixels of the input image. A scan is a point cloud of the whole scene, captured from many views. We therefore project the scan into the input camera and take the nearest depth along each camera ray as the first visible surface. Then we apply a depth threshold:

  • a reference point is visible if its depth is within 10 cm of this first surface
  • it is occluded if it lies behind it.

Each predicted point is assigned to be either visible or occluded. We score each part with the same alignment and evaluation region as Table 1. Each cell gives CD ↓ / F1 ↑. As in Table 1, we omit VolFill on ScanNet++ because it is trained on this dataset, which would be an unfair comparison to the other baselines.

MethodT&T visibleT&T occludedScanNet++ visibleScanNet++ occluded
EvoScene4.41 / 0.2665.04 / 0.1994.13 / 0.7608.45 / 0.507
Extend3D8.68 / 0.1707.83 / 0.1484.02 / 0.7528.87 / 0.448
3D-RE-GEN6.92 / 0.1297.58 / 0.15014.41 / 0.30916.03 / 0.277
Lyra 2.02.60 / 0.4174.06 / 0.3326.08 / 0.54610.83 / 0.280
VolFill2.56 / 0.3733.28 / 0.319––
GenRecon5.05 / 0.2346.47 / 0.1769.34 / 0.45713.71 / 0.316
World Tracing2.96 / 0.3713.55 / 0.2813.67 / 0.7518.05 / 0.519
Ours1.44 / 0.5702.91 / 0.4133.15 / 0.8027.94 / 0.501

The visible part corresponds to the first case. The occluded part corresponds mainly to the second case. It also contains the fully hidden geometry of the third case, because we cannot separate objects in a real scan. All methods are less accurate on the occluded part than on the visible part. In Tanks and Temples, our improvement holds on both parts. That is, the gain comes from both the reconstruction of observed surfaces and the completion of occluded ones. In ScanNet++, we are the best on the visible part and on the occluded CD.

The third case. We agree that CD and F1 cannot capture the plausibility of alternative completions. When there is little or no visual evidence, a plausible completion that differs from the scan counts as an error for every method.

On our validation set (Tables 2, 3, and 4), we limit this effect by removing from the reference every object that is visible in fewer than 80 pixels of the 1024×1024 input. This is not possible on real scans.

In Table 1, the user study covers this case in part. Each result is shown as a looping video of normal-map renderings, so the participants also see the regions that are hidden in the input, and they are asked to consider the coverage of the scene and how clean the geometry is, in addition to the faithfulness to the input. They prefer our result over every baseline, in 73% to 94% of the judgments.

We added the visible / occluded table to the paper as Table A3 (Appendix A.6), and the details of the user study to Appendix A.5 (Table A2).

[1] ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion

[2] Helix4D: Complex 4D Mesh Generation

[3] Fast 4D Mesh Generation by Spatio-Temporal Attention Chains

[W1] Generation extent, evaluation region, and ICP settings are unclear

Sorry for the confusion. We provide the details below and have clarified the paper.

The protocol is the same for every method.

  • Region: the reference is the scan inside the frustum of the input camera. The predictions are cropped to this frustum and to the pixels where the scan has data.
  • Generation extent: because geometry outside this region is cropped, a method that generates more geometry is neither rewarded nor penalized for it.
  • Alignment: we search a global scale over 25 log-spaced values in [0.5, 2] around the ratio of the median depths. For each scale, we run a point-to-plane ICP on the translation, and we keep the scale with the lowest symmetric capped Chamfer distance. We then refine the scale per axis. We do not estimate a rotation, since every method predicts in the frame of its input camera.

We added the evaluation region to Section 5, and the sampling and the alignment to Appendix A.10.

[W2] Smaller nearby chunks should provide finer physical resolution, while larger distant chunks provide broader coverage at coarser resolution

We added Fig. A7 (Appendix A.8) to show this on the generated geometry. For each scene, it shows the sparse structure colored by chunk size, and close-ups of a nearby and a far region, as mesh normals and as voxels.

Scenes. The effect is hard to see when the region near the camera is mostly ground, as for the Colosseum, or when the scene has no nearby region, as for Notre-Dame (its smallest chunks are 48 m). We therefore show the bridge and the Jeep, and add a scene with a detailed object in front and a building behind it (a knight in front of a gate).

Resolution follows the chunk size. Every chunk has the same number of voxels, so the voxel size grows with the chunk size:

Chunk size3 m6 m12 m24 m
Voxel size4.7 cm9.4 cm19 cm38 cm

Nearby objects lie in 3 m chunks and far structures in 24 m chunks, and the detail follows the voxel size. In Fig. A7, the stones of the arch are coarser than the armor of the knight, and the roof of the pagoda is coarser than the leaves of the pine. For the Jeep, the tread of the front tire is visible, while the house behind it is coarser. The effect is smaller there, because the far chunks are only twice as large (6 m).

Reviewer_P4hYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a method for generating complete 3D scene meshes from a single image in a chunk-based manner. The main idea is to extend the pretrained object-centric generator Trellis 2 to scene-level generation while preserving its learned 3D shape priors. To achieve this, the authors introduce adaptive chunking, which uses smaller chunks for nearby regions and larger chunks for distant structures. Next, the 2D-3D conditioning is used to lift image features onto surfaces and generate the unobserved geometry based on the visibility information. The scene is generated in a chunk-by-chunk manner, with overlapping latents reused to maintain consistency. Experiments on Tanks and Temples, ScanNet++, and in-the-wild images show improvements over existing methods in geometric accuracy.

Critical strengths
  • The task of generating complete 3D scenes from a single image is interesting, particularly for outdoor environments. The paper addresses a limitation existing object-centric 3D generation methods and indoor scene generation methods.
  • Although adaptive chunking is a relatively straightforward idea, I think it is a practical solution for large-scale outdoor scenes. The approach balances geometric detail and computational efficiency, with clear improvements demonstrated in Table 2.
  • The paper provides extensive comparisons with existing methods across indoor and outdoor scenes, showing improvements in reconstruction quality.
Critical weaknesses
  • Global consistency. While the method reuses overlapping latents to maintain consistency between neighboring chunks, it is unclear whether this is sufficient to ensure global scene consistency. Since chunks are generated autoregressively, errors in earlier chunks may propagate to subsequent ones without global refinement.
  • Robustness to geometry priors. The proposed method relies heavily on estimated point maps and camera parameters for both adaptive chunking and 2D-3D conditioning. Errors in these predictions could affect chunk allocation, feature alignment, and visibility estimation. This is also worth investigating because the model uses ground-truth geometry during training but relies on estimated geometry at inference time.
  • Evaluation of unobserved geometry. One of the main goals of the paper is to generate complete 3D scenes, including regions not visible in the input image. While the paper reports strong overall reconstruction results, it does not separately evaluate observed and unobserved geometry. I think such an evaluation would be helpful to better understand whether the improvements come primarily from more accurate reconstruction of visible surfaces or better completion of unseen regions.
Questions for authors
  1. Global consistency. Since the scene is generated chunk by chunk, I wonder how well the method maintains global consistency, especially when large structures span multiple chunks. Could errors in earlier chunks affect the generation of later ones?

  2. Cross-scale consistency. How does the method handle the boundaries between chunks of different physical scales? It would be helpful to see whether latent resampling introduces any discontinuities in the final mesh.

  3. Robustness to depth estimation errors. The method relies heavily on estimated point maps and camera parameters, while using ground-truth values during training. How robust is the method when these predictions are inaccurate? The authors can show how depth estimation errors affect the final reconstruction.

  4. Evaluation of unobserved geometry. Could the authors separately evaluate the reconstruction quality of visible and invisible regions? This would help clarify how much of the improvement comes from better reconstruction of observed surfaces versus completion of unseen geometry.

  5. Could the proposed framework be extended to support multiple input images? It would be interesting to understand whether the current chunk-based generation and 2D–3D conditioning could naturally incorporate multi-view observations.

5 comments

We thank the reviewer for the detailed review and recognizing our method's contribution. We address each concern below.

[W1 + Q1] Global consistency; can errors in earlier chunks propagate to later ones?

We agree that sequential generation can propagate errors from one chunk to the next, as acknowledged in Appendix A.3. However, two aspects of our design help mitigate this issue.

First, every chunk is anchored to the input. Error accumulation is severe in outpainting and in autoregressive video models, because the model is conditioned only on previous outputs. In our method, every chunk is also conditioned on the input image and on the point map of the whole scene. A large structure that spans several chunks is placed by the same point map in each of them, so its layout does not depend on the previous chunks.

Second, the context is relatively small. A chunk only receives the latents of its neighbors inside the overlap, which keeps the surfaces continuous between the chunks of the same depth region. The rest of the chunk is generated from the input.

The point map only covers the visible surfaces. For the hidden surfaces, a chunk knows what its neighbor generated only through the overlap, so this is where an error can propagate. To measure it, we inject an error on purpose. We corrupt the sparse-structure latent of the first chunk by mixing it with noise, at 50% and at 100% (pure noise), and give it to the later chunks as their overlap context. Everything else is fixed, including the noise of the later chunks, so a change in a later chunk is caused by the corrupted context alone.

The table gives, for 16 in-the-wild images (8 indoor, 8 outdoor), the occupancy IoU of the sparse structure with the normal run, per depth region (1 means that the geometry is unchanged). A depth region is the set of chunks of one scale, and each farther region uses larger chunks, so the second and third columns measure the propagation across scales. In the first depth region, the corrupted chunk itself is excluded, so the column covers only the other chunks of the same scale (images whose first region lies almost entirely inside the corrupted chunk are left out of this column).

Corruption of the first chunkFirst depth region, other chunks (IoU)Second depth region (IoU)Third depth region (IoU)
50%0.8840.9130.966
100% (pure noise)0.8510.8940.964

Even when the context from the first chunk is pure noise, most of the geometry of the later chunks is unchanged. The change is largest in the chunks next to the corrupted one and decreases with distance, to an IoU of 0.96 in the third depth region. It also barely grows with the size of the error. Going from 50% to 100% corruption lowers the IoU by at most 0.03. The later chunks therefore follow the image and the point map more than the overlap, so an error stays local and does not accumulate.

Training with context. Training each chunk with its neighbors as context could reduce the propagation further, but the model must then be trained on its own generated neighbors to avoid exposure bias, as in Self Forcing [1]. We leave this to future work.

[1] Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion

[Q2] Boundaries between chunks of different physical scales. Does latent resampling introduce discontinuities?

There are two kinds of boundaries, and only the one between different scales is visible in the mesh. Its discontinuity comes from the decoding, not from the latent resampling. We added Fig. A6 (Appendix A.7), which shows close-ups of both kinds of boundaries on the final meshes.

Same scale: no boundary. All chunks of a depth region have the same voxel size and lie on one lattice. The decoder is convolutional, so it is translation-equivariant and works for any spatial extent. We therefore decode all chunks of a region jointly in one pass and extract a single mesh. The chunks share vertices, with no cracks and no duplicated surfaces (Fig. A6, blue).

Different scales: the resampling is context only. When a farther depth region is generated, it must know what the nearer region already generated in the overlap. We give it the latents of the nearer region, trilinearly resampled onto the farther lattice and kept fixed during denoising. These latents are only an input to the generation. They are never decoded into the mesh, so the resampling cannot introduce a discontinuity. Both regions are conditioned on the same image and point map, so a visible surface that crosses the boundary is placed at the same depth on both sides.

Where the discontinuity comes from. A convolution layer inside the decoder assumes one voxel size, so two regions cannot be decoded jointly. We decode them separately and, in the overlap, keep only the mesh of the nearer (finer) region, so no surface appears twice. The two meshes then meet at the boundary but share no vertices, which leaves a gap and a change in the level of detail (Fig. A6, red). The gap widens as the voxels of the farther region grow. It is a thin crack between the 6 m and 12 m chunks of the bridge, and a visible gap between the 24 m and 51.5 m chunks of the Colosseum. This is a limitation of our decoder, not of the representation.

Prior works that generate large scenes chunk by chunk also observe similar problems at the transitions between chunks, for example SynCity [3] and NuiScene [4].

Future work. Decoding all depth regions jointly would require a representation with several voxel sizes, such as an octree [1, 2], which we leave to future work. We added this limitation to Appendix A.3.

[1] Dual Octree Graph Networks for Learning Adaptive Volumetric Shape Representations

[2] OctFusion: Octree-based Diffusion Models for 3D Shape Generation

[3] SynCity: Training-Free Generation of 3D Worlds

[4] NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes

[W2 + Q3] Robustness to estimated point maps and camera parameters; ground truth at training, estimates at inference

We measured this on our validation set. We ran the same model twice, once with the ground-truth point map and camera (as in training) and once with the MoGe-3 estimates (as at inference). The ground-truth row is the last row of Tables 3 and 4.

Input depth and cameraIndoor CD ↓ / F1 ↑Outdoor CD ↓ / F1 ↑
Ground truth3.54 / 0.7936.90 / 0.501
MoGe-34.39 / 0.7668.22 / 0.324

Indoors, the estimated geometry costs little, and the F1 drops by 0.03. Outdoors, the drop is larger, 0.18, because the error of a monocular depth estimate grows with distance, and the outdoor scenes are much deeper.

The chunk allocation, the projected features, and the visibility test all use the same point map, so they stay consistent with each other when the depth is wrong. A depth error therefore mainly displaces the generated surfaces, and the scene is still generated around the estimated geometry.

Other methods that lift a single image to 3D with an estimated depth or point map observe the same dependence [1, 2, 3]. Our method is agnostic to the estimator, so a newer and more accurate model can be swapped in later without retraining.

[1] WonderJourney: Going from Anywhere to Everywhere

[2] WonderWorld: Interactive 3D Scene Generation from a Single Image

[3] ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

Reviewer_dfS1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts an object-centric Trellis 2 generator to reconstruct scene meshes from a single RGB image. It combines depth- and height-adaptive chunks, surface-aligned image features with free-space conditioning, and approximately 4,000 synthetic outdoor scenes for training. Chunks are generated sequentially with overlap-latent reuse. Experiments on Tanks and Temples, ScanNet++, and in-the-wild images report improved geometry and perceptual quality, while ablations examine chunking, conditioning, and outdoor supervision.

Critical strengths
  1. The paper addresses a useful extension of object-level 3D priors to indoor, outdoor, and large-scale scene geometry without requiring multiple input views. Table 1 reports consistent advantages over a diverse set of reconstruction approaches. On Tanks and Temples, the reported CD improves from the strongest baseline's 2.63 to 1.99, and F1 improves from 0.388 to 0.506. The qualitative comparisons also support better preservation of scene layout across varied inputs. This breadth makes the system contribution practically interesting even though its components build on existing generative representations.

  2. The three design choices form a coherent solution and receive targeted empirical support. Adaptive chunks allocate detail near the camera while extending distant coverage; surface-aligned features and signed-depth conditioning distinguish observed surfaces from free and occluded space; rerendering assembled synthetic scenes provides internally consistent supervision. Table 2 compares chunking strategies using the same trained model and reports 5 versus 15 minutes for adaptive versus fixed-size chunks. Tables 3-4 separately examine conditioning and additional outdoor data, with outdoor F1 increasing from 0.317 to 0.501 when the data are added. This is more informative than an unablated end-to-end pipeline.

Critical weaknesses
  1. The evaluation does not yet clearly separate observed-surface reconstruction from the central claim of completing unobserved geometry. Sections 5 and 5.1 report aggregate CD/F1 and image-based scores, but do not specify the visibility/cropping support well enough to establish which hidden surfaces contribute. The four-view normal-map evaluation is also unclear: the target normals are estimated from the input image, so the correspondence between novel rendered views and the reference needs explanation. Input-image agreement and semantic similarity alone do not establish correct hidden geometry. This limits how strongly the reported metrics support the completion claim, rather than merely improved visible reconstruction.

  2. Important experimental details remain insufficiently specified for assessing the size and attribution of the gains. For example, the adaptive configuration in Table 2 has outdoor CD/F1 of 37.4/0.240, whereas the full configuration in Table 3 reports 6.90/0.501, without explaining which validation population or settings differ. The treatment of the originally multi-view GenRecon baseline under single-image input also needs clarification. Only 7 Tanks and Temples and 32 ScanNet++ scenes are used, and the comparatively small ScanNet++ gains lack uncertainty estimates. The user study omits participant and judgment counts. These gaps do not negate the reported improvements, but reduce confidence in their robustness and reproducibility.

Questions for authors
  1. Please specify the evaluated spatial support, visibility masks, point sampling, and ICP transformation for CD/F1, including how monocular scale is handled. How are the four rendered normal views compared with the input-image normal estimate in CLIP-N, and what render/reference pair is used for DreamSim? If the scan geometry permits, a visible-versus-occluded breakdown would clarify whether the improvement extends to hidden-surface completion; otherwise, please state that evaluation limitation explicitly.

  2. What validation scenes, checkpoints, coordinate normalizations, and chunking settings underlie Tables 2-4, and what explains the different full-method results? Please also describe the single-image adaptation of GenRecon, including its camera/geometry inputs, and how the projection-only control relates to its spatial conditioning. These details would clarify the incremental benefit of the proposed surface/visibility conditioning and adaptive allocation under comparable settings.

  3. Can you provide per-scene paired uncertainty or the scene-level results for the reported geometry gains, especially on ScanNet++? For the user study, please report the number of raters and judgments, scene coverage, presentation randomization, and tie handling, and confirm that each Table 1 win rate denotes preference for your method over the listed baseline. These details could materially affect confidence in the comparative conclusions.

7 comments

We thank the reviewer for their feedback and for recognizing the interesting contribution of our system, which extends object-level 3D priors to large-scale scene generation.

[W1 + Q1] Scoring visible and occluded surfaces separately

The scan geometry permits this breakdown, which separates reconstruction from generation.

How we split. The scans are not aligned with the pixels of the input image, so we project the scan into the input camera and take the nearest depth along each camera ray as the first visible surface. A reference point is visible if its depth is within 10 cm of this first surface, and occluded if it lies behind it. Each predicted point is assigned in the same way, and we score each part with the same alignment and region as Table 1. Each cell gives CD ↓ / F1 ↑. VolFill is trained on ScanNet++, so we do not report it there, as in Table 1.

MethodT&T visibleT&T occludedScanNet++ visibleScanNet++ occluded
EvoScene4.41 / 0.2665.04 / 0.1994.13 / 0.7608.45 / 0.507
Extend3D8.68 / 0.1707.83 / 0.1484.02 / 0.7528.87 / 0.448
3D-RE-GEN6.92 / 0.1297.58 / 0.15014.41 / 0.30916.03 / 0.277
Lyra 2.02.60 / 0.4174.06 / 0.3326.08 / 0.54610.83 / 0.280
VolFill2.56 / 0.3733.28 / 0.319––
GenRecon5.05 / 0.2346.47 / 0.1769.34 / 0.45713.71 / 0.316
World Tracing2.96 / 0.3713.55 / 0.2813.67 / 0.7518.05 / 0.519
Ours1.44 / 0.5702.91 / 0.4133.15 / 0.8027.94 / 0.501

On Tanks and Temples, ours is the best on both the visible and the occluded surfaces. On ScanNet++, ours is the best on the visible surfaces and has the best CD on the occluded ones, while its occluded F1 (0.501) is level with World Tracing and EvoScene. All methods are less accurate on the occluded part than on the visible part.

This table is the direct measure of the hidden geometry, while CLIP-N and DreamSim measure consistency with the input image [W1+Q1, part 3]. The in-the-wild images have no reference geometry, so this split is not possible there.

We added this table to the paper as Table A3 (Appendix A.6).

[W1 + Q1] CD / F1 protocol

The protocol of Table 1 is the following.

What is scored. The reference is the scan inside the frustum of the input camera, and for Tanks and Temples also inside the crop volume of the benchmark. Each prediction is cropped to the same region, so what a method generates outside it is neither rewarded nor penalized. Otherwise, a method would be penalized for geometry that the scan does not contain, such as the ground and the trees around the object in Tanks and Temples. We use no visibility mask: the frustum is a volume, so the surfaces that are hidden from the input view, such as the back of an object, stay in the reference, since completing them is what the task asks for.

How it is scored. We sample 500k points for the reference and for each prediction. Meshes are sampled on the surface with area weighting, point clouds use the predicted points, and Lyra 2.0 uses the centers of its Gaussians. A single image does not determine the scale, so we fit it in the same way for every method: we try 25 global scales between half and twice the ratio of the median depths, align the translation with a point-to-plane ICP for each scale, keep the scale with the lowest Chamfer distance, and then refine the scale per axis. We do not fit a rotation, because the predictions are in the frame of the input camera. CD is normalized by the median depth of the reference, and F1 uses a threshold of 10 cm.

We added the evaluation region to Section 5, and the sampling and the alignment to Appendix A.10.

[W1 + Q1] CLIP-N from four rendered views against an input-image normal estimate. Which pair does DreamSim compare?

What is compared. Both metrics compare the same pair: a normal map rendered from the generated mesh, and the normal map that Lotus-2 [1] estimates from the input image. CLIP-N is the cosine similarity between the CLIP embeddings of the two, and DreamSim [2] is the perceptual distance between them. Neither metric uses the RGB image. We render each mesh from four cameras: the method's own estimate of the input camera, and three cameras that look at the same point and are shifted to the left, to the right, and upward by 25% of the median rendered depth. The scores are averaged over the four views, and then over the images.

What it measures. A shifted view has no pixel correspondence with the reference, so both metrics compare the two images as a whole, through their embeddings. This measures consistency with the input image, and not whether the hidden geometry is correct. The scores still say something about the occluded part. When the camera moves and reveals regions that are hidden in the input, a good generation still agrees with the input image, while holes, flat geometry, or distorted surfaces make the rendering differ from it. This measure is not perfect, but it complements CD and F1: it also applies to the in-the-wild images, which have no reference geometry, and it has been used before in Trellis 2 [3] and Helix4D [4].

We clarified the pair that each metric compares in Section 5, and added how we render 4 views and this discussion to Appendix A.10.

[1] Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model

[2] DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

[3] Native and Compact Structured Latents for 3D Generation

[4] Helix4D: Complex 4D Mesh Generation

Reviewer_3rvEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes single-image scene generation using object-centric generators to work on both indoor and outdoor scenes. To this end, the paper adapts Trellis 2 for single image scene mesh generation through three components: adaptive chunks whose physical scale increases with distance and scene height, 2D-3D image conditioning using surface-aligned image features and a depth-based visibility measure, and 4000 synthetic outdoor training scenes. Experiments cover Tanks and Temples, ScanNet++, and in-the-wild images, with ablations of chunking, conditioning, and training data.

Critical strengths
  1. The representation addresses a practical limitation with large scale scene reconstruction. Allocating smaller chunks nearby and larger ones farther away is well motivated based on perspective projection.
  2. The reported reconstruction improvements are significant. On Tanks and Temples, CD improves from the strongest baseline’s 2.63 to 1.99, while F1 increases from 0.388 to 0.506. The method also outperforms the closely related GenRecon baseline on both Tanks and Temples and ScanNet++ under the single-image evaluation protocol. Qualitative examples show improved scene layout and geometric detail across diverse inputs.
  3. The paper includes useful ablations supporting the design choices. The proposed conditioning improves over the tested alternatives, and adding outdoor supervision increases outdoor F1.
Critical weaknesses
  1. The ablation evaluation seems to be missing some information. Section 5 mentions using a separate validation set in Ln 354. The full method results are different between Table 2 and Tables 3 and 4.
  2. Section 5 describes ICP alignment and geometric metrics but does not explain how the generated surfaces outside reference-scan coverage are handled. The user study details, e.g., number of participants, are also missing.
  3. The novelty of the proposed method is incremental. GenRecon already combines Trellis 2, overlapping scene chunks, and projected image conditioning. The contribution is mainly variable-scale chunking and depth-aware conditioning. The ablations are useful, but the paper should clearly distinguish the contributions from prior work.
Questions for authors
  1. What is the source and size of the ablation validation set mentioned in ln 354? What’s the reason for the differences in Table 2 vs Tables 3 and 4?
  2. How are generated surfaces outside the reference scan’s coverage handled when computing CD and F1?
  3. How many participants and judgments per comparison were included in the user study?
  4. How is the method different from GenRecon?
5 comments

We thank the reviewer for recognizing the motivation of adaptive chunking and the significant gains over the baselines that support our design choices. We address each point below.

[W1 + Q1] Ablation validation set

The ablations use a validation set of 120 views, with one view per scene. It includes 64 indoor views (32 each from Infinigen-v1 and v2) and 56 outdoor views from our synthetic scenes. These scenes are synthetic, so the complete geometry is available. We remove objects with fewer than 80 visible pixels in the 1024×1024 view, since the image barely shows them. We keep the visible objects whole, so that the metrics also measure the completion of their hidden parts. We added this description to Sec. 5 (Datasets) of the revision.

[W1 + Q1] Why does the full method differ between Table 2 and Tables 3–4?

All three tables use the same model and checkpoint. In the submission, the results differed because the evaluation settings were different in two ways:

1. Input depth. Table 2 used MoGe-3 estimated depth and camera parameters, matching our inference setting. Since this table compares chunk layouts, we used estimated depth to evaluate how each layout performs in practice.

Tables 3–4 instead use ground-truth depth and camera parameters to compare conditioning mechanisms and training data. This removes depth estimation errors, allowing us to isolate the effects of each component. We will clarify the writing.

2. Evaluation region. Table 2 evaluated against the entire reference scene because different layouts cover different depth ranges. This penalized layouts that fail to cover distant geometry, especially outdoors, where the reference extends far beyond the generated region.

Direct comparison. We re-evaluated all three layouts from Table 2 using the settings of Tables 3–4 (ground-truth depth and the same evaluation region). The adaptive layout corresponds exactly to the "Ours" row in Tables 3–4.

LayoutIndoor CD ↓ / F1 ↑Outdoor CD ↓ / F1 ↑
Single chunk4.47 / 0.7457.26 / 0.424
Fixed 3 m4.36 / 0.7677.17 / 0.486
Adaptive3.54 / 0.7936.90 / 0.501

The adaptive layout has the best CD and F1, indoors and outdoors. In the revision, Table 2 uses the settings of Tables 3–4 (Sec. 5.2), so the full method has the same scores in all three tables.

[W2 + Q2] How generated surfaces outside the reference scan's coverage are handled in CD / F1

Generated surfaces outside the region covered by the reference are removed before scoring, so they are neither rewarded nor penalized. The same region and the same alignment are used for every method.

  • Tanks and Temples: the reference is the laser scan inside the official crop volume of the benchmark, which contains the object of interest (for example Truck, Ignatius, or Caterpillar), and inside the frustum of the input camera.
  • ScanNet++: the reference is the room mesh inside the same frustum.
  • Predictions: they are cropped to the same frustum, to the crop volume, and to the pixels of the input view where the scan has data. For example, the trees that the object-centered scans of Tanks and Temples never captured are excluded.

What remains in the evaluation. The frustum is a volume. Surfaces inside it that are hidden from the input view, such as the back of an object or the geometry behind an occluder, stay in the evaluation.

We added this description to Sec. 5 (Metrics) of the revision.

Comments on the paper

1 comment

We thank all the reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers find adaptive chunking well motivated and practical for single-image scene generation, and the comparisons broad, with significant gains over the baselines. We uploaded a revised PDF, in which all changes are marked in blue. We summarize the main points below and answer each reviewer in detail in the individual responses.

  • Visible and occluded geometry (6aCg, P4hY, dfS1): We split the scores of Tab. 1 into a visible part and an occluded part (Tab. A3). On Tanks and Temples, ours is the best on both parts. On ScanNet++, ours is the best on the visible part and has the best CD on the occluded part.
  • Evaluation protocol (3rvE, 6aCg, dfS1): Every method is cropped to the same region and aligned in the same way, so geometry outside the reference is neither rewarded nor penalized. We clarified the region, the alignment, and the image pair that CLIP-N and DreamSim compare (Sec. 5, Appendix A.10).
  • Chunks of different scales (6aCg, P4hY): We added three figures. Chunks of the same scale are decoded jointly into one mesh, without cracks or duplicated surfaces (Fig. A6, Fig. A8), and smaller chunks give finer detail (Fig. A7). Regions of different scales are decoded separately, which can leave a gap at their boundary. We added this limitation to Appendix A.3.
  • Sequential generation (6aCg, P4hY): The model is trained on individual chunks, and the previous chunks enter only through the overlap at inference. With pure noise as the first chunk's context, later chunks keep an occupancy IoU of around 0.9 with the normal run, so the perturbation has a limited effect on them.
  • Novelty and baselines (3rvE, 6aCg, dfS1): Unlike GenRecon, we incorporate variable-scale chunks with surface-aware conditioning to generate scenes from a single image. Our approach also supports large outdoor scenes. We added the pretrained Trellis 2 to Tab. 1 and Tab. 4. It is far behind our model, which shows the benefit of adapting it to scenes.
  • Consistency of the ablations (3rvE, dfS1): Tab. 2–4 use the same model and checkpoint. The scores of the full method differed because Tab. 2 used estimated depth and a different evaluation region. Under the settings of Tab. 3–4, the adaptive layout remains the best indoors and outdoors, and the revised Tab. 2 uses these settings.
  • Per-scene results and user study (3rvE, dfS1): We added the CD and F1 of every scene (Tab. A4 and A5), with a paired comparison per scene. We also added the details of the user study (Tab. A2), in which our result is preferred over every baseline.
  • Other additions: the effect of estimated depth and camera (P4hY), and the samples and statistics of the training set next to the in-the-wild test images (6aCg, Fig. A9, Fig. A10, Tab. A1).

We hope that our responses address the reviewers' concerns. We are aware that our response comes late in the discussion period, as the additional experiments took time to complete. We will continue to monitor the discussion closely until it closes, and we would be happy to address any further questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.