- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper adapts TRELLIS.2 for generating 3D scene meshes from a single image. It combines adaptive chunking, depth and visibility conditioning, and approximately 4,000 synthetic outdoor training scenes to reconstruct observed surfaces and complete unobserved geometry.
Critical strengths
The paper addresses a challenging task across indoor and large outdoor scenes. Adaptive chunking provides a promising balance between geometric quality and efficiency. Experiments include ablations examining chunking, conditioning, and the contribution of additional training data.
Critical weaknesses
-
The evaluation does not sufficiently distinguish reconstruction from generation. Generating a complete scene mesh from a single image involves several cases: reconstructing observed objects, completing partially observed objects, and generating objects with little or no direct visual evidence. Examples include completing the partially visible pavilion in the bridge image in Figure 1 and generating additional background buildings in the supplementary Notre-Dame example. However, CD and F1 measure alignment between predicted mesh/points and ground truth mesh/points. Those two metrics are okay for the observed objects, but do not fully capture the plausibility of alternative completions in unobserved regions. Although the paper also reports perceptual metrics and a user study, the evaluation does not clearly separate observed object reconstruction from unobserved region generation. The generation extent, evaluation region, and ICP settings also need clarification, particularly when different methods generate different amounts of additional geometry.
-
The mesh level effects of adaptive chunking are insufficiently demonstrated. Figure A1 visualizes chunk allocation over the MoGE-3 point map, and Table 2 provides quantitative comparisons of chunking strategies. However, the resulting mesh geometry associated with individual chunks is difficult to identify in the visualizations. For the bridge, car, and Notre-Dame examples, I would like to see the final meshes colored by chunk, with the physical chunk sizes and close-ups around boundaries. Smaller nearby chunks should provide finer physical resolution, while larger distant chunks provide broader coverage at coarser resolution. The current visualizations do not clearly demonstrate how this allocation affects local detail or objects spanning multiple chunks.
-
Cross scale latent assembly and mesh decoding are not sufficiently explained. Section 4.3 states that latents from previously generated chunks are reused in overlapping regions and trilinearly resampled when chunk scales differ. However, it remains unclear how these latents, associated with different chunk sizes, are assembled and decoded into the final mesh. Are they first resampled onto a common grid and decoded jointly, or are chunks decoded separately and their meshes subsequently merged? Since TRELLIS.2 uses O-Voxel geometry, how are edges generated across different voxels in different chunk sizes? GenRecon’s uniform chunk scale avoids this particular cross resolution issue. The paper should explain how its adaptive representation handles these interfaces and demonstrate whether cracks, duplicate surfaces, or discontinuities occur.
Questions for authors
-
Could you provide more visualizations and statistics for the proposed outdoor dataset? Figures 4 and A5 only illustrate the same building scene. It would be helpful if the author could show representative samples across the stated 20 categories, including non-building scenes, and report their distribution. I would like to see the visual differences between the newly collected dataset and the current in-the-wild visualizations.
-
Could you include the original pretrained TRELLIS.2 as a baseline? Table 4 evaluates training with and without the additional outdoor data, but neither setting represents TRELLIS.2 without additional scene fine-tuning. GenRecon includes this baseline, but training-free approaches such as SynCity demonstrate that TRELLIS based scene generation can achieve promising results without additional finetuning. Qualitative and quantitative comparisons with the original pretrained TRELLIS.2 would therefore help clarify the benefits of scene specific adaptation in this task.
-
Does generating an object across multiple chunks affect its geometry quality? In the supplementary bridge, Notre-Dame, Jeep, and Taj Mahal examples, I observed holes and missing geometry. I did one test by myself, I send the same reference images into TRELLIS.2 to run a single object mesh generation. Amazingly, it sometimes produced meshes with fewer holes and better looking local geometry. Then I raise a question that could splitting an object across multiple chunks lead to some kind of perfomrance degradation? A comparison of the same object generated within one chunk, across multiple same-scale chunks, and across different-scale chunks would help clarify this question.
-
Could you clarify the training and inference procedures for sequential chunk generation? This world "autoregressive" might be a little bit misleading. This paper copies overlapping latents from previously generated chunks into the current chunk, which sounds like conditional chunk generation, such as NuiScene. It is unclear whether the model is trained to condition on previous chunks, how much historical context it receives, or whether this dependency is introduced only during inference. Does the denoiser use overlap masks and known latent regions during training? At each denoising timestep of the current chunk, are the inherited latents kept fully denoised, or are they first converted to a noisy state corresponding to the current timestep?
9 comments
[W1] Generation extent, evaluation region, and ICP settings are unclear
Sorry for the confusion. We provide the details below and have clarified the paper.
The protocol is the same for every method.
- Region: the reference is the scan inside the frustum of the input camera. The predictions are cropped to this frustum and to the pixels where the scan has data.
- Generation extent: because geometry outside this region is cropped, a method that generates more geometry is neither rewarded nor penalized for it.
- Alignment: we search a global scale over 25 log-spaced values in [0.5, 2] around the ratio of the median depths. For each scale, we run a point-to-plane ICP on the translation, and we keep the scale with the lowest symmetric capped Chamfer distance. We then refine the scale per axis. We do not estimate a rotation, since every method predicts in the frame of its input camera.
We added the evaluation region to Section 5, and the sampling and the alignment to Appendix A.10.
[W2] Smaller nearby chunks should provide finer physical resolution, while larger distant chunks provide broader coverage at coarser resolution
We added Fig. A7 (Appendix A.8) to show this on the generated geometry. For each scene, it shows the sparse structure colored by chunk size, and close-ups of a nearby and a far region, as mesh normals and as voxels.
Scenes. The effect is hard to see when the region near the camera is mostly ground, as for the Colosseum, or when the scene has no nearby region, as for Notre-Dame (its smallest chunks are 48 m). We therefore show the bridge and the Jeep, and add a scene with a detailed object in front and a building behind it (a knight in front of a gate).
Resolution follows the chunk size. Every chunk has the same number of voxels, so the voxel size grows with the chunk size:
| Chunk size | 3 m | 6 m | 12 m | 24 m |
|---|---|---|---|---|
| Voxel size | 4.7 cm | 9.4 cm | 19 cm | 38 cm |
Nearby objects lie in 3 m chunks and far structures in 24 m chunks, and the detail follows the voxel size. In Fig. A7, the stones of the arch are coarser than the armor of the knight, and the roof of the pagoda is coarser than the leaves of the pine. For the Jeep, the tread of the front tire is visible, while the house behind it is coarser. The effect is smaller there, because the far chunks are only twice as large (6 m).
We thank the reviewer for the thoughtful and constructive feedback, and for recognizing that adaptive chunking offers a promising approach. We address each weakness and question below.
[W1] Evaluation does not separate reconstruction from generation
We agree that generating a scene from a single image involves three cases:
CD and F1 in Table 1 are computed against the complete scan inside the evaluation region, so they cover the three cases together.
Why CD and F1 are meaningful beyond the observed surfaces. For a partially observed object, the visible part constrains the hidden part. For example, the visible half of a car or of the pavilion largely determines the other half. The scan is a valid target for the hidden part too, and the distance to it measures the quality of the completion. These are the common metrics for completion from partial observations, used for example in ActionMesh [1], Helix4D [2], and Fast4DMesh [3], which report CD and F1 against the complete ground-truth geometry.
Separating visible and occluded geometry in Table 1. To make the distinction clear, we split the scores of Table 1 into a visible part and an occluded part. In Tanks and Temples, the scans are not aligned with the pixels of the input image. A scan is a point cloud of the whole scene, captured from many views. We therefore project the scan into the input camera and take the nearest depth along each camera ray as the first visible surface. Then we apply a depth threshold:
Each predicted point is assigned to be either visible or occluded. We score each part with the same alignment and evaluation region as Table 1. Each cell gives CD ↓ / F1 ↑. As in Table 1, we omit VolFill on ScanNet++ because it is trained on this dataset, which would be an unfair comparison to the other baselines.
The visible part corresponds to the first case. The occluded part corresponds mainly to the second case. It also contains the fully hidden geometry of the third case, because we cannot separate objects in a real scan. All methods are less accurate on the occluded part than on the visible part. In Tanks and Temples, our improvement holds on both parts. That is, the gain comes from both the reconstruction of observed surfaces and the completion of occluded ones. In ScanNet++, we are the best on the visible part and on the occluded CD.
The third case. We agree that CD and F1 cannot capture the plausibility of alternative completions. When there is little or no visual evidence, a plausible completion that differs from the scan counts as an error for every method.
On our validation set (Tables 2, 3, and 4), we limit this effect by removing from the reference every object that is visible in fewer than 80 pixels of the 1024×1024 input. This is not possible on real scans.
In Table 1, the user study covers this case in part. Each result is shown as a looping video of normal-map renderings, so the participants also see the regions that are hidden in the input, and they are asked to consider the coverage of the scene and how clean the geometry is, in addition to the faithfulness to the input. They prefer our result over every baseline, in 73% to 94% of the judgments.
We added the visible / occluded table to the paper as Table A3 (Appendix A.6), and the details of the user study to Appendix A.5 (Table A2).
[1] ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusion
[2] Helix4D: Complex 4D Mesh Generation
[3] Fast 4D Mesh Generation by Spatio-Temporal Attention Chains