acceptodds
Under review as a conference paper at ICLR 2027

WorldRecon: Generative Multi-View Mesh Reconstruction for Large-Scale 3D World

Abstract

Reconstructing complete, high-quality meshes of large-scale scenes from unposed multi-view images requires faithful geometric grounding while plausibly completing surfaces that are unobserved or only sparsely captured. We present WorldRecon, a generative multi-view reconstruction framework for recovering complete, large-scale 3D scene meshes from unposed RGB images. WorldRecon uses a coarse, noisy global point cloud and camera poses estimated from the input images by a reconstruction model as a global scaffold for whole-scene generation and partitions it into overlapping spatial blocks. To robustly leverage these imperfect geometric priors, we equip the native 3D generative model TRELLIS.2 with a dedicated reconstruction-based conditioning architecture that generates each block from its local point-cloud crop and block-relevant image observations, producing sparse structure, detailed geometry, and PBR materials. To maintain scene-level coherence, we introduce neighbour-aware latent inpainting and a causal world-generation procedure that propagates structural, geometric, and material context across overlapping blocks. Together, these designs scale multi-view generative reconstruction to large-scale scenes, yielding coherent, unified meshes. Extensive experiments demonstrate that WorldRecon achieves state-of-the-art reconstruction performance, reducing Chamfer distance by 14.9% and 43.1% while improving F-score by 17.3% and 12.0% over the strongest baselines on synthetic and real-world indoor scenes, respectively, thereby demonstrating faithful and complete generative reconstruction as well as strong sim-to-real generalization.

Primary area:
generative models
Keywords:
Multi-View 3D Reconstruction, 3D Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FuUfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a generative reconstruction pipeline to recover large-scale indoor PBR meshes from unposed multi-view RGB images. It leverages VGGT-Ω to obtain coarse point cloud and camera priors, partitions scenes into overlapping spatial blocks, and finetunes TRELLIS.2 with reconstruction based conditioning to generate per-block geometry and materials. Experiments are conducted on synthetic 3D FRONT and real world ScanNet++ datasets, showing improved quantitative metrics over several baselines.

Critical strengths
  1. The paper targets a practical and important problem, namely large-scale scene mesh reconstruction from casually captured unposed multi-view images.
  2. Implementation details are relatively complete, including dataset curation, training recipes and ablation studies for key modules.
  3. Quantitative comparisons against strong baselines (VGGT-Ω+TSDF, MonoSDF, FineRecon, GenRecon) on both synthetic and real indoor datasets demonstrate clear performance gains.
Critical weaknesses
  1. This work assembles existing components into a working pipeline for unposed multi-view scene reconstruction. While the system works empirically, there are few genuinely new technical insights.
  2. The whole pipeline relies on VGGT-Ω to provide coarse camera poses and global point cloud scaffold. Although the paper claims robustness against imperfect priors, failure cases clearly show that severe pose/geometry errors from VGGT-Ω propagate and lead to missing or distorted reconstruction. It would be valuable to quantify performance degradation under different levels of pose noise in controlled experiments.
  3. Among all compared methods, only GenRecon adopts a 3D generative prior to perform structured completion for occluded scene regions. Currently the evaluation pool mixes high capacity generative models and pure reconstruction pipelines with different capability boundaries and computational complexity, which makes overall metric gains harder to interpret.
  4. There is no comparison of computational overhead, including model size, training memory cost and inference runtime.
  5. Current limitation analysis is brief; please elaborate more about failure modes besides VGGT-Ω error propagation, e.g. thin geometry, reflective surfaces, repeated structures hallucination.
Questions for authors
  1. This work assembles existing components into a working pipeline for unposed multi-view scene reconstruction. While the system works empirically, there are few genuinely new technical insights.
  2. The whole pipeline relies on VGGT-Ω to provide coarse camera poses and global point cloud scaffold. Although the paper claims robustness against imperfect priors, failure cases clearly show that severe pose/geometry errors from VGGT-Ω propagate and lead to missing or distorted reconstruction. It would be valuable to quantify performance degradation under different levels of pose noise in controlled experiments.
  3. Among all compared methods, only GenRecon adopts a 3D generative prior to perform structured completion for occluded scene regions. Currently the evaluation pool mixes high capacity generative models and pure reconstruction pipelines with different capability boundaries and computational complexity, which makes overall metric gains harder to interpret.
  4. There is no comparison of computational overhead, including model size, training memory cost and inference runtime.
  5. Current limitation analysis is brief; please elaborate more about failure modes besides VGGT-Ω error propagation, e.g. thin geometry, reflective surfaces, repeated structures hallucination.
2 comments
AuthorsOpenReview ↗

[1/1] Response to Reviewer FuUf

We sincerely thank Reviewer FuUf for thoughtful review and insightful questions. We appreciate the reviewer’s positive assessment of our problem’s practical importance and the clear performance gains. Below is our point-by-point response. Changes are marked in blue in the revised manuscript.

(W1, Q1) Technical insights

WorldRecon extends overlapping-block generation to reconstruction from unposed multi-view images, providing an end-to-end pipeline from casual captures to complete scene meshes with materials. Our reconstruction-based block generators fuse multi-view observations and coarse geometric priors through learned attention-based conditioning, accommodating imperfect poses and geometry. Our causal pipeline first assembles a global sparse structure, then generates geometry and material latents conditioned on it and previously generated neighbours, adapting established clamping and blending mechanisms accordingly for cross-block consistency.

(W2, Q2) Robustness to pose noise

Tab. R1 compares WorldRecon with GenRecon under progressively stronger perturbations, starting from GT poses and depth. We use rotation/translation/depth RMS levels of Light (1°, 5 cm, 2.5 cm), Medium (2°, 10 cm, 5 cm), and Heavy (4°, 20 cm, 10 cm). WorldRecon achieves better overall reconstruction metrics across all perturbation levels on both datasets and exhibits smaller degradation. From Clean to Heavy, CD increases by only 0.0603/0.0525 for Ours on 3D-FRONT/ScanNet++, compared with 0.1838/0.2362 for GenRecon, supporting our robustness to imperfect geometric priors.

Table R1: Ablation on Input Pose and Depth Perturbation Levels

DatasetNoise LevelMethodCD↓F-score↑NC↑Complete.↑RMSE↓AbsRel↓Angular↓RGB CLIP↑
3D-FRONTCleanGenRecon0.12490.73910.70920.96790.37290.108212.95870.9140
CleanOurs0.09610.85120.76830.97640.32100.068912.12330.9294
LightGenRecon0.16150.58820.61640.96200.46270.154715.67030.9074
LightOurs0.09560.83030.75800.98660.30300.072013.84140.9326
MediumGenRecon0.19680.46470.49600.96340.59080.226519.22210.8969
MediumOurs0.10660.74970.72240.98870.35840.089016.88790.9247
HeavyGenRecon0.30870.29950.30020.94661.01710.434027.64040.8834
HeavyOurs0.15640.55620.54900.97600.50940.163424.58100.8991
ScanNet++CleanGenRecon0.10990.66510.70900.97720.38860.145918.00720.8274
CleanOurs0.05110.87940.81880.99660.29040.077817.04640.8304
LightGenRecon0.14900.54000.65870.97670.48830.198220.28450.8184
LightOurs0.05570.85370.80350.99410.30630.084019.12710.8289
MediumGenRecon0.22190.39140.51570.96420.69990.311323.93640.8038
MediumOurs0.06480.79870.75910.99250.36700.112723.55610.8081
HeavyGenRecon0.34610.24590.37520.89351.11520.541930.33540.7662
HeavyOurs0.10360.67370.63940.98520.57530.208832.00960.7791

(W3, Q3) Fair comparison of reconstruction and generative completion

Thank you for your question. We evaluate both reconstruction-only and generative methods against GT using the same metrics for reconstruction fidelity and completion. Pure reconstruction baselines indicate what can be recovered from limited visual observations, while GenRecon provides a generative reconstruction baseline that also completes unseen regions. Tab. 1–2 and Appendix Tab. S2 show that WorldRecon achieves better performance than GenRecon on both datasets under predicted and GT pose/depth settings. Tab. R2 below further shows WorldRecon achieves a lower average runtime than GenRecon.

(W4, Q4) Computational overhead

Tab. R2 compares computational overhead across methods. Since peak training memory usage is not consistently reported, we provide the training hardware configuration as an indication of the maximum available GPU memory, rather than actual memory consumption.

Table R2: Comparison on computational cost

MetricTSDFFineReconMurreGenReconMonoSDFOurs
Model size (M)08.801,289.984,057.6212.918,197.74
Training hardware—8 V1008 A1008 H20*1 RTX 309032 H20
Training cost (GPU-hours)02882006,714*—20,811
Average runtime (min/scene)0.631.5216.0744.85581.1833.20

* GenRecon does not report its training hardware or GPU-hours. We report our local GPU configuration and estimate its training cost.

(W5, Q5) More failure modes

We provide additional failure cases in Appendix D.3, Fig. S4 of the revised manuscript, illustrating failures involving thin geometry, reflective surfaces, and hallucinated repeated structures.

Thank you again for reviewing our paper and sharing your comments. As the discussion period draws to a close, we would appreciate any further feedback on our responses. Please let us know if anything needs further clarification.

Reviewer_rSCPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reconstruction of complete scene-level meshes with PBR materials from unposed multi-view RGB images of indoor scenes. VGGT-Ω\Omega estimates cameras and a global point cloud, which is split into overlapping 4m blocks. Each block is generated by a fine-tuned TRELLIS.2, conditioned on the block's point-cloud crop (sparse structure stage only) and up to 4 in-block views with Plucker camera tokens. Blocks are generated sequentially: inpainting variants of all three stages condition each new block on overlap latents from already generated neighbors, and at inference the known overlap is replaced with its forward-noised latent (sparse structure) or feather-blended (geometry and material). Trained on 3D-FRONT and SAGE-10k, the method reports lower CD and higher F-score than reported baselines on 30 3D-FRONT and 30 ScanNet++ scenes.

Critical strengths
  • Large gains on real data with synthetic-only training. On ScanNet++ (Tab 2), CD drops from 0.1121 (best baseline, VGGT-Ω\Omega+TSDF) to 0.0638 and F-score rises from 0.7467 to 0.8363, with the best values on all 2D depth and normal metrics. The same trend holds on 3D-FRONT. As all baselines use the same VGGT-Ω\Omega cameras, the gain does not come from a better pose front end.
  • A working pipeline from casual captures to complete scene meshes with materials. Fig. 3 and Fig. S1 show multi-room scenes in which regions missing from the VGGT-Ω\Omega point cloud are filled in, which is useful in practice for building assets from captured or generated videos.
Critical weaknesses

W1. Limited technical novelty beyond combining existing components (Sec 3.2, 3.3, contribution list in Sec 1).

Each part follows an established recipe. Splitting a scene into overlapping blocks and generating them progressively is prior work that the paper cites (BlockFusion, WorldGrow, LT3SD), and GenRecon also uses overlapping chunks for scene reconstruction. The point-cloud encoder follows Hunyuan3D-Omni. Inpainting through extra mask and known-latent input channels with zero-initialized projections is the standard inpainting finetuning recipe. The trajectory-clamping mechanism (Eqn 3-4), which the paper says it introduces, replaces the known region with the forward-noised clean latent at every step -- this is the replacement strategy of RePaint [1] and blended latent diffusion [2]. The distance-weighted averaging in Eqn 5-6 is the usual overlap blending used in tiled diffusion [3]. None of [1-3] are cited. What separates this work from prior block-wise methods is the conditioning signal, and the conditioning modules are themselves standard.

W2. The ablation does not isolate the claimed contributions (Tab 3). The rows follow the TRELLIS.2 stages, not the proposed ideas. "+SS" turns on point-cloud conditioning, multi-view conditioning, camera tokens, and block-level fine-tuning at once, and is compared against vanilla single-image TRELLIS.2, which the paper notes has no consistent world orientation, so the CD drop from 0.4003 to 0.0996 indicates little about any single design. "Stitch" combines SS clamping (Eqn 3-4) and Geo/Mat blending (Eqn 5-6). Adding Inpaint alone worsens the 3D metrics (CD 0.0954 to 0.1023, F-score 0.7624 to 0.7482), and most of the consistency gain arrives with Stitch (F-score 0.7482 to 0.8226). This raises the question of whether the six trained inpainting models (Tab S1) are needed, since clamping and blending can also be applied to the base block generator without inpainting training.

Finally, the stated advantage over GenRecon is that projecting image features with estimated poses makes reconstruction sensitive to pose errors, while this method instead feeds images and camera tokens through cross-attention. This is supported only by qualitative examples (Fig 3, Fig S3), with no controlled ablations.

References

[1] RePaint: Inpainting using denoising diffusion probabilistic models, Lugmayr et al., CVPR 22.

[2] Blended latent diffusion, Avrahami et al., SIGGRAPH 23.

[3] MultiDiffusion: Fusing diffusion paths for controlled image generation, Bar-Tal et al., ICML 23.

Questions for authors
  1. In the Inpaint without Stitch row of Tab 3, how are overlapping blocks merged, and is the known region enforced at all? In the +SS row, which Geo & Mat models are used, and with which image condition?

  2. Was GenRecon retrained on the same 3D-FRONT + SAGE-10k block data, or run with released weights? Appendix C1 only says official pipeline. A training data mismatch could explain part of the gap in Tab 1-2.

  3. Are the 15 + 15 ablation scenes a subset of the scenes in Tab 1-2, and why does Tab 3 use different metrics (MAE, RGB CLIP) from the main tables?

  4. Additional ablations for W2: (a) a Stitch without Inpaint row, i.e., clamping and blending applied to independently generated blocks; (b) separate rows for point-cloud, image, and camera-token conditioning; (c) Tab 3 reported separately on 3D-FRONT and ScanNet++; (d) a controlled experiment that perturbs the input poses by increasing amounts and compares against GenRecon. Row (a) in particular would show whether the trained inpainters are a necessary part of the method.

  5. PBR materials are listed as a contribution, but appearance is only measured by RGB CLIP in the ablation. An appearance comparison on held-out views against baselines that produce color would support this claim.

I am open to raising my score if the rebuttal clarifies weaknesses and questions.

4 comments
AuthorsOpenReview ↗

[1/3] Response to Reviewer rSCP

We sincerely thank Reviewer rSCP for the careful review and insightful questions and suggestions. We are grateful for the reviewer’s recognition of our pipeline’s practical value. Below is our point-by-point response. Changes are marked in blue in the revised manuscript.

(W1) Technical novelty and attribution

Thank you for your insightful question. We have added the missing references and clarified the adaptation of established mechanisms in Sec. 3. WorldRecon extends overlapping-block generation to reconstruction from unposed multi-view images, providing an end-to-end pipeline from casual captures to complete scene meshes with materials. Our reconstruction-based block generators fuse multi-view observations and coarse geometric priors through learned attention-based conditioning, accommodating imperfect poses and geometry. Our causal pipeline first assembles a global sparse structure, then generates geometry and material latents conditioned on it and previously generated neighbours, adapting established clamping and blending mechanisms accordingly for cross-block consistency.

(Q1) Implementation details of the ablation settings

Inpaint w/o Stitch: Blocks are processed sequentially, with SS, Geo, and Mat generation completed for each block before proceeding to the next. Overlapping blocks are merged in a shared world latent. Each block's latent is written only to unoccupied voxels, without clamping and blending. The known region is softly guided by the inpainting conditioning (Eq. 2).

+SS: Geo & Mat models use the vanilla pretrained TRELLIS.2 models, conditioned on the single image with the highest voxel coverage per block.

(Q2) GenRecon training setup and comparison fairness

GenRecon was evaluated with its released weights. However, both methods train on SAGE and 3D-FRONT. Our method uses the full dataset (10k SAGE, 6k 3D-FRONT), while GenRecon performs its own dataset curation, selecting a smaller subset (5k SAGE, 1.2k 3D-FRONT). The training distributions largely overlap.

To further assess the effect of training data, we fine-tuned GenRecon on our training set. As shown in Tab. R1, fine-tuning yields modest improvements on ScanNet++ and mixed results on 3D-FRONT, while WorldRecon maintains stronger overall performance on both datasets.

Table R1: Comparison with released and fine-tuned GenRecon.

DatasetMethodCD↓F-score↑NC↑Complete.↑D-RMSE↓D-AbsRel↓D-LPIPS↓D-CLIP↑N-Angular↓N-LPIPS↓N-CLIP↑
3D-FRONTGenRecon0.1861‾\underline{0.1861}0.61960.61280.95870.53480.1666‾\underline{0.1666}0.2548‾\underline{0.2548}0.9490‾\underline{0.9490}17.9822‾\underline{17.9822}0.3139‾\underline{0.3139}0.9543‾\underline{0.9543}
GenRecon finetuned0.18730.6426‾\underline{0.6426}0.6229‾\underline{0.6229}0.95910.5319‾\underline{0.5319}0.16690.26740.944518.04340.32280.9495
WorldRecon (Ours)0.15830.72710.68390.9589‾\underline{0.9589}0.39100.11260.19660.958216.40440.29510.9601
ScanNet++GenRecon0.12270.59010.59690.97220.68370.30480.38620.929131.57220.47110.9172
GenRecon finetuned0.1092‾\underline{0.1092}0.6117‾\underline{0.6117}0.6172‾\underline{0.6172}0.9828‾\underline{0.9828}0.6489‾\underline{0.6489}0.2771‾\underline{0.2771}0.3762‾\underline{0.3762}0.9306‾\underline{0.9306}30.6130‾\underline{30.6130}0.4695‾\underline{0.4695}0.9250‾\underline{0.9250}
WorldRecon (Ours)0.06380.83630.78400.99180.33310.09920.21390.952220.04290.36310.9320

(Q3) Ablation datasets and evaluation metrics

The ablation scenes are a subset of Tab. 1–2 (15 3D-FRONT + 15 ScanNet++). MAE instead of RMSE was an oversight, which we have corrected in Tab. 3. The aligned RMSE values are 1.5211, 0.3609, 0.3589, 0.3214, and 0.3057. RGB CLIP was added to ablate appearance.

AuthorsOpenReview ↗

[2/3] Response to Reviewer rSCP

(W2, Q4) Disentangled ablations and pose robustness

(1) Regarding Q4(a) and Q4(c)

We add Stitch without Inpaint and report Tab. 3 separately on 3D-FRONT and ScanNet++ (Tab. R2). Trained inpainters improve 2D metrics over stitching alone on both datasets, while on 3D-FRONT, these improvements come with a modest decline in 3D metrics.

Fig. S7 of the revised manuscript complements these quantitative results with qualitative ablations of trained inpainting, SS trajectory clamping, and Geo/Mat latent blending. In the illustrated example, removing trained inpainting introduces pronounced floor discontinuities, highlighting its role in maintaining local surface continuity.

Table R2: Ablation Study on Key Components.

DatasetSSGeo+MatInpaintStitchCD↓F-score↑NC↑Complete.↑RMSE↓AbsRel↓Angular↓RGB CLIP↑
3D-FRONT××××0.36140.24770.19750.84261.47250.616447.68760.7877
✓×××0.1024‾\underline{0.1024}0.79110.72850.97240.37760.093816.74090.8902
✓✓××0.09780.81260.74620.97840.39340.093615.49290.9249‾\underline{0.9249}
✓✓✓×0.10930.78230.72470.98030.3364‾\underline{0.3364}0.0830‾\underline{0.0830}15.09800.9159
✓✓×✓0.10300.7954‾\underline{0.7954}0.7335‾\underline{0.7335}0.9818‾\underline{0.9818}0.34710.084214.8715‾\underline{14.8715}0.9171
✓✓✓✓0.10870.78190.72680.98570.30520.080213.99510.9282
ScanNet++××××0.43910.15480.13890.63281.56960.615549.60660.6632
✓×××0.09680.69650.70610.97200.34430.102222.54590.7587
✓✓××0.0930‾\underline{0.0930}0.71230.72360.9932‾\underline{0.9932}0.32440.084420.02900.7968
✓✓✓×0.09530.71420.71580.99330.3065‾\underline{0.3065}0.0830‾\underline{0.0830}19.3944‾\underline{19.3944}0.8059‾\underline{0.8059}
✓✓×✓0.05730.8594‾\underline{0.8594}0.7983‾\underline{0.7983}0.99300.31110.083819.40890.8039
✓✓✓✓0.05730.86330.80130.99300.30610.082819.16260.8061

(2) Regarding Q4(b)

We fine-tune the SS stage for each conditioning variant in Tab. R3. Both point-cloud and image conditioning improve reconstruction quality, with point-cloud conditioning being critical for spatial grounding. Camera tokens provide limited benefits, possibly due to inaccuracies in the predicted input camera poses.

Table R3. Ablation study on conditioning inputs.

DatasetMethodCD↓F-score↑NC↑Complete.↑RMSE↓AbsRel↓Angular↓RGB CLIP↑
3D-FRONTw/o Point Cloud0.22520.45990.37460.92161.26640.509544.46410.8228
w/o Image0.11250.77430.72290.98130.3366‾\underline{0.3366}0.083314.68640.9281
w/o Camera Tokens0.10530.79550.73270.9848‾\underline{0.9848}0.33690.0831‾\underline{0.0831}14.6071‾\underline{14.6071}0.9303
Full0.1087‾\underline{0.1087}0.7819‾\underline{0.7819}0.7268‾\underline{0.7268}0.98570.30520.080213.99510.9282‾\underline{0.9282}
ScanNet++w/o Point Cloud0.14460.57700.49030.90091.49190.552647.69070.6970
w/o Image0.05830.85410.79660.9928‾\underline{0.9928}0.3082‾\underline{0.3082}0.083419.34360.8054‾\underline{0.8054}
w/o Camera Tokens0.0575‾\underline{0.0575}0.8583‾\underline{0.8583}0.80300.99300.30980.0829‾\underline{0.0829}19.2884‾\underline{19.2884}0.8040
Full0.05730.86330.8013‾\underline{0.8013}0.99300.30610.082819.16260.8061
AuthorsOpenReview ↗

[3/3] Response to Reviewer rSCP

(3) Regarding Q4(d)

Tab. R4 compares our method with GenRecon under progressively stronger perturbations, starting from GT poses and depth. We use rotation/translation/depth RMS levels of Light (1°, 5 cm, 2.5 cm), Medium (2°, 10 cm, 5 cm), and Heavy (4°, 20 cm, 10 cm). Our method achieves better reconstruction metrics across all perturbation levels on both datasets and exhibits smaller degradation. From Clean to Heavy, CD increases by only 0.0603/0.0525 for Ours on 3D-FRONT/ScanNet++, compared with 0.1838/0.2362 for GenRecon, supporting our robustness to imperfect geometric priors.

Table R4: Ablation on Input Pose and Depth Perturbation Levels

DatasetNoise LevelMethodCD↓F-score↑NC↑Complete.↑RMSE↓AbsRel↓Angular↓RGB CLIP↑
3D-FRONTCleanGenRecon0.12490.73910.70920.96790.37290.108212.95870.9140
CleanOurs0.09610.85120.76830.97640.32100.068912.12330.9294
LightGenRecon0.16150.58820.61640.96200.46270.154715.67030.9074
LightOurs0.09560.83030.75800.98660.30300.072013.84140.9326
MediumGenRecon0.19680.46470.49600.96340.59080.226519.22210.8969
MediumOurs0.10660.74970.72240.98870.35840.089016.88790.9247
HeavyGenRecon0.30870.29950.30020.94661.01710.434027.64040.8834
HeavyOurs0.15640.55620.54900.97600.50940.163424.58100.8991
ScanNet++CleanGenRecon0.10990.66510.70900.97720.38860.145918.00720.8274
CleanOurs0.05110.87940.81880.99660.29040.077817.04640.8304
LightGenRecon0.14900.54000.65870.97670.48830.198220.28450.8184
LightOurs0.05570.85370.80350.99410.30630.084019.12710.8289
MediumGenRecon0.22190.39140.51570.96420.69990.311323.93640.8038
MediumOurs0.06480.79870.75910.99250.36700.112723.55610.8081
HeavyGenRecon0.34610.24590.37520.89351.11520.541930.33540.7662
HeavyOurs0.10360.67370.63940.98520.57530.208832.00960.7791

(Q5) Appearance evaluation

Tab. R5 reports RGB appearance metrics on the same 30 scenes per dataset and 50 held-out views per scene used in Tab. 1–2, comparing WorldRecon with baselines that produce color. Furthermore, qualitative comparisons and additional appearance and material evaluations are provided in Fig. S6 and Tab. S5 of the revised manuscript, respectively.

Table R5: Appearance evaluation

DatasetMethodPSNR ↑SSIM ↑LPIPS ↓CLIP ↑
3D-FRONTVGGT-Ω+TSDF14.39740.60640.57020.7841
MonoSDF14.18760.66060.52590.8169
GenRecon15.3616‾\underline{15.3616}0.6766‾\underline{0.6766}0.4389‾\underline{0.4389}0.9018‾\underline{0.9018}
WorldRecon (Ours)16.23060.69750.41230.9199
ScanNet++VGGT-Ω+TSDF11.8744‾\underline{11.8744}0.5963‾\underline{0.5963}0.5397‾\underline{0.5397}0.7194
MonoSDF7.08920.36610.66900.7075
GenRecon10.84330.58040.56990.7801‾\underline{0.7801}
WorldRecon (Ours)12.12770.61280.51810.7948

Thank you again for reviewing our paper and sharing your comments. As the discussion period draws to a close, we would appreciate any further feedback on our responses. Please let us know if anything needs further clarification.

Reviewer_fLfbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

WorldRecon reconstructs complete, scene-scale indoor meshes with PBR materials from unposed multi-view RGB images. VGGT-Ω provides cameras and a global point cloud. A fine-tuned TRELLIS.2 cascade (sparse structure, geometry, material) generates each block from fused DINOv3 and Plücker multi-view tokens. Inpainting variants condition each block on its overlaps with up to two previously generated neighbours. Trajectory clamping and latent feathering stitch blocks in a causal order. Trained on 117K synthetic blocks (3D-FRONT, SAGE-10k), it outperforms five baselines on 30 3D-FRONT and 30 ScanNet++ scenes.

Critical strengths
  1. The paper sets up a practically improtant problem statement. Complete large scene meshes from unposed captures are useful. Conditioning through tokens and an estimated point cloud avoids projecting image features with noisy poses, which is the failure mode of volume-projection methods such as GenRecon.
  2. The proposed model presents strong sim-to-real results. On ScanNet++, CD falls from 0.1121 (the VGGT-Ω+TSDF scaffold the method starts from) to 0.0638, and F-score rises from 0.7467 to 0.8363. 2D depth and normal metrics improve consistently, despite synthetic-only training.
Critical weaknesses
  1. The experiments do not isolate the source of the gains, and appearance is not evaluated.

    • All baselines run on VGGT-Ω cameras (C.1). On ScanNet++, all four other baselines score below plain VGGT-Ω+TSDF. The comparison therefore mainly measures robustness to pose and depth errors. A GT-pose/depth setting, available on both datasets, would separate the two.
    • No appearance evaluation. PBR materials are a headline claim (abstract, first contribution, Fig. 1), yet Tables 1–2 and Fig. 3 are geometry-only. The only appearance number is RGB CLIP in the ablation, although 50 held-out RGB views per scene and GT materials on 3D-FRONT exist.
    • The test results on 3D-FRONT scene are not strong enough. Held-out 3D-FRONT scenes likely share 3D-FUTURE furniture with training scenes, and Sec. 4.4 states that TRELLIS.2 was pretrained on 3D-FRONT scene data.
  2. Novelty relative to other block-wise world generation is unclear.

    • Generating overlapping blocks with context from already-generated neighbours is a well-developed line of work. The paper itself lists BlockFusion (latent tri-plane extrapolation), LT3SD, SynCity, TRELLISWorld and WorldGrow. However, it dismisses them as text- or layout-conditioned generators and never discusses which of its own components they already contain. There are more missed works such as WorldGrow (AAAI 2026) and NuiScene (ICCV 2025). It seems the new element is conditioning this machinery on reconstruction cues (multi-view tokens and a point-cloud scaffold) for unposed reconstruction. Please position the work against these papers and attribute the borrowed components.
  3. Some assumptions limit the claimed generality.

    • At inference, metric scale comes from rescaling every scene to a 2.9 m vertical extent (B.4). This is a fixed ceiling-height prior that breaks for other room heights or when the floor or ceiling is not captured. The evaluation’s Sim(3) alignment to GT cameras (C.2) hides the resulting scale error, and the main text says only that the reconstruction is “aligned and rescaled”.
    • Layout priors. Orientation assumes gravity plus Manhattan-style dominant wall directions (B.4). The vertical axis is a single 4 m block (indoor and single-story only, which the paper acknowledges). “Arbitrary scene scales” and “arbitrary viewpoints · unordered · sparse” (Fig. 1) are tested only with 32 views per scene. There is no study of view count, scale error, or runtime as scene size grows.
Questions for authors
  1. It will enrich the experiments that eport Tables 1–2 with GT poses, and GT depth where a method uses depth, for all methods including WorldRecon? This would separate pose robustness from reconstruction quality.
  2. Can you report RGB metrics (PSNR, SSIM, LPIPS) on the 50 held-out views, and material accuracy against the 3D-FRONT GT? A baseline that projects the input images onto WorldRecon’s own geometry would show whether the generated materials stay faithful to the observations.
  3. Do the held-out 3D-FRONT scenes share 3D-FUTURE furniture with the training scenes, and did TRELLIS.2 pretraining include them?
  4. How does WorldRecon differ technically from other block-wise world generation works, in the inpainting formulation, the block ordering and the conditioning?
  5. How do results change with the number of input views (e.g., 8, 16, 64)? What are per-scene runtime and memory as a function of scene size?
  6. It would strengthen the paper with more ablations including:
    • point-cloud-only vs. image-only conditioning in the structure stage (the jump from TRELLIS.2 to +SS conflates the two, and its starting point is single-image TRELLIS.2)
    • clamping vs. blending within Stitch
    • view count V, block size and overlap.
5 comments
AuthorsOpenReview ↗

[1/4] Response to Reviewer fLfb

We are grateful to Reviewer fLfb for the detailed and constructive feedback and thoughtful questions. We also appreciate the reviewer’s recognition of the practical value of our problem setting and our strong sim-to-real results. Below is our point-by-point response. Changes are marked in blue in the revised manuscript.

(W1-1, Q1) Evaluation under GT poses and depth

Qualitative comparisons under GT poses and depth are provided in Fig. S5 of the revised manuscript. Tab. R1 reports Tab. 1–2 with GT poses and depth for all methods. WorldRecon achieves the best overall performance on both datasets, demonstrating high reconstruction quality beyond robustness to input errors.

Table R1: Comparisons on Synthetic and Real-World Scenes with GT Poses and Depth.

DatasetMethodCD↓F-score↑NC↑Complete.↑D-RMSE↓D-AbsRel↓D-LPIPS↓D-CLIP↑N-Angular↓N-LPIPS↓N-CLIP↑
3D-FRONTVGGT-Ω+TSDF0.25110.54030.66400.76310.42790.08330.37690.91058.44210.38880.9337
Murre0.35450.41090.56820.60940.67800.15130.44880.864716.87300.60510.8785
MonoSDF0.23940.55130.63150.90410.47650.10330.21780.923012.89360.30180.9150
FineRecon0.18250.64730.7065‾\underline{0.7065}0.86090.3758‾\underline{0.3758}0.0809‾\underline{0.0809}0.26610.917111.0644‾\underline{11.0644}0.34310.9088
GenRecon0.1648‾\underline{0.1648}0.7095‾\underline{0.7095}0.69540.9555‾\underline{0.9555}0.39830.11100.2001‾\underline{0.2001}0.9578‾\underline{0.9578}13.41850.2566‾\underline{0.2566}0.9613‾\underline{0.9613}
WorldRecon (Ours)0.13450.82460.75520.95640.35450.08010.15210.964912.75600.23910.9644
ScanNet++VGGT-Ω+TSDF0.20120.61530.63740.58580.3348‾\underline{0.3348}0.08310.53290.869419.68620.54280.8824
Murre0.12280.69960.70800.61430.36520.09010.49060.879721.14570.52600.8811
MonoSDF0.08480.75820.77331.00000.34320.12000.1984‾\underline{0.1984}0.933918.45540.33890.8946
FineRecon0.0830‾\underline{0.0830}0.8114‾\underline{0.8114}0.7858‾\underline{0.7858}0.94480.34590.10300.22690.924718.4486‾\underline{18.4486}0.36690.8932
GenRecon0.10720.67970.71990.97750.41000.16010.22190.9545‾\underline{0.9545}18.70690.33540.9295‾\underline{0.9295}
WorldRecon (Ours)0.05400.87420.81060.9937‾\underline{0.9937}0.30890.0862‾\underline{0.0862}0.19190.956718.36600.3357‾\underline{0.3357}0.9370
AuthorsOpenReview ↗

[2/4] Response to Reviewer fLfb

(W1-2, Q2) Appearance and material evaluation

Tab. R2 reports RGB appearance metrics on the same 30 scenes per dataset and 50 held-out views per scene used in Tab. 1–2, comparing WorldRecon with baselines that produce color.

Table R2: Comparison on RGB Metrics

DatasetMethodPSNR ↑SSIM ↑LPIPS ↓CLIP ↑
3D-FRONTVGGT-Ω+TSDF14.39740.60640.57020.7841
MonoSDF14.18760.66060.52590.8169
GenRecon15.3616‾\underline{15.3616}0.6766‾\underline{0.6766}0.4389‾\underline{0.4389}0.9018‾\underline{0.9018}
WorldRecon (Ours)16.23060.69750.41230.9199
ScanNet++VGGT-Ω+TSDF11.8744‾\underline{11.8744}0.5963‾\underline{0.5963}0.5397‾\underline{0.5397}0.7194
MonoSDF7.08920.36610.66900.7075
GenRecon10.84330.58040.56990.7801‾\underline{0.7801}
WorldRecon (Ours)12.12770.61280.51810.7948

Tab. R3 reports material accuracy against the 3D-FRONT GT on the same 30 scenes and 50 held-out views. WorldRecon achieves better material reconstruction across all three parameters. Note that metallic values in the rendered GT maps are predominantly zero, so metallic results are provided for reference only.

Table R3: Comparison on Material accuracy

Material parameterMethodPSNR ↑SSIM ↑MSE ↓
Base colorGenRecon10.87770.52900.114631
WorldRecon (Ours)11.56830.55850.105961
RoughnessGenRecon5.90220.49040.281801
WorldRecon (Ours)15.78690.62890.044824
MetallicGenRecon58.16210.96770.005590
WorldRecon (Ours)92.51620.99920.000001

Tab. R4 compares WorldRecon with a baseline that projects input images onto WorldRecon’s geometry on 3D-FRONT test set. WorldRecon achieves a 4.41 dB higher PSNR, recovering appearance in observed regions and filling in unseen areas.

Table R4: Comparison with projected RGB Baseline.

MethodPSNR ↑SSIM ↑LPIPS ↓CLIP ↑
Projected RGB11.82130.59240.52460.8793
WorldRecon (Ours)16.23060.69750.41230.9199

Fig. S6 in the revised manuscript provides qualitative appearance comparisons, including the Projected RGB baseline.

(W1-3, Q3) Training furniture assets overlap

The held-out 3D-FRONT scenes share furniture assets with the training scenes. Tab. 1 evaluates generalization to held-out scenes and layouts, rather than unseen furniture. As shown in Tab. R5, we additionally report results on 30 SAGE scenes whose furniture asset IDs are disjoint from those in our training set, extending our evaluation to include furniture-level generalization.

Table R5: Comparisons on Synthetic SAGE Scenes with Unseen Furniture.

MethodCD ↓F-score ↑NC ↑Complete. ↑Depth RMSE ↓Depth AbsRel ↓Depth LPIPS ↓Depth CLIP ↑Normal Angular ↓Normal LPIPS ↓Normal CLIP ↑
VGGT-Ω+TSDF0.0543‾\underline{0.0543}0.8481‾\underline{0.8481}0.80730.93110.1553‾\underline{0.1553}0.02920.33920.884314.75080.50530.8776
Murre0.16650.60260.71790.70180.35790.05590.44460.851115.62620.57690.8748
MonoSDF0.05900.82300.8779‾\underline{0.8779}1.00000.18010.04110.1440‾\underline{0.1440}0.93747.86470.19860.9383
FineRecon0.05790.84290.87760.96670.16570.03290.20620.91439.64620.29130.9053
GenRecon0.06420.81260.85550.9912‾\underline{0.9912}0.16810.04750.14800.9659‾\underline{0.9659}7.6366‾\underline{7.6366}0.1739‾\underline{0.1739}0.9682‾\underline{0.9682}
WorldRecon (Ours)0.03660.98890.90851.00000.14730.0326‾\underline{0.0326}0.07830.97717.61850.15100.9741

Regarding TRELLIS.2 pretraining, we apologize for the incorrect statement in Sec. 4.4. Unlike TRELLIS, TRELLIS.2 excludes 3D-FUTURE furniture from its pretraining data. We have corrected the statement accordingly.

AuthorsOpenReview ↗

[3/4] Response to Reviewer fLfb

(W2, Q4) Comparison with prior block-wise generation methods

Thank you for this insightful question. We have added the missing references to Sec. 2. WorldRecon extends established overlapping-block generation to reconstruction from unposed multi-view images, providing an end-to-end pipeline from casual captures to complete scene meshes with materials. Our reconstruction-based block generators are designed to accommodate imperfect poses and geometry. Our causal pipeline first assembles a global sparse structure, then generates geometry/material latents conditioned on it and generated neighbours, adapting established clamping and blending mechanisms accordingly for cross-block consistency. We clarify below the difference in inpainting, ordering, and conditioning.

Inpainting. We adapt WorldGrow’s inpainting formulation to TRELLIS.2 under reconstruction conditioning, using a quarter-block overlap width to reduce computation. In the Sparse Structure stage, our trained inpainter generates the unknown region, while known-region replacement following RePaint enforces consistency along the known overlap’s forward-flow trajectory. For geometry and material, we blend clean latents after each block completes generation, unlike the per-step fusion in TRELLISWorld and LT3SD.

Ordering. LT3SD expands breadth-first from a corner, while WorldGrow and NuiScene use raster scans. We start from the block with the highest reconstructed voxel occupancy and prioritize subsequent blocks by occupancy, maintaining a connected frontier with each new block conditioned on one neighbouring edge or two perpendicular edges.

Conditioning. LT3SD and NuiScene perform unconditional scene generation, BlockFusion supports layout conditioning, and TRELLISWorld and WorldGrow use text conditioning. SynCity uses generated images and rendered context, while GenRecon uses projection-based multi-view conditioning. WorldRecon directly fuses observed image and estimated-camera tokens through attention, with additional point-cloud conditioning for sparse structure. This grounds generation in the observed scene while accommodating imperfect poses and geometry.

(W3-1) Scale and layout assumptions

We thank the reviewer for highlighting these limitations. We have clarified these assumptions and limitations in Sec. 5 and will investigate more diverse scene layouts in future work.

(W3-2, Q5) Input view count and computational efficiency

Tab. R6 ablates the number of scene-level input views on the same 30 scenes used in Tab. 3. Reconstruction quality improves as the number of input views increases, with gains largely saturating at 32 views.

Table R6: Ablation Study on input views.

Input viewsCD↓F-score↑NC↑Complete.↑RMSE↓AbsRel↓Angular↓RGB CLIP↑
80.38390.52620.54070.75210.71150.245228.27120.7986
160.10310.75930.72110.97610.43190.123919.80520.8583
320.08300.82260.76400.98940.30570.081516.57890.8671
640.08670.82160.78030.98260.31360.081116.54020.8661

WorldRecon runtime mainly depends on block count rather than input views, averaging 3.82 min/block. Sequential block generation keeps peak GPU memory approximately constant across block counts, averaging 7.67 GiB. In our runtime experiments, the smallest scenes contain 1 block, averaging 6.39 minutes per scene, while the largest contain 15 blocks, averaging 55.29 minutes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank the Area Chairs and all reviewers for their insightful comments and constructive feedback. We appreciate their recognition of the practical importance of reconstructing complete scene meshes from casual captures (fLfb, rSCP, FuUf), the clear performance gains and strong sim-to-real results (fLfb, rSCP, FuUf), and the detailed implementation (FuUf).

Following the reviewers’ suggestions, we revised the manuscript with additional experiments and clarifications. Changes are marked in blue. The main updates are:

1. Additional Comparisons and Generalization Evaluation

  • [Appendix E.1, Tab. S2, Fig. S5] GT Poses and Depth (fLfb): We repeat the main comparisons using GT cameras and, where required, GT depth, with the same RGB inputs and evaluation views. WorldRecon lowers CD (↓) by 18.4% on 3D-FRONT (0.1648 to 0.1345) and by 34.9% on ScanNet++ (0.0830 to 0.0540) relative to the strongest baseline on each dataset, demonstrating reconstruction gains beyond robustness to errors in the predicted priors.
  • [Appendix E.1, Tab. S3] SAGE Test Set (fLfb): We evaluate 30 SAGE scenes with furniture asset IDs disjoint from training to explicitly assess generalization to unseen furniture. WorldRecon improves F-score (↑) by 16.6% (0.8481 to 0.9889) and lowers RMSE (↓) by 5.2% (0.1553 to 0.1473) over the strongest baseline.
  • [Appendix E.1, Tab. S4] GenRecon Fine-tuning (rSCP): In addition to its released weights, we evaluate GenRecon fine-tuned on our training data. Fine-tuning yields mixed results on 3D-FRONT and improvements on ScanNet++, while WorldRecon retains better overall performance.
  • [Appendix E.1, Tab. S5, Fig. S6] Appearance and Materials (fLfb, rSCP): Quantitative evaluations show improved RGB appearance on both datasets and more accurate PBR materials on 3D-FRONT, complemented by qualitative comparisons. With geometry held fixed, WorldRecon achieves a PSNR gain of 4.41 dB (11.82 to 16.23 dB) over input-image projection on 3D-FRONT, demonstrating the benefit of generated appearance independently of geometric differences.

2. Additional Ablation Studies

  • [Appendix E.2.1, Tab. S6, Fig. S7] Inpainting and Stitching (fLfb, rSCP): We add the Stitch without Inpaint variant and report component ablations separately for each dataset. Qualitative ablations further illustrate the contributions of trained inpainting, SS trajectory clamping, and Geo/Mat latent blending to cross-block continuity.
  • [Appendix E.2.1, Tab. S7] Reconstruction-based Conditioning (fLfb, rSCP): We ablate point-cloud, image, and camera-token conditioning by fine-tuning the SS stage for each variant. Results highlight the role of point clouds in spatial grounding and the additional benefits of image conditioning.
  • [Appendix E.2.2, Tab. S8] Pose and Depth Perturbations (rSCP, FuUf): We compare with GenRecon under progressively stronger pose and depth perturbations starting from GT poses and depth. From Clean to Heavy noise, WorldRecon’s relative reduction in CD over GenRecon grows from 23.1% to 49.3% on 3D-FRONT and from 53.5% to 70.1% on ScanNet++, indicating greater robustness to imperfect geometric priors.
  • [Appendix E.2.3, Tab. S9–S11] Input Views, Local Conditioning Views, and Block Overlap (fLfb): We add ablation studies on the number of scene-level input views, local conditioning views per block, and block overlap ratios.

3. Clarifications

  • [Appendix E.3, Tab. S12] Computational Cost (fLfb, FuUf): We report model size, training hardware, training cost, and inference runtime.
  • [Sec. 2–3] Method Positioning and Attribution (fLfb, rSCP, FuUf): We add missing citations, clarify WorldRecon’s relationship to prior work, and explain how we adapt and extend existing approaches for our reconstruction task.
  • [Sec. 4.4, Tab. 3, Appendix E.2] Evaluation Clarifications (fLfb, rSCP): We correct the statement about TRELLIS.2 pretraining data and amend Tab. 3 to report RMSE with the corresponding values.
  • [Sec. 5, Appendix D.3, Fig. S4] Limitations and Failure Modes (fLfb, FuUf): We clarify the limitations arising from our scale and layout assumptions and present additional failure cases involving thin geometry, reflective surfaces, and hallucinated repeated structures.

Detailed results and explanations, along with additional clarifications, are provided in our point-by-point responses. We are grateful for the reviewers’ constructive feedback, which helped us make the experiments more comprehensive and the presentation of WorldRecon clearer. We hope our revisions and responses address the concerns raised and would appreciate any further comments or suggestions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.