- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aims to do multi-view images to 3D mesh by using latent forcing and correspondence-aware attention. For visible surfaces, they voxelize the point maps to guide each voxel latent in sparse structure model and use noise for the occluded voxels. Then they do sparse attention to localize the appearance to each voxel. They evaluate on Forge3Dbench, Toys4K, OmniObject3D.
Critical strengths
- This approach seems to help even without training as in Table 3.
Critical weaknesses
- I think the paper doesn't state what the problem they are trying to solve clearly. Is it multi-view RGB or RGBD of objects to 3D mesh along with object pose, similar to SAM3D? Are these scene images or objects?
- Other baselines, Hunyuan3D-2MV, Pixal3D don't use depth as the input. It is unclear how much gain is from the approach itself. My point is it might be unfair for other baselines. It would be better if the authors can add it in Table 1 using depth for the baselines, maybe during the inference.
- The author didn't mention about GenRecon [1] which is multiview images to 3D, and could have discussed the design choices why the author choose this design over GenRecon which is inspired from IBR and is simpler. It also seems to be effective which can handle occlusion and scene generation without using point maps.
[1] GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction
Questions for authors
- How do you fix the self attention for the voxels that are invisible from the images as you can't find the correspondences. Are you letting it attend to every patch of every view? Do you see the degradation without letting the voxel see global information?
- Why averaging the image patch when there are multiple patches for each voxel? Why not let it attend all as this is sparse attention?
- Why starting from SAM3D? Isn't your approach doesn't output object pose? Isn't it be better to use Pixal3D or Trellis2?
6 comments
Response to Reviewer orNx
[Q1] Attention for voxels that are invisible in a view
Q1-a. Self-attention
The bias is applied only in the cross-attention between voxel tokens and image tokens. The self-attention among voxel tokens is unchanged, so every voxel receives information from all other voxels, including the visible ones.
Q1-b. How invisible voxels are handled in the cross-attention
A voxel that is visible in an auxiliary view gets zero bias around its projection in that view. A voxel that no view observes gets −10 on every auxiliary-view patch. It relies on the anchor view and on the other voxels through self-attention.
Q1-c. Can a voxel attend to every patch of every view
Yes. The bias is soft. It adds −10 to the attention logits outside a 3×3 window and does not mask them. The anchor view has no bias, so every voxel attends to all of its patches freely. Invisible voxels therefore keep access to global image information for completing the unobserved regions.
Q1-d. Degradation without global information
We did not see any. Results are the same from the loosest setting to the tightest one we tried, with window sizes from 5×5 to 1×1 and bias from −5 to −20. A model fine-tuned without the bias, where every voxel attends to all patches equally, also gives the same results.
| Window | LPIPS ↓ | PSNR ↑ | SSIM ↑ | CD-L1 ↓ | [email protected] ↑ |
|---|---|---|---|---|---|
| 1×1 | 0.0554 | 24.67 | 0.938 | 0.0079 | 0.850 |
| 3×3 | 0.0554 | 24.72 | 0.939 | 0.0080 | 0.845 |
| 5×5 | 0.0554 | 24.64 | 0.938 | 0.0080 | 0.849 |
| Dataset | Bias | LPIPS ↓ | PSNR ↑ | SSIM ↑ | CD-L1 ↓ | [email protected] ↑ |
|---|---|---|---|---|---|---|
| Toys4K | −5 | 0.0556 | 24.69 | 0.9382 | 0.00794 | 0.8503 |
| Toys4K | −10 | 0.0554 | 24.72 | 0.939 | 0.0080 | 0.845 |
| Toys4K | −20 | 0.0553 | 24.70 | 0.9383 | 0.00794 | 0.8495 |
| OmniObject3D | −5 | 0.1646 | 18.39 | 0.8731 | 0.01352 | 0.7274 |
| OmniObject3D | −10 | 0.1637 | 18.37 | 0.8722 | 0.01355 | 0.7265 |
| OmniObject3D | −20 | 0.1642 | 18.35 | 0.8716 | 0.01355 | 0.7264 |
We have further elaborated on the multi-view attention mechanism in Appendix C.
[Q2] Averaging of image patches
Q2-a. Where the averaging happens
Averaging is used only to build the initial appearance latent in Eq. (2). The pretrained SLAT encoder expects one feature per voxel, so we average the features of the views that observe the voxel. Figure 5(b) in Appendix C shows this step.
Q2-b. Why not attend to all patches
The model does attend to all of them. During generation, every voxel token attends to all patch tokens of all views through cross-attention. Averaging sets the starting point only.
[Q3] Choice of SAM 3D
Q3-a. Why SAM 3D
FORGE3D applies to TRELLIS-style architectures in general. We chose SAM 3D as a representative backbone of this family that was available when the project began.
Q3-b. Object pose
We do not use the pose output of SAM 3D. As in W1-b, the mesh is generated in the frame of the observed points.
Q3-c. Alternative backbone
Following the suggestion, we applied training-free latent forcing to TRELLIS.2. It improves all five metrics (table in W2-a). TRELLIS.2 encodes material attributes instead of image features, so we build the appearance seed from the observed colors and from materials estimated by SuperMat. We thank the reviewer for suggesting this experiment, which shows that the method applies beyond SAM 3D.
| Method | LPIPS ↓ | PSNR ↑ | SSIM ↑ | CD-L1 ↓ | [email protected] ↑ |
|---|---|---|---|---|---|
| TRELLIS.2 | 0.1163 | 18.98 | 0.9054 | 0.02050 | 0.4770 |
| TRELLIS.2 + training-free latent forcing | 0.0815 | 22.51 | 0.9205 | 0.00850 | 0.7711 |
Gentle Reminder
Dear Reviewer orNx,
We sincerely appreciate your time and effort in reviewing our manuscript. As the discussion period ends within a day, we would like to gently ask whether our response has addressed your concerns. If you have any further questions, please do not hesitate to let us know.
Best regards,
The Authors
Response to Reviewer orNx
We thank the reviewer for the time and effort spent on our paper. We respond to each comment below. In the revised draft, we mark our revisions as “blue”.
[W1] Problem scope
W1-a. Input
The input is multi-view RGB-D. Each view has an RGB image, a depth map, camera parameters, and a mask of the target object. The depth can come from a sensor or from an estimator such as VGGT (Table 5).
W1-b. Output and object pose
The output is a complete textured mesh of one object. We do not predict object pose as a separate output. The mesh is generated in the frame of the observed 3D points, so it is already aligned with the input views. Object pose can then be estimated from the mesh with an external estimator, as in Table 2.
W1-c. Scene images or objects
Our method is object-centric. It reconstructs one object at a time, and scene-level reconstruction is outside our scope. The input images may still show a full scene. On FORGE3DBench the object sits in a cluttered scene, and the mask selects the target.
We will state this scope at the start of Section 3.
[W2] Depth in the baselines
W2-a. Adding depth to the baselines at inference
Our central contribution is to align generated meshes with the observed geometry by encoding depth directly into the latent space of the generator. The RGB-conditioned baselines have no depth encoder, so depth cannot be given to them without changing their architecture.
Latent forcing makes this possible without training. The second row of Table 3 applies geometry and appearance forcing to SAM 3D with frozen weights. Compared with the first row, [email protected] rises from 0.524 to 0.762 and CD-L1 falls from 0.0159 to 0.0097.
W2-b. Baseline that uses depth
We also compared with RecGen[1], which takes depth through a point-map encoder.
FORGE3D outperforms RecGen on all five metrics, although both methods receive depth. These results demonstrate that initializing generation from the encoded depth is more effective than using depth as a conditioning signal while starting from pure noise. The advantage is therefore attributable to latent forcing rather than to the availability of depth.
[1] Zadaianchuk, Andrii, et al. "Reconstruction by generation: 3d multi-object scene reconstruction from sparse observations.", ECCV 2026
[W3] GenRecon
We agree that GenRecon is an efficient way to encode multi-view observations. It aggregates image features with an IBRNet-style aggregator, and the resulting features lie outside the latent space of the pretrained generator. It therefore needs additional trained modules and cannot be used without training.
In contrast, latent forcing reuses the pretrained encoders of the backbone, so it needs no new module. It already works without training: in the second row of Table 3, forcing on the frozen backbone raises [email protected] from 0.524 to 0.762 and lowers CD-L1 from 0.0159 to 0.0097.
We compared FORGE3D with GenRecon on Toys4K with 4 views. FORGE3D is better on all five metrics. We will cite and discuss GenRecon in the related work.