acceptodds
Under review as a conference paper at ICLR 2027

FORGE3D: Faithful Observation-Grounded 3D Mesh Generation via Latent Forcing

Abstract

Recent 3D mesh generators produce visually compelling assets, yet faithfully reconstructing specific objects from real-world observations remains challenging. A practical mesh generation framework must jointly preserve observed 3D geometry and pixel-aligned appearance from one or more views while plausibly completing unseen regions. However, these observations often serve only as coarse conditioning signals, allowing generated geometry and appearance to deviate from the input. Moreover, existing 3D mesh benchmarks provide limited coverage of realistic scenes and challenging observation conditions needed to assess reconstruction faithfulness in real-world scenarios. To address these gaps, we introduce FORGE3D (Faithful Observation-Grounded 3D Mesh Generation), an observation-faithful mesh generation pipeline, and FORGE3DBench, a benchmark for evaluating observation faithfulness under realistic multi-view conditions. FORGE3D directly encodes geometric and visual observations into the generative latent space through geometry and appearance latent forcing, initializing generation from observed evidence. Applied to a pretrained generator, latent forcing already improves geometric fidelity without additional training, and finetuning with correspondence-aware multi-view attention further improves appearance. FORGE3DBench features more than 300 unique assets across 20 simulated backgrounds with 24 camera viewpoints, encompassing varied object placements, lighting conditions, and occlusions. Across three benchmarks, FORGE3D achieves state-of-the-art multi-view reconstruction performance in both geometry and appearance and improves downstream 6D pose estimation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_orNxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to do multi-view images to 3D mesh by using latent forcing and correspondence-aware attention. For visible surfaces, they voxelize the point maps to guide each voxel latent in sparse structure model and use noise for the occluded voxels. Then they do sparse attention to localize the appearance to each voxel. They evaluate on Forge3Dbench, Toys4K, OmniObject3D.

Critical strengths
  • This approach seems to help even without training as in Table 3.
Critical weaknesses
  • I think the paper doesn't state what the problem they are trying to solve clearly. Is it multi-view RGB or RGBD of objects to 3D mesh along with object pose, similar to SAM3D? Are these scene images or objects?
  • Other baselines, Hunyuan3D-2MV, Pixal3D don't use depth as the input. It is unclear how much gain is from the approach itself. My point is it might be unfair for other baselines. It would be better if the authors can add it in Table 1 using depth for the baselines, maybe during the inference.
  • The author didn't mention about GenRecon [1] which is multiview images to 3D, and could have discussed the design choices why the author choose this design over GenRecon which is inspired from IBR and is simpler. It also seems to be effective which can handle occlusion and scene generation without using point maps.

[1] GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction

Questions for authors
  • How do you fix the self attention for the voxels that are invisible from the images as you can't find the correspondences. Are you letting it attend to every patch of every view? Do you see the degradation without letting the voxel see global information?
  • Why averaging the image patch when there are multiple patches for each voxel? Why not let it attend all as this is sparse attention?
  • Why starting from SAM3D? Isn't your approach doesn't output object pose? Isn't it be better to use Pixal3D or Trellis2?
6 comments
AuthorsOpenReview ↗

Response to Reviewer orNx

We thank the reviewer for the time and effort spent on our paper. We respond to each comment below. In the revised draft, we mark our revisions as “blue”.


[W1] Problem scope

W1-a. Input

The input is multi-view RGB-D. Each view has an RGB image, a depth map, camera parameters, and a mask of the target object. The depth can come from a sensor or from an estimator such as VGGT (Table 5).

W1-b. Output and object pose

The output is a complete textured mesh of one object. We do not predict object pose as a separate output. The mesh is generated in the frame of the observed 3D points, so it is already aligned with the input views. Object pose can then be estimated from the mesh with an external estimator, as in Table 2.

W1-c. Scene images or objects

Our method is object-centric. It reconstructs one object at a time, and scene-level reconstruction is outside our scope. The input images may still show a full scene. On FORGE3DBench the object sits in a cluttered scene, and the mask selects the target.

We will state this scope at the start of Section 3.


[W2] Depth in the baselines

W2-a. Adding depth to the baselines at inference

Our central contribution is to align generated meshes with the observed geometry by encoding depth directly into the latent space of the generator. The RGB-conditioned baselines have no depth encoder, so depth cannot be given to them without changing their architecture.

Latent forcing makes this possible without training. The second row of Table 3 applies geometry and appearance forcing to SAM 3D with frozen weights. Compared with the first row, [email protected] rises from 0.524 to 0.762 and CD-L1 falls from 0.0159 to 0.0097.

W2-b. Baseline that uses depth

We also compared with RecGen[1], which takes depth through a point-map encoder.

MethodLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
RecGen0.115719.480.9060.01710.482
FORGE3D (full model)0.055424.720.9390.00800.845

FORGE3D outperforms RecGen on all five metrics, although both methods receive depth. These results demonstrate that initializing generation from the encoded depth is more effective than using depth as a conditioning signal while starting from pure noise. The advantage is therefore attributable to latent forcing rather than to the availability of depth.

[1] Zadaianchuk, Andrii, et al. "Reconstruction by generation: 3d multi-object scene reconstruction from sparse observations.", ECCV 2026


[W3] GenRecon

We agree that GenRecon is an efficient way to encode multi-view observations. It aggregates image features with an IBRNet-style aggregator, and the resulting features lie outside the latent space of the pretrained generator. It therefore needs additional trained modules and cannot be used without training.

In contrast, latent forcing reuses the pretrained encoders of the backbone, so it needs no new module. It already works without training: in the second row of Table 3, forcing on the frozen backbone raises [email protected] from 0.524 to 0.762 and lowers CD-L1 from 0.0159 to 0.0097.

We compared FORGE3D with GenRecon on Toys4K with 4 views. FORGE3D is better on all five metrics. We will cite and discuss GenRecon in the related work.

MethodLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
GenRecon0.113918.710.90690.040110.4705
FORGE3D (full model)0.055424.720.9390.00800.845
AuthorsOpenReview ↗

Response to Reviewer orNx

[Q1] Attention for voxels that are invisible in a view

Q1-a. Self-attention

The bias is applied only in the cross-attention between voxel tokens and image tokens. The self-attention among voxel tokens is unchanged, so every voxel receives information from all other voxels, including the visible ones.

Q1-b. How invisible voxels are handled in the cross-attention

A voxel that is visible in an auxiliary view gets zero bias around its projection in that view. A voxel that no view observes gets −10 on every auxiliary-view patch. It relies on the anchor view and on the other voxels through self-attention.

Q1-c. Can a voxel attend to every patch of every view

Yes. The bias is soft. It adds −10 to the attention logits outside a 3×3 window and does not mask them. The anchor view has no bias, so every voxel attends to all of its patches freely. Invisible voxels therefore keep access to global image information for completing the unobserved regions.

Q1-d. Degradation without global information

We did not see any. Results are the same from the loosest setting to the tightest one we tried, with window sizes from 5×5 to 1×1 and bias from −5 to −20. A model fine-tuned without the bias, where every voxel attends to all patches equally, also gives the same results.

WindowLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
1×10.055424.670.9380.00790.850
3×30.055424.720.9390.00800.845
5×50.055424.640.9380.00800.849
DatasetBiasLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
Toys4K−50.055624.690.93820.007940.8503
Toys4K−100.055424.720.9390.00800.845
Toys4K−200.055324.700.93830.007940.8495
OmniObject3D−50.164618.390.87310.013520.7274
OmniObject3D−100.163718.370.87220.013550.7265
OmniObject3D−200.164218.350.87160.013550.7264

We have further elaborated on the multi-view attention mechanism in Appendix C.


[Q2] Averaging of image patches

Q2-a. Where the averaging happens

Averaging is used only to build the initial appearance latent in Eq. (2). The pretrained SLAT encoder expects one feature per voxel, so we average the features of the views that observe the voxel. Figure 5(b) in Appendix C shows this step.

Q2-b. Why not attend to all patches

The model does attend to all of them. During generation, every voxel token attends to all patch tokens of all views through cross-attention. Averaging sets the starting point only.


[Q3] Choice of SAM 3D

Q3-a. Why SAM 3D

FORGE3D applies to TRELLIS-style architectures in general. We chose SAM 3D as a representative backbone of this family that was available when the project began.

Q3-b. Object pose

We do not use the pose output of SAM 3D. As in W1-b, the mesh is generated in the frame of the observed points.

Q3-c. Alternative backbone

Following the suggestion, we applied training-free latent forcing to TRELLIS.2. It improves all five metrics (table in W2-a). TRELLIS.2 encodes material attributes instead of image features, so we build the appearance seed from the observed colors and from materials estimated by SuperMat. We thank the reviewer for suggesting this experiment, which shows that the method applies beyond SAM 3D.

MethodLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
TRELLIS.20.116318.980.90540.020500.4770
TRELLIS.2 + training-free latent forcing0.081522.510.92050.008500.7711
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer orNx,

We sincerely appreciate your time and effort in reviewing our manuscript. As the discussion period ends within a day, we would like to gently ask whether our response has addressed your concerns. If you have any further questions, please do not hesitate to let us know.

Best regards,
The Authors

Reviewer_Um6COpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the challenge of achieving observation-faithful 3D mesh reconstruction in real-world scenarios. The authors propose a generative mesh pipeline named FORGE3D, along with a dedicated benchmark, FORGE3DBench, for evaluating multi-view faithfulness. The core mechanism directly incorporates observed geometry and appearance into the generative source through “latent forcing.” The authors conduct extensive evaluations across three dimensions: generation faithfulness, novel-view synthesis capability, and downstream 6D pose estimation. Quantitative results demonstrate substantial improvements over established baselines, indicating that the proposed approach consistently outperforms existing methods. Finally, comprehensive ablation studies confirm the individual contributions of each architectural component.

Critical strengths

1.Strong motivation and practical impact. The motivation driving this research is remarkably compelling. Fidelity in 3D mesh generation remains a bottleneck, and addressing this problem holds substantial industrial value for 3D reconstruction and 3D generative world models. The core research question is of paramount importance to the 3D mesh generation community, as maintaining faithfulness to observations is central to practical deployment. Evaluating multi-view reconstruction in both geometry and appearance, alongside downstream tasks such as 6D pose estimation, connects academic research with real-world industrial applications.

2.Superior empirical performance. The experiments demonstrate outstanding performance. The authors provide a comprehensive supplementary website showcasing 3D reconstruction results, where FORGE3D significantly outperforms existing state-of-the-art baselines. Importantly, the evaluation goes beyond reconstruction metrics: Table 2 uses the generated meshes in an Any6D/FoundationPose-based 6D pose estimation pipeline without requiring ground-truth CAD models. On HO3D, FORGE3D achieves an ADD score of 76.0 and an AR of 73.6, compared with the strongest baseline values of 47.0 and 53.5, respectively. These results demonstrate a substantial margin of improvement.

3.Professional presentation. The paper is well-written and maintains high professional standards. The underlying logic is clear and easy to follow, with thorough justifications for both the motivation and the technical design choices. The authors also provide a comprehensive website for convenient access to their 3D results. Its Method, Proof, Compare, Inspect, FORGE3DBench, and Notes sections are clearly organized and provide sufficient detail.

4.A valuable benchmark contribution and extensive evaluation. The contribution of both a robust generation pipeline and a rigorous benchmark is a major asset to the field. FORGE3DBench supports observation-faithful 3D mesh generation evaluation across 120 multi-object scenes assembled from more than 300 unique assets, representing a commendable scale. The evaluation spans 638 distinct generation scenarios and includes comparisons with modern state-of-the-art baselines. The experiments evaluate geometry and appearance on Toys4K, OmniObject3D, and FORGE3DBench against ReconViaGen, Pixal3D, Amodal3R, Hunyuan3D-2mv, and CUPID. FORGE3D achieves the best reported values across all five metrics on each dataset in Table 1. For example, on FORGE3DBench, it achieves LPIPS of 0.0912, PSNR of 21.73, and [email protected] of 0.701, compared with the strongest baseline values of 0.1644, 17.56, and 0.401, respectively.

Critical weaknesses

1.The intuition behind latent forcing needs further explanation. The mechanism by which latent forcing yields such a pronounced performance improvement remains somewhat counterintuitive. A deeper conceptual explanation of why this particular initialization prevents degradation over time would strengthen the manuscript. Table 3 also reveals an important distinction: training-free forcing improves geometry but reduces PSNR from 20.24 to 19.36 and SSIM from 0.912 to 0.903. Thus, adding training-free forcing to the pretrained model actually lowers these appearance metrics. A more detailed explanation of this geometry–appearance difference, together with an analysis of when observed information is retained or altered during generation, would strengthen the interpretation of the core mechanism.

2.Inference-time computational costs are insufficiently discussed. The manuscript lacks a detailed discussion of efficiency. It is unclear how the initialization method affects memory usage and execution speed, and a comparative computational-efficiency analysis against existing baselines is missing. Although the paper reports its training resources—four NVIDIA A100 GPUs and approximately four days—it does not provide a comparable analysis of inference latency or peak GPU memory.

Questions for authors

1.Table 3 shows that training-free latent forcing improves geometry but reduces PSNR and SSIM. Why does one-time initialization produce this geometry–appearance trade-off, and how does subsequent fine-tuning mitigate it?

2.Table 5 shows that replacing ground-truth depth with VGGT point maps reduces [email protected] from 0.846 to 0.570. Which geometric errors most strongly affect latent forcing, and can the model correct inaccurate observations rather than propagate them into the final mesh?

3.What are the end-to-end inference latency and peak GPU memory relative to the pretrained backbone and competing methods under matched settings? How do these costs scale from one to eight input views, including observation aggregation and correspondence-aware attention?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Um6C

We thank the reviewer for the time and effort spent on our paper. We respond to each comment below. In the revised draft, we mark our revisions as “blue”.


[W1, Q1] Latent forcing and the geometry–appearance trade-off

Q1-a. Intuition behind latent forcing

Our intuition is that latent forcing improves alignment with the observations by starting generation from their encoded geometry and appearance. A generator that takes observations only as conditions starts from pure noise and has to recover the observed shape during sampling. With latent forcing, the observed part is already encoded in the starting latent, and the pretrained prior only has to complete the rest.

This is possible because we reuse the encoders of the pretrained model (E_SS and E_SLAT), which are normally discarded at inference. The gain on the frozen backbone in Table 3, where [email protected] rises from 0.524 to 0.762, supports this intuition.

Q1-b. Why training-free forcing lowers PSNR and SSIM

The drop comes from the appearance forcing stage. Geometry forcing alone improves all five metrics on the frozen backbone (second row below). The drop appears when appearance forcing is added (third row). We attribute it to a distribution mismatch, as the pretrained SLAT flow was not trained to start from observation-derived latents.

Setting (Toys4K, 4 views)LPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
Pretrained model0.103020.240.9120.01590.524
+ geometry forcing0.083121.330.9180.01120.714
+ appearance forcing (training-free forcing in Table 3)0.090519.360.9030.00970.762
+ delayed release0.076620.730.9160.01060.769
+ fine-tuned SLAT flow0.068123.000.9270.00950.771
+ fine-tuned SLAT and SS flows0.055424.720.9390.00800.845

Q1-c. How fine-tuning mitigates the trade-off

Fine-tuning addresses the mismatch directly. The flows are trained to start from forced latents, and both appearance and geometry improve (last two rows).

The trade-off can also be reduced without training. We experiment with delayed release, where we hold the forced tokens at their encoded values during the early part of sampling and release them later. This improves all five metrics over the pretrained model (fourth row).

Q1-d. When observed information is retained or altered

We added qualitative examples to Appendix D (Figure 6). FORGE3D reconstructs observed lighting and texture faithfully, but it may lose fine details such as printed text and logos. We attribute this partly to the patch-level image features, which are too coarse to capture such patterns.


[Q2] Estimated geometry

Q2-a. Which geometric errors affect latent forcing most

A key source of error is inaccurate depth near object boundaries, where flying pixels put misplaced surface points into the geometry seed. We attribute the drop on Toys4K partly to its synthetic, unlit images, which are out of distribution for VGGT.

On the real images of OmniObject3D, with the same random seed for all methods, FORGE3D with VGGT depth remains the best on LPIPS, PSNR, SSIM, and [email protected], and is on par with Pixal3D on CD-L1.

MethodLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
ReconViaGen0.203816.800.8470.02960.369
Pixal3D0.195016.870.8570.02590.358
Amodal3R0.211416.000.8240.03020.294
Hunyuan3D-2mv0.207816.600.8580.02710.339
CUPID0.218416.100.8470.03840.320
FORGE3D + VGGT depth0.169618.140.8710.02580.414

Q2-b. Can the model correct inaccurate observations

Partly. To reduce the propagation of unreliable observations, we interpolate the observation-derived latents with noise according to the prediction confidence of VGGT when we initialize the SS flow. This lets the model denoise uncertain regions with its own prior. With the same VGGT predictions on Toys4K, both appearance and geometric metrics improve.

MethodLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
Ours + VGGT0.082922.290.9240.02060.570
Ours + VGGT Confidence0.078122.720.9270.01780.603
AuthorsOpenReview ↗

Response to Reviewer Um6C

[W2, Q3] Inference cost

Q3-a. Latency and peak GPU memory

We measured end-to-end latency and peak GPU memory on 50 Toys4K samples with a single RTX 6000 GPU.

Latency (s)

Method1 view2 views4 views8 views
ReconViaGen7.8 ± 1.77.6 ± 1.28.2 ± 1.59.2 ± 2.5
Pixal3D34.8 ± 22.333.6 ± 21.635.8 ± 21.041.3 ± 22.3
Amodal3R10.4 ± 3.39.8 ± 3.210.0 ± 3.110.3 ± 3.5
Hunyuan3D-2mv26.1 ± 0.828.6 ± 0.734.0 ± 0.9×
CUPID7.5 ± 1.010.5 ± 1.516.3 ± 2.226.9 ± 1.8
FORGE3D4.5 ± 1.34.4 ± 1.14.9 ± 1.36.7 ± 3.4

Peak GPU memory (GB)

Method1 view2 views4 views8 views
ReconViaGen20.621.320.621.4
Pixal3D40.847.646.948.0
Amodal3R12.713.013.013.0
Hunyuan3D-2mv16.216.216.2×
CUPID13.513.814.213.5
FORGE3D23.522.022.422.7

FORGE3D has the lowest latency at every view count. Its peak memory is higher than that of four of the five baselines, and it fits on a single GPU. Hunyuan3D-2mv does not support eight views.

Q3-b. Scaling from one to eight views

From 1 to 8 views, the latency of FORGE3D grows from 4.5 s to 6.7 s, and its peak memory stays between 22.0 and 23.5 GB.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer Um6C,

We sincerely appreciate your time and effort in reviewing our manuscript. As the discussion period ends within a day, we would like to gently ask whether our response has addressed your concerns. If you have any further questions, please do not hesitate to let us know.

Best regards,
The Authors

Reviewer_fxtqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper does observation-faithful textured-mesh reconstruction from calibrated views with masks plus depth or estimated point maps, keeping observed surfaces while completing unseen regions. FORGE3D builds on the two-stage SAM 3D Objects generator and initialises its flows from encoded observations instead of noise. The sparse-structure flow starts from an occupancy grid aggregated from observed points; the SLAT flow starts from encoded DINOv2 features with noise at unobserved voxels. Both are fine-tuned with flow matching, and SLAT also gets auxiliary-view tokens, ray embeddings and a correspondence-biased attention. The paper introduces FORGE3DBench, with more than 300 assets in simulated scenes covering clutter, lighting changes and robot-grasp occlusion, captured from 24 calibrated views. FORGE3D reports the best point estimates on the three reconstruction benchmarks and the best pose metrics, and its ablation shows geometry improving from training-free forcing.

Critical strengths
  1. Latent forcing is a simple use of the backbone's own encoders, and it pays off at fixed weights: CD-L1 drops from 0.0159 to 0.0097 and [email protected] rises from 0.524 to 0.762, with the PSNR drop reported openly. This supports the observation-initialization package without claiming it isolates the injection mechanism.
  2. FORGE3DBench covers clutter, illumination and robot-grasp occlusion with calibrated views and geometric annotations. The pose probe is an extra practical check, with large margins over the best baseline on both HO3D metrics.
Critical weaknesses
  1. The comparative scope is under-specified. One table contrasts ground-truth depth with estimated VGGT point maps on a single dataset, and the VGGT row is worse than the main table's Pixal3D row on all five metrics. But no fully matched cross-table protocol is established, and there is no per-method input inventory for the main tables. Depth is a legitimate input here; the problem is reading the ranking as equal-information superiority.
  2. Attention is not isolated. The paper describes auxiliary-view tokens, ray embeddings and the attention bias, but does not say which are active in each ablation row, and there is no bias-on/off control. The combined gain is supported; attribution to the bias alone is not. Uncertainty is also missing for narrow margins in some tables.
Questions for authors
  1. Which depth or point-map source, camera information and mask type does each method receive, and which seed produced the single-view results? If an RGB-only claim is intended, a matched comparison on one representative setting would help; otherwise scoping the claim is enough.
  2. Which auxiliary-view tokens, ray embeddings and attention biases are active in each ablation row? A matched bias-on/off comparison would separate correspondence bias from the rest of the fine-tuning.
  3. For appearance metrics, how are masks and backgrounds handled and which held-out views are used? For geometry, how are meshes aligned and sampled? And what counts as one of the 638 generation scenarios?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fxtq

We thank the reviewer for the time and effort spent on our paper. We respond to each comment below. In the revised draft, we mark our revisions as “blue”.


[W1, Q1] Input requirements and reconstruction with estimated depth

Q1-a. Inputs of each method

The table lists what each method receives in Table 1. We will clarify the inputs for each method in the manuscript.

MethodImagesMaskCameraDepth
ReconViaGen4 viewsobject✗✗
Amodal3R4 viewsamodal + visible✗✗
Hunyuan3D-2mv4 viewsobject✗✗
CUPID4 viewsobject✗✗
Pixal3D4 viewsobjectground truth✗
FORGE3D4 viewsobjectground truthground truth

Q1-b. Scope of the claim

We do not claim superiority under equal information. Depth is often available in practice, from a sensor or from an estimator. The compared generators do not take it as a geometric constraint, and adding it would require changing their architectures. Our claim is that FORGE3D can use it on top of a pretrained generator without such changes. We will state this scope in Section 5.1 and in the caption of Table 1.

Q1-c. The VGGT row is below Pixal3D

We attribute this partly to Toys4K’s synthetic, unlit images with removed backgrounds, which differ from VGGT’s training distribution and reduce the accuracy of its estimated geometry.

To test estimated depth on real images, we ran FORGE3D with VGGT depth on OmniObject3D.

MethodLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
ReconViaGen0.203816.800.8470.02960.369
Pixal3D0.195016.870.8570.02590.358
Amodal3R0.211416.000.8240.03020.294
Hunyuan3D-2mv0.207816.600.8580.02710.339
CUPID0.218416.100.8470.03840.320
FORGE3D with VGGT depth0.169618.140.8710.02580.414

With the same random seed for all methods, FORGE3D is best on LPIPS, PSNR, SSIM, and [email protected] with estimated depth, and on par with Pixal3D on CD-L1.

Q1-d. Seed of the single-view results

The single-view results (Table 6 in Appendix B) use the front view. The geometry seed is built from the ground-truth depth of that view. The random seed is 42.


[W2, Q2] Attention components and the bias

Q2-a. Components in each ablation row

All configurations are listed in the table below (Toys4K, 4 views). "Training" means that the flow is fine-tuned, and otherwise the pretrained flow is used. "MV attn." means that the patch tokens of all input views are concatenated and attended jointly.

#SS forcingSS trainingSLAT forcingSLAT trainingMV attn.Ray emb.BiasLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
1✗✗✗✗✗✗✗0.103020.240.9120.01590.524
2✓✗✗✗✗✗✗0.083121.330.9180.01120.714
3✓✗✓✗✗✗✗0.090519.360.9030.00970.762
4✓✗✓✓✓✓✓0.068123.000.9270.00950.771
5✓✓✗✗✗✗✗0.063623.380.9330.00810.822
6✓✓✗✗✓✗✗0.063123.520.9330.00810.822
7✓✓✗✗✓✗✓0.061423.640.9340.00810.822
8✓✓✓✗✓✗✗0.079720.700.9120.00820.839
9✓✓✓✗✓✗✓0.079820.700.9120.00820.839
10✓✓✓✓✓✓✗0.055224.710.9380.00800.848
11✓✓✓✓✓✓✓0.055424.720.9390.00800.845

Rows 1, 3, 4, and 11 are the four rows of Table 3. Rows 1 and 3 use the original single-view attention of SAM 3D, with no multi-view attention, ray embeddings, or bias. Rows 4 and 11 use all three components.

Q2-b. Matched bias on and off

Rows 10 and 11 differ only in the bias, with the same fine-tuning budget. The two models perform on par. So the gains in Table 3 come from latent forcing and fine-tuning.

The bias gives a modest improvement when the SLAT flow is frozen and appearance forcing is disabled (rows 6 and 7). We will revise the paper to present the bias as optional when the SLAT flow is fine-tuned.

AuthorsOpenReview ↗

Response to Reviewer fxtq

[W2] Uncertainty

Table 1 in the revised manuscript now reports the mean and standard deviation over three random seeds, for every method on all three datasets. The margins are far larger than the standard deviations. On Toys4K, FORGE3D reaches 24.70 ± 0.01 dB in PSNR and 0.8447 ± 0.0003 in [email protected]. The strongest baseline, Pixal3D, reaches 23.37 ± 0.04 dB and 0.7494 ± 0.0016.

The standard deviations of FORGE3D are also among the smallest in the table. In CD-L1 they are more than ten times smaller than those of every baseline on all three datasets. This follows from latent forcing. The baselines start from pure noise, so the seed changes the whole starting point. FORGE3D starts from the encoded observations, and random noise remains only on the voxels that no view observes. The seed therefore changes only a small part of the starting point.


[Q3] Evaluation protocol

Q3-a. Appearance

We evaluate the object only. Each image is composited over a white background with the object mask. Input views are rendered from the input cameras and compared with the masked input images. Held-out views are 24 views that are not among the inputs. The reported value is averaged over all of these views, 28 per object on Toys4K including the 4 input views.

Q3-b. Geometry

All meshes are scored in the ground-truth frame. FORGE3D and Pixal3D produce meshes in this frame and are not aligned. The other baselines are scaled to the ground-truth bounding box, aligned by an orientation search, and refined by ICP. We then sample 100,000 surface points from each mesh and compute CD-L1 and [email protected].

Q3-c. Generation scenario

Each generation scenario corresponds to reconstructing one target object within a scene. A scene containing multiple objects therefore accounts for multiple scenarios.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer fxtq,

We sincerely appreciate your time and effort in reviewing our manuscript. As the discussion period ends within a day, we would like to gently ask whether our response has addressed your concerns. If you have any further questions, please do not hesitate to let us know.

Best regards,
The Authors

Reviewer_bWxHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FORGE3D uses a two-stage 3D generation framework. In the sparse-structure stage, encoded partial occupancy replaces the initial noise to guide geometry generation. In the SLAT stage, multi-view image features at observed voxels and noise at unobserved voxels are encoded to initialize geometry and appearance generation. Both flows are fine-tuned to complete partial observations, with correspondence-aware multi-view attention added to the SLAT flow.

The paper also introduces FORGE3DBench, covering realistic scene backgrounds, lighting variation, and occlusion. Experiments show that training-free latent forcing substantially improves geometric fidelity, while fine-tuning further improves texture quality. Comparisons across three benchmarks, view-scaling results, and downstream 6D pose estimation support the effectiveness of the complete method.

Critical strengths
  1. A well-developed benchmark and evaluation framework for multi-view-conditioned 3D generation. FORGE3DBench includes diverse assets in realistic simulated scenes with varied lighting, clutter, and occlusion, together with calibrated views, depth, object masks, and ground-truth meshes (Section 4). The evaluation covers both geometric fidelity and texture/appearance quality through complementary metrics (Section 5.1). This provides a useful framework for assessing whether generated assets faithfully reconstruct the observed objects under challenging conditions, making the benchmark a valuable contribution.

  2. Convincing empirical improvements and effective use of additional views. The experiments provide substantial evidence for the effectiveness of the complete method: Table 1 shows consistent improvements in both geometry and appearance over the compared baselines across three benchmarks, supported by qualitative comparisons in Figure 4. The view-scaling results in Table 4 show further improvements as the number of input views increases from one to two, four, and eight. These results convincingly demonstrate improved reconstruction quality and indicate that the method effectively exploits complementary information from additional views.

Critical weaknesses
  1. The treatment of unobserved geometry in latent forcing is insufficiently specified. Section 3.2 defines the partial occupancy grid OO by voxelizing observed 3D points and uses ESS(O)E_{\mathrm{SS}}(O) to replace the geometry flow's initial noise (Eq. (1)), but does not specify the values assigned to voxels outside the observed surface or how unobserved regions are distinguished from known empty space. With a single input view, much of the object's surface may be unobserved. If all remaining voxels are set to zero, the encoder input conflates missing geometry with empty space. Unlike the appearance initialization in Eq. (2), the geometry initialization provides no explicit description of an observation mask or separate treatment of unknown regions. This omission concerns the central initialization mechanism and makes it difficult to assess how it supports faithful reconstruction and completion, particularly in the training-free setting.

  2. The contribution of correspondence-aware multi-view attention is not isolated experimentally. Section 3.3 introduces a geometric attention bias, auxiliary-view tokens, and camera embeddings alongside SLAT flow fine-tuning. However, Table 3 only reports cumulative latent forcing and flow fine-tuning settings, without specifying the attention configuration in each row or comparing otherwise identical models with and without correspondence bias. The reported appearance gains therefore do not establish the independent benefit of the proposed attention mechanism over ordinary multi-view attention. The paper also provides no sensitivity analysis for the fixed 3×33\times3 neighborhood and −10-10 bias. Since the geometric bias itself has no learned parameters, a controlled comparison on a frozen backbone would help establish whether its benefit requires fine-tuning; no such comparison is reported. This limits the evidence for the mechanism's contribution and the justification of its design choices.

  3. Minor: the evaluation does not break down performance by occlusion severity. FORGE3DBench includes varied occlusion conditions, but Table 1 reports aggregate results without showing how reconstruction quality, particularly geometric fidelity, varies with the degree of occlusion. Reporting geometry metrics across occlusion levels would strengthen the analysis of when observation-grounded latent forcing is most effective and where completion becomes difficult as less of the object is visible. Such an analysis could also identify useful directions for subsequent research. I consider this a secondary limitation of the experimental analysis, with less weight in my assessment than W1 and W2.

Questions for authors
  1. Construction of the geometry seed (related to W1). How exactly is OO in Eq. (1) constructed from a single input view? Please specify the values assigned to voxels containing observed surface points, known free space, and unobserved regions behind the visible surface, and whether any visibility or observation mask is used. If unobserved regions are zero-filled, how does the geometry flow distinguish missing geometry from empty space, especially when latent forcing is applied to the frozen pretrained backbone without fine-tuning?

  2. Independent effect and design of correspondence attention (related to W2). Which attention components are enabled in each row of Table 3? Can you provide a controlled comparison with and without correspondence bias, keeping latent forcing, input views, camera embeddings, and the fine-tuning budget fixed, together with sensitivity results for neighborhood size and bias magnitude? Can you also compare ordinary multi-view attention against correspondence-biased attention on a frozen backbone, using the shared pretrained image projection and explicitly stating how the newly introduced camera-embedding projection is handled? These comparisons would clarify the bias's independent contribution and whether fine-tuning is necessary for its effectiveness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bWxH

We thank the reviewer for the time and effort spent on our paper. We respond to each comment below. In the revised draft, we mark our revisions as “blue”.


[W1, Q1] Construction of the geometry seed

Thank you for pointing out that this part was underspecified. We answer each point below and will add these details to Section 3.2.

Q1-a. Values assigned to voxels

O is a binary 64³ occupancy grid, as in SAM 3D. A voxel that contains an observed surface point is 1. All other voxels are 0. This includes both known free space and unobserved regions behind the visible surface. For a single input view, we back-project the masked depth into 3D and voxelize the resulting points.

Q1-b. Visibility or observation mask

The geometry seed uses no 3D visibility mask and no unknown-state channel. The only mask is the 2D object mask. It selects the depth pixels that are back-projected, and it is given to the backbone as conditioning, as in SAM 3D.

This differs from Eq. (2) for a structural reason. The appearance latent has one token per voxel, so we can force the observed voxels and leave the others as noise. The geometry latent is a dense low-resolution grid that encodes the whole occupancy grid jointly, so a per-voxel mask cannot be applied to it.

Q1-c. How the flow distinguishes missing geometry from empty space

As noted, we assign 0 to both empty and unobserved voxels as the seed. The flow decides which of them to fill.

Such seed serves only as a starting point of the flow. Zero-valued voxels are not explicitly constrained during sampling, allowing the flow to predict occupancy in initially empty regions. This is possible because the pretrained backbone has already learned to infer complete object geometry from a single view. Thus, the flow can fill in unobserved regions with plausible geometry.

We tested this scheme on the frozen pretrained backbone and found that geometry forcing alone improves both geometry metrics on Toys4K. We further fine-tune the geometry flow to generate complete occupancy from the encoded partial occupancy, allowing FORGE3D to learn this completion directly.

We also explored an alternative 3D voxel design, where we assign 0.5 instead of 0 to surface voxels that no camera observes. We evaluated this design on Toys4K with artificial occlusion, but it degraded geometry quality. We attribute this to the pretrained encoder being trained only on binary occupancy grids, making the intermediate value out of distribution. We therefore retain the binary seed and the latent initialization.

Frozen backboneCD-L1 ↓[email protected] ↑
Pretrained0.01590.524
+ geometry forcing0.01120.714
Unobserved voxelsCD-L1 ↓[email protected] ↑
0 (ours)0.01230.717
0.50.01380.675

[W2, Q2] Independent effect of the correspondence bias

Summary

We agree that table 3 reports cumulative settings and does not isolate the bias. We ran the requested comparisons on Toy4k. The results indicate that the gains in Table 3 come from latent forcing and fine-tuning.

The bias gives a modest improvement when the SLAT flow is frozen and appearance forcing is disabled (rows 6 and 7). With appearance forcing, or after fine-tuning, models with and without the bias perform on par. We will revise the paper to present the bias as optional when the SLAT flow is fine-tuned.

All configurations are listed in the table below. "Training" means that the flow is fine-tuned, and otherwise the pretrained flow is used. "MV attn." means that the patch tokens of all input views are concatenated and attended jointly, “Bias” means that the soft mask was active, and “Ray emb” means camera ray embeddings were used for auxiliary views.

#SS forcingSS trainingSLAT forcingSLAT trainingMV attn.Ray emb.BiasLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
1✗✗✗✗✗✗✗0.103020.240.9120.01590.524
2✓✗✗✗✗✗✗0.083121.330.9180.01120.714
3✓✗✓✗✗✗✗0.090519.360.9030.00970.762
4✓✗✓✓✓✓✓0.068123.000.9270.00950.771
5✓✓✗✗✗✗✗0.063623.380.9330.00810.822
6✓✓✗✗✓✗✗0.063123.520.9330.00810.822
7✓✓✗✗✓✗✓0.061423.640.9340.00810.822
8✓✓✓✗✓✗✗0.079720.700.9120.00820.839
9✓✓✓✗✓✗✓0.079820.700.9120.00820.839
10✓✓✓✓✓✓✗0.055224.710.9380.00800.848
11✓✓✓✓✓✓✓0.055424.720.9390.00800.845
AuthorsOpenReview ↗

Response to Reviewer bWxH

Q2-a. Attention components in each row of Table 3

Rows 1, 3, 4, and 11 are the four rows of Table 3. Rows 1 and 3 use the original single-view attention of SAM 3D, with no multi-view attention, ray embeddings, or bias. Rows 4 and 11 use all three components. The three components exist in the SLAT flow only. The SS flow keeps the original anchor-view conditioning in every row.

Q2-b. Controlled comparison with and without the bias

Rows 10 and 11 differ only in the bias. Latent forcing, input views, ray embeddings, and the fine-tuning budget of 60k iterations at batch size 12 are the same. The two models perform on par.

Q2-c. Sensitivity to neighborhood size and bias magnitude

We fine-tuned the SLAT flow with window sizes 1×1, 3×3, and 5×5 at bias −10. We additionally evaluate bias magnitudes of −5, −10, and −20, keeping the window size fixed at 3×3. We evaluated each checkpoint on toys4k.

WindowLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
1×10.055424.670.9380.00790.850
3×30.055424.720.9390.00800.845
5×50.055424.640.9380.00800.849
DatasetBiasLPIPS ↓PSNR ↑SSIM ↑CD-L1 ↓[email protected] ↑
Toys4K−50.055624.690.93820.007940.8503
Toys4K−100.055424.720.9390.00800.845
Toys4K−200.055324.700.93830.007940.8495
OmniObject3D−50.164618.390.87310.013520.7274
OmniObject3D−100.163718.370.87220.013550.7265
OmniObject3D−200.164218.350.87160.013550.7264

Performance remains stable across the tested settings.

Q2-d. Ordinary versus biased multi-view attention on a frozen backbone

Rows 6 to 9 use the frozen pretrained SLAT flow with multi-view attention. The tokens of the additional views pass through the shared pretrained image projection. The camera-embedding projection is zero-initialized, so it outputs zero in the frozen flow. The SS flow is the fine-tuned one in these rows. It does not use the bias, and the geometry is the same within each pair.

Without appearance forcing, the bias improves appearance over ordinary multi-view attention (rows 6 and 7). PSNR rises from 23.52 to 23.64 and LPIPS falls from 0.0631 to 0.0614.

With appearance forcing, the two rows are the same (rows 8 and 9). Appearance forcing writes the observed features into the initial latent of each voxel, so the model no longer relies on the bias once forcing is on.

On the frozen SLAT flow, appearance forcing improves geometry ([email protected] from 0.822 to 0.839) but lowers PSNR, because the pretrained flow was not trained to start from a forced latent. Fine-tuning removes this trade-off and gives the best results (rows 10 and 11).


[W3] Reconstruction quality under occlusion in FORGE3DBench

To analyze reconstruction quality under different degrees of occlusion, we stratify FORGE3DBench by the anchor view’s occlusion ratio. We evaluate all baseline methods and report geometric fidelity across three occlusion ranges.

Method0–25%: CD-L1 ↓0–25%: [email protected] ↑25–50%: CD-L1 ↓25–50%: [email protected] ↑50%+: CD-L1 ↓50%+: [email protected] ↑
ReconViaGen0.02220.40730.03440.31460.04780.2066
Pixal3D0.04730.26560.05000.24590.06980.1461
Amodal3R0.02560.34550.02660.32590.02620.3206
Hunyuan3D-2mv0.02400.38380.03620.27880.03530.2383
CUPID0.04870.24070.05590.21440.08140.1331
FORGE3D0.01260.70620.01550.63720.01620.6613

FORGE3D achieves the lowest CD-L1 and highest [email protected] across all three ranges, demonstrating robustness under the evaluated occlusion conditions. Even when more than half of the object is occluded in the anchor view, FORGE3D maintains substantially better geometric fidelity than the compared methods, supporting the effectiveness of our method for reconstructing partially visible objects.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer bWxH,

We sincerely appreciate your time and effort in reviewing our manuscript. As the discussion period ends within a day, we would like to gently ask whether our response has addressed your concerns. If you have any further questions, please do not hesitate to let us know.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

Dear reviewers and Area Chair,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

As reviewers (R1: bWxH, R2: fxtq, R3: Um6C, R4: orNx) highlighted, we address the important problem (R3) of observation-faithful 3D mesh generation, and our method, latent forcing, is simple (R2) and effective even without training (R1, R2, R4). Reviewers pointed out that FORGE3DBench is a well-developed (R1) and valuable (R1, R3) benchmark. We achieve convincing improvements across three benchmarks (R1, R3) and downstream 6D pose estimation (R2, R3), with effective use of additional views (R1).

We appreciate your constructive comments on our manuscript. In response to the comments, we provide the following additional discussions and experiments in our responses and the revised manuscript:

  • Mechanism of latent forcing and construction of the geometry seed (R3-W1/Q1, R1-W1/Q1)
  • Training-free latent forcing on SAM 3D and TRELLIS.2 (R3-Q1, R4-Q3)
  • Extended component ablation and elaboration of multi-view attention (R1-W2/Q2, R2-W2/Q2, R4-Q1; Appendix C)
  • Inference latency and memory, and results over three random seeds (R3-W2/Q3, R2-W2; Table 1)
  • Further comparison with RecGen and GenRecon (R4-W2/W3)
  • Results with predicted (VGGT) depth on the real images of OmniObject3D (R2-W1/Q1, R3-Q2)
  • Input setting of each method, problem scope, and evaluation protocol (R2-W1/Q1/Q3, R4-W1)
  • Reconstruction quality under different occlusion levels on FORGE3DBench (R1-W3)

In the revised manuscript, we highlighted the updates in "Blue" for your convenience. We sincerely hope our responses and revision address all the reviewers' concerns.

Thank you very much,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.