acceptodds
Under review as a conference paper at ICLR 2027

UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space

Abstract

Generating complete 3D assets from sparse, unposed images requires preserving observed geometry while completing unseen regions. Joint reconstruction and generation offers a way to address both goals: reconstruction recovers the geometry observed in the input images, while generation contributes learned shape priors. However, combining camera-relative reconstruction with canonical-space generation introduces a coordinate mismatch that makes direct geometric interaction more difficult. Moreover, conditioning each 3D location on all views without accounting for their relevance may mix useful geometric evidence with unrelated image content, hindering local detail recovery. We present UniForge3D, which jointly models reconstruction and generation in a shared anchor-guided space, resolving their coordinate mismatch. This space follows the anchor view's azimuth while keeping the object upright. The two branches exchange features bidirectionally to predict per-view point maps and a complete sparse voxel structure. We further condition point-map prediction on the generated structure to establish pixel-to-voxel correspondences and select the most informative input view for each voxel. Features from the selected views then guide detailed shape generation, helping preserve observed geometric details. Our method achieves state-of-the-art appearance and geometry results on Toys4K and GSO, generating complete 3D assets with improved fidelity.

Primary area:
generative models
Keywords:
3D Reconstruction, 3D Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_kbHzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a strong 3D asset reconstruction method that jointly performs 3D reconstruction and generation from sparse, unposed multi-view images. Specifically, it unify reconstruction and generation in a shared anchor-guided coordinate space, enabling bidirectional feature exchange between a VGGT-based reconstruction branch and a TRELLIS.2-based 3D generator. Experiments show state-of-the-art 3D object reconstruction quality.

Critical strengths

This paper aims at the timely 3D object reconstruction/generation subject and achieves state-of-the-art results in geometry and appearance fidelity. The difference of relative camera space and canonical object space is a long lasting issue when unifying 3D reconstruction and generation, which is addressed here using anchor-guided shared space. The idea is novel and results are good, which might be a good foundation work if the qualitative examples are not extremely selected cases.

Critical weaknesses

The method currently operates on synthetic renderings, it may face significant degradation when it comes to real-world unposed images (with background filtered).

Questions for authors

I would like to know how is the training views prepared, especially the sample distribution. Does each object randomly sample from the same distribution, or from a larger range.

2 comments

We thank the reviewer for recognizing the shared-space formulation and the strong geometry and appearance results.

Generalization to real-world images.

Our method generalizes to unposed real-world images, with results presented in Fig. 5 and the supplementary video.

Training view preparation.

For each object, we randomly sample two azimuth angles and rotate the asset about its vertical axis to each corresponding orientation. For each rotated asset, we render six candidate anchor views and 48 images from randomly sampled viewpoints. Each training sample uses one randomly selected anchor view and three additional views.

Training augmentations.

To simulate variations in real-world image capture and background removal, we apply the following augmentations:

  1. Rotation, scaling, and translation. To simulate camera roll and changes in framing, we apply random in-plane rotation, scaling, and translation about the foreground center.

  2. Lighting and exposure perturbations. To simulate different lighting and exposure conditions, we randomly adjust white balance, brightness, contrast, saturation, gamma, and black and white clipping levels.

  3. Lens distortion. To simulate deformation introduced by real camera lenses, we apply radial and tangential distortion using randomly sampled coefficients. Views within the same sample share these coefficients.

  4. Image-quality degradation. To simulate reduced sharpness, compression artifacts, and camera noise, we apply Gaussian blur, downsampling followed by upsampling, JPEG compression, and additive Gaussian noise. Under regular augmentation settings, the blur standard deviation is 0.4–1.4 pixels, downsampling factors are 0.55–0.90, JPEG quality is 55–95, and noise standard deviation is 0.003–0.015.

  5. Foreground boundary perturbations. To simulate object boundaries being mistakenly removed or background regions being incompletely removed, we randomly erode or dilate foreground masks. We then soften mask boundaries using Gaussian filtering and extend foreground colors outward, creating color fringes around the boundaries to simulate imperfect foreground extraction.

These perturbations are randomly combined at different strengths. After augmentation, images are cropped around the foreground and resized to the network input resolutions.

Thanks for the additional information. I have no major concerns thus I will retain the positive score.

Reviewer_aP3HOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a novel approach to multi-view 3D reconstruction/generation: reconstructive where observed, generative where occluded. To do so, they propose a three-stage approach: (1) joint per-view point cloud reconstruction and global sparse structure DiT, (2) structure-conditioned VGGT, (3) dense DiT with voxel-wise view selection. The method and data + training are outlined, and both a comparison against baselines and an extensive ablation study shows the benefit of the proposed pipeline.

Critical strengths
  • The method is relatively novel and its high level is clearly outlined.
  • the extensive ablation study is very much appreciated.
  • the related work section is complete and correct to my knowledge
  • the results (both quantitatively and qualitatively) are convincing
Critical weaknesses
  • the method is quite engineered, with three subsequent steps building on each other and -- possibly -- suffering from upstream errors. Specifically the authors' own call-out that "errors in these upstream reconstruction estimates can propagate into generation" (L83-84) seems to equally apply to their own pipeline, which is not discussed.

  • the abstract does a poor job outlining the three-stage pipeline (e.g. L49 talks about point-map prediction again, which is a distinct step from the point map prediction earlier in the abstract, but this is not clear when reading the abstract)

  • i'm missing one ablation study: what would the benefit of the joint-anchor-guided-space training be over joint-first-camera-space training? Is the benefit of a canonical up-axis significant, or negligible?

  • see below, but due to many vagueties of the manuscript, I do not rate reproducibility very high -- the authors mention they might release code, though.

  • why is the method quantitatively compared to Hunyuan 3D v2.1, but not qualitatively?

  • math/notation errors: --> never defined (only implicitly): phi^l, D_v, C^l_mv

  • minor errors: --> Team 2025 for Hunyuan 3D 2.1 is the wrong reference name

Questions for authors

There are a large amount of extreme vagueties throughout the paper, mostly in the method and data section. Please address these:

  • L223: "we spatially align the reconstruction tokens with DINOv3 patch features." Does this just mean that the downsampling etc architecture is identical? Or is there more complex 3D alignment going on?
  • L256: "we encode V into a sequence of tokens". How?
  • L273: again talks about aligning the reconstruction features with DINOv3, this time with an adapter. No details are given beyond that sentence.
  • L285: what subset of TRELLIS-500K and how was it selected?

Some choices are poorly motivated:

  • Eqn 5: why limit to only one view? the cross attention is flexible enough, you could easily attend to any number of views that properly see a given voxel
  • L271: why force occluded voxels to attend to the anchor view? Couldn't they just skip the cross-attention (which, per Eqn 3, is residual) completely?
  • Why the need to evaluate two versions of ReconViaGen?
3 comments

We sincerely thank the reviewer for the insightful comments and positive assessment of our method’s novelty, extensive ablation study, and convincing quantitative and qualitative results. We address each concern below.

Q1: How do we reduce the risk of error propagation across stages?

We agree that errors can accumulate across stages. Two components of our pipeline help limit their propagation:

  • Joint reconstruction and generation: We jointly train the reconstruction and generation branches in a shared anchor-guided space, allowing them to guide each other directly without relying on a separate upstream reconstruction. This reduces the risk of propagating reconstruction errors into generation.
  • Structure-guided reconstruction: We condition reconstruction on both the input images and the generated sparse structure to improve point-map alignment with the structure.

On GSO (Table 3 in the main paper), the overall point-map error decreases from 4.846 × 10⁻³ with image-only reconstruction to 4.152 × 10⁻³ with joint training and 3.313 × 10⁻³ with structure-guided reconstruction. Fig. 6 in the supplementary material provides the corresponding qualitative comparison.

Q2: Why preserve the canonical vertical axis rather than use the first-camera coordinate system?

Geometry in first-camera space appears at varying tilts, which can make it harder for the model to learn plausible support structures and may sometimes lead to geometric distortions. Our anchor-guided space maintains a consistent vertical axis, allowing the model to learn part layouts and support relationships from upright geometry.

As a qualitative comparison, Pixal3D generates geometry in a camera-aligned space. In the examples shown, its outputs exhibit structural distortions with either estimated or ground-truth camera parameters used during generation, whereas our results show more coherent part arrangements and support structures. We have added this discussion and comparison in Sec. A.2.6 (L1006–1013) and Fig. 12 of the supplementary material.

Q3: Which subset of TRELLIS-500K is used, and how is it selected?

We use a subset containing 210K objects. We first select assets with usable PBR materials and then assess the geometric quality of meshes reconstructed from dual grids. Specifically, we use connectivity, non-manifold edge ratios, and variations in adjacent-face normals to exclude samples with severely fragmented meshes or abnormal surface pitting.

Q4: Why select only one view per voxel rather than attend to multiple views that observe it?

Different views provide different observations of the same 3D location: some clearly reveal its structure and details, whereas others observe it obliquely or with occlusion. We therefore select the most informative input view for each voxel based on pixel–voxel correspondences and use it to guide local detail generation. We alternate this selected-view attention with global multi-view attention.

Table 4 in the main paper compares this design with full multi-view conditioning using the same predicted sparse structure. CD decreases from 1.4739 to 1.4047 (×10⁻²), while F-score increases from 0.8758 to 0.8897, supporting the effectiveness of the proposed conditioning scheme.

Q5: Why use the anchor view for unsupported voxels rather than skip cross-attention?

We retain cross-attention to keep the computation consistent across voxels. An unsupported voxel inherits the selected view of a directly supported neighbor, falling back to the anchor view if no such neighbor exists. It then attends to all image tokens of that view, retaining image context even without a direct pixel–voxel correspondence.

Q6: Hunyuan comparison and ReconViaGen versions.

The Hunyuan baseline evaluated in Table 1 of the main paper is Hunyuan3D-2MV. We have added its qualitative comparison in Sec. A.2.7 (L1016–1018) and Fig. 13 of the supplementary material. We have also corrected the author-name formatting for the Hunyuan3D 2.1 reference.

ReconViaGen provides v0.2 and v0.5 as open-source releases, and we include both as baselines in Table 1 of the main paper.

Q7: Clarifying the pipeline and implementation.

We have revised the abstract (L046–053). We also clarify the following implementation details:

  • Reconstruction-to-generation alignment and notation: Alignment refers to corresponding positions on the same view’s 2D patch grid, without additional 3D alignment. The learnable adapter ϕb\phi_b maps VGGT features to the DINOv3 feature dimension before addition to the corresponding patch features. Dv\mathbf{D}_v denotes the DINOv3 feature sequence of view vv, and Cmvb\mathbf{C}_{\mathrm{mv}}^b is the concatenated multi-view condition for generation block bb (Sec. 3.1.2, L230–239).

  • Structure-token construction: Occupied voxel-center coordinates undergo Fourier positional encoding followed by two successive MLPs. The resulting tokens serve as cross-attention keys and values, with reconstruction features as queries (Sec. 3.2.1, L265–268).

  • Feature adaptation for shape generation: Multi-level reconstruction features are concatenated along the channel dimension. A learnable adapter matches the DINOv3 feature dimension and patch-grid resolution, with upsampling when needed, before addition to the corresponding DINOv3 patch tokens (Sec. 3.2.2, L282–285).

thank you for these additional details and explanation. I will be retaining my positive score.

Comments on the paper

3 comments

Thank you for your time and effort in reviewing our paper and for your insightful comments. We address each of your concerns below.

Q1: Ablations on bidirectional interaction and view conditioning.

Bidirectional interaction. We have added a jointly trained one-way variant that retains reconstruction-to-generation conditioning but removes the reverse interaction. Our bidirectional model outperforms this one-way variant in both sparse-structure generation and point-map reconstruction.

MethodVoxel IoU (%) ↑Voxel CD (×10⁻²) ↓Point-map Overall (×10⁻³) ↓
One-way joint training55.611.16864.503
Bidirectional joint training (Ours)58.271.05634.152

These results have been added to Sec. A.2.9 and Table 6 of the supplementary material.

View conditioning. We have added comparisons with Mix3R-style correspondence-based attention bias and a learned view router at the Shape1024 stage. Our method achieves better geometry and normal-map metrics than both alternatives.

MethodMesh CD (×10⁻²) ↓F-score ↑Normal PSNR (dB) ↑Normal SSIM ↑Normal LPIPS ↓
Mix3R-style Bias1.47080.878526.8550.96070.04769
Learned Router1.46490.878126.8480.96130.04753
Ours1.40470.889727.2070.96570.0388

These results have been added to Sec. A.2.10 and Table 7 of the supplementary material. Fig. 15 in the supplementary material shows examples where our selected views more clearly reveal the queried regions than those selected by the learned router.

Q2: How is the upright, anchor-guided coordinate convention realized at inference for unposed images?

At inference, we use the first input image as the anchor, and the network directly predicts point maps and a sparse structure in the anchor-guided space. During training, both outputs are supervised in this space, with a fixed canonical upright direction and a horizontal orientation determined by the anchor view’s azimuth. No known camera poses or additional coordinate transformations are required at inference.

Q3: Are the reconstructed point maps and generated meshes aligned before registration?

To directly quantify native alignment, we measure the nearest Euclidean distance from foreground points predicted by Stage1 and structure-conditioned VGGT to the triangle surfaces of the final generated mesh on 50 GSO objects, without registration, scale fitting, or additional independent normalization.

Here, hh denotes the voxel spacing. [email protected] and Hit@h denote the percentages of predicted points whose distances are at most half and one voxel spacing, respectively.

Point-map sourceMean distance (×10⁻³) ↓Mean distance / h ↓[email protected] ↑Hit@h ↑
Stage11.91970.122997.77%99.71%
Structure-conditioned VGGT1.43680.09298.83%99.75%

These results show good geometric alignment between the reconstructed point maps and generated meshes.

Q4: How sensitive is the method to the choice of anchor view?

For each of 50 GSO objects, we keep the four input images and sampling seed fixed and perform inference with each image serving as the anchor. We use the mesh generated with View 1 as the reference, align the other three meshes to it, and compute their Chamfer distances (CD).

Anchor viewMean CD to View 1 (×10⁻²) ↓
View 21.077
View 31.155
View 41.092

These results suggest that the overall generated shape is relatively stable to anchor selection after alignment. We provide the table and qualitative comparison in Sec. A.2.8 of the supplementary material.

Q5: Evaluation protocol and reproducibility.

Novel-view rendering evaluation. The RGB metrics in Table 1 are computed at four held-out viewpoints that do not overlap with the input views. From the randomly rendered views, we use four images as inputs and four additional views for evaluation. All methods receive the same input images and use the same test viewpoints. After aligning each predicted mesh to the ground truth using GICP + Sim(3), we render it from the four novel viewpoints and compute PSNR, SSIM, and LPIPS on the full RGB images with a white background.

Inference settings. VGGT uses 448 × 448 inputs. DINOv3 uses 512 × 512 inputs for Stage1 and Shape512, and 1024 × 1024 inputs for Shape1024 and texture generation. Shape generation follows a 512 → 1024 cascade. The pipeline’s default sampling parameters are listed below.

ModuleSampling stepsCFG strengthGuidance intervalGuidance rescalerescale_t
Stage1127.5[0.6, 1.0]0.75
Shape512203All steps01
Shape1024206.5All steps0.31
Texture202.5All steps01.5

End-to-end runtime. We measure runtime on a single NVIDIA A800 80GB GPU with no concurrent workloads, after warm-up. Results are averaged over 10 samples, each containing four input views.

ComponentMean runtime (s/sample)
Stage1 conditioning and sampling5.27
Shape512 conditioning and sampling6.06
Shape1024 conditioning and sampling31.57
Texture conditioning and sampling31.49
End-to-end generation212.95

The end-to-end runtime includes all stages above, decoding, mesh processing, UV unwrapping, texture baking, and GLB export.

Structure conditioning during training. The structure-guided reconstruction network is trained using ground-truth sparse structures and uses predicted structures at inference.

Thanks for your providing supplementary experiments and clarification, which addressed most of my concerns. Therefore, I maintain my score

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.