acceptodds
Under review as a conference paper at ICLR 2027

Coarse3D: Controllable and Faithful Sparse Structure Generation for Image-to-3D

Abstract

Coarse-to-fine 3D native generation relies heavily on the first-stage sparse structure, which determines the global orientation and overall geometry for subsequent refinement. However, existing coarse generators suffer from two key limitations: their orientation is difficult to control due to ambiguous semantic canonicalization, and their structures are often inconsistent with input images due to weak 2D-3D grounding. We propose Coarse3D, a sparse-structure generation model that produces orientation-controllable and structurally faithful voxel geometry from arbitrary unposed images. Coarse3D introduces three designs: a first-frame anchored canonical space that aligns generation with the first input view while preserving object-centric normalization; a dual-tower DiT that couples generation with auxiliary feed-forward reconstruction to learn spatially grounded features; and spatial-prior guided attention that regularizes cross-attention with pixel-to-voxel correspondence. Coarse3D can directly replace the coarse stage of existing pipelines. Experiments show that it substantially improves sparse structure quality and consistently enhances final 3D generation, achieving state-of-the-art performance.

Primary area:
generative models
Keywords:
Image-to-3D Generation, Sparse Structure Generation, Reconstruction-Guided Generation, Orientation-Controllable Generation, Cross-Attention Grounding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w6KuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper improves the coarse voxel structure generated in the first stage of image-to-3D generation. It addresses two problems: the output may have an unwanted orientation, and its shape may not match the input images. The method uses the first image to set the object’s azimuth while keeping it centered, normalized in scale, and upright. It also connects a VGGT-based image tower with a TRELLIS2-based geometry tower at each layer and trains them together for reconstruction and generation. During training, ground-truth geometry guides the attention between voxel and image tokens. Results on GSO and Toys4K show better coarse structures. Replacing the coarse stage in three existing pipelines also improves their final geometry without retraining the later stages.

Critical strengths
  1. The results on GSO and Toys4K are strong. Replacing only the coarse stage improves the final geometry in all three tested pipelines, while keeping the refinement models unchanged.

  2. The spatial-prior-guided attention provides a reasonable way to encourage pixel-to-voxel correspondence during training, and the ablation results show that it improves structure quality.

Critical weaknesses
  1. The dual-tower design has limited novelty. ReconViaGen and Mix3R already combine reconstruction features with 3D generation.

  2. Aligning the output with the input view has already been explored by Pixal3D. Coarse3D aligns only the azimuth and keeps the object upright, centered, and normalized in scale. This may be useful, but the paper does not directly test whether it is better than full view alignment.

  3. Coarse3D uses about one million private training assets in addition to public data. The comparison does not control for this extra data, so it is hard to tell whether the gains come from the method or the training data. The paper also has no reproducibility statement and does not say whether the code, model weights, or private data will be released.

Questions for authors
  1. Which cross-attention layers use the spatial prior loss? Is it applied to every attention head and every denoising timestep?

  2. Can the authors compare their method with the baselines using the same public training data? This would help show how much of the improvement comes from the method rather than the extra private data.

  3. Does mixing the spatial prior with the teacher’s attention work better than using either one alone, given the same training budget? How much do the results change with different values of β, and why was 0.2 chosen?

5 comments

We thank the reviewer for recognizing the strong results, compatibility with frozen refinement stages, and effectiveness of spatial-prior-guided attention. We address the concerns below, combining W3 and Q2 on training data. Revisions are marked in red in the updated PDF.

W1. Novelty of the dual-tower design

We agree that combining reconstruction features with generation is not itself new. We have added the previously omitted Mix3R to the related work and distinguish our specific design as follows:

  • Compared with ReconViaGen: Rather than injecting features from a separately trained reconstruction model, Coarse3D jointly optimizes the image tower through reconstruction and generation objectives, with layer-wise conditioning.
  • Compared with Mix3R: Mix3R jointly trains reconstruction and generation with bidirectional token interaction and predicts a similarity transformation between their coordinate spaces. Coarse3D differs in two respects:
    • Unidirectional conditioning: Geometry tokens query image features without feeding noisy geometry back into the image tower. Image features are independent of the denoising state and can be computed once and reused throughout denoising, while generation gradients still update the image tower.
    • Shared coordinate supervision: Coarse3D directly supervises reconstruction and generation in the same first-frame anchored space, eliminating the need for the cross-space alignment branch used in Mix3R.

We added two ablations to test these choices.

Interaction direction

We compare our unidirectional conditioning with a Dual-DiT variant that adds reverse cross-attention, feeding geometry features back into the image tower.

InteractionIoU ↑CD ↓RRE ↓RTE ↓
Bidirectional0.45710.01337.390.0805
Unidirectional (ours)0.46520.01175.880.0692

Unidirectional conditioning improves both generation and reconstruction: CD decreases by 12.0%, RRE from 7.39 to 5.88, and RTE from 0.0805 to 0.0692. These results support keeping image conditioning independent of noisy geometry in our framework. Note that this is an interaction-direction ablation within Dual-DiT, not a reproduction of or direct comparison with Mix3R’s full MoT architecture.

Reconstruction supervision. With architecture, initialization, coordinate space, data, and schedule fixed, enabling the reconstruction loss gives:

Reconstruction lossIoU ↑CD ↓
Without0.44370.0153
With0.46520.0117

These results support the value of explicit reconstruction supervision beyond simply adding a pretrained image tower (updated Table 4).

W2. Azimuth anchoring versus full view alignment

We agree that Pixal3D already explores input-view-aligned generation. Our distinction is azimuth-only anchoring, not view alignment itself. Retaining an upright, centered, scale-normalized representation provides:

  • Refinement compatibility: Three pretrained refinement stages accept our coarse structures without retraining.
  • Upright assets: Preserving the canonical up-axis avoids camera-induced tilt, simplifying 3D product visualization for e-commerce and XR placement.

In our evaluated multi-view configuration, Coarse3D also outperforms Pixal3D with ground-truth poses while using unposed inputs: GSO IoU/CD are 0.5101/0.0124, versus 0.4419/0.0181.

However, this end-to-end comparison does not isolate azimuth-only versus full alignment. We present our coordinate convention as a design suited to upright coarse-to-fine generation, not as proven universally superior to full camera alignment.

W3 / Q2. Private data, matched comparisons, and reproducibility

Removing private assets. We trained a variant using only approximately 1M public assets from TRELLIS-500K and the filtered TexVerse subset, excluding the approximately 1M in-house assets from our training stages.

Training assetsGSO IoU ↑GSO CD ↓Toys4K IoU ↑Toys4K CD ↓
Public only, ~1M0.49430.01370.61920.0108
Public + in-house, ~2M0.51010.01240.63150.0091

The public-only model still outperforms all compared baselines in coarse IoU and CD, including Pixal3D with ground-truth poses. The observed advantage therefore does not require the additional in-house assets.

This controls private-data use within Coarse3D, but does not retrain every baseline on identical data or match their capacity and pretraining. “Public-only” refers to our training-stage assets; both variants retain TRELLIS2/VGGT initialization.

Reproducibility. We recognize the importance of public access for reproducibility. While company policy prevents us from sharing the approximately 1M in-house assets, we are actively pursuing internal disclosure review and open-source approval for the inference code and model weights. Their release remains subject to approval, and we are not yet able to commit to a release date.

Q1. Which layers, heads, and timesteps are supervised?

The submitted model applies spatial-prior supervision to layers 16–30 (1-based), all 12 heads, and every sampled training timestep. These details are documented in Sec. 3.3 and Appendix A.2.3, pp. 15–16.

Layer selection

Starting from the same checkpoint, all variants receive 100K additional training steps:

Supervised layersIoU ↑CD ↓Peak memory ↓
None0.55990.0103~50 GB
1–300.63180.0089~110 GB
11–300.64330.0087~94 GB
16–30 (default)0.63920.0087~86 GB

All tested ranges improve over the unguided control. Our default achieves the same reported CD as layers 11–30, with only 0.0041 lower IoU and approximately 8.5% less memory, making it a practical quality–memory trade-off.

The attention visualizations in Fig. 6, p. 16 show more localized voxel–image correspondence in deeper layers, motivating this choice. upervised layers explicitly compute attention maps; other layers retain FlashAttention. Memory measurements use gradient checkpointing and microbatch size 1 with four accumulation steps.

Head selection

Supervised headsIoU ↑CD ↓
First 6 of 120.62530.0092
All 120.63920.0087

All-head supervision performs better. Each head’s target includes its own teacher attention, retaining head-specific associations rather than imposing an identical purely geometric target.

Timestep coverage

The loss is applied at every sampled training timestep, without restricting the noise range. Our attention-map inspection did not reveal a consistent timestep-dependent pattern that would justify limiting supervision to a particular noise range. We therefore apply spatial guidance throughout the sampled denoising trajectory.

At inference, all layers use FlashAttention, with no spatial-prior construction or attention-supervision overhead.

Q3. Mixed targets and sensitivity to β

We compare targets under the same training schedule: the same 200K-asset subset and evaluation split as Table 4, the same starting checkpoint, and 100K fine-tuning steps at a learning rate of 1e-5. Only β changes:

Attention target = (1 − β) × teacher attention + β × spatial prior

βTargetIoU ↑CD ↓
0.0Teacher only0.56390.0103
0.1Mixed0.62150.0092
0.2Mixed (default)0.63920.0087
0.5Mixed0.63840.0089
1.0Spatial only0.61770.0094

Mixing outperforms either target alone. The default reduces CD by 15.5% versus teacher-only and 7.4% versus spatial-only supervision. The spatial prior provides geometric guidance, while teacher attention retains head-specific associations. Note that β = 0 still applies teacher-attention supervision; it is not the no-attention-loss control.

Why β = 0.2? It was the best tested value in our pre-submission ablation. However, β = 0.5 performs similarly, and all tested mixtures outperform both endpoints. Thus, the benefit is not restricted to a single precisely tuned value; we do not claim that 0.2 is uniquely optimal.

Dear Reviewer w6Ku,

Thank you again for your thoughtful questions about spatial-prior-guided attention. The newly uploaded supplement (pp. 15–17) addresses layer selection, head selection, timestep coverage, and sensitivity to β, including compute-matched comparisons of mixed and unmixed attention targets. We hope these additions clarify the design and its effectiveness.

As the discussion period is nearing its end, we would appreciate your feedback and would be happy to clarify any remaining questions.

Best regards,
The Authors

Reviewer_YMn3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Coarse3D, a method designed to address two challenges in coarse-to-fine image-to-3D generation: uncontrollable orientation and insufficient structural faithfulness. The proposed approach introduces a first-frame anchored canonical space, a reconstruction-guided dual-tower DiT, and spatial-prior-guided attention. Coarse3D does not require camera poses at inference time and can serve as a drop-in replacement for the coarse generation stage of existing frameworks. Experiments on GSO and Toys4K demonstrate improvements in sparse structure accuracy and final 3D shape quality.

Critical strengths
  1. The paper addresses important and practically relevant research problems.

The paper focuses on the coarse structure generation stage, with particular emphasis on orientation controllability and structural faithfulness, both of which are important challenges in 3D generation. Since the coarse generation stage determines the global orientation and geometric layout of an object, errors introduced at this stage may be difficult to fully correct during subsequent refinement. Therefore, improving this stage is a meaningful research direction with practical implications for the quality of generated 3D assets.

  1. The proposed technical designs are well motivated and appropriately address the identified challenges.

To mitigate orientation ambiguity, the paper introduces a First-frame Anchored Canonical Space, which aligns the generated orientation with the input view while preserving object-centric normalization and scale consistency, thereby avoiding reliance on an ambiguous semantic front. To improve structural faithfulness, the proposed Dual-DiT incorporates an auxiliary 3D reconstruction task to enhance the geometric information encoded in image features. In addition, Spatial-prior-guided Attention encourages more accurate correspondences between image features and 3D voxels. These components have clear technical motivations and are well aligned with the challenges identified in the paper.

  1. The quantitative and qualitative results are promising.

The paper reports promising results on GSO and Toys4K, with improvements in both coarse structure accuracy and final 3D shape quality. The qualitative visualizations further demonstrate the potential of Coarse3D to generate more structurally faithful 3D assets. These results suggest that the proposed approach is effective and has practical potential for improving existing coarse-to-fine 3D generation pipelines.

Critical weaknesses
  1. The paper overstates the novelty of the identified challenges.

Although the authors discuss Orientation Matters and Pixal3D in the related work section and appendix, the introduction still describes uncontrollable orientation and unfaithful structures as "two overlooked failure modes." This claim is misleading, as prior work has already investigated orientation ambiguity and 2D–3D geometric correspondence. The contribution of Coarse3D lies in proposing new solutions to these existing problems in coarse structure generation, rather than identifying previously overlooked challenges. The authors should revise this claim to more accurately reflect the novelty of their work.

  1. The proposed framework lacks a clear unified motivation.

The paper addresses orientation controllability and structural faithfulness using different technical components. The First-frame Anchored Canonical Space resolves orientation ambiguity, while Dual-DiT and Spatial-prior-guided Attention improve 2D–3D geometric correspondence. However, the paper does not clearly explain how these components are conceptually connected or why they should be integrated into a unified framework. As presented, the method appears more like a combination of independently motivated techniques than a framework derived from a unified problem formulation. A clearer explanation of their relationship and potential synergy would strengthen the methodological contribution.

  1. Controlled comparisons with alternative orientation modeling strategies are lacking.

The First-frame Anchored Canonical Space is a key component for achieving orientation controllability. Although the paper provides a reasonable motivation for this design and demonstrates its benefits through ablation studies, its advantages over existing orientation modeling strategies have not been sufficiently established. In particular, the dataset-level orientation alignment strategy of Orientation Matters and the camera-aligned coordinate system of Pixal3D are closely related to the objective of this work. Controlled comparisons of these coordinate system or orientation alignment strategies under the same model architecture, training data, and evaluation protocol would provide stronger evidence for the necessity and effectiveness of the proposed first-frame anchoring strategy.

  1. The experimental comparisons do not sufficiently disentangle the contributions of the proposed designs from model capacity and training resources.

Coarse3D employs a dual-tower architecture with approximately 2B parameters, leverages pretrained initialization from TRELLIS2 and VGGT, and is trained on approximately 2 million 3D assets, including around 1 million in-house assets. However, the paper does not sufficiently clarify the differences between Coarse3D and the competing baselines in terms of model capacity, pretraining conditions, and training data distributions. Consequently, it remains difficult to fully distinguish the improvements attributable to the proposed methodological innovations from those potentially arising from larger model capacity or greater training resources. Furthermore, Spatial-prior-guided Attention is introduced through an additional 100K fine-tuning steps after the initial training stage. However, the paper does not provide a compute-matched control in which the model is trained for the same additional number of steps without the spatial prior. As a result, the observed improvements cannot be fully attributed to the proposed geometric guidance mechanism, since additional optimization may also contribute to the performance gains.

  1. The evaluation of out-of-distribution generalization remains limited.

The paper evaluates Coarse3D on GSO and Toys4K, providing some evidence of cross-dataset generalization. However, given the large-scale and heterogeneous training data used in this work, the model's generalization ability under more diverse out-of-distribution conditions remains insufficiently explored. Additional evaluations on real-world captured images, unseen object categories, or datasets with substantially different distributions would help assess the robustness of the proposed method in more challenging practical scenarios.

  1. A systematic analysis of the proposed method's failure modes is missing.

The paper presents numerous successful generation results and illustrates the importance of the identified challenges through failure cases of baseline methods. However, the failure modes and limitations of Coarse3D itself are not systematically analyzed. It would be valuable to include representative failure cases and investigate the method's behavior under challenging conditions, such as complex geometry, severe occlusion, ambiguous viewpoints, or inaccurate geometric correspondences. Such an analysis would help clarify the applicability and limitations of the proposed approach.

Questions for authors

My main concerns are outlined in the Weaknesses section above. I am willing to raise my rating if the authors can adequately address these concerns through further explanations or additional experiments.

6 comments

We thank the reviewer for recognizing the importance of coarse-stage generation and the motivation and effectiveness of our designs. We have clarified our claims, added targeted controls, and expanded the robustness and failure analyses. Revisions are marked in red in the updated PDF.

W1. Novelty of the identified challenges

We agree that “overlooked” overstates the novelty of the problems. Orientation ambiguity and 2D–3D correspondence have been studied in prior work, including Orientation Matters and Pixal3D. Our contribution lies in new solutions for coarse sparse-structure generation, not in identifying these challenges for the first time.

We have replaced “two overlooked failure modes” with “two persistent failure modes.” Here, “persistent” refers to their continued occurrence in the evaluated coarse-stage baselines, not a lack of prior investigation.

W2. Unified motivation

Coarse3D aims to ground image-conditioned coarse generation in a shared, well-defined 3D space. Its components address three complementary requirements:

  • Coordinates: First-frame anchoring defines a common frame for generation targets, reconstructed point maps, and camera supervision, without requiring a semantic-front convention.
  • Features: Reconstruction supervision makes image conditioning geometrically informative; the generation tower accesses these features through layer-wise cross-attention.
  • Correspondences: Spatial-prior-guided attention supervises pixel-to-voxel associations constructed in that same frame.

The coordinate system establishes the geometric reference, reconstruction enriches the features, and attention guidance encourages their spatially appropriate use. Table 4 supports these complementary roles: reconstruction improves geometry but does not resolve orientation ambiguity; anchoring improves orientation consistency; spatial guidance further improves structural accuracy.

W3. Alternative orientation strategies

1. How the orientation strategies differ

  • Orientation Matters: semantic alignment.
    It improves dataset-level consistency by aligning assets to semantic orientations. Coarse3D instead defines output azimuth relative to the first input view, avoiding the need to infer an ambiguous semantic front.

  • Pixal3D: camera-dependent pixel alignment.
    It establishes pixel-aligned correspondences using supplied or predicted cameras. Coarse3D anchors only azimuth while retaining object centering, scale normalization, and the canonical up-axis, preserving compatibility with canonical-space refinement.

2. Effectiveness of first-frame anchoring

We compare two TRELLIS2 variants using the same TRELLIS-500K training assets, model architecture, and parameter count, changing only the coordinate-space definition. Neither includes an additional image tower or spatial attention supervision.

As shown in the table below, anchoring increases IoU by 0.0623 / 0.0749 and reduces CD by 55.2% / 34.5% on GSO / Toys4K. These gains isolate the benefit of our coordinate design from additional model capacity or training data.

3. Single-view comparison

The table also compares the anchoring-only TRELLIS2 variant—not the full Coarse3D model— with existing methods under the same single-view evaluation protocol. Pixal3D uses predicted cameras; our variant requires no supplied poses.

MethodGSO IoU ↑GSO CD ↓Toys4K IoU ↑Toys4K CD ↓
Orientation Matters0.09920.11910.12790.0916
Pixal3D0.06380.13180.05400.1456
TRELLIS20.23140.07630.27260.0501
TRELLIS2 + our anchoring0.29370.03420.34750.0328

The anchoring-only variant achieves the best IoU and CD on both datasets. The TRELLIS2 pair provides a controlled ablation, while comparisons with Orientation Matters and Pixal3D demonstrate end-to-end performance advantages rather than architecture- or training-matched evidence.

W4. Capacity, training resources, and additional optimization

We agree that end-to-end comparisons alone cannot isolate methodological gains from resource differences. We therefore added three targeted controls.

(a) Excluding the additional in-house assets

We train a public-only variant using approximately 1M assets from TRELLIS-500K and the filtered TexVerse subset, excluding the approximately 1M in-house assets from our training stages.

Training assetsGSO IoU ↑GSO CD ↓Toys4K IoU ↑Toys4K CD ↓
Public only, ~1M0.49430.01370.61920.0108
Public + in-house, ~2M0.51010.01240.63150.0091

The public-only model still outperforms the compared baselines in coarse IoU and CD. The advantage therefore does not depend on access to the additional in-house assets, although those assets provide further improvements.

(b) Matching capacity and pretrained initialization

The updated Table 4 includes a control with the same ~2B-parameter Dual-DiT architecture and TRELLIS2/VGGT initialization, but without the reconstruction loss. Its image tower remains trainable through the generation objective. Architecture, initialization, training data, and schedule are held fixed.

SettingIoU ↑CD ↓
Dual-DiT without reconstruction loss, original canonical space0.44370.0153
+ Reconstruction loss supervision0.46520.0117
+ First-frame anchoring0.55860.0104

Reconstruction supervision reduces CD by 23.5%; anchoring further increases IoU by 0.0934, without adding parameters. These gains cannot be attributed to capacity or initialization differences between the ablated variants.

This control isolates our supervision and coordinate designs within the same architecture; it does not fully isolate the benefit of adopting that architecture relative to external baselines.

(c) Matching additional training steps

Starting from the same checkpoint, both variants receive 100K additional steps with identical data, batch size, and learning-rate schedule. The unguided control retains generation and reconstruction losses but omits spatial attention supervision.

SettingIoU ↑CD ↓
Base model0.55860.0104
+ 100K steps, without spatial guidance0.55990.0103
+ 100K steps, with spatial guidance0.63920.0087

Additional optimization alone gives little improvement. At the same training length, spatial guidance increases IoU by 0.0793 and reduces CD by 15.5% over the unguided control.

This is training-step-matched, not strictly FLOP-matched, because teacher-based attention-target construction adds computation. Nevertheless, it directly shows that longer training alone does not explain the gains.

W5. Out-of-distribution generalization

We added four sets of real-world smartphone captures in Appendix A.6, Fig. 7, p. 19, covering challenging illumination, elevation changes, and self-occlusion. No camera poses are supplied at inference.

The results qualitatively preserve first-view-conditioned orientation and visible structural details beyond benchmark-rendered inputs. We describe these as additional real-capture robustness evidence, not a comprehensive OOD evaluation or proof of generalization to unseen categories.

W6. Failure modes and limitations

We added two representative failures and their analysis in Appendix A.6, Fig. 8, p. 19:

  • Front–back reversal: Similar front/rear shapes and repetitive patterns lead to a reversed structure relative to the first view. First-frame anchoring removes the need for a semantic-front convention, but does not eliminate perceptual ambiguity.
  • Distorted geometry under wide-FoV capture: A rectangular box is reconstructed with skewed geometry, which persists through refinement. The model may conflate camera-induced appearance changes with object geometry; this is a plausible explanation rather than an experimentally isolated cause.

These cases identify limitations in orientation disambiguation and robustness to challenging camera conditions. We have expanded the limitations discussion accordingly, while recognizing that two examples do not constitute a systematic failure benchmark.

I thank the authors for their detailed response and additional experiments, which have addressed most of my previous concerns. However, my main concern regarding methodological novelty and the conceptual connection between the proposed contributions remains. The method addresses two previously studied challenges through largely separable technical designs. While the revised motivation provides a clearer explanation of how these components complement each other, it does not sufficiently clarify the methodological novelty beyond combining these independently motivated designs. As a result, I still view the paper primarily as an integration of several effective technical improvements into a single system, with its contributions leaning more toward engineering integration than substantial methodological innovation. This remains my primary reservation about the work despite the improvements made during the rebuttal. Could the authors further clarify whether the main contribution lies primarily in combining incremental technical improvements, or whether the proposed method offers a more fundamental methodological innovation?

Reviewer_JpYKOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on coarse voxel structure generation in the first stage of two-stage 3D-native frameworks such as TRELLIS and TRELLIS2. It first anchors the azimuth of the generated geometry to the first input view, substantially reducing ambiguity in output orientation. The proposed Dual-DiT incorporates an auxiliary VGGT-based reconstruction branch to provide geometry-aware features for generation, improving geometric fidelity. Finally, it uses pixel-to-voxel correspondences to regularize cross-attention between 2D image features and 3D voxel tokens, strengthening their spatial alignment. The reported experiments demonstrate consistent improvements over the evaluated baselines on geometric metrics, with impressive overall performance.

Critical strengths
  1. The proposed Dual-DiT framework effectively incorporates geometric information from a VGGT-based reconstruction branch into the generation branch, substantially improving the quality of the generated geometry.

  2. The method strengthens pixel-to-voxel alignment in cross-attention, enabling image features to interact more precisely with voxel tokens and producing geometry that is more faithful to the input images.

Critical weaknesses

1)The contribution of reconstruction grounding is not sufficiently isolated. Although Table 4 controls the training data and schedule, introducing Dual-DiT simultaneously adds substantial pretrained model capacity and auxiliary reconstruction supervision. There is no control using the same dual-tower architecture and initialization but trained without the reconstruction losses. Consequently, the reported improvements do not clearly establish how much benefit comes specifically from reconstruction grounding, which is central to the paper’s methodological contribution.

2)The evaluation provides limited direct evidence of orientation controllability. Section 4.2.1 primarily supports this claim through the gap between IoU and PA-IoU. However, this gap also depends on geometric errors and the reliability of PCA-based registration, making it an indirect measure of orientation accuracy. Direct azimuth-error measurements and tests that change the first reference view while keeping the input image set fixed would better establish whether the output follows the intended orientation while preserving consistent geometry. This leaves a central claimed capability insufficiently validated.

Questions for authors

1.The description of the CD evaluation protocol is inconsistent. Section 4.1 (p. 6) states that all CD metrics are computed after rigid alignment, whereas Appendix A.3 (p. 14) states that Umeyama alignment removes both pose and scale ambiguity. Could the authors clarify whether scaling is allowed during alignment, and whether the same alignment procedure is applied consistently to all compared methods and CD-based evaluations?

2.Could the authors clarify why first-frame anchoring is limited to rotation about the vertical axis, without accounting for elevation and roll?

3.RecGen3D also generates complete 3D shapes from unposed multi-view images and reports results on GSO and Toys4K. Could the authors clarify why it is included only in the auxiliary reconstruction evaluation, but not in the comparison of final shape quality? Even if Coarse3D cannot directly replace its coarse stage, comparing the final outputs under the same evaluation protocol would help substantiate the claimed geometric improvements over this closely related method.

1 comment

We thank the reviewer for the positive assessment and constructive suggestions.

W1. Isolating reconstruction supervision

We conducted the requested control using identical Dual-DiT architectures, parameter counts, TRELLIS2/VGGT initialization, coordinate conventions, training data, and schedules. Only the auxiliary reconstruction loss differs. Without it, the image tower still receives gradients from the generation objective.

Recon. supervisionIoU ↑PA-IoU ↑
Without0.44370.5086
With0.46520.5329

Reconstruction supervision improves IoU by 0.0215 and PA-IoU by 0.0243, isolating its benefit beyond added pretrained capacity. We have added this control to Table 4 and Sec. 4.3.

W2. Reference-view intervention and orientation consistency

We agree that the IoU–PA-IoU gap is not a direct orientation measure. We added a reference-view intervention and a reprojection test, neither using PCA registration.

For each GSO object, we fix four input images and vary the first/reference view using three selection seeds. Voxel IoU is evaluated against GT in the corresponding first-view-anchored frame.

We also render predicted voxels with the selected reference camera: azimuth is set to zero in the anchored frame, while elevation and intrinsics are retained. Rendered silhouettes are compared with the reference-image foreground masks at 256 × 256 resolution. Identically rendered GT voxels provide a reference for voxelization/rendering discrepancies.

RunVoxel IoU ↑Pred. mask IoU ↑GT mask IoU ↑
10.50670.86380.89
20.51240.87740.90
30.50880.86120.88

Stable voxel IoU and high silhouette agreement support reference-view consistency, but do not establish exact azimuth accuracy or per-object geometric invariance, especially for symmetric objects. We will describe these as intervention and reprojection-consistency tests and qualify the original IoU–PA-IoU interpretation.

Q1. Rigid versus similarity alignment

Scaling is allowed. CD uses similarity alignment—uniform scaling, rotation, and translation—not rigid alignment. All compared methods within each CD-based evaluation use the same procedure. We will correct Sec. 4.1 and relevant captions and specify the transformations in Appendix A.3. The voxel IoU and PA-IoU protocols are unchanged.

Q2. Why anchor only azimuth?

Our goal is to resolve semantic-front ambiguity while preserving a canonical upright direction. In our asset setting, the up-axis is generally better defined than the semantic front, so we anchor only azimuth to the first view.

Anchoring elevation and roll would make the output orientation fully camera-dependent. These factors are not ignored: their effects remain represented in the observations and transformed camera supervision.

Preserving the upright direction across reference views facilitates 3D product visualization and XR placement. We will clarify that this design targets upright assets and assumes a meaningful canonical up-axis.

Q3. Final-shape comparison with RecGen3D

We agree that RecGen3D merits comparison regardless of coarse-stage compatibility. We ran its official implementation on GSO and Toys4K using the same input views, similarity alignment, and metric definitions as the other methods. We also include the original ReconViaGen baseline to show the effect of replacing its coarse stage.

GSO — Final-shape quality

Method  CD ↓    F-Score ↑    ULIP ↑    Uni3D ↑  
ReconViaGen0.07850.38350.22450.2707
ReconViaGen + Coarse3D0.01320.82640.23910.2836
RecGen3D0.02210.67290.21450.2742

Toys4K — Final-shape quality

Method  CD ↓    F-Score ↑    ULIP ↑    Uni3D ↑  
ReconViaGen0.04900.50960.25190.3687
ReconViaGen + Coarse3D0.01300.85370.26700.3745
RecGen3D0.01700.77940.25230.3708

Compared with RecGen3D, ReconViaGen + Coarse3D reduces CD by 40.3% on GSO and 23.5% on Toys4K, while improving F-Score and both semantic-similarity metrics. Only ReconViaGen’s coarse stage is replaced; its pretrained refinement stage is unchanged.

We have added final-shape results to Table 2 and surface-voxel results to Table 1. All methods in Table 1 use the same surface-voxel IoU protocol, rather than RecGen3D’s original volumetric IoU protocol.

Comments on the paper

2 comments

Dear Reviewers,

Thank you again for your time and thoughtful feedback. As the rebuttal period closes tomorrow, we wanted to check whether any points in our responses remain unclear or would benefit from further clarification. We would be happy to address any remaining questions.

Best regards, The Authors

Dear Reviewers,

Thank you for the discussion. As a final clarification regarding methodological novelty, we would like to point to the concurrent ICLR 2027 submission UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space.

UniForge3D adopts essentially the same azimuth-anchored shared-space definition as Coarse3D. Its Introduction and §3.1 identify the coordinate mismatch in Mix3R—camera-relative reconstruction versus canonical-space generation—and address it by defining both tasks in a shared, input-anchored space that preserves object centering, scale normalization, and the canonical upright direction. Its Table 2 further reports improvements over canonical-space and cross-space joint-training alternatives. While this concurrent work does not itself establish our novelty, it provides relevant context for our methodological motivation: the shared-space formulation resolves a coordinate mismatch between reconstruction and generation, rather than merely controlling output orientation.

The overlap is substantial, and we will acknowledge it explicitly. However, the methods differ in two important respects:

  • Task coupling. UniForge3D uses bidirectional interaction, including time-gated feedback from generation tokens into reconstruction (§3.1.2). Coarse3D uses asymmetric forward coupling: reconstruction provides layer-wise conditioning without receiving noisy geometry tokens, while both objectives jointly optimize its features.

  • Correspondence-guided generation. UniForge3D uses predicted point-map correspondences to select views for subsequent detailed shape and texture generation (§3.2; Appendix A.1.2). Coarse3D instead uses GT-derived correspondences to directly supervise coarse-stage cross-attention during training, with teacher-mixed targets. Our updated supplement (pp. 14–16) examines layer and head selection, timestep coverage, and target mixing under controlled settings.

Coarse3D therefore unifies reconstruction and generation through coordinate space, architectural coupling, and geometric supervision of their interaction. Its coarse-stage contribution is also evaluated through integration into three pipelines with their refinement models frozen. We respectfully ask that the final assessment consider this specific formulation and evidence, beyond the established general idea of combining reconstruction and generation.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.