acceptodds
Under review as a conference paper at ICLR 2027

Structured Voxel Diffusion for 3D Generation

Abstract

Recent 3D generative models often rely on multiview intermediates or learned latent representations, separating the generated variables from the final geometry. We introduce GRAVER, an image-conditioned framework that combines compact latent modeling of coarse structure with direct generation of fine geometry in an explicit sparse representation. Direct high-resolution geometric generation is challenging because volumetric cost grows cubically while the surface signal is highly sparse. GRAVER addresses this challenge with a staged block-UDF formulation that predicts (i) active blocks on a global grid, (ii) local surface-support masks within those blocks, and (iii) fine unsigned distance values only near the predicted support. The first two stages organize sparse structure, whereas the final stage predicts explicit UDF values without a geometry autoencoder. This decomposition concentrates computation near surfaces, accommodates open and non-watertight geometry, and reconstructs meshes through sparse marching cubes. The same spatially indexed representation also enables native local editing by preserving selected support and UDF values, and supports a preliminary adaptation to blockwise PBR prediction. Experiments show competitive image-shape alignment, validate the support-conditioned fine-UDF design, and demonstrate strong preservation of fixed regions during editing.

Primary area:
generative models
Keywords:
3D Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1ZVnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GRAVER, an image-conditioned 3D generation framework that directly produces fine geometry in an explicit sparse voxel representation (block-UDF). The method factorizes generation into three stages: global block layout on a 64³ grid, local surface-support masks (8³ per block) via a sparse occupancy VAE, and fine unsigned distance values (16³ per block) through support-conditioned flow matching without a geometry autoencoder. The spatially indexed representation further enables native 3D editing with hard preservation of unedited regions. Experiments show semantic alignment competitive with TRELLIS2 and Hunyuan3D 2.1, and editing results demonstrate strong fixed-region fidelity.

Critical strengths
  1. Clean hierarchical decomposition that concentrates computation near surfaces. The three-stage factorization (where → support → geometry) is well-motivated: Stage 1 eliminates empty space, Stage 2 localizes the surface neighborhood, and Stage 3 only generates UDF values where the gate is active. This avoids the infeasible cost of dense 960³ generation while keeping the final generative state in explicit geometric variables. The ablation in Table 2(c) confirms that the support mask is critical — removing it degrades MSEmcMSE_{mc} by nearly 4×.
  2. Native editing with hard preservation guarantees. Because the representation is spatially indexed, unedited regions are preserved by direct value copying rather than a soft consistency loss. This is a structural advantage over latent-space editing methods where preservation can only be encouraged, not enforced. Table 3 validates this: Fix@100 achieves 99.998% F-score on model-generated assets, demonstrating near-lossless round-tripping.
Critical weaknesses
  1. Insufficient experimental evidence to support the central claim. The paper's core argument is that direct UDF generation in explicit sparse space yields better geometry than latent-decoded alternatives. However, Table 1 reports only semantic alignment metrics (CLIP-N, ULIP-2, Uni3D) and an LLM quality rating — standard geometric accuracy metrics (Chamfer Distance, F-score, IoU, Hausdorff Distance) against ground-truth meshes are entirely absent. Moreover, GRAVER does not significantly outperform any baseline on the reported metrics: TRELLIS2 leads on CLIP-N and Quality-LLM, TRELLIS leads on ULIP-2, and GRAVER's only first-place result (Uni3D: 0.3568 vs 0.3538) falls within one standard deviation. For a new representation and generation paradigm, parity on proxy metrics without demonstrated geometric superiority is insufficient to justify the added complexity of a three-stage pipeline.

  2. Resolution comparison is unspecified and scalability is unclear. The paper does not state at what resolution the baseline methods produce their meshes for evaluation. For instance, TRELLIS2 is capable of generating at 1536³ resolution, which is substantially higher than GRAVER's effective 960³. It remains unclear whether GRAVER's advantage (if any) persists at matched or higher resolutions, whether the block-UDF hierarchy can scale beyond 960³, and what the corresponding VRAM cost would be. Without this information, the comparison may not be conducted on equal footing.

Questions for authors
  1. Can you report geometric accuracy metrics (Chamfer Distance, F-score, IoU) against ground-truth meshes for the baseline comparison in Table 1? Without these, it is unclear whether direct UDF generation yields more faithful geometry than latent-decoded approaches.

  2. What is the inference time and peak memory usage of the full three-stage pipeline, compared to TRELLIS2 and Hunyuan3D 2.1? Given the 960³ effective resolution and 20-layer sparse transformer in Stage 3, computational cost is a practical concern.

  3. Stage 1 and Stage 2 both rely heavily on pretrained TRELLIS2 components (sparse-structure VAE, occupancy VAE, flow model). Could you clarify what proportion of the total system is newly trained versus inherited? Would the method still work if these components were trained from scratch on your data?

Reviewer_abXHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to model UDFs directly in raw space, avoiding the potential information loss introduced by VAE compression. Experimental results demonstrate competitive image-to-3D generation performance on image–shape alignment.

Critical strengths
  • The motivation to avoid information loss caused by VAE compression is clear, and the proposed method is straightforward and easy to understand.
  • The method achieves competitive generation quality compared with existing approaches.
Critical weaknesses
  • The method does not fully eliminate the information loss introduced by VAEs. Although the proposed approach avoids VAE compression for fine-grained UDF generation, the surface support grid at a resolution of (512^3) still relies on a VAE, which cannot guarantee lossless reconstruction. Errors in this stage may propagate to the final geometry and cannot necessarily be recovered by the subsequent UDF generation stage. For example, some generated results in Figure 3 still exhibit holes.

  • The contribution of "Native 3D Editing" is unclear. The proposed editing approach appears to primarily exploit the inherent properties of spatially indexed distance-field representations and apply existing techniques for local field manipulation. It is unclear what novel methodological contribution is introduced beyond these standard operations.

  • The necessity and effectiveness of raw-space UDF generation are not sufficiently justified. For continuous and smooth fields, VAE compression can provide an effective and compact representation. I am concerned that directly modeling high-dimensional UDFs without a VAE may introduce additional learning difficulties, particularly due to noise and high-frequency variations in UDF/SDF representations. A related observation is that pixel-space image generation has often been less effective than latent-space generation.

  • The introduction of a bottleneck raises further questions about the necessity of the proposed design. Although the method aims to avoid VAE compression, it introduces a 64-dimensional bottleneck within the diffusion model. This raises the question of whether the benefits come from raw-space modeling itself.

Questions for authors

See the weakness.

Reviewer_ytytOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

GRAVER is an image-to-3D generator built on a block-UDF hierarchy. Stage 1 generates active 16³ blocks on a 64³ grid, using the pretrained TRELLIS 2 sparse-structure VAE. Stage 2 generates a binary near-surface support mask at 512³ with its own sparse occupancy VAE and latent flow. Stage 3 generates truncated UDF values (effective resolution 960) directly, without a geometry autoencoder, using clean-data prediction with a velocity-weighted loss; samples outside the dilated support are clamped to the far-field value throughout sampling. Meshes are extracted by sparse marching cubes. Trained on about 679K assets, the model is reported to be on par with TRELLIS 2 in semantic alignment (Table 1). Because the final field is explicit, local editing can copy unedited UDF values exactly, which Table 3 measures against a mask-only variant.

Critical strengths
  • The factorization into where the surface lies (blocks and support) and its exact distance values is clear, and the gating makes field-level local editing straightforward. Fixed regions are preserved almost exactly (Table 3b, model-generated assets at Fix@50: F-score 99.985% vs. 96.285% for the mask-only variant).
  • The paper provides a useful data point at scale: dropping the geometry autoencoder in the final stage still yields semantic alignment comparable to TRELLIS 2.
  • The data pipeline and architectures are described in detail (App. A–C).
Critical weaknesses
  1. No geometric evaluation supports the central claim. Contribution 1 is that generating explicit UDF values places supervision directly on the final geometry. However, Table 1 contains only semantic embedding scores and a mean LLM rating, whose model and prompt are not specified. The differences from TRELLIS 2 are −0.0041 (CLIP-N), +0.0004 (ULIP-2), +0.0030 (Uni3D) and −0.40 (Quality-LLM), which the paper itself describes as small relative to the standard deviations (p.7). Although the evaluation set, Toys4k, comes with ground-truth meshes (App. B, p.15), no Chamfer distance, F-score or normal consistency against ground truth is reported, so the claim of better fine geometry is untested.

  2. The contribution of raw-space generation is not isolated. Where the surface lies at 512³ is still generated in a learned latent space (Stage 2), and Stage 1 reuses the pretrained TRELLIS 2 sparse-structure VAE (Table 4). Stage 3 refines distances only inside the dilated support band. Without the support mask, the Stage 3 error is 3.8× higher (8.3 vs. 2.2 ×10⁻⁴, Table 2c), so the latent support does much of the work. The paper does not compare Stage 3 with a matched latent alternative (a VAE on 16³ UDF blocks plus latent flow at equal compute), which is the control the central claim requires. Table 2(d) also shows that a 64-dimensional per-block bottleneck beats no compression (2.5 vs. 12.1 ×10⁻⁴), which sits uneasily with the "no geometry autoencoder" framing.

  3. Novelty and attribution.

    • Generating distance values directly in the original space after localizing the surface has precedents: LAS-Diffusion (Zheng et al., SIGGRAPH 2023) diffuses an occupancy grid and then SDF values inside occupied voxels, and Mosaic-SDF (Yariv et al., CVPR 2024) trains a flow model directly on local SDF grids near the surface. LAS-Diffusion is not cited, and Mosaic-SDF appears only in a citation list (p.3).
    • The Stage 3 recipe follows JiT (Li & He, 2025) closely: clean-data prediction with a velocity-weighted loss, a logit-normal time shift with μ = −0.8 as the best setting, clipping of 1 − t at 0.05, and a low-dimensional bottleneck embedding. JiT is cited only in a list (p.3) and is not credited in the method or ablation sections.
    • Accommodating open and non-watertight geometry is not a distinguishing capability: TRELLIS 2's O-Voxel representation already handles open and non-manifold surfaces at resolutions up to 1536³.
    • SparseFlex and Sparc3D, which the paper calls "especially close in spirit" (p.3), and Direct3D-S2 are cited but not compared. UltraShape (a Table 1 baseline) and Toys4k (the evaluation set) have no reference entries.
  4. The editing evaluation measures only what holds by construction. Exact preservation of unedited values is guaranteed by Eq. 11, and Table 3 compares only against an internal mask-only variant. TRELLIS (Xiang et al.) already supports training-free local deletion, addition and replacement, but no external editing baseline is included, and there is no measure of edit success or seam quality. The paper also notes that the guarantee holds only before mesh cleanup and smoothing (p.5).

Questions for authors
  1. Do the Table 2 ablations use ground-truth or predicted support masks? Since MSE_mc measures only predicted-to-ground-truth distance, could you also report the reverse direction or a symmetric metric?
  2. For Table 1, what are the input images and the number of evaluated assets? Which LLM and prompt define Quality-LLM, and which resolution and settings were used for TRELLIS 2?
  3. Meshes are extracted at the UDF isovalue 0.2, one global voxel (p.17). For an open sheet, this level set is a closed shell offset by one voxel on both sides. Does the cleanup step remove one layer? Could you show outputs for open surfaces?
  4. For artist-authored assets at Fix@100, where nothing is regenerated, Mask + Feats reaches only 88.3% F-score with a drift of 0.98h (Table 3). Is this the round-trip error of the representation and mesh extraction, and does it limit editing of real assets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.