- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Align-VAE studies geometry–attention alignment in VecSet VAEs for 3D reconstruction. A key mechanism predicts continuous spatial anchors from decoder-refined tokens and uses a distance-based attention bias when decoding TSDF queries. The anchors are supervised using the corresponding FPS centers.
Critical strengths
My overall impression is positive. I find the proposed improvement within the VecSet framework worthwhile, particularly the use of explicit spatial anchors to guide the decoding of geometry. My main reservation concerns the practical motivation for choosing this representation relative to alternatives.
Critical weaknesses
My central concern is the present-day competitiveness of VecSet. Improvements within this framework are valuable, but they do not fully answer when VecSet should be preferred over sparse-voxel representations. A comparison under comparable resource budgets would help clarify this point. The budget should account for actual latent storage and computational costs, rather than token count alone. I do not expect superiority in every setting, but would appreciate a clearer picture of the quality–cost trade-offs and the settings in which this representation is advantageous.
Some final reconstructions also appear somewhat over-smoothed to me. I would like to better understand the remaining limitations in recovering fine geometric detail, without assuming that these limitations necessarily arise from the VecSet representation itself.
Questions for authors
-
How does the method compare with a representative sparse-voxel approach under comparable storage and computational budgets? If such a comparison is not available, could the authors clarify the intended use cases and practical advantages of retaining VecSet?
-
Could the authors discuss the remaining smoothing in the qualitative results, and whether it is primarily associated with latent compression, field decoding, or surface extraction?
-
For a potential latent-diffusion extension, when do the predicted anchor positions stabilize relative to the decoded SDF geometry during denoising? Does spatial organization emerge early, followed by geometric refinement, or do both continue to evolve together? For reference-based denoising, positional accuracy could be measured against the target; for free generation, stabilization relative to the final sample would be the relevant observation. I recognize that the current paper evaluates reconstruction, so this is a question about its generative implications rather than a demonstrated limitation of the reconstruction method.
4 comments
Predicted anchors during denoising.
We test whether anchors settle early or continue moving as the generated geometry is refined. DiT evolves latent features, and the frozen VAE predicts coordinates from those features. For this diagnostic, we additionally decode the current prediction of the finished shape at selected steps, represented by the clean latent . The intermediate anchors reveal the sampler's current geometric estimate; they are not fed back into DiT and do not change sampling.
We use fixed conditions from the evaluation list, with 2048 tokens and 50 sampling steps. At each recorded step, we measure two gaps to that trajectory's final generated output: (i) mean Euclidean displacement of anchors, matching the same latent-token indices; and (ii) SDF MAE at fixed query points near the final surface, taken from a 65³ grid. Thus, the two measurements track how anchor locations and the decoded surface field approach their final states. For each sample and metric, we divide by its first-step gap before averaging across conditions: 100% is the initial gap, and 0% is the final reference by construction.
| Sampling evaluation / 50 | Anchor gap (% of initial) | SDF gap (% of initial) |
|---|---|---|
| 1 | 100.0 | 100.0 |
| 10 | 62.2 | 89.4 |
| 20 | 27.2 | 30.2 |
| 30 | 3.9 | 6.9 |
| 40 | 1.2 | 2.2 |
| 50 | 0.0 | 0.0 |
Anchors and the decoded field co-evolve, with anchor displacement generally contracting earlier. By evaluation 30, the remaining gaps are 3.9% and 6.9%; both continue changing afterward.
The generation examples are in Section 6.2, Figure 7 (p. 9), with additional comparisons and sampling settings in Appendix E, Figure 11 (p. 27). These demonstrate the intended image-to-mesh pipeline without supplied spatial coordinates.
References
[1] Xiang et al. Structured 3D Latents for Scalable and Versatile 3D Generation. CVPR, 2025.
[2] Ren et al. XCube: Large-Scale 3D Generative Modeling Using Sparse Voxel Hierarchies. CVPR, 2024.
Thank you for the additional experiments.My main question about TRELLIS remains. Its first stage uses very few tokens, so omitting it may save little time overall. What fraction of end-to-end generation time does that stage actually account for? And the cause of the visible smoothing also remains unclear. The resolution sweep shows how CD changes, but does the missing detail reappear at higher extraction resolutions, or is it already absent from the decoded field?
Thank you for the follow-up. We clarify the role of the first stage and the distinction between field fidelity and extraction resolution below.
The role of the first stage.
Our main contribution is improved first-stage geometry at the same latent budget, rather than an end-to-end speedup over TRELLIS. We have not profiled TRELLIS's stage-wise runtime, so we do not have a measured fraction to report. Our motivation does not depend on its first stage accounting for a large share of generation time.
Align generates a mesh from a self-contained latent set without requiring pre-generated occupancy or spatial coordinates. This also makes it relevant to pipelines that refine an initial mesh: Ultra3D [1] voxelizes a VecSet-generated coarse mesh to obtain coordinates for its second stage; LATTICE [2] and UltraShape [3] likewise use coarse generated geometry to guide subsequent refinement. Improving the initial geometry could therefore provide more faithful spatial support for a later stage, including when that geometry is voxelized at finer resolutions. Our image-to-mesh examples are in Section 6.2, Figure 7.
Smoothing and extraction resolution.
We agree that CD alone does not distinguish whether a particular missing feature is under-resolved or absent from the field. At a fixed checkpoint and latent code, the sweep changes the discretization of the same field:
Here R denotes extraction resolution. Higher resolution can reveal features already represented by the field, but cannot recover geometry that the field does not encode.
For a controlled comparison with VecSet and our LATTICE baseline, reconstruction evaluation uses uniformly sampled encoder inputs, without additional sharp-edge samples. This can undersample narrow edge regions and is a possible contributor to reduced sharpness. Evaluating sharp-edge-aware inputs and targeted edge-preservation objectives, matched across methods, is a natural extension to improve detail fidelity.
References
[1] Chen et al. Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attention. arXiv:2507.17745, 2025.
[2] Lai et al. LATTICE: Democratize High-Fidelity 3D Generation at Scale. arXiv:2512.03052, 2025.
[3] Jia et al. UltraShape 1.0: High-Fidelity 3D Shape Generation via Scalable Geometric Refinement. arXiv:2512.21185, 2025.
We thank the reviewer for recognizing the value of improving VecSet VAEs and introducing explicit spatial anchors. We provide our pointwise replies below.
When to retain VecSet rather than a sparse-voxel VAE.
Our target is single-stage generation through a self-contained latent set: image/noise → latent set → decoded field → mesh. No input 3D shape or FPS coordinates exist at generation time. Align predicts the anchors from the code, rather than requiring them as an external input.
In a TRELLIS-style pipeline [1], the first stage predicts coarse occupancy to determine the active voxel coordinates, and the second stage generates latent features at those coordinates. Its sparse decoder requires both coordinates and features. Without the first-stage occupancy prediction, the decoder lacks its required spatial coordinates. Comparing this pipeline with Align therefore needs to include occupancy generation and its cost.
We have added the XCube coarse VAE variant that requires no pre-generated occupancy [2]. Using its released model and native mesh extraction, the Table 1 comparison is reproduced below as CD (×10³).
We report feature storage together with coordinate storage. At 16 bits per feature, our 4096×64 and 8192×64 codes occupy 0.5 and 1.0 MiB, with no stored coordinate array. In our saved SparseFlex reconstruction results, the median representation contains 57,191×8 feature tokens: 0.873 MiB of features plus 0.654 MiB of int32 coordinates. For computation, matched measurements on the same platform give VecSet→Align latent-refinement times of 54.06→56.58 ms at 4096 tokens (batch 1, BF16), excluding mesh extraction. Matched FP16 forward/backward/AdamW steps take 943.5→977.1 ms, with 13.63→15.83 GiB peak memory (batch 1, excluding I/O, EMA and distributed communication). Align provides a fixed-size first-stage interface that needs no separate spatial-support generator and can also supply a later refinement stage.
Remaining smoothing and extraction resolution.
To examine the effect of extraction resolution, we evaluate the same checkpoints at three resolutions. CD (×10³) on the in-the-wild test set is below. 518 reuses published Table 1 results; 768/1024 use the same checkpoints and original BF16 evaluation path, without the later FP32 coordinates/PE modification.
Each method aggregates its valid results; shapes with insufficient active voxels are skipped. Align has the lowest CD at every matched resolution and latent budget. The improvement persists across all three extraction resolutions. This comparison is added in Appendix C.1, Table 10 (p. 20).