acceptodds
Under review as a conference paper at ICLR 2027

Align-VAE: Aligning Attention and Geometry in VecSet VAEs for 3D Reconstruction

Abstract

Set-based variational autoencoders (VecSet VAEs) represent 3D shapes as compact sets of latent tokens. Although each token originates from a query at a sampled surface location, whether these latents truly align with geometry remains unclear. We investigate this alignment and identify four issues: (i) attention-sink-like tokens that attract queries from across the shape; (ii) spatially periodic repetitions in decoder attention responses; (iii) mismatched normal information in encoder and decoder query features; and (iv) missing supervision of the intrinsic relation between signed distance fields and surface normals. To address these issues, we introduce Align-VAE with four corresponding modifications: (i) a shared bank of learnable decoder tokens provides a workspace for global computation; (ii) distance-biased cross-attention guides retrieval toward nearby predicted spatial anchors; (iii) normal-free encoder queries align the two stages' access to normal information while retaining normals in the encoder keys and values; and (iv) a screened Poisson-inspired regularizer supervises surface values and normal-direction derivatives. These changes align latent retrieval and field prediction with geometry without enlarging the per-shape code. On the ABO dataset, Align-VAE reduces Chamfer distance by 26.6% compared to the standard VecSet VAE.

Primary area:
generative models
Keywords:
3D reconstruction, VAE

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_14TUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Align-VAE studies geometry–attention alignment in VecSet VAEs for 3D reconstruction. A key mechanism predicts continuous spatial anchors from decoder-refined tokens and uses a distance-based attention bias when decoding TSDF queries. The anchors are supervised using the corresponding FPS centers.

Critical strengths

My overall impression is positive. I find the proposed improvement within the VecSet framework worthwhile, particularly the use of explicit spatial anchors to guide the decoding of geometry. My main reservation concerns the practical motivation for choosing this representation relative to alternatives.

Critical weaknesses

My central concern is the present-day competitiveness of VecSet. Improvements within this framework are valuable, but they do not fully answer when VecSet should be preferred over sparse-voxel representations. A comparison under comparable resource budgets would help clarify this point. The budget should account for actual latent storage and computational costs, rather than token count alone. I do not expect superiority in every setting, but would appreciate a clearer picture of the quality–cost trade-offs and the settings in which this representation is advantageous.

Some final reconstructions also appear somewhat over-smoothed to me. I would like to better understand the remaining limitations in recovering fine geometric detail, without assuming that these limitations necessarily arise from the VecSet representation itself.

Questions for authors
  1. How does the method compare with a representative sparse-voxel approach under comparable storage and computational budgets? If such a comparison is not available, could the authors clarify the intended use cases and practical advantages of retaining VecSet?

  2. Could the authors discuss the remaining smoothing in the qualitative results, and whether it is primarily associated with latent compression, field decoding, or surface extraction?

  3. For a potential latent-diffusion extension, when do the predicted anchor positions stabilize relative to the decoded SDF geometry during denoising? Does spatial organization emerge early, followed by geometric refinement, or do both continue to evolve together? For reference-based denoising, positional accuracy could be measured against the target; for free generation, stabilization relative to the final sample would be the relevant observation. I recognize that the current paper evaluates reconstruction, so this is a question about its generative implications rather than a demonstrated limitation of the reconstruction method.

4 comments

We thank the reviewer for recognizing the value of improving VecSet VAEs and introducing explicit spatial anchors. We provide our pointwise replies below.


When to retain VecSet rather than a sparse-voxel VAE.

Our target is single-stage generation through a self-contained latent set: image/noise → latent set → decoded field → mesh. No input 3D shape or FPS coordinates exist at generation time. Align predicts the anchors from the code, rather than requiring them as an external input.

In a TRELLIS-style pipeline [1], the first stage predicts coarse occupancy to determine the active voxel coordinates, and the second stage generates latent features at those coordinates. Its sparse decoder requires both coordinates and features. Without the first-stage occupancy prediction, the decoder lacks its required spatial coordinates. Comparing this pipeline with Align therefore needs to include occupancy generation and its cost.

We have added the XCube coarse VAE variant that requires no pre-generated occupancy [2]. Using its released model and native mesh extraction, the Table 1 comparison is reproduced below as CD (×10³).

MethodFeature codeObjaverseABOIn-the-wild
XCube coarse4096×834.492934.982037.2420
Standard VecSet4096×644.79725.14253.0173
Align4096×643.92284.01112.5230

We report feature storage together with coordinate storage. At 16 bits per feature, our 4096×64 and 8192×64 codes occupy 0.5 and 1.0 MiB, with no stored coordinate array. In our saved SparseFlex reconstruction results, the median representation contains 57,191×8 feature tokens: 0.873 MiB of features plus 0.654 MiB of int32 coordinates. For computation, matched measurements on the same platform give VecSet→Align latent-refinement times of 54.06→56.58 ms at 4096 tokens (batch 1, BF16), excluding mesh extraction. Matched FP16 forward/backward/AdamW steps take 943.5→977.1 ms, with 13.63→15.83 GiB peak memory (batch 1, excluding I/O, EMA and distributed communication). Align provides a fixed-size first-stage interface that needs no separate spatial-support generator and can also supply a later refinement stage.


Remaining smoothing and extraction resolution.

To examine the effect of extraction resolution, we evaluate the same checkpoints at three resolutions. CD (×10³) on the in-the-wild test set is below. 518 reuses published Table 1 results; 768/1024 use the same checkpoints and original BF16 evaluation path, without the later FP32 coordinates/PE modification.

TokensResolutionVecSetLATTICEAlign
40965183.01733.77272.5230
40967682.92883.47222.1975
409610242.80763.33092.3176
81925182.47883.12192.2379
81927682.20052.92651.6758
819210242.25382.77461.8859

Each method aggregates its valid results; shapes with insufficient active voxels are skipped. Align has the lowest CD at every matched resolution and latent budget. The improvement persists across all three extraction resolutions. This comparison is added in Appendix C.1, Table 10 (p. 20).


Predicted anchors during denoising.

We test whether anchors settle early or continue moving as the generated geometry is refined. DiT evolves latent features, and the frozen VAE predicts coordinates from those features. For this diagnostic, we additionally decode the current prediction of the finished shape at selected steps, represented by the clean latent z^0(t)=zt−tvCFG(zt,t,c)\hat z_0(t)=z_t-tv_{\mathrm{CFG}}(z_t,t,c). The intermediate anchors reveal the sampler's current geometric estimate; they are not fed back into DiT and do not change sampling.

We use fixed conditions from the evaluation list, with 2048 tokens and 50 sampling steps. At each recorded step, we measure two gaps to that trajectory's final generated output: (i) mean Euclidean displacement of anchors, matching the same latent-token indices; and (ii) SDF MAE at fixed query points near the final surface, taken from a 65³ grid. Thus, the two measurements track how anchor locations and the decoded surface field approach their final states. For each sample and metric, we divide by its first-step gap before averaging across conditions: 100% is the initial gap, and 0% is the final reference by construction.

Sampling evaluation / 50Anchor gap (% of initial)SDF gap (% of initial)
1100.0100.0
1062.289.4
2027.230.2
303.96.9
401.22.2
500.00.0

Anchors and the decoded field co-evolve, with anchor displacement generally contracting earlier. By evaluation 30, the remaining gaps are 3.9% and 6.9%; both continue changing afterward.

The generation examples are in Section 6.2, Figure 7 (p. 9), with additional comparisons and sampling settings in Appendix E, Figure 11 (p. 27). These demonstrate the intended image-to-mesh pipeline without supplied spatial coordinates.

References

[1] Xiang et al. Structured 3D Latents for Scalable and Versatile 3D Generation. CVPR, 2025.

[2] Ren et al. XCube: Large-Scale 3D Generative Modeling Using Sparse Voxel Hierarchies. CVPR, 2024.

Thank you for the additional experiments.My main question about TRELLIS remains. Its first stage uses very few tokens, so omitting it may save little time overall. What fraction of end-to-end generation time does that stage actually account for? And the cause of the visible smoothing also remains unclear. The resolution sweep shows how CD changes, but does the missing detail reappear at higher extraction resolutions, or is it already absent from the decoded field?

Thank you for the follow-up. We clarify the role of the first stage and the distinction between field fidelity and extraction resolution below.


The role of the first stage.

Our main contribution is improved first-stage geometry at the same latent budget, rather than an end-to-end speedup over TRELLIS. We have not profiled TRELLIS's stage-wise runtime, so we do not have a measured fraction to report. Our motivation does not depend on its first stage accounting for a large share of generation time.

Align generates a mesh from a self-contained latent set without requiring pre-generated occupancy or spatial coordinates. This also makes it relevant to pipelines that refine an initial mesh: Ultra3D [1] voxelizes a VecSet-generated coarse mesh to obtain coordinates for its second stage; LATTICE [2] and UltraShape [3] likewise use coarse generated geometry to guide subsequent refinement. Improving the initial geometry could therefore provide more faithful spatial support for a later stage, including when that geometry is voxelized at finer resolutions. Our image-to-mesh examples are in Section 6.2, Figure 7.


Smoothing and extraction resolution.

We agree that CD alone does not distinguish whether a particular missing feature is under-resolved or absent from the field. At a fixed checkpoint and latent code, the sweep changes the discretization of the same field:

MR=ExtractR(fθ(⋅,z)).\mathcal{M}_R = \mathrm{Extract}_R \left(f_\theta(\cdot,z)\right).

Here R denotes extraction resolution. Higher resolution can reveal features already represented by the field, but cannot recover geometry that the field does not encode.

For a controlled comparison with VecSet and our LATTICE baseline, reconstruction evaluation uses uniformly sampled encoder inputs, without additional sharp-edge samples. This can undersample narrow edge regions and is a possible contributor to reduced sharpness. Evaluating sharp-edge-aware inputs and targeted edge-preservation objectives, matched across methods, is a natural extension to improve detail fidelity.

References

[1] Chen et al. Ultra3D: Efficient and High-Fidelity 3D Generation with Part Attention. arXiv:2507.17745, 2025.

[2] Lai et al. LATTICE: Democratize High-Fidelity 3D Generation at Scale. arXiv:2512.03052, 2025.

[3] Jia et al. UltraShape 1.0: High-Fidelity 3D Shape Generation via Scalable Geometric Refinement. arXiv:2512.21185, 2025.

Reviewer_xzdzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Align-VAE to improve the alignment between attention and local geometry in VecSet VAEs for 3D reconstruction without enlarging the per-shape latent code. By combining a learnable decoder token bank, distance-biased cross-attention, normal-free encoder queries, and a screened Poisson-inspired field regularizer, Align-VAE reduces Chamfer distance on the ABO dataset by 26.6% over the standard VecSet VAE.

Critical strengths
  1. The proposed modifications are logically grounded and directly correspond to the specific geometric and attention mismatches identified in the baseline analysis. Achieving these improvements while keeping the per-shape latent storage fixed is a practical design choice.
  2. The method demonstrates substantial quantitative gains over standard VecSet VAE and LATTICE baselines across multiple datasets, alongside improved preservation of fine local structures.
Critical weaknesses
  1. The manuscript makes it challenging for the reader to quickly grasp the primary motivation before reaching the method section. Streamlining the diagnostic presentation would significantly improve accessibility.
  2. Some foundational claims in the introduction are overstated and lack direct empirical support. For instance, the paper asserts that "Increasing output sampling density cannot recover detail that the code fails to preserve or the decoder fails to retrieve". While latent capacity is certainly important, output sampling resolution (e.g., marching cubes grid density) also plays a critical role in surface extraction, and the paper provides no ablation across varying decoding resolutions to substantiate this claim or isolate decoding density from latent retrieval.
  3. Figure 2, particularly panel (a), which visualizes encoder input selection, suffers from low visual resolution and severe clutter from overlapping points and lines. It is difficult to interpret the physical meaning of Figure 2(a) or visually verify the claimed attention convergence without relying heavily on the summary bar chart in Figure 2(b).
  4. Because Align-VAE integrates four additional components, computational overhead is a natural concern. While Section 6.1 and Table 10 in Appendix C.2 briefly report inference latency, a clearer analysis in the main text is needed. Specifically, the training-time overhead introduced by the three-point normal supervision probes and anchor regression is not quantified, and the anomalous query-time gap in Table 10 requires clarification.
Questions for authors
  1. In Tables 3 and 4, adding the normal supervision regularizer Lnormal\mathcal{L}_{\mathrm{normal}} in variant A4 increases the field-normal angular error from 10.82 (A3) to 14.48, making it worse than the baseline A0 (12.83). Although Section 6.3 and Appendix C.5 note that surface accuracy and gradient orientation are complementary, and that a directional slope constraint does not constrain tangential components, this outcome remains counter-intuitive, given that the normal supervision regularizer was explicitly introduced to address field-normal misalignment. Could the authors provide a clearer, high-level geometric explanation for why supervising normal derivatives may actually worsen overall surface normal orientation? Does this suggest an optimization conflict between zero-level-set fitting and gradient directionality?

  2. Could the authors provide empirical evidence across different output query resolutions to substantiate the claim regarding output sampling density in the Introduction? Additionally, could the authors clarify the training-time overhead and explain the query-latency discrepancy between VecSet VAE† and A0: VecSet in Table 10?

3 comments

We thank the reviewer for recognizing the targeted design, fixed latent storage, and improvements across datasets and fine structures. We provide our pointwise replies below.


Motivation and Figure 2.

The practical setting is to generate a fixed-size latent set from image conditioning and noise, then decode it into a mesh. There is no input point cloud or externally supplied FPS support during generation. The question is how to preserve and retrieve more geometry through this fixed-size interface. In Figure 2(a), each link joins an FPS encoder query to the input point receiving its highest head-averaged attention weight. Many links sharing an endpoint indicate shared information aggregation; they are not geometric correspondences. Figure 2(b) quantifies this behavior across shapes.


Output resolution.

We evaluate reconstruction at matched extraction resolutions. Below are CD values (×10³) on the in-the-wild test set. 518 reuses published Table 1 values; 768/1024 evaluate the same checkpoints using the original BF16 path, without the later FP32 coordinates/PE modification.

TokensResolutionVecSetLATTICEAlign
40965183.01733.77272.5230
40967682.92883.47222.1975
409610242.80763.33092.3176
81925182.47883.12192.2379
81927682.20052.92651.6758
819210242.25382.77461.8859

Each method aggregates its valid results; shapes with insufficient active voxels are skipped. Align achieves the lowest CD in all six matched resolution/budget settings, reducing CD relative to VecSet by 9.7–25.0%. These results show better reconstruction at the same extraction resolution and latent budget. This comparison is added in Appendix C.1, Table 10 (p. 20).


Why does A4 have higher normal angular error?

A3→A4 improves CD from 2.723 to 2.526 but increases angular error from 10.82° to 14.48°. The finite-difference loss constrains a directional slope, not the full gradient orientation. Writing the rescaled gradient as g=an+tg=a n+t, with t⊥nt\perp n, constrains aa but leaves tangential components unconstrained. For example, g=(b,0,1)g=(b,0,1) and n=(0,0,1)n=(0,0,1) have the correct normal derivative but angle arctan⁡∣b∣\arctan|b|. Thus, a lower directional loss is compatible with a higher angular error.


Computational cost and the query-time discrepancy.

We remeasured matched inputs on the same platform with batch size 1, BF16, 4096 tokens, and 4096 query locations, using five warm-up calls and 20 synchronized repetitions:

ModelLatent refinement (ms)Query (ms)
A054.072.212
Standard VecSet54.062.202
Align56.582.191

The A0/full-VecSet query-time gap disappears under matched measurement; these values replace the earlier inconsistent timings in Table 12. Align's latent refinement adds 4.7%, while stored latents remain 512 KiB.

We also measure FP16 forward/backward/AdamW steps on the same platform with matched inputs (batch 1, 4096 tokens, 4096 volume + 4096 near-surface queries; 2048 surface samples when normal supervision is enabled). After five warm-up steps, we average ten steps per input; I/O, EMA and distributed communication are excluded.

Model / normal termStep time (ms)Peak memory (GiB)
A3: no normal term1616.415.272
A4: directional finite difference1628.015.572
IGR: full-gradient normal term1650.517.086
Full-scale VecSet943.513.631
Full-scale Align977.115.829

The first three rows isolate normal-term cost on the same A3 backbone and weights. Directional and IGR supervision add 0.7% and 2.1% step time, respectively; full-scale Align adds 3.6% over VecSet. The training comparison is now discussed in Section 6.3 and tabulated in Appendix C, Table 13 (p. 21).

Thanks for your responses and additional experiments, which thoroughly address my concerns and strengthen my confidence in accepting this paper. I will raise my score.

Thank you very much for your positive reassessment and willingness to raise your score! We are glad that the additional experiments and clarifications addressed your concerns, and we appreciate your constructive feedback.

Reviewer_nXw4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper takes an in-depth look at VecSet-style VAEs, which have their main use case in 3D shape generation. The authors identified several patterns in the inner workings of such attenton-based VAEs, which are interpreted to be sub-optimal. Consequently, the authors propose remidies for each of the 4 identified issues:

  • A) To fight against attention-sink like behvior of a few select tokens, the authors propose to use a bank of register-like tokens in the decoder
  • B) To fight periodic attention behvior in the decoder cross attention, the authors propse to bias the attntion based on the distance from query to anchor.
  • C) to remove the information dis-balance between encoder and decoder queries, the authors propose the zero-out the normal information in the FPS queries fed into the encoder
  • D) To focus supervision more on the surface orientation, the authors propose to use finite differences to align the SDF with the surface normal

the authors compare against several VAE-baselnes with fixed token budgets, across several different datasets, and the conduct detailed ablations, aiming to validate each individual addtion. Compari

Next to typical metrics, the paper also proposes several investigative measures which reveal their findings (A)-(D), which are also analyzed in their ablation experiments.

Critical strengths
  • Reconstruction improvement on VecSet-style VAEs is a worthwhile research are, although it is mainly motivated by the downstream generative models, which are not concidered in this paper. Nevertheless a thourough look at the VAE is certainly interesting. Ultimately, the experiments show significant improvements in reconstruction quality. (Although the downstream gains for generation remain unknown).
  • Secondly, and less importantly according to my intuition, are the analytical tools which are proposed for the identifaction of issues (A)-(D) and for validation the ablations. Ultimately, they are interesting insights/tools, but it remains questionable how important it is that a modern transformer behaves as we, as humans, would like it to behave. In this case, the metrics validate the claims (especially the token bank).
Critical weaknesses
  • My main concern with the paper is the fact, that the single cross-attention path from decoder query to output is assumed fixed. The only way that the queries can interact with the encoding is through this single cross attention.
    • e.g. in combination with periodic position encoding, finding (B) is quite expected for example.
    • more importantly, given a complex/fine-grained geometry, the vector set will not be dense enought to represent it with simple query-to-sdf decoding. consequently, this single cross attention, has to represetn a complex information retrieval from all the tokens (stemming from the decoder self-attention). Since a simple cross-attention layer cannot model such things. Thus, my assumption is that the transformer finds alternative ways of representing fine details, e.g. by strong information non-locally. Partially, my intuition here is supported by the big performance increase from the token bank. So far, I have not thought about register tokens as a means for performance increase, but rather to improve the latent quality or feature distribution. Thus, the big performance increase presented in the paper surprised my. I assume that the added tokens, free of any positional information, can help the decoder with its almost ill-posed task. Compared to (B)-(D), hich all have marginal improvements, (A) is by far more important. A more through analysis on why that is the case would be more interesting to me.
  • The other main concern I have with the paper is the lack of generative modelling experiments. The main motivation for the investigation of such VAEs are their downstream use in generative models. One could even claim that better generative performance is even more important than better reconstruction performance. E.g. I would say that visually pleaseing attention patterns are less important, the inducing a more well-behavaed latent distribution, which can be better learned using a diffusion model, compared to e.g. vanilla VecSet-VAEs. Ultimately, the comunity would benefit much more form such an analysis, e.g. researches working on shape generation still do not know whether it is worth it to use this model for generation.
Questions for authors
  • In ablation Table 3, it seems like A0 performs much worse then VecSet. Matching the numbers to the main table, the ablation experiments are presumable executed on a subset of the in-the-wild dataset. However, I am wondering why A0 is so much worse then VecSet. Can you provide additional information on this?
  • How does you finitie-difference-based normal loss compare to e.g. the normal component of the IGR loss?
  • Have you tried to use registers in encoder? I am aware, that this would change the latent size, but this could be adjusted for.
  • Why does the method not keep position of the FPS points, and instead predicts anchor positions? AIR-Nets for example keeps the positions? Is the reason that positions are less suitable for downstream generative tasks?
  • Have you compared to less compressive VAEs, such as (sparse) grid approaches, e.g. used in TRELLIS (2.0)? How do you justify not comparing against such approaches?
6 comments

We thank the reviewer for recognizing the value of our VecSet analysis, the reconstruction improvements, and the diagnostic tools. We provide our pointwise replies below.


Decoder capacity and the learnable bank.

Our decoder uses a single query-to-latent cross-attention block. The bank adds computation and shared parameters while preserving the per-shape latent code. A0→A1 reduces CD (×10³) from 3.723 to 2.960, and the attention diagnostics show a redistribution of attention.

We tested an A0-based decoder with two consecutive query-to-latent cross-attention blocks, no registers, and the same 4096×64 code. CD (×10³) is:

Method (4096×64 code)ObjaverseABOIn-the-wild
Standard VecSet (Table 1)4.79725.14253.0173
Align (Table 1)3.92284.01112.5230
A0 + second cross-attention block10.595241.347050.8106

The second block does not improve CD over A0 (in-the-wild: 3.723 → 50.811). Evaluation includes the original meshes and their floating components. The Table 1 rows provide the published full-data baselines; the A0/A1 ablation comparison is given above.


DiT generative evaluation.

We compare both methods on the same conditioning images, using 2048 latent tokens, the same DiT architecture, and matched sampling settings. Mean scores are:

MethodDINOv3 ↑ULIP-2-I ↑
VecSet + DiT0.63360.3987
Align + DiT0.65730.4033

The added qualitative comparison is in Section 6.2, Figure 7 (p. 9), directly following the quantitative reconstruction results; further examples and sampling details are in Appendix E, Figure 11 (p. 27). These samples are generated from images and latent noise, without input point clouds or retained coordinates.


A0 versus the main-table VecSet.

Table 1 uses the full training dataset, while Table 3 uses a curated subset to accelerate ablation experiments. All variants in Table 3 are trained on the same subset, ensuring fair comparisons within the ablation study.


Encoder registers and token count.

We tested 4096 FPS tokens plus 256 learnable encoder registers, discarding the 256 registers before the KL bottleneck. Thus, the stored code remains exactly 4096×64, and no additional tokens are generated by DiT. CD (×10³; FP16, extraction resolution 518) is:

MethodObjaverseABOIn-the-wild
Standard VecSet (Table 1)4.79725.14253.0173
Align (Table 1)3.92284.01112.5230
Encoder registers, discarded before KL5.42865.91723.2486

The encoder-register variant performs worse than A1's decoder learnable latents on in-the-wild (CD: 3.2486 vs. 2.960).


Normal supervision versus IGR.

IGR's normal term [1] supervises the full spatial gradient vector. Our finite-difference variant constrains the derivative along the reference normal and avoids backpropagating through input gradients.

We replaced the directional term with IGR’s full-gradient normal term, retaining the TSDF, KL, anchor, and surface losses and adding no Eikonal term. Results are:

DatasetCD (×10³) ↓[email protected] (%) ↑
Objaverse9.266416.78
ABO11.344714.49
In-the-wild5.558727.94

The in-the-wild normal-loss comparison is reproduced below; angular error follows the original field-normal diagnostic:

Normal termCD (×10³) ↓[email protected] (%) ↑Angular error (°) ↓
None (A3)2.72346.7110.82
Directional finite difference (A4)2.52649.1214.48
Full gradient (IGR)5.55927.9412.71

IGR improves angular error over A4, while A4 achieves better surface reconstruction. This comparison is added as Table 4 in Section 6.3. On the same platform, matched FP16 optimizer steps (batch 1, 4096 tokens) take 1616.4 / 1628.0 / 1650.5 ms for no-normal / directional / IGR losses, with 15.27 / 15.57 / 17.09 GiB peak memory. These are forward/backward/AdamW measurements over the same inputs, excluding I/O, EMA and distributed communication.


Predicted versus retained anchors.

Our setting is single-stage latent generation: image/noise → latent set → decoded field → mesh. The shape encoder is not run at generation time, so there is no input point cloud and no FPS coordinates to retain. AIR-Nets [2] obtains spatial anchors from an observed point cloud. A decoder requiring those anchors cannot execute from our generated latent features alone. Retaining FPS coordinates is therefore not a drop-in alternative in this setting; it would require an additional coordinate generator or a different joint representation. Align instead predicts anchors from the generated code itself.


Why not substitute a sparse-voxel VAE?

The relevant distinction is the generation stage. In a TRELLIS-style pipeline [3], the first stage predicts coarse occupancy, which determines the active voxel coordinates; the second stage generates latent features at those coordinates. The sparse decoder then receives both the coordinates and features. A comparison with this VAE must therefore account for the preceding occupancy-generation stage and its cost. Align directly decodes a fixed-size, self-contained latent set into a mesh, predicting its anchors from the latent features.

We additionally compare with the XCube coarse VAE variant that requires no pre-generated occupancy [4], using its released model and native mesh extraction. The added Table 1 comparison is reproduced here as CD (×10³):

MethodFeature codeObjaverseABOIn-the-wild
XCube coarse4096×834.492934.982037.2420
Standard VecSet4096×644.79725.14253.0173
Align4096×643.92284.01112.5230

References

[1] Gropp et al. Implicit Geometric Regularization for Learning Shapes. ICML, 2020.

[2] Giebenhain and Goldlücke. AIR-Nets: An Attention-Based Framework for Locally Conditioned Implicit Representations. arXiv:2110.11860, 2021.

[3] Xiang et al. Structured 3D Latents for Scalable and Versatile 3D Generation. CVPR, 2025.

[4] Ren et al. XCube: Large-Scale 3D Generative Modeling Using Sparse Voxel Hierarchies. CVPR, 2024.

Thank you for providing a detailed rebuttal, and several new experiments. I have some follow-up questions.

  • Small transfer to generative task. The DinoV3 metric improved by ~3.7%, while we do not know how both VAEs behave at 2048 tokens, at 4096 tokens the CD dropped by 36%. What is your take on this? is the latent distribution of the bank to complicated to learn? or is that an issue of the metrics. Visually, your examples look better, but they are few and can be easily cherry picked.
  • X-Cube: why use 8x less latent size? why not use 16x16x16 spatial and 64 latent? or 32x32x32 with 8 feature channels? The comparison does not look fair.
  • How did you implement the second cross attention block? Did you use the standard best practices from the literature for transformers?It looks like the model either competely breaks during training (CD of 50.8 on in-the-wild) or it is heavily overfitting.
  • why do you think that adding registers to the encoder deminishes all the gains compared to the baseline, and even becomes significantly worse than the baseline?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the Area Chair and Reviewers nXw4, xzdz, and 14TU for their constructive feedback. In response, we added the following experiments and clarifications, with corresponding updates to the main text and appendix:

  1. DiT generation and anchor dynamics (§6.2, Fig. 7; Appendix E, Fig. 11). Under matched DiT settings and the same conditioning images, Align improves DINOv3 from 0.6336 to 0.6573 and ULIP-2-I from 0.3987 to 0.4033. We added qualitative comparisons and analyzed how predicted anchors and the decoded field evolve together during denoising.

  2. Architecture and normal-supervision ablations. We tested 256 encoder registers discarded before the KL bottleneck, a second query-to-latent cross-attention block, and IGR’s full-gradient normal term. These address register placement, decoder capacity, and normal supervision while preserving the latent-code size. The normal-loss comparison is added in §6.3, Table 4: directional supervision achieves lower reconstruction CD than IGR (2.526 vs. 5.559, ×10³), while IGR gives lower angular error. We also clarified the full-data versus ablation-subset training settings.

  3. Sparse-voxel comparison and generation setting (Table 1). We added the XCube coarse VAE requiring no pre-generated occupancy across all three evaluation datasets. We clarified why retained FPS coordinates are unavailable during generation and how our self-contained latent representation differs from TRELLIS-style occupancy-conditioned decoding. Our focus is first-stage geometric fidelity at a fixed latent budget.

  4. Extraction-resolution scaling (Appendix C.1, Table 10). At resolutions 518, 768, and 1024, Align achieves the lowest CD in all six matched resolution/budget settings, improving over VecSet by 9.7–25.0%. We also clarified the distinction between finer mesh extraction and detail represented by the decoded field.

  5. Computational cost (Appendix C, Tables 12–13). Matched measurements on the same platform resolve the earlier A0/full-VecSet timing discrepancy and quantify normal-supervision overhead. At 4096 tokens, Align adds 4.7% latent-refinement time and 3.6% training-step time over VecSet.

As the discussion deadline approaches, we would greatly appreciate any further feedback and would be more than happy to address any remaining questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.