acceptodds
Withdrawn from ICLR 2027

Structured 2D Atlases via Geodesic Optimal Transport for 3D Representation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent 3D generation pipelines commonly employ Variational Autoencoders (VAEs) to learn compact latent representations of 3D shapes. However, learning high-resolution 3D representations at scale is highly GPU-memory intensive, and existing Shape VAEs often rely on substantial surface subsampling, coarse 3D discretization, or large model capacity to make training tractable. We propose Structured 2D Atlases, which reorganize dense oriented 3D surface samples into a regular image-space representation while retaining their original 3D positions and surface normals. Our method establishes a bijection between surface samples and fixed Fibonacci-sphere targets using hierarchical optimal transport guided by intrinsic geodesic distances, followed by deterministic equirectangular indexing into square position and normal maps. An atlas stores oriented surface samples, enabling an atlas-based transformer Shape VAE to encode and decode 3D surface geometry through a regular 2D domain rather than a 3D-native representation. Experiments show that our atlas-based Shape VAE uses to fewer parameters than the evaluated 3D-native Shape VAEs, providing a favorable trade-off between model size and fine-scale geometric detail. These results demonstrate that Structured 2D Atlases provide a parameter-efficient alternative for dense 3D surface representation learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Geodesic Optimal Transport, Structured 2D Atlas, Image-Space Shape VAE, 3D Representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tmopOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores structured 2D atlases for 3D representation learning. It uses geodesic optimal transport to organize sampled surface positions and normals into a 2D representation, and evaluates reconstruction using an image-space Shape VAE.

Critical strengths

I find this an interesting and worthwhile exploration of a representation direction. Although the results are not uniformly state of the art, I learned something useful from the approach. The relatively small 33.1M-parameter reconstruction network and its reconstruction results are encouraging. Overall, I lean toward acceptance because of the value of this exploration.

Critical weaknesses

The final reconstructions still appear to lose substantial fine geometric detail. My main concern is whether the high-frequency variation in the atlas, particularly discontinuities in XYZ coordinates, can be faithfully preserved through VAE compression and decoding. Smoothing across such discontinuities could introduce incorrect intermediate positions rather than merely a small visual loss of sharpness. This is a possible explanation for the observed detail loss, not an established cause.

The overall reconstruction results do not isolate how accurately these discontinuities survive compression. More targeted evidence would help clarify the limitations of the representation and its reconstruction model. Its implications for generative modeling are also uncertain: successful reconstruction alone does not establish that the resulting representation is easy for a diffusion model to learn.

Questions for authors
  1. How accurately are XYZ discontinuities preserved after VAE encoding and decoding? Could the authors provide localized input-versus-reconstruction comparisons or errors around discontinuities, together with the corresponding 3D geometry?

  2. What is the contribution of the continuity masking and boundary/seam losses to preserving these structures? Can the authors distinguish detail lost during VAE reconstruction from the effects of subsequent surface reconstruction or refinement?

  3. Do the authors have evidence or analysis of how the high-frequency atlas structure affects generative modeling? In particular, can the VAE preserve the relevant detail while producing latents that are tractable for a diffusion model? I recognize that the current experiments focus on reconstruction and regard this as a question about the approach's future generative potential.

Reviewer_pMeXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a structured 2D atlas representation that reorganizes dense, unstructured 3D surface samples into regular 2D position and normal maps, enabling the direct application of 2D image architectures to 3D shape modeling. The core technical contribution is a hierarchical Gromov-Wasserstein (GW) assignment that approximates full pairwise geodesic optimal transport using K=16 landmarks and recursive balanced partitioning, mapping surface patches to fixed Fibonacci-sphere target blocks at an 8×8 resolution with 64×64 leaf assignments. A deterministic equirectangular indexing scheme — latitude ranking followed by longitude ranking — assigns each surface patch a unique 2D pixel address that is consistent across shapes. To maximize geometric information within a fixed sample budget, the paper introduces importance-based oriented sampling that allocates points between broad coverage (uniform area-weighted) and fine geometric detail (unsigned dihedral angle density) using a tunable balance. The decoded atlas feeds into a Swin Transformer-based Shape VAE (33.1M trainable parameters, 4 non-overlapping 4×4 patches → 128×128 tokens of width 192, 64×64×128 latent, no feature skips bypassing the bottleneck), trained on 174,149 shapes from Objectverse, ShapeNetCore, HSSD, ABO, Thingi10K, and GSO with a composite loss combining weighted Chamfer (L_pair), bidirectional surface (L_surf), point-to-plane (L_plane), and boundary-aware atlas regularization (L_reg) with KL penalty.

Critical strengths

S1. The structured 2D atlas formulation is a novel and principled contribution to 3D shape representation. This work formulates 3D surface parameterization as a hierarchical geodesic optimal transport problem to a fixed Fibonacci-sphere target, producing a canonical 2D atlas that is consistent across shapes by construction. The deterministic equirectangular indexing is an elegant solution to the correspondence problem that avoids learned alignment or template deformation. This formulation directly enables the use of mature 2D image architectures (Swin Transformer, U-Net variants) for 3D shape modeling without architectural modification, which is a meaningful practical benefit. The atlas representation is likely to inspire follow-up work on geometry-aware 2D parameterizations for other 3D tasks (e.g., texture synthesis, shape editing).

S2. Remarkable parameter efficiency with robust reconstruction stability. At 33.1M trainable parameters — 5.3× smaller than TripoSF (178.8M) and 25.0× smaller than TRELLIS.2 (828.6M) — the model achieves a 99.8% reconstruction success rate on Toys1k, matching or approaching the stability of heavily parameterized 3D-native baselines (99.3%–100.0%). This demonstrates that despite drastically reducing the parameter footprint by up to 25×, the structured 2D parameterization maintains remarkable reconstruction stability without collapsing into degenerate shapes. For resource-constrained settings where memory footprint and deployment efficiency are prioritized over ultra-fine details, this represents a highly pragmatic trade-off.

Critical weaknesses

W1. Narrow evaluation domain with unverified topological robustness. The quantitative evaluation is conducted exclusively on the held-out Toys1k benchmark (1,000 synthetic toy shapes). Given that the Shape VAE is trained on a massive and diverse corpus of 174k assets, restricting the benchmark to toys is overly narrow and risks masking fundamental representation limits:

  • The mapping assigns surfaces bijectively to a genus-0 Fibonacci sphere. However, complex 3D shapes frequently feature non-zero genus (e.g., handles, through-holes), interior structures, open boundaries, or disconnected components.
  • For disconnected parts, the authors rely on heuristic "virtual Euclidean bridges" (Appendix A.2) to allow graph traversal. There is no category-specific analysis or cross-dataset evaluation on architecturally intricate or topologically challenging benchmarks (e.g., ShapeNetCore test splits or complex Objaverse subsets) to demonstrate how the atlas handles non-spherical topology in practice.

W2. Uncontrolled trade-off between model capacity and representation efficiency. The paper claims substantial parameter efficiency (5.3× to 25.0× fewer parameters than baselines), but this is evaluated across fundamentally mismatched systems without isolating network capacity:

  • The proposed model employs a compact 33.1M Swin Transformer, whereas baselines like TRELLIS.2 (828.6M) and TripoSF (178.8M) utilize substantially larger networks.
  • Concurrently, the method trails top baselines significantly on reconstruction fidelity: CD-L1 is roughly 2× worse than TRELLIS.2 (0.0085 vs. 0.0042), HD95 is over 3× worse (0.0268 vs. 0.0082), and [email protected] shows a stark drop (77.1 vs. 95.9).
  • Without a capacity-matched baseline (e.g., downscaling a 3D-native baseline to ~33M parameters, or scaling the Swin atlas model up to ~180M), it remains unclear whether the parameter compactness is a genuine advantage of the 2D atlas representation or simply the trivial consequence of evaluating an under-parameterized model that sacrifices fine-scale geometric detail.
Questions for authors

Q1. Capacity-Matched Baseline Comparison: To disentangle whether the 5.3×–25.0× parameter reduction stems from the intrinsic efficiency of the 2D atlas representation or simply from evaluating an under-parameterized model, could you provide a capacity-matched comparison? Specifically:

  • What happens to the reconstruction fidelity (CD-L1, HD95, and particularly [email protected]) if your 2D Swin Transformer architecture is scaled up to ~180M parameters (comparable to TripoSF)? Does it close the quality gap with top 3D-native baselines, or does performance saturate due to the fixed 5122512^2 atlas resolution?
  • Alternatively, how does a standard 3D-native Shape VAE baseline perform when its backbone is intentionally scaled down to a comparable ~33M budget?

Q2. Cross-Dataset Generalization Beyond Toys1k: Given that the Shape VAE was trained on a massive multi-source corpus of 174,149 shapes containing complex CAD and everyday objects, why was evaluation confined exclusively to the Toys1k synthetic benchmark?

  • Can you provide zero-shot evaluation results on a more architecturally diverse, held-out test split?
  • Does the performance gap between your method and 3D-native baselines widen or narrow when evaluated on shapes with sharper industrial features, thin planar boundaries, and internal parts?
Reviewer_F3ZwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The proposed work converts mesh input into structured 2D atlases of six channels (positions and normals), then trains an Image-space shape VAE. It aligns importance-based sampled mesh points to fixed Fibonacci-sphere targets using a hierarchical geodesic optimal transport and assigns targets to regular image pixels. It is trained on 174,149 shapes and on held-out Toys1k, the 33.1M-parameter model reports 99.8% reconstruction success, CD-L1 0.0085, HD95 0.0268, and [email protected] 77.1 (Table 2, p. 9). These are a parameter-quality trade-off: the model uses 5.3×–25× fewer reported parameters than the evaluated 3D-native VAEs, but ranks third on CD-L1 and HD95 and trails the highest F-score methods. The authors identify smoothing of fine details and approximate transport as limitations (§5, p. 9; Appendix A).

Critical strengths
  • The atlas constructor is deterministic and bijective. It reorders the sampled points to form target pixels without interpolating or dropping any samples. This provides a dense representation of oriented geometry for the shape VAE.
Critical weaknesses

1. Parameter-efficiency headline

  • Parameter count comparison isn’t enough, there should be comparison based on FLOPS or throughput --> Report peak GPU memory and throughput for encoding and decoding, for the proposed model and the open-source baselines.
  • The latent compression is just 3×3\times from (512×512×6512\times512\times6) to (64×64×12864\times64\times128), which is less than the 4$\times$ reduction in TripoSG’s latent --> Train one reduced-latent variant, such as 64×64×6464\times64\times64 or 64×64×3264\times64\times32, to show quality at a latent budget comparable to the baselines.
  • The 25×25\times fewer parameters wrt TRELLIS.2 (5123512^3) come at a cost of 2×2\times on CD-L1 (0.0042 vs 0.0085), about 3×3\times on HD95 (0.0082 vs 0.0268), and +18.8 on [email protected]. A more justified claim would be on a method where it is also able to beat all the reconstruction metrics (for eg. TRELLIS (SLat) and Dora-VAE with 5.3–5.8× fewer parameters). Instead of the inflated 25$\times$ fewer parameters --> Bring the abstract and contributions in line with what Table 2 supports.

2. Claimed geodesic OT is not compared against Euclidean OT on the trained model

  • Tables 5- 6 claim that Geodesic OT is better on S95S_{95} metric, but its influence on the overall trained model is not reported --- Train the same VAE on random, Euclidean-OT and geodesic-OT atlases. This is crucial for establishing the novel use of complex geodesic computation in atlas creation.
  • Section 3.2 should be explained before Section 3.1

3. Evaluation doesn’t compare like with like

  • Table 2: The proposed work compares the raw decoded points (inherits importance-sampling) with baseline’s points sampled from extracted meshes (inherits mesh extraction error). Chamfer distance and F-score are sensitive to point density. --> Compare proposed and baseline on points from extracted meshes.
  • How the 100k evaluation points are drawn from those 262,144 is never said, and neither is how the reference points are sampled --> Specify this detail
  • Training shapes are used for visual comparison in Fig.8 and 9. Although it has been specified in the A.5 Qualitative scope. --> Held out test shapes should be used for visual comparison.

4. The baseline numbers need a sanity check.

  • Hunyuan3D 2.1 looks broken. It has the same parameter count as 2.0 (327.7M) but is 3.2$\times$ worse on CD (0.0347 vs 0.0109) and 3.8$\times$ worse on HD95 (0.1955 vs 0.0510).
  • No alignment raises the stakes. Because evaluation uses no alignment at test time, any mismatch in a baseline’s orientation or scale convention counts entirely against that baseline. Per-baseline settings (decode resolution, token count, normalization) aren’t reported.
  • There are no confidence intervals. The paper’s margin over TripoSG and TRELLIS on CD is only about 10% (0.0085 vs 0.0093 and 0.0094). “Ranks third” may not hold up statistically.
  • F-score and Chamfer disagree. The model beats TripoSG on CD but trails it by 12.5 points on [email protected] (77.1 vs 89.6). My interpretation, not established: many decoded points may sit just beyond the 0.01 threshold, a uniform off-surface “fuzz.” F-scores at several thresholds would show this.
Questions for authors
  1. Which elements are claimed as novel relative to Gaussian Atlas, SPGen, and Geometry Images? Please distinguish the fixed-target permutation of sampled mesh points and hierarchical graph-geodesic matching from prior sphere-to-grid or image-domain representations.
  2. What checkpoints, inference settings, training-data scope, and parameter-counting rules were used for the Table 2 baselines? How is reconstruction success defined, and are upstream feature extractors or generative components excluded consistently?
  3. Other questions are marked after --> in Critical weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.