acceptodds
Withdrawn from ICLR 2027

CanonAE: Scalable Pose Synchronization for 3D Shape Representation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Many 3D generative pipelines rely on autoencoders (AEs) whose reconstruction quality depends on training-shape orientations. Pose variation complicates learning, while semantic alignment need not optimize geometric representation. We propose CanonAE, which learns canonical orientations from collection geometry and the chosen AE’s reconstruction objective without prescribed semantic frames. Geometric pose synchronization initializes consistent poses, followed by joint pose and AE refinement. Our analysis characterizes local pose sensitivity to changes in matching losses and gives conditions for small subsequent AE-driven pose adjustments. Motivated by this analysis, our hierarchical, symmetry-aware synchronization retains multiple alignment hypotheses and scales to hundreds of thousands of shapes. Experiments on Thingi10K, CanoVerse, and protein densities show consistently lower test reconstruction error across two AE backbones. Ablations attribute most gains to synchronization, with modest benefits from joint refinement. These results support scalable geometric canonicalization for 3D representation learning and motivate its use in latent-space generative modeling.

Primary area:
generative models
Keywords:
3D Shape Generation, Canonical Pose, Auto-Encoder

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6jZMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CanonAE, a framework that learns consistent orientations for 3D shapes to improve autoencoder-based representation learning. The method first constructs overlapping clusters using rotation-invariant descriptors, then performs hierarchical rotation synchronization through weighted rotation propagation, rotation clustering, and MRF-based selection. The synchronized poses are further refined through distributed optimization and used to initialize joint pose-autoencoder learning. The authors also provide theoretical analyses of pose stability and refinement. For validation, experiments are conducted on Thingi10K, ProteinDensity, and CanoVerse with two autoencoder architectures. The results show that CanonAE reduces geometric alignment errors and improves reconstruction accuracy across the evaluated datasets.

Critical strengths
  1. The paper presents a framework for handling ambiguous pose alignment in large 3D shape collections. By retaining multiple relative rotation hypotheses and enforcing consistency within and across overlapping clusters, the method accounts for ambiguities arising from shape symmetries. The hierarchical design also enables distributed synchronization, and the method is evaluated on the approximately 320K-shape CanoVerse dataset.

  2. The experimental evaluation covers diverse datasets and autoencoder architectures. The authors evaluate CanonAE on Thingi10K, ProteinDensity, and CanoVerse using two different AE backbones. The paper also includes ablation studies examining the effects of multiple rotation hypotheses, continuous refinement, and joint optimization, providing further evidence for the contributions of the main components.

Critical weaknesses
  1. The reconstruction results are based on a single training run per configuration. This leaves the stability of the reported improvements unclear, particularly when some of the gains from joint refinement are relatively small.
Questions for authors
  1. Could the authors comment on the stability of the reported reconstruction improvements across different training runs? The current results are based on a single run per configuration, and some of the improvements, particularly those from joint refinement, are relatively small. It would be helpful to understand whether the authors have observed similar trends across different runs.

  2. Could the authors further clarify the contribution of joint pose-AE refinement? The ablation shows only a 0.3%-0.5% improvement over using fixed synchronized poses, suggesting that most of the reconstruction gain comes from the initial pose synchronization. It would be helpful to better understand the additional benefit provided by joint refinement.

  3. Could the authors clarify the test-time pose estimation procedure for large training collections? Each test shape is aligned to its 16 nearest training shapes in the rotation-invariant descriptor space, which may require retrieval over a large training collection such as CanoVerse. It would be helpful to understand how this retrieval is implemented in practice and how the relative rotation estimates from the retrieved shapes are aggregated to initialize the test pose.

Reviewer_PCgZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes that synchronizing poses in 3D datasets so that inputs are geometrically aligned improve the representations learned by encoder-decoder architectures. To allow for an scalable synchronization process, the authors introduce a three-stage framework based on clustering. The authors provide empirical evaluations on two 3D shape datasets and one protein density dataset.

Critical strengths
  • The hypothesis that geometrically-aligned poses might ease learning and lead to better representations is explored adequately across several experimental scenarios.
Critical weaknesses
  • The usefulness of the method is not entirely supported, due to many factors. While the obtained reconstructions are indeed slightly better, let's recall that a representation's quality is usually measured by how useful it is for solving downstream tasks; most representation learning methods are usually benchmarked on metrics such as classification accuracy or generation quality. However, the paper only evaluates its learned representations in downstream tasks in the generation experiment of Section 5.2, which presents several weaknesses. Firstly, note how the reconstructions in Table 1 compare e.g. a 3DS2VS baseline trained by the authors, against the CanonAE-version of 3DS2VS also trained by the authors, which is correct. For generation, however, the authors pick an off-the-shelf checkpoint of 3DS2VS for their baseline computations, compared against their CanonAE version of 3DS2VS. This in itself is a problem because we can't attribute gains to either of the architectures since they were trained in potentially different settings (different learning rates, training iterations, batch sizes, optimization differences, etc). This weakens the comparison in Section 5.2. Not only that, but the generation quality is assessed only qualitatively. We don't have a quantitative metric comparing CanonAE against the 3DS2VS baseline. Finally, the generation is studied (for some reason) on ShapeNet, which is not among the three datasets presented as the scope of the paper's empirical evaluation. Why is there no corresponding ShapeNet reconstruction comparison? The paper also gives no experimental details or quantitative results on how the ShapeNet CanonAE poses and latents were obtained. Overall, this weakens the only downstream-task evidence that the paper provides. Furthermore, such evidence covers a single dataset (ShapeNet), a narrow-scoped evaluation.

  • Table 1 and ablation results report no error bars or standard deviations over multiple seeds. Many differences are very small, and therefore they might be within noise. It is therefore unclear whether the improvements are real, and whether some components of the method are doing real work. In this regard, the ablation on 5.4.2 on no alternate optimization shows that this component barely bears any difference in reconstruction. A more simple version of this method would be that of using pose synchronization as a preprocessing step, and then running any kind of AE normally. This is arguably simpler and more elegant, and provides most if not all the benefits.

  • The main body of the paper lacks mathematical clarity. Several formulas are introduced without prior definitions of their components. For instance, equations (1) and (2) appear in the introduction with many undefined terms, and Sections 3 and 4 (which represent the in-depth method sections) never reintroduce them formally. The confusion carries through the paper; for instance, Section 4 opens by proposing a “scalable approach” to solving equation (2), but the reader never had a chance to grasp equation (2), since it was never properly derived or defined. A critique in the same spirit is that the problem setting is never made explicit, which makes every section hard to follow. For instance, Section 3.1 is the first method section, where the setting should be established. It starts by assuming a fixed connected weighted graph, with no explanation of whether the graph represents e.g. a datapoint or some other encoder property of the data that the authors aim to study. In this same section, losses and the “stationary alignment” are also defined informally, and it is unclear why they are introduced or how they connect to the exact problem that the authors aim to solve, since the problem itself has not yet been defined clearly and formally. The reader can probably infer the method despite these obstacles, but overall the writing and clarity are of very poor quality.

  • The paper shows how pose synchronization can plausibly aid learning in vanilla autoencoders. On the other hand, I believe group-invariant/equivariant autoencoders attack the same problem through architectural invariance rather than explicit pose alignment, e.g. IE-AE [1], SGM [2], and RECON [3] (incidentally, all these works are missing from the related work section and are relevant to both the “Joint shape-and-pose learning” and “Canonicalization of 3D collections” subsections). Since these methods use a group-invariant encoder, their reconstruction loss is be independent of input orientation (I believe this to be true?), in which case CanonAE would provide no benefit. A comparison is needed, at a minimum theoretically and ideally empirically: does CanonAE bring any advantage over such methods, and how large is the practical gap in representation learning performance? This also relates to L123: “These methods jointly model intrinsic shape variation and pose, making optimization susceptible to pose ambiguity and poor local minima.” This may be true in theory. But does it translate into empirical disadvantages relative to pose synchronization? In practice, such methods might be competitive with approaches that build in group equivariance/invariance priors. Showing this is important to convince the reader of the method’s value.

  • Appendix D.1 introduces several critical assumptions whose failure modes are not discussed. Where can they fail? For instance, are there specific instances in any of the experiments where they break, and how does that affect the behavior of the method?

  • No analysis of the number of neighbors k is provided. It is mentioned that k is chosen large enough to satisfy the convergence conditions of App. H, but it is unclear how to choose k in arbitrary scenarios. A sensitivity analysis on k would enable practitioners to apply the method succesfully. So would a discussion of what happens, theoretically and empirically, when a “wrong” k violates the convergence assumptions.

  • Section 5.1 mentions that, at test time, each shape is aligned to its 16 nearest training shapes, unlike the single forward pass of other canonicalization/AE methods. This test-time cost should be reported as part of the price paid, e.g., throughput of this method vs. the baselines.


References:

[1] Winter, R., Bertolini, M., Le, T., Noe, F., & Clevert, D. A. (2022). Unsupervised learning of group invariant and equivariant representations. Advances in Neural Information Processing Systems, 35, 31942-31956.

[2] Allingham, J. U., Mlodozeniec, B. K., Padhy, S., Antorán, J., Krueger, D., Turner, R. E., ... & Hernández-Lobato, J. M. (2024). A generative model of symmetry transformations. Advances in Neural Information Processing Systems, 37, 91091-91130.

[3] Urbano, A., Romero, D. W., Zimmer, M., & Pokutta, S. (2026, April). RECON: Robust symmetry discovery via explicit canonical orientation normalization. In International Conference on Learning Representations (Vol. 2026, pp. 7131-7163).

Questions for authors

1- What does “geometric compatibility” mean (L34)? The introduction would benefit from more explicit explanations, ideally with a running example.

2 - Only train/test splits are described (L370-371). Without a validation set, how were hyperparameters (e.g., k) tuned, and how was the reported checkpoint selected for each configuration?

3 - Can the authors provide standard deviations over multiple seeds for Table 1 and the ablations?

4 - Can the authors provide quantitative generation metrics, with a baseline AE trained under the same protocol as CanonAE?

5 - How does CanonAE compare, theoretically and/or empirically, to invariant/equivariant autoencoders?

6 - What are the complexity and runtime relative to naive synchronization? Discussing this would speak to the "scalability" claim in a more direct manner. And what is the test-time optimization cost of the method, relative to the baselines?

Reviewer_BW22OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes learning canonical orientations from collection geometry and using them to initialize poses for autoencoder training, with the goal of reducing reconstruction error. CanonAE first constructs overlapping clusters of similar shapes, then synchronizes shape rotations within and across clusters by propagating multiple rotation candidates and selecting among them with an MRF, followed by continuous refinement. The synchronized poses are used for AE training in place of the original poses, with network parameters and training poses updated alternately. The method is evaluated on two shape datasets and one protein dataset with two autoencoder backbones, and achieves the lowest alignment error and test MAE in all settings.

Critical strengths

None

Critical weaknesses
  • The evaluation is done only with MAE, while previous methods like 3DShape2VecSet use IoU/Chamfer/F-score to assess reconstruction quality, and a quantitative comparison with previous methods is missing.
  • In Table 1, the alignment error and test MAE are not necessarily aligned. For example, “no sync” shows a larger alignment error but better MAE compared to the other baselines, which seems to weaken the core motivation of this paper.
Questions for authors
  • Besides MAE, have the authors tried to evaluate reconstruction quality?
  • Could the authors explain more about the correlation of the alignment error and test MAE?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.