acceptodds
Under review as a conference paper at ICLR 2027

What Makes a Latent Space Shared? Direct and Recoverable Compatibility in Motion VAEs

Abstract

A shared latent space is often assumed to make representations from different schemas directly interchangeable, yet sharing parameters, dimensionality, or a prior does not determine how corresponding instances are expressed in latent coordinates. We show that this assumption conflates recoverable correspondence, direct compatibility, and decoder reuse. In a controlled multi-schema VAE on HumanML3D, even identical inputs yield only – raw Recall@1 and – paired-token cosine, despite correspondence being almost perfectly recoverable after linear calibration. Cycle consistency strengthens recoverability but still leaves direct retrieval and token agreement near zero. In contrast, only paired Rotations clips raise raw Recall@1 to – and token cosine to approximately , making most post-hoc alignment unnecessary, although crossed decoding remains substantially worse than native reconstruction. We further find that reducing measured marginal discrepancy does not imply schema invariance. These results show that a shared backbone or prior does not ensure directly compatible latent coordinates, while limited paired correspondence can substantially improve direct compatibility in the tested Rotations setting.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
human motion representation, shared latent spaces, representation identifiability, cross-schema learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FKLiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies shared latent spaces in motion VAEs. It finds shared backbones or priors cannot guarantee directly‑compatible latent codes. Unpaired methods enable post‑hoc recoverable matching but poor direct alignment. A small number of paired motion clips greatly improves direct compatibility, yet cross‑decoder reuse still remains imperfect. It stresses evaluating actual functions instead of just shared model structures.

Critical strengths
  1. The paper untangles mixed definitions of shared latent space common in multi‑schema representation research. It draws distinct boundaries between four concepts: direct compatibility, recoverable correspondence, marginal similarity and decoder reuse. This framework fixes loose, unexamined assumptions and gives structured evaluation guidelines for future motion representation work. (86 words)

  2. It uses modified HumanML3D motion datasets and a suite of diagnostic metrics including Procrustes, CCA, retrieval and decoder‑reuse tests. Experiments prove shared VAE backbone or prior cannot guarantee interoperable latents. Controlled conditions separate schema variations, producing trustworthy results for cross‑skeleton motion model developers. (84 words)

Critical weaknesses
  • All experiments rely on synthetic HumanML3D variants with ground‑truth data alignments. Real‑world motion data is noisy, unaligned and comes from different capture sources. Key results are valid mainly for the Rotations setting, while fully disjoint feature schemas require complete paired supervision. It remains unclear whether these findings translate to real‑world use cases and reduces the paper’s overall impact.

  • The paper treats decoder reuse as an important independent property but only tests one‑way B‑to‑A decoding. It does not investigate root causes behind stable cross‑decoder performance gaps, even when latents are well‑aligned. No mitigation strategies are validated. Since cross‑schema transfer hinges on decoder interoperability, a vital practical problem stays unresolved.

Questions for authors
  1. Your strong direct‑compatibility result with few paired samples is shown only for the Rotations schema pair. Please explain which structural or feature‑overlap properties let Rotations work well under limited‑pair supervision. Also outline what obstacles stop this low‑pair effect from appearing for the Disjoint schema setting.

  2. You see a steady B→A cross‑decoder performance gap for Anchor‑64, despite strong agreement across latent tokens. Within your controlled test environment, do you have experimental evidence to tell whether this gap comes from schema‑driven decoder bias, latent‑space manifold mismatch, or other identifiable sources?

  3. Your calibration‑budget analysis shows flexible CCA delivers good retrieval results even for random encoders when plenty of paired data exists. For real‑world evaluation without ground‑truth correspondences, how should practitioners separate true learned cross‑schema structure from side‑effects caused by strong post‑hoc calibration?

Reviewer_Gt5LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This controlled study distinguishes direct latent compatibility, correspondence recoverable through calibration, marginal similarity, and decoder reuse. Two schema-specific interfaces share a VAE backbone and prior while receiving derived HumanML3D views. Across reconstruction, cycle, marginal, and paired-anchor objectives, strong calibrated retrieval can coexist with poor direct token agreement. Limited paired supervision substantially improves direct compatibility in Rotations, but does not restore native-quality cross-decoding. The paper presents diagnostics and a qualified coordinate-reparameterization analysis rather than a new general-purpose alignment algorithm.

Critical strengths
  1. A useful distinction backed by complementary measurements. Retrieval after fitting a map is not treated as interchangeability. Clip retrieval, synchronized-token comparisons, and matched-checkpoint cross-decoding reveal different failure modes. In particular, the remaining decoder gap after strong Anchor-64 alignment makes the distinction consequential for building interoperable motion interfaces.

  2. Careful controls and appropriately bounded conclusions. Training-only calibration, random encoders, shuffled pairs, mirror-family handling, paired-subset sensitivity, alternative pair losses, and a temporal-backbone check address substantial alternative explanations. The authors disclose repeated exposure to paired clips, unequal compute, the shared-channel Rotations setting, and the limits of the invariance argument. This experimental discipline makes the study informative without requiring a large new model.

Critical weaknesses
  1. The demonstrated generality remains deliberately limited. All views come from the same underlying corpus; the low-pair result uses a feature subset of the native schema, whereas Disjoint uses full pairing. The TCN check broadens architectural coverage but is not parameter-matched. These acknowledged boundaries do not invalidate the controlled findings, but limit their immediate transfer to independently collected or asynchronous motion domains.

  2. The conceptual novelty is incremental relative to established alignment and identifiability principles. Coordinate ambiguity, insufficient cycle constraints, and the limitations of marginal matching already have substantial precedents. The strongest contribution is therefore the operational diagnostic hierarchy and its empirical separation in motion VAEs, not a new identifiability theorem or alignment objective. A sharper synthesis of which interface-design decisions these experiments change would strengthen the paper's significance.

Questions for authors
  1. Contribution clarification: Which empirical result or combination of diagnostics provides the most consequential advance beyond existing representation-alignment analyses? Please connect it to a concrete choice among post-hoc calibration, paired-coordinate alignment, and direct decoder compatibility, without claiming a universal decision rule.

  2. Practical interpretation: What conclusion should a practitioner draw from strong Anchor-64 retrieval but persistently worse crossed decoding? Please distinguish numerical alignment from information missing in Rotations; the Identical control and token-map results are valuable existing evidence, not a request to repeat them.

  3. Reproducibility: Can you clarify the availability of schema-construction and evaluation code, exact paired subsets, saved configurations/checkpoints, and per-run hardware and runtime? The detailed protocol is helpful, but the local submission materials do not include an executable package.

Reviewer_UBcfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether a shared VAE produces directly compatible latent representations for different forms of motion data. The authors test direct matching, matching after alignment, and decoder reuse. The experiments show that sharing a model does not mean that its latent representations can be used interchangeably. Good matching also does not guarantee good reconstruction with another decoder.

Critical strengths
  1. The paper provides a useful insight that sharing a model does not mean its latent representations can be used interchangeably. It clearly distinguishes similar latent distributions, matching after alignment, direct matching, and decoder reuse.
  2. The carefully designed experiments show that poor direct matching can occur even with identical inputs. The authors also test token agreement and decoder reuse, so the findings do not rely only on retrieval scores.
Critical weaknesses

None

Questions for authors
  1. Given that all data forms come from HumanML3D and the low-pair result is tested only in the Rotations setting, as noted in Appendix I, could the authors clarify whether these findings also hold across different datasets, skeletons, or noisy observations?
  2. Could the authors clarify how these findings can benefit the research community?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.