acceptodds
Under review as a conference paper at ICLR 2027

Retargeting Motions to Diverse Skeletons via Learnable Flattening

Abstract

Cross-structural motion retargeting aims to transfer motion between different skeletal topologies. Despite recent progress, existing state-of-the-art models struggle with reliability in zero-shot settings, i.e. skeletons with different topologies which were unseen during training, and recent Transformer-based attempts have failed to outperform specialized geometric methods. We bridge this gap with a Transformer Autoencoder that learns a topology- and translation-invariant latent space. Our core contribution is a learnable flattening of skeletal graphs that captures both local dependencies and global structure. Unlike the standard transformer architecture, which adds positional information to token content, we integrate graph-based positional encodings multiplicatively, a design choice that follows directly from our flattening formulation. The resulting model handles diverse skeletal topologies within a single unified architecture and trains in a fully unsupervised manner, requiring no paired retargeting data. Ablation studies show, that the graph encodings, multiplicative formulation, and Transformer backbone is critical for the performance. In zero-shot evaluations, our method reduces global joint position error by % over current benchmarks. A user study (), including expert animators, further ranks our approach highest in motion alignment and physical plausibility (). These results demonstrate that our model design is key to making transformer architectures effective for motion retargeting, outperforming existing approaches.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Motion retargeting, unsupervised learning, transformer, graph neural networks, positional encoding, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aSCrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an unsupervised Transformer autoencoder for motion retargeting between humanoid skeletons with different joint counts and connectivity. Its learnable flattening operation multiplies projected joint features by skeleton-dependent masks generated by GraphSAGE and aggregates them into a fixed-dimensional pose representation. The model represents the root trajectory separately and predicts it in global coordinates. Training combines cycle consistency, adversarial learning, and augmentation consistency without paired retargeting data. Experiments evaluate reconstruction and retargeting on unseen datasets, alongside architectural ablations and a perceptual user study.

Critical strengths
  • The model handles unseen humanoid skeletons without paired training data. Although it was not trained on Mixamo motions or skeletons, it outperforms SAME in joint-position, joint-rotation, and root-trajectory reconstruction on all three evaluation datasets (Table 1). For instance, on Bandai-Namco, the joint-position error falls from 12.40 to 2.63 cm. The reported inference cost is also low (Appendix A.4).
  • The multiplicative-versus-additive comparison provides useful design evidence. Replacing multiplicative encoding with additive encoding raises intra and cross-structure GJP from 1.45/1.28 to 2.06/2.01 (Table 4). The augmentation and individual-loss ablations further help distinguish the contributions of the architecture and training procedure.
Critical weaknesses
  • Table 2 does not provide convincing evidence for the claimed 43–47% improvement under comparable conditions. Appendix A.5 specifies 95 Mixamo clips per character, but the Aberman values in Table 2, 2.76/2.25, exactly match the published SAN results from the original 106-clip protocol. The PALUM values, 2.72/5.67, also match its published height-normalized MSE values multiplied by 1,000, whereas this paper defines GJP using mean Euclidean distance. PALUM additionally averages across different seen/unseen character and motion settings. It is unclear whether these values were recomputed using the current paper’s metric and evaluation subset.
  • The paper overstates what the ablations and user study establish. The flattening decomposition motivates multiplicative masking, and Table 4 shows it outperforms the tested additive variant. This is a useful result, but the broader claim that the formulation is what makes Transformers effective for retargeting needs more support. The no-PE and GAT variants have very large errors, with much of the GAT error attributed to root-trajectory prediction, making it difficult to isolate the role of structural encoding. Variability across training runs is not reported either. The abstract also overstates the user-study findings: Table 3 shows no significant alignment improvement over Aberman, and the measured criterion was perceptual quality rather than physical plausibility.
Questions for authors
  • Does the multiplicative-versus-additive gap hold across independent training runs? Were the architecture and hyperparameters selected using validation data separate from the Mixamo evaluation set?
  • Which evaluated skeleton pairs have different numbers of joints along corresponding chains? For example, how is motion distributed when the target has five spine joints, and the source has two? An example of the resulting rotations and learned masks would help clarify this behavior, including how twist is handled beyond what position-based GJP captures.
  • What is known about how Adobe produced the Mixamo ground-truth pairs? I would also appreciate clarification of the practical advantage over conventional retargeters such as MotionBuilder/HumanIK, particularly the required joint correspondences and preparation effort. This asks for context on the intended use case rather than a new benchmark.
2 comments

We thank Reviewer aSCr for the thorough review and for highlighting the strong zero-shot generalization results and the value of the multiplicative vs. additive comparison as design evidence. Please find our responses below.


RE: GJP definition and baseline comparability

See general response.


RE: Abstract — "physical plausibility" and user study overstating

See general response.


RE: Local joint position GJP — isolating root-trajectory contribution

See general response.


RE: Variance of multiplicative vs. additive gap across training runs

The training variance across runs has not been reported, and the robustness of the multiplicative vs. additive gap is therefore unknown. Currently the training is still running and will be added afterwards. We additionally note that hyperparameter selection on a validation set separate from the Mixamo evaluation set will be clarified.


RE: Joint-chain distribution example — differing chain lengths across skeletons

We evaluate this directly using a controlled experiment on the 95 motion clips from our standard retargeting evaluation set. We retarget from BigVegas to two target variants: (1) the original BigVegas skeleton with 3 spine joints, and (2) the same skeleton with 2 additional spine joints (5 total). We then compare the spine curves of the two retargeted outputs directly, fitting a cubic spline through each spine normalized by total spine length, and report mean L2L_2 (average point-by-point deviation) and Fréchet distance (worst-case shape deviation), both as fractions of spine length.

FréchetMean L2L_2
Original (3 spine) vs. Added (5 spine)0.0095±0.00360.0095 \pm 0.00360.0058±0.00250.0058 \pm 0.0025

The two retargeted outputs differ by less than 1% of total spine length on average, confirming that the model smoothly distributes spinal motion across the additional joints.


RE: Comparison with MotionBuilder / HumanIK — required correspondences and setup effort

HumanIK requires per-character setup that our method avoids. Every source and target skeleton must first be characterized: this means manually mapping its joints onto HumanIK's canonical template (15 mandatory nodes in a fixed hierarchy), posing it in a prescribed T-stance, and entering foot and hand sizes by hand if floor contact is used. Retargeting is then a full-body IK solve driven by reach and pull weights for each body part. By default only the ankles are tightly constrained — this handles foot sliding well, but other contacts (such as a hand on the hip) are lost unless an artist tunes these weights or keys IK effectors by hand.


RE: How was Mixamo dataset created (Retargeting pairs)

The exact method used to produce the Mixamo retargeting pairs is not publicly disclosed by Adobe; the retargeting algorithm is proprietary. We therefore cannot provide details on how the ground-truth pairs were generated, and to our knowledge no publication or documentation from Adobe describes the specific pipeline used.

AuthorsOpenReview ↗

RE: Variance of multiplicative vs. additive gap across training runs

We ran 33 independent seeds for both our full model and the additive PE variant. The results, including the original run from the paper, are as follows:

ModelIntra GJP (runs)
Additive PE2.06, 58.33, 76.53, 1.872.06,\ 58.33,\ 76.53,\ 1.87
Ours (multiplicative)1.45, 2.14, 1.47, 2.161.45,\ 2.14,\ 1.47,\ 2.16
ModelCross GJP (runs)
Additive PE2.01, 58.33, 76.56, 1.922.01,\ 58.33,\ 76.56,\ 1.92
Ours (multiplicative)1.28, 1.75, 1.34, 1.901.28,\ 1.75,\ 1.34,\ 1.90

Two of the three new additive PE runs collapsed to poor local minima (GJP >50> 50), while all multiplicative runs converged stably. For the collapsed additive runs, we decomposed GJP into local joint position error and root trajectory error, finding that both components are affected (local position error >10> 10, root trajectory error >43> 43). The two additive runs that did converge achieved intra GJP of 1.871.87 and 2.062.06 (mean 1.97±0.101.97 \pm 0.10), which on intra alone is comparable to or marginally better than some of our multiplicative runs (mean 1.81±0.371.81 \pm 0.37). However, on cross-structure retargeting our multiplicative formulation outperforms every individual converged additive run (mean 1.97±0.051.97 \pm 0.05, best 1.921.92 and worst 2.012.01) compared to our multiplicative runs (mean 1.57±0.281.57 \pm 0.28, best 1.281.28, worst 1.901.90). Together, the multiplicative formulation provides not only better average performance but also greater training stability.

Reviewer_88GoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a Transformer Autoencoder framework for zero-shot cross-structural 3D motion retargeting across diverse skeletal topologies. The core problem addressed is transferring motion between characters with differing kinematic structures without relying on paired retargeting data, per-skeleton-pair models, or textual joint labels, while avoiding error accumulation in global trajectories. To solve this, the authors propose a learnable skeletal graph flattening mechanism that reformulates conventional joint feature flattening as a combination of learnable tiling and masking. The model incorporates graph-based positional encodings generated via a GraphSAGE network and integrates them multiplicatively with joint features. Motion is disentangled into two separate latent spaces: a topology-invariant pose token and a global root trajectory token. The framework is trained end-to-end in an unsupervised manner using cycle consistency, adversarial losses, and specific data augmentations (global translation, bone scaling, joint removal, and rest pose perturbation).

The experimental results demonstrate that the model achieves state-of-the-art zero-shot performance on cross-structural retargeting and reconstruction. On the Mixamo benchmark, the proposed method reduces Global Joint Position (GJP) error by 43–47% compared to existing baselines. In zero-shot reconstruction across unseen datasets, it eliminates catastrophic trajectory drift observed in delta-based methods. A user study confirms that the method ranks significantly higher than existing approaches in motion alignment and visual quality (p<0.05p < 0.05).

Critical strengths
  1. The transition from classical vector flattening to learnable flattening is interesting. Expressing standard concatenation as an interaction between a broadcasting projection matrix TT and a sparse positional mask MM provides a solid theoretical justification for modulating content features with graph positional embeddings via element-wise multiplication rather than standard additive positional encodings.
  2. The proposed method achieves the SOTA performance. On unseen evaluation datasets (Table 1), the method reduces Joint Position (JP) error on Bandai-Namco by 78% (from 12.40 cm for SAME to 2.63 cm). In Table 2, the proposed model achieves lower Global Joint Position (GJP) error in both Intra-structure (1.45) and Cross-structure (1.28) settings compared to SAME (2.91 Intra / 2.47 Cross) and Aberman et al. (2.76 Intra / 2.25 Cross).
  3. The experiment is extensive. The ablation experiments in Table 4 thoroughly validate every key design component. The authors also provide empirical proof of latent space properties.
Critical weaknesses
  1. The claim “A Transformer Autoencoder can retarget motion between arbitrary humanoid skeletons — with varying numbers of joints and edges — within a single unified model” in line 075 is not suitable. As can be seen in equation 4 and equation 5, the mapping from source joint feature (with dimension d) to latent feature (with dimension D) and the mapping from latent feature to target joint feature is fixed. These fixed feature mappings indicate that the joint corresponding mapping between source motion and target motion is fixed. A fixed model can only handle fixed number of joint mapping. If the number of joint of source or target character changes, a new model with corresponding dd need to be re-built. Considering the limitted number of sample of characters with specific number of joints, the model may face overfitting issue during training.
  2. The symbol MM in equation 5 seems wrong. The size of zpz_p is dd, while the size of matrix MM is J×DJ \times D. Their dimension difference introduces conflict in the Hadamard (element-wise) product.
  3. The decomposing flattening is interesting. However, it introduce redundant element repetition during the operation unavoidably;
  4. Absence of Surface Mesh Geometry and Physical Constraints. The framework operates strictly on skeletal graphs and joint rotations. It does not incorporate character surface mesh geometry or volume information.
  5. Minor Writing, Formatting, and Citation Issues. Figure 2 caption on page 4 contains a grammatical typo: "...Schematic of the our model architecture."
Questions for authors

Please see the weakness part for rebuttal.

1 comment

We thank Reviewer 88Go for the thorough review and for recognizing the theoretical grounding of the learnable flattening, the state-of-the-art results, and the thoroughness of the ablation study. Please find our responses below.


RE: Figure 2 caption typo

"Schematic of the our model architecture" will be corrected to "Schematic of our model architecture."


RE: Variable joint counts — architecture handles arbitrary skeletons

The key clarification concerns TencT_{\text{enc}}: it is not a single linear layer applied to all joints simultaneously (which would fix JJ). Rather, Tenc:Rd→RDT_{\mathrm{enc}} : \mathbb{R}^d \to \mathbb{R}^D is a learned projection applied independently to each joint jj, mapping its individual feature vector to a DD-dimensional token, making it fully agnostic to JsrcJ_\text{src}. The spatial mask M(Ssrc)∈RJsrc×DM(S_\text{src}) \in \mathbb{R}^{J_\text{src} \times D} is then computed dynamically by GraphSAGE for whatever skeleton topology is provided, and multiplied element-wise with each token. A Transformer then aggregates the JsrcJ_\text{src} tokens into a single latent zp∈RDz_p \in \mathbb{R}^D. In the decoder, M(Stgt)∈RJtgt×DM(S_\text{tgt}) \in \mathbb{R}^{J_\text{tgt} \times D} is computed dynamically by GraphSAGE from the target skeleton topology, zpz_p is broadcast across JtgtJ_\text{tgt} and multiplied element-wise, yielding a Jtgt×DJ_\text{tgt} \times D matrix which is passed through a Transformer, and each of the resulting JtgtJ_\text{tgt} output tokens is then independently projected to the desired per-joint features via a learned shared projection. JsrcJ_\text{src} and JtgtJ_\text{tgt} are thus fully independent, which is precisely what enables a single model to handle arbitrary skeleton topologies without retraining.


RE: Equation 5 — broadcasting of zpz_p not stated

The reviewer correctly notes that zp∈RDz_p \in \mathbb{R}^D and M(S)∈RJ×DM(S) \in \mathbb{R}^{J \times D} cannot be multiplied element-wise without broadcasting. This is indeed the case: zpz_p is broadcast along the JJ dimension before the Hadamard product. We will add an explicit note stating this in Equation 5.


RE: Redundancy in the flattening decomposition

While the standard flattening decomposition (Eq. 1–3) does indeed involve sparse tiling and masking operations with redundant element repetitions, our learnable flattening replaces both with dense embeddings. Rather than constructing the explicit sparse tiling and masking matrices, both TT and M(S)M(S) are dense learned embeddings, analogous to replacing one-hot encodings with dense learned representations. In practice, the joint count in humanoid skeletons remains modest (up to ∼100{\sim}100 joints), keeping the memory footprint small. The learned dense formulation thus avoids the redundancy of the sparse construction entirely while remaining computationally efficient.


RE: Absence of Surface Mesh Geometry and Physical Constraints

We appreciate this observation. As noted in our limitations section, the current framework operates strictly on skeletal graphs and joint rotations and does not incorporate surface mesh geometry or volume information. We consider the extension to mesh-aware retargeting a natural and important direction for future work.

Reviewer_C85gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript investigated motion retargeting between different skeletons. It uses learnable flattening to encode motions into fixed-size pose and trajectory tokens, then decodes them with the target skeleton. The model combines graph-based multiplicative encoding with a Transformer and is trained without paired retargeting data.

Critical strengths
  1. Investigating an interesting topic: motion retargeting across different skeletons and propose a learnable flattening technique to bridge different skeletons.

  2. The encoding ablation is informative. Multiplicative encoding performs better than additive encoding under the reported setting.

Critical weaknesses
  1. The motivation is not very clear. For example, in Lines 041 to 047, the status of motion retargeting is not sufficiently reviewed, which makes the claim “cross-skeleton transfer difficult” too generic or vague.

  2. It also lacks justification why transformer encoder is introduced, among many possible solutions, which limits the technical contributions. Note that MoMa already uses a Transformer autoencoder for retargeting. Its assumptions are different, but the paper should explain this difference.

  3. PALUM reports MSE under a different evaluation protocol, while this paper defines GJP as mean Euclidean distance. The reported SAN numbers also match its original results on a different test subset. It is unclear whether these methods were rerun.

  4. The large improvement over SAME and the poor GAT results do not fully isolate the benefit of learnable flattening.

  5. There is a rich literature on motion retargeting and many relevant ones are missed in Section 2.1 Motion Retargeting, such as T. Guo, et al., Ultrafast and Controllable Online Motion Retargeting for Game Scenarios, ACM TOG, 44(6): 261, Dec 2025.

  6. Presentation can be further improved. For example, Figure 1 is too brief to convey necessary details of the proposed method.

Questions for authors
  1. Provide better justification on the novelty and technical contributions.

  2. What is the exact GJP formula, and how is it made comparable with PALUM's MSE?

  3. How does the method differ from MoMa in handling unseen skeletons?

  4. How much improvement remains when evaluating local joint positions after removing root-trajectory errors?

1 comment

We thank Reviewer C85g for the thorough review and for finding the topic of cross-skeleton motion retargeting interesting and the encoding ablation informative.

Please find our responses below.


RE: Isolating the benefit of learnable flattening

We would appreciate clarification from the reviewer on what additional ablation or experiment would be considered sufficient to fully isolate the benefit of learnable flattening.


RE: GJP decomposed into local joint position and root-trajectory error

See general response.


RE: Missing literature (Guo et al., MOMA)

See general response.


RE: Figure 1

See general response.


RE: GJP formula and comparability with PALUM's MSE

See general response.


RE: SAN numbers — 106-clip vs. 95-clip protocol

See general response.


RE: Comparison with MoMa

See general response.


RE: Transformer encoder choice — justification

See general response.


RE: Ethics — human ethics approval for user study

The user study involved only passive viewing of animation clips and rating them on a Likert scale, with no deception and no identifiable participant information retained in the collected data. Participants were recruited through personal contacts across multiple research groups via informal invitation, participation was entirely voluntary and unpaid, and all participants provided informed consent prior to taking part. We will add an explicit ethics statement to the revised manuscript clarifying this.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comments

We appreciate the thorough and constructive reviews and are glad to see that the reviewers found our learnable flattening formulation interesting (C85g, 88Go, aSCr), our quantitative results state-of-the-art and convincing (88Go, aSCr), our ablation study thorough and informative (88Go, aSCr), and our method practically applicable with low inference cost (aSCr).

However, the reviewers also raised some concerns including:

  • Metric comparability: GJP definition and baseline protocols (C85g, aSCr)
  • Missing comparison with MoMa (C85g)
  • Missing literature — Transformer justification: Guo et al., ACM TOG 2025, MOMA (C85g)
  • Figure 1 redesign (C85g)
  • Abstract corrections: physical plausibility claim and user study overstating (aSCr)
  • Local Position and Root Trajectory Disentanglement (C85g, aSCr)

Re 1: GJP Metric Definition

We will clarify in the revision that GJP is computed as MSE on height-normalised joint positions, multiplied by 10001000 — identical to the PALUM, SAN and SAME protocol. The description "mean Euclidean distance" in the current draft is imprecise and will be corrected.


Re 2: SAN / Aberman Baseline — Protocol Mismatch

Both SAN and PALUM train directly on Mixamo data, while we do not, due to licensing and prohibition to train ML models on it. Our zero-shot results are therefore obtained without having seen any Mixamo motions or skeletons during training.

The SAN values in Table 2 (2.76/2.252.76 / 2.25 intra/cross) originate from the original 106-clip protocol, whereas both we and SAME evaluate on 95 clips. We re-evaluated the cross-skeleton setting on the correct 95-clip subset, where the score updates from 2.25→1.612.25 \rightarrow 1.61, and our method achieves 1.281.28, a further reduction. For intra, a fair re-evaluation is not possible: SAN trains a separate model per skeleton topology, and the only released model is their intra-structure model, which is specialised to that configuration and does not generalise to cross-skeleton retargeting. Since we cannot retrain their model on Mixamo due to licensing restrictions, obtaining a correctly evaluated intra model is not feasible. We will add a footnote to Table 2 making this explicit.

PALUM reports cross and intra GJP scores, which are what we report in Table 2. However, their paper does not specify which character set or clip subset was used for evaluation and their code is not publicly available, so we cannot verify whether their scores are directly comparable to ours. We will note this in the footnote as well.


Re 3: Comparison with MoMa

Motion retargeting requires capturing interactions between joints far apart in the kinematic tree but close in space, with interactions changing across poses. RNNs make these interaction dependent on arbitrary joint ordering, while GNNs require multiple message-passing layers (≈7\approx 7 hops for hand–hip), risking over-smoothing. Using a fully connected graph with dynamic learned connections — a GAT with pose-dependent attention — is essentially equivalent to a Transformer with global self-attention, which is the principle our model adopts.


Re 4: Missing Literature — Guo et al. (ACM TOG 2025)

We will include Guo et al. (2025) "Ultrafast and Controllable Online Motion Retargeting for Game Scenarios" in the related work. MOMA will also be added.


Re 5: Figure 1 — Insufficient Detail

We will redesign Figure 1 to include a more detailed schematic illustrating the learnable flattening step, the Transformer encoder/decoder, and the mask generation via GraphSAGE.


Re 6: Abstract Corrections

We will replace "physical plausibility" with "perceptual quality" and restate the user study finding more carefully: our method shows a significant improvement in perceptual quality, while the alignment improvement over Aberman is not statistically significant and will be framed accordingly.


Re 7: GJP Decomposed into Local Joint Position and Root-Trajectory Error

To further disentangle the sources of improvement, we decompose GJP into local joint position error and root-trajectory error for key ablations:

ModelIntra local ↓\downarrowIntra traj ↓\downarrowCross local ↓\downarrowCross traj ↓\downarrow
GAT15.94233.2313.71227.40
No PE14.1546.5314.1546.53
Ours0.725.460.666.69

The no-PE variant reduces trajectory error over GAT but leaves local joint position error largely unchanged. Our full model improves both by a large margin, confirming the improvement is not an artefact of either component alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.