acceptodds
Under review as a conference paper at ICLR 2027

Compact Motion Embeddings for Feed-forward Unified Human Animation

Abstract

Animating a reference image with a driver's motion requires a motion representation that captures face, hand, and body articulation but excludes the driver's appearance. Most body animation methods take motion from external pose estimators and render it with many denoising steps; compact learned motion representations mostly cover the face. We present FUHA, a real-time-capable feed-forward transformer that learns a compact motion embedding of the whole human and uses it for face, upper-body, and full-body animation. A Q-Former compresses the driver's features from a frozen visual encoder into this embedding. To disentangle motion from appearance, a novel contrastive loss pulls it toward the motion embedding of the driver's rendered keypoint image and away from that of the same person in another pose. At inference time, FUHA needs no external pose estimators, keypoints, or parametric body models. As a retrieval key, the motion embedding matches body pose better than the learned global embeddings of ten vision encoders. Its same-person retrieval rate is 0.1%, against 9.5% for its own DINOv3 backbone. On our upper-body benchmark, FUHA outperforms all baselines on every self-reenactment metric and on cross-reenactment identity similarity. It is competitive on face and full-body and runs at up to 38.9 FPS at 512x512 on a single RTX 4090.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Motion Representation Learning, Human Animation, Disentangled Representation Learning, Contrastive Learning, Feed-forward Networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LYGTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FUHA, a feed-forward human image animator using a compact learned representation of per-frame pose/expression. A frozen DINOv3 encoder and Q-Former compress a driver image into two 512-dimensional motion tokens. A separate appearance encoder and transformer decoder synthesize the reference person in the driver's configuration through a frozen RGB tokenizer. A triplet objective pulls driver embeddings toward embeddings of rendered keypoints and away from another pose of the same identity, while reconstruction and identity losses train the renderer. Keypoint estimators are used during training/data preparation, but not at inference. A single model handles face, upper-body, and full-body settings. Validation includes extensive reconstruction/cross-person comparisons, component ablations, retrieval/interpolation probes, a diffusion-decoder variant, and timing. The faster variant reaches 38.9 model FPS at 512x512 on a 4090 with the reported optimized inference setup. Upper-body self-reenactment and cross-person identity scores are strong, while cross-person/full-body perceptual video quality is more mixed.

Critical strengths
  1. This paper proposes a useful quality/throughput regime for unified learned-motion animation. FUHA extends a stateless, single-pass renderer across face, hands, upper body, and full body. In upper-body experiments it combines strong self-reenactment scores and high cross-person identity similarity with a much faster inference regime than the evaluated multi-step alternatives.

  2. This paper presents several complementary tests to support the usefulness of the learned motion representation. The contrastive-loss ablation causes a major degradation, motion retrieval improves body-pose similarity over global encoder embeddings, and the frozen embedding can condition a separate diffusion renderer.

Critical weaknesses
  1. The methodological novelty is limited, with the contribution primarily consisting of a combination of existing system components(X-UniMotion, IM-Animation).
Questions for authors

See weaknesses.

2 comments

Thank you for the review and the positive assessment.

FUHA starts from what current animation methods leave open. Compact learned motion latents mostly cover the face. Body methods take skeletons from pose estimators, which carry little expression and fail under occlusion. The two whole-body embeddings, X-UniMotion and IM-Animation, rely on crops, synthetic or generated identity pairs, and multi-step diffusion. FUHA is designed to close these gaps in one framework: one embedding for face, hands and body, no per-frame preprocessing, identity separation learned from real video, and single-pass rendering.

The table compares FUHA with these two methods:

X-UniMotionIM-AnimationFUHA
Embedding4 region tokens; face, hands from crops32 tokens; face from X-NeMo encoderOne 1024-d embedding, full frame
SupervisionKeypoint heatmap regressionKeypoint heatmap regressionTriplet: keypoint image positive, same person in another pose negative
Cross-identity dataSynthetic (rigged 3D characters)Generated (Kling, Unreal Engine)None (real video only)
Renderer7B diffusion transformer5B diffusion (Wan2.2)Single-pass transformer, 12.4 / 38.9 FPS (FUHA-C / -T) at 512²

At inference, FUHA takes only the reference image and the driver frame: no pose estimator, no face or hand crops, nothing to localize per frame. X-UniMotion reports that one global descriptor misaligns gaze, lips and fingers. FUHA's single embedding nonetheless gives the lowest upper-body self-reenactment hand and expression errors (AKD-H 48.5 vs. 65.6, AED 0.315 vs. 0.467; Table 1).

Keypoints enter only in training, as the triplet's positive, and reconstruction trains the embedding end to end, so it is not limited to what they encode: driving FUHA with keypoint images instead raises self AKD-H from 48.5 to about 85 (reply to spE1, Q1). The same loss keeps identity out using real video only. In retrieval, the embedding returns the query's own person in 0.1% of queries, against 9.5% for DINOv3 (Table 6); without the loss, cross-ID falls from 0.701 to 0.057 (Table 4).

Finally, the embedding is not tied to its renderer. Kept frozen, it also drives Wan-Animate (FUHA-W, App. A.3), whereas X-UniMotion trains its tokens jointly with its generator.

Some ingredients are known on their own, such as DINOv3 features, Q-Former bottlenecks and triplet losses. Our contribution is how they are combined to address these gaps, and Table 4 and the new controls show that each part contributes. We will add this discussion to the revision.

Thank you again for your review. As the discussion period closes on Nov 18, we would be glad to answer any remaining questions.

Reviewer_eikXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses reference-image human animation, where transferring a driver’s facial and body movements while preserving the reference person’s appearance is challenging, by learning a compact motion representation with contrastive supervision and rendering the animation through a feed-forward model.

Critical strengths
  1. The model generates each frame in a single forward pass without iterative denoising or an external pose estimator at inference, reducing the processing stages needed for interactive animation.

  2. The same model handles facial, upper-body and full-body animation without region-specific fine-tuning, avoiding separate animation models for different body regions.

Critical weaknesses
  1. The two motion tokens receive joint supervision without distinct assigned roles, so the token-count ablation identifies a preferred capacity setting but does not explain whether the tokens learn complementary motion factors or why a two-token representation is needed. (Sections 3.2–3.3 and 4.3.)

  2. The appearance branch combines RGB reconstruction latents and DINO features while the motion branch uses only DINO features, but removing DINO from the appearance branch tests feature utility rather than whether this asymmetric input design improves motion–appearance separation. (Sections 3.1 and 4.3.)

  3. The motion decoder expands the motion tokens before the appearance decoder combines them with appearance, but removing this intermediate decoder also reduces network capacity, so the reported comparison does not establish why serial decoding is preferable to a capacity-matched decoder that directly uses both embeddings. (Sections 3.2 and 4.3.)

  4. The motion decoder uses learned queries tied to a fixed spatial grid, but no procedure is given for adapting those queries across resolutions, leaving it unclear whether one trained model can generate at different native resolutions without retraining. (Appendices D.1 and E.)

Questions for authors

No additional questions.

2 comments

Thank you for the precise review.

W1. No roles are assigned to the two tokens, and we do not claim that they specialize. They are two 512-d Q-Former queries, flattened into a single 1024-d embedding on which the triplet loss acts (Eq. 1), so the token count sets capacity only. The size follows from the bottleneck argument (Sec. 3.1): whole-body motion needs more room than the face alone, but far less than appearance. Precedent agrees: X-NeMo uses 512-d for the face alone, X-UniMotion 4 × 512-d for face, hands and body. Table 4 bounds the size from above, as four tokens worsen every self metric and cross-AED (0.686 → 0.761). We will add this rationale to Sec. 3.2.

W2. We agree: that row tests the utility of DINO features in the appearance branch, not the asymmetry itself. We therefore trained the missing control, feeding summed tokenizer latents and DINOv3 features to both branches:

FUHA-Cw/o DINO in appearanceSymmetric input
Self LPIPS ↓0.1060.1250.134
Self ID ↑0.8200.8110.809
Self FID ↓15.218.318.1
Self FVD ↓38.959.471.9
Cross ID ↑0.7010.7120.721
Cross AED ↓0.6860.7320.773
Cross AKD-B ↓51.555.861.2
Cross AKD-H ↓88.891.193.0
Cross FID ↓55.956.153.2
Cross FVD ↓171.7202.4222.1

Symmetric input is worse on every self metric and on cross-reenactment motion accuracy (AED 0.686 → 0.773, AKD-B 51.5 → 61.2). Its cross-ID rises slightly (0.701 → 0.721), but this is not a better trade-off: four tokens (Table 4) reach higher cross-ID (0.733) at lower AED (0.761). The asymmetry thus serves motion accuracy and fidelity, while separation comes mainly from the training target (without the contrastive loss, cross-ID falls to 0.057). We will add this row to Table 4.

W3. The motion map serves as the appearance decoder's queries, so motion fixes the spatial layout and appearance detail is fetched per location. The ablation (Sec. 4.3) removes only the 8 motion-decoder layers; the appearance decoder and its 64×64 queries are unchanged. Serial decoding improves self-LPIPS (0.106 vs. 0.116) and cross-FVD (171.7 vs. 193.0), but the variant is also smaller (104M vs. 138M per frame), so the comparison is not capacity-matched. We will report both counts and add a capacity-matched direct decoder to the final version; it did not fit into the discussion period.

W4. No. Each trained model renders at a single resolution, 512² here, because the motion decoder's learned queries match the 64×64 latent grid. Other resolutions would require interpolating these queries or fine-tuning. We will list this as a limitation, and fix the "256×256" typo in App. D.1.

We would be glad to clarify any point that remains open.

Thank you again for your review. As the discussion period closes on Nov 18, we would be glad to answer any remaining questions.

Reviewer_spE1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FUHA, a feed-forward transformer for face, upper-body and full-body animation with one model. It uses a frozen DINOv3 to extract features from the driver image and compresses them into a compact motion embedding with a Q-Former. The reference image is encoded into appearance embeddings, and a transformer decoder combines them to render the output in a single pass without denoising or external pose estimator at inference. To disentangle motion from appearance, they use a contrastive loss that pulls the motion embedding toward the rendered keypoint image of the driver and away from the reference. The model runs up to 38.9 FPS at 512 resolution, and shows good results on their upper-body benchmark, face and full-body datasets.

Critical strengths
  1. The paper makes real-time full-body reenactment work with a single feed-forward model. Previous methods are either face only or need multi-step diffusion, and this paper covers face, upper-body and full-body at up to 38.9 FPS. This is valuable for interactive avatar applications.
  2. The motion embedding is well analyzed. The retrieval experiment in Table 6 and the latent walk in Figure 5 show that the embedding keeps the pose but carries little identity, and the FUHA-W experiment shows it can also drive a diffusion decoder.
Critical weaknesses
  1. The method section only describes the network structure, but does not explain the insight of the design. Why only DINOv3 for the motion branch, why two tokens, why a 2D motion decoder, these are not discussed, and the alternatives are not compared.
  2. The contrastive loss is not novel. Many previous methods such as MegaPortraits, LIA and X-NeMo already use similar losses to learn a motion embedding. The only new part is using the keypoint image as the positive sample, which is actually distilling a pose estimator into the encoder, and it brings the shape bias of the driver into the result as shown in the demo.
Questions for authors
  1. Can you train the same decoder with the keypoint image as the condition instead of the learned motion embedding? This will show whether the learned embedding is really better than external keypoints.
  2. Since the positive sample is the rendered keypoint image, the motion encoder is basically learning to do pose estimation. How is this different from using a pose estimator directly, except for the runtime?
  3. The driver’s body shape sometimes appears in the result, as shown in the demo. Is this caused by the keypoint image used in the contrastive loss? Can you show cross-reenactment between people with very different body shapes?
  4. The 1 token variant fails even in self-reenactment with ID 0.134. Is this a capacity problem or a training problem?
  5. The authors mention the same people and footage may appear in both training and test data. Can you report the results on a test set with the identities removed from training?
3 comments

Thank you for the close read and the concrete questions.

W1. Three choices shape the embedding. Only DINOv3 features enter the motion branch, so the tokenizer's photometric detail does not reach it directly; DINOv3's patch features carry pose and articulation of face, hands and body without region-specific tuning. Two 512-d tokens form a 1024-d bottleneck, enough for whole-body motion yet far smaller than the appearance embeddings. The 2D motion decoder expands the embedding into a spatial map that serves as the appearance decoder's queries: motion fixes the layout, and cross-attention fetches appearance detail at each location. Table 4 tests the last two choices. Four tokens worsen cross-AED and every self metric; removing the motion decoder worsens self-LPIPS and cross-FVD. Our reply to eikX tests the input. We agree this rationale belongs in the paper and will add it to Sec. 3.1.

W2. Contrastive motion losses are indeed not new. Of the cited works, MegaPortraits applies one, but only to an expression latent: head pose comes from a separate pre-trained network, and negatives come from other videos, so they also differ in identity. Our loss acts on all motion, head and body pose included, and its negative is the same person in another pose. LIA and X-NeMo use no contrastive term. Our claim is limited to this choice of positive and negative (Sec. 3.3), and we will scope the abstract and Sec. 1 to match. On distillation: keypoints come from offline estimators and are used only in training, as the triplet's positive (weight 0.2). We do not regress them, and L1 and LPIPS (1.0 each) train the embedding end to end. A distilled estimator could carry no more than the keypoint image; Q1 shows the embedding carries more. On the driver's shape bias, please see Q3.

Q1. We trained the suggested control: the keypoint image replaces the driver frame, with encoder, bottleneck, decoder and training recipe unchanged. The triplet loss is disabled, as its anchor and positive would coincide. We also fed keypoint images to FUHA-C without retraining.

Upper-body benchmark:

Keypoints, no retrainingKeypoints, retrainedFUHA-C
Self ID ↑0.7940.3780.820
Self AED ↓0.4370.4560.315
Self AKD-B ↓44.944.327.1
Self AKD-H ↓84.885.048.5
Cross ID ↑0.7220.2290.701
Cross AED ↓0.7400.6570.686
Cross AKD-B ↓59.757.151.5
Cross AKD-H ↓110.6108.988.8

Identity holds without retraining, yet the two keypoint models agree within 3 px on AKD-B and AKD-H, both well behind FUHA-C. The embedding carries motion detail that a keypoint image lacks.

Q2. The Q1 table measures this directly: with keypoint images in place of the embedding, self AKD-B rises from 27.1 to 44–45, AKD-H from 48.5 to about 85, and AED from 0.315 to 0.44–0.46. Two more things differ. Estimators drop or misplace occluded and truncated joints, whereas the embedding is read from the whole frame and has no missing entries. Per-frame estimates also jitter; FUHA-C has the best temporal consistency on face and upper-body (Table B.1).

Q3. FUHA handles moderate differences in build, since body shape, clothing and identity come from the reference. It does not adjust bone lengths. Because training reconstructs the driver frame itself, the driver's limb proportions carry over with or without the keypoint positive. Keypoint-driven methods without explicit retargeting share this, and we will state it in the limitations.

Q4. On revisiting the 1-token run, we found its validation loss noisier than FUHA-C's, so the failure may stem from training rather than capacity. As the run differs from FUHA-C only in token count, we will rerun it for the final version and revise Sec. 4.3 to match. The 1024-d budget follows precedent: X-NeMo uses 512-d for the face alone, and X-UniMotion 4 × 512-d for face, hands and body.

Q5. FUHA's lead does not depend on people or footage shared between training and test. We defined an unseen set of 74 upper-body clips from videos not used in training, matching the benchmark in framing, motion and hand visibility. For every clip, the closest training clip has a clip-averaged ArcFace similarity of at most 0.43. We reran the two strongest baselines on all 74 (bold: best).

Wan-AnimateUniAnimate-DiTFUHA-C
Self SSIM ↑0.6870.7090.815
Self LPIPS ↓0.2240.1820.116
Self ID ↑0.6460.7010.791
Self AED ↓0.4720.5430.328
Self AKD-B ↓34.536.425.3
Self AKD-H ↓69.272.345.0
Self FID ↓26.816.216.6
Self FVD ↓122.564.350.4
Cross ID ↑0.4700.6170.688
Cross AED ↓0.8120.8870.694
Cross AKD-B ↓48.553.349.9
Cross AKD-H ↓99.4109.895.0
Cross FID ↓64.055.160.5
Cross FVD ↓220.1164.2192.9

The ranking of the three methods matches Table 1 on 13 of the 14 metrics. We will add this table to the revision.

Thanks for the reply. Most of my concerns have been addressed. Regarding the method explanation, I recommend that the authors revise the submission to convey the insight behind their design in the main text, so that readers understand why each component is designed this way. Q3 could be left as a direction for future improvement, or the authors could consider incorporating an implicit motion code as in HunyuanPortrait. Finally, I will keep my current rating.

Thank you very much for the follow-up and for your detailed review.

We will revise the method section to better explain the reasoning behind each design choice, and note the driver's shape bias (Q3) as future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reviews.

Strengths noted by the reviewers

  • One model for face, upper-body and full-body. spE1: "Previous methods are either face only or need multi-step diffusion, and this paper covers face, upper-body and full-body at up to 38.9 FPS. This is valuable for interactive avatar applications." eikX adds that this holds "without region-specific fine-tuning".
  • Inference cost. LYGT points to "a much faster inference regime than the evaluated multi-step alternatives".
  • Analysis of the motion embedding. spE1 finds that it "keeps the pose but carries little identity" and "can also drive a diffusion decoder".

Our contributions

FUHA addresses gaps that current animation methods leave open. Compact learned motion latents mostly cover the face. Body methods take skeletons from pose estimators, which carry little expression and fail under occlusion. The whole-body embeddings of X-UniMotion and IM-Animation rely on face and hand crops, synthetic or generated identity pairs, and multi-step diffusion.

FUHA reads face, hands and body from the full frame into one 1024-d embedding. At inference it takes only the reference image and the driver frame: no pose estimator, no crops, nothing to localize per frame. X-UniMotion reports that a single global descriptor misaligns gaze, lips and fingers; FUHA's single embedding nonetheless has the lowest upper-body self-reenactment hand and expression errors of all methods (Table 1).

Keypoints are used only in training, as the positive of a triplet loss whose negative is the same person in another pose. FUHA does not regress them, and reconstruction trains the embedding end to end, so it is not limited to what keypoints encode. Driving the model with keypoint images instead, with or without retraining, raises self-reenactment AKD-H from 48.5 to about 85 (reply to spE1, Q1). The same loss keeps identity out using real video only, with no synthetic pairs; removing it drops cross-reenactment identity from 0.701 to 0.057 (Table 4).

Decoding takes a single feed-forward pass, at up to 38.9 FPS at 512², and the frozen embedding also conditions a diffusion renderer (FUHA-W, App. A.3), so the representation is not tied to its generator.

The ablations in Table 4 and the new experiments test each of these design choices.

New experiments

  • Keypoint-driven control (spE1, Q1).
  • Symmetric encoder input (eikX, W2): giving the motion branch the tokenizer latents as well degrades every self-reenactment metric and cross-reenactment motion accuracy.
  • Unseen footage (spE1, Q5): on 74 upper-body clips from videos not used in training, FUHA-C leads on every self-reenactment metric except FID, and on cross-reenactment ID, AED and AKD-H.

All new results and the comparison with X-UniMotion and IM-Animation will be added to the revision.

We thank the AC and the reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.