acceptodds
Withdrawn from ICLR 2027

How do diffusion models learn personalized expressions? Global Representation Learning for Talking Portrait Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In recent years, diffusion models have shown great potential in talking portrait generation. However, limitations still exist in personalized representation, particularly in appearance and motion, which we attribute to the current learning mechanisms. Regarding portrait appearance, the model tends to favor high-probability samples from the training data, leading to averaged results and weakened individual characteristics. In terms of motion, the model tends to produce locally plausible yet independent motion patterns, making it difficult to capture coordinated relationships in cross-identity portrait animation. To address these issues, we propose a talking portrait generation method based on global representation learning, which consists of two core components: alignment from group to individual (AG2I) and fusion of geometry and motion (FG&M). AG2I constructs the global appearance representations and performs group-to-individual alignment, thereby avoiding averaged appearance. FG&M constructs the global motion representations and leverages the fusion of geometric structures with multi-scale motion, enhancing motion coordination. Experimental results demonstrate that the proposed method effectively enhances the personalization and realism of the generated results compared to state-of-the-art methods.

Primary area:
generative models
Keywords:
Diffusion Models, Video Generation, Personalization, Fusion, Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o7avOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a diffusion-based framework for personalized talking portrait generation. Its two main components are group-to-individual appearance alignment (AG2I) and geometry–motion fusion (FG&M). These components combine feature decomposition, individual-guided modulation, and multi-scale motion modeling. Experiments report improvements over six diffusion-based baselines, supported by component ablations, a user study, and a lightweight distilled variant.

Critical strengths

This paper addresses a talking head problem: preserving distinctive appearance details while generating coordinated facial motion. The reported quantitative improvements span visual quality, identity similarity, and lip synchronization. The component ablations also suggest complementary roles for appearance alignment and motion fusion. Including perceptual evaluation and a distilled model broadens the assessment beyond image reconstruction quality alone.

Critical weaknesses

The method is insufficiently specified, with questionable objective formulations. Although the task takes a portrait image and audio, both appearance and motion representations are defined using a frame sequence. It remains unclear where these frames originate during inference and how audio enters the computational pipeline. The operational distinction between “individual” and “group” features is also unclear; decorrelation alone does not establish the claimed semantic separation. More concretely, the entropy loss in Eq. (19), repeated in Eq. (80), is positive entropy. Minimizing it with a positive coefficient encourages concentrated scale weights, contrary to the stated goal of preventing scale collapse. The negative squared-distance term in Eq. (13) also permits unbounded separation unless additional constraints are imposed. These issues prevent a reliable assessment of the implemented method.

The evaluation does not sufficiently establish generalization or explain the reported gains. The hybrid dataset is divided 7:3, but the paper does not specify whether identities and source videos are disjoint, or how overlap among constituent datasets is removed. Consequently, performance on unseen identities remains uncertain. The substantial reconstruction gains require a clearly defined reference-image selection and evaluation protocol. Baseline training descriptions do not adequately distinguish pretrained initialization from retraining or specify comparable inference settings. There is also a concrete reporting inconsistency: the AniPortrait and MoDiTalker result rows are exchanged between Tables 1 and 3. Finally, the ablations remove entire modules without comparable-capacity replacements, so they do not isolate the proposed mechanisms from additional model capacity.

Questions for authors

What exactly is available during inference, and how are the frame sequences, audio features, and group representations obtained without access to the target video?

What loss signs, coefficients, and feature constraints are actually implemented for Eqs. (13) and (19)? Can corrected formulations and corresponding ablations be provided?

Are evaluation identities and source videos excluded from training, and which AniPortrait/MoDiTalker results are correct?

Reviewer_NgVUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies audio-driven talking portrait generation. It argues that diffusion models wash out person-specific details such as teeth, wrinkles and eye regions, and that they produce poorly coordinated motion across facial regions. Two conditioning branches are added to a diffusion denoiser. AG2I encodes frames, splits the features into "individual" and "group" parts, decorrelates them, and modulates the group part with the individual part through channel-wise scale and shift. FG&M extracts four-scale motion/geometry features from frames, decorrelates each scale, mixes the scales with a Transformer and an entropy term, and fuses them coarse-to-fine. The two outputs form the condition c of ϵθ(xt,t,c)\epsilon_\theta(x_t, t, c), trained with the diffusion loss plus eight auxiliary losses (Eq. 101).

Critical strengths
  1. The problem is concrete and easy for users to see. Losing a person's tooth shape, brow wrinkles or gaze behavior is a recognizable failure of current portrait animators, and Fig. 4 and the supplementary video pick cases where this failure is visible.
  2. The ablation covers all eight sub-modules, and the two branches affect different metrics. Removing an AG2I module lowers PSNR from 37.81 to 32.57–33.62, while removing an FG&M module leaves PSNR almost unchanged and raises LMD from 1.46 to 2.05–2.88 (Table 2). This pattern matches the roles the authors assign to the two branches.
  3. The limitations are stated plainly. Section 4.4 and Appendix D.3 say that details absent from the input come from data priors, and Appendix D.4 with Fig. 11 shows texture blur and audio-visual drift beyond about 45 seconds.
Critical weaknesses
  1. The paper does not specify the generative backbone or its initialization. The manuscript defines the denoiser only as ϵθ(xt,t,c)\epsilon_\theta(x_t, t, c), without identifying its architecture, whether diffusion operates in pixel or latent space, or whether the model is initialized from pretrained weights or trained from scratch. The supplied code nominally defines a custom 2D pixel-space conditional U-Net (modules/diffusion_model.py), and its model-construction path contains no explicit pretrained-backbone loading (train.py). The submitted code conditions the model on its own target frames and has no audio path.

  2. The test-time input and the audio path are not defined. Similiar to W1, the test protocol is confusing. Fig. 1 lists audio and a portrait image as inputs, and Fig. 3 draws a wav2vec block inside FscaleF_{scale}. No equation nor description in the main paper nor the appendix takes audio.

  3. The protocol behind Table 1 is unclear, and Fig. 4 makes it harder to read. In both panels of Fig. 4, the Ground Truth is a different person from the input image: a woman in each Ground Truth row, a man in each input and output. Lines 299–301 say the Ground Truth is the video belonging to the input audio, so Fig. 4 is cross-identity driving. The ablation in Fig. 5 instead shows the same person in Ground Truth and output. Table 1 does not say which protocol it uses. If it is cross-identity, PSNR, LPIPS and LMD against the Ground Truth have no pixel correspondence. If it is self-reenactment, then whether AG2I and FG&M read the ground-truth frames becomes the deciding question; PSNR 37.65 and LPIPS 0.01 at 720p are reconstruction-level values and fit the case where those frames are available. Either way, the margins in Table 1 cannot be attributed to the proposed modules.

  4. The code does not implement the training objective and cannot run as shipped. losses/total_loss.py (L47–127) sums the eight auxiliary losses but never reads noise and noise_pred, which train.py returns at L175–176, so the diffusion loss L(c)\mathcal L(c) in Eq. 101 is absent.

  5. The reported numbers are inconsistent across tables. Table 3 in the appendix repeats Table 1, but the MoDiTalker and AniPortrait rows are swapped in full.

Questions for authors
  1. At test time, what exactly does the model receive? Please state whether any ground-truth video frames, or features, landmarks or poses derived from them, enter AG2I or FG&M, and give the equation where audio enters KmotionK_{motion}.
  2. What exact denoising backbone produced the reported results, and does it operate in pixel or latent space? Was it trained from random initialization or initialized from a pretrained checkpoint? Please identify the checkpoint/source if applicable, state which components were frozen or trained, and provide the matching experiment configuration. Is the custom ConditionalUNet in the supplementary code the backbone used in those experiments?
  3. Is Table 1 computed under self-reenactment or under cross-identity driving as in Fig. 4? If cross-identity, what is the pixel reference for PSNR, LPIPS and LMD? Is the 7:3 split identity-disjoint, and were the baselines run with official checkpoints or retrained?
  4. Is the supplementary code the code used for the paper?
Reviewer_vFThOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a talking portrait generation method based on global representation learning.

Specifically, the paper proposes the alignment from group to individual (AG2I) to enhance the individual characteristics.

In addtion, the paper introduces FG&M(fusion of geometry and motion) strategy to improve motion coordination.

However, the authors need to add more experiments and rewrite the paper. Otherwise, it will be considered AI-generated.

Critical strengths

This paper proposes a talking portrait generation method based on global representation learning.

Specifically, the paper proposes the alignment from group to individual (AG2I) to enhance the individual characteristics.

In addtion, the paper introduces FG&M(fusion of geometry and motion) strategy to improve motion coordination.

Critical weaknesses
  1. Limited evaluation on diverse datasets. The authors should conduct additional experiments on publicly available datasets, such as MEAD and CelebV-HQ, to further demonstrate the generalization of the proposed method, particularly in cross-identity portrait animation and diverse facial expressions.

  2. Insufficient comparisons with recent methods. Table 1 lacks comparisons with recent talking portrait generation methods [1][2][3]. The authors should include these methods under consistent evaluation settings to better demonstrate the effectiveness of the proposed approach.

  3. Insufficient comparisons with identity-preserving methods. The paper repeatedly emphasizes identity preservation as one of its main contributions. However, comparisons with recent identity-preserving methods [4][5][6] are missing. The authors should provide additional experimental comparisons and clarify the differences between the proposed AG2I module and these methods in the Introduction or Related Work section.

[1] CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning

[2] Mamba-Enhanced Implicit Motion Learning for Audio-Driven Portrait Animation. 2026

[3] DiTalker: A Unified DiT-Based Framework for High-Quality and Style-Controllable Portrait Animation

[4] HiFi-Portrait: Zero-shot Identity-preserved Portrait Generation with High-fidelity Multi-face Fusion

[5] AsymTalker: Identity-Consistent Long-Term Talking Head Generation via Asymmetric Distillation

[6] Audio-driven Facial Animation in the Wild Using a Multi-Style Identity Preservation Framework

Questions for authors

Are the images in Figures 1–3 AI-generated? They are truly unsightly. Regardless of their origin, I request that the author revise them.

Furthermore, were the formulas in the appendix (20–100) written by AI? I read the main text in its entirety, but upon reaching the appendix, I lost all interest in continuing. I suggest that the author reduce the use of AI-generated content.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.