acceptodds
Under review as a conference paper at ICLR 2027

PlanTalk: MLLM-Based Motion Planning with Hierarchical Audio Guidance for Talking Head Generation

Abstract

Audio-driven talking head generation is challenging because facial articulation is identity-dependent and fine-grained audio-visual correspondence is difficult to preserve across multi-stage generation. In this paper, we propose PlanTalk to exploit joint speech-identity context through an MLLM-based motion planner, together with hierarchical audio guidance. The motion planner jointly interprets the driving speech and speaker-specific visual characteristics to organize an identity-aware facial motion sequence. Hierarchical audio guidance then progressively strengthens audio-visual correspondence at different representation levels: audio-landmark fusion refines the alignment between facial geometry and speech articulation, while denoising-stage audio attention preserves speech-consistent facial dynamics during video synthesis. The proposed cross-modal planning strategy effectively captures identity-dependent articulation, while hierarchical audio guidance maintains fine-grained speech correspondence throughout the generation pipeline. Extensive experiments demonstrate that PlanTalk achieves state-of-the-art performance over existing methods in terms of lip synchronization and identity consistency while maintaining high visual fidelity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Driven Talking Head Generation, Multimodal Large Language Models, Audio-Visual Correspondence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_AwzROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PlanTalk addresses audio-driven talking-head generation with a multi-stage pipeline. Instead of regressing landmarks from audio with identity as a conditioning signal, it formulates facial motion generation as identity-aware cross-modal planning: a frozen Vicuna-7B MLLM with frozen ImageBind encoders takes the identity frame and driving audio and autoregressively produces landmark hidden states. Hierarchical audio guidance then re-injects speech at two levels: audio–landmark MM-Attention fusion refining the planned motion, and denoising-stage audio cross-attention inside a MoDiTalker-style Motion-to-Video diffusion generator. Training uses landmark reconstruction/residual/multi-scale-velocity/identity losses plus standard diffusion loss. On HDTF and CelebV-HQ, PlanTalk reports the best LMD/LSE-D/PSNR/CSIM, competitive FID, and the fastest inference. Ablations show that both guidance levels contribute.

Critical strengths
  1. Persistent audio guidance is a real and well-validated idea. The denoising-stage audio attention lets speech cues correct errors that landmark prediction already baked in — addressing a genuine weakness of multi-stage pipelines. The ablation (Table 3) shows consistent, monotone improvements from each guidance level.
  2. Competitive efficiency despite an MLLM. Replacing iterative motion diffusion with a single autoregressive pass while retaining diffusion only for pixels is a sensible design point; 20.1 s/clip with the best lip-sync numbers is a practically attractive combination, if the timing protocol holds up.
Critical weaknesses
  1. No ablation study of the core component. The paper's title and first contribution is the MLLM motion planner, yet nothing isolates it: no comparison against a same-size non-LLM transformer planner, no test of whether the LM's linguistic knowledge matters at all (vs. ImageBind features alone), and the Fig. 3 landmark comparison is qualitative only. Given that Vicuna-7B and ImageBind are frozen and only projectors/fusion are trained, it is plausible the MLLM is just an expensive generic encoder. This directly undermines the paper's central claim.
  2. Evidence strength does not match the SOTA claim. Improvements over MoDiTalker are within noise on most metrics, FID is not better on CelebV-HQ, test sets are small (98/50 clips, in-domain, self-reconstruction), and there is no human evaluation of lip sync or identity. For a generation paper at this venue, the evaluation is thin.
Questions for authors
  1. Please provide an ablation replacing the MLLM planner with a parameter-matched non-pretrained transformer (and with ImageBind features + a small temporal module, no LLM). Does the planning advantage survive?
  2. How many tokens does the MLLM emit per clip, and what is the per-stage runtime breakdown (planner vs. fusion vs. diffusion) for the 20.1 s figure? Which GPU and batch size?
  3. Do results hold with variance across multiple sampling seeds, and on cross-identity audio?
  4. The authors are recommended to discuss in the future work section: can the MLLM planner accept textual emotion/style instructions?
Reviewer_wETQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PlanTalk, an MLLM-based framework for talking-head generation. The method jointly feeds the driving audio and an identity image into an MLLM to generate identity-aware facial motion plans. It further enhances audio–visual correspondence at both the motion representation and video synthesis levels through audio–landmark fusion and audio attention during the video denoising process. Experimental results show that PlanTalk achieves strong lip synchronization, identity consistency, and visual quality on the HDTF and CelebV-HQ datasets, outperforming existing SOTA methods on several metrics.

Critical strengths
  1. The proposed hierarchical audio-guidance mechanism continuously injects audio information at both the motion-planning and video-denoising stages, helping mitigate the gradual weakening of audio–visual correspondence in multi-stage generation.
  2. The paper uses an MLLM for cross-modal motion planning and achieves strong lip synchronization, identity consistency, and visual quality on the HDTF and CelebV-HQ datasets.
Critical weaknesses

The paper does not sufficiently demonstrate the necessity of using an MLLM compared with a conventional cross-modal Transformer, nor does it clearly verify whether the model genuinely learns identity-specific articulation patterns. In addition, the reported end-to-end inference efficiency may still not qualify as real-time generation in the strict sense. I recommend that the authors include fairer comparisons with alternative model architectures, experiments on disentangling identity from articulation, and a more comprehensive efficiency analysis. The paper may consider conducting these additional experiments, and I would be happy to discuss these issues further.

Questions for authors

as the comment

Reviewer_PV1xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PlanTalk for audio-driven talking head generation, combining an MLLM-based motion planner with audio–landmark fusion and denoising-stage audio attention. Experiments on HDTF and CelebV-HQ report competitive visual quality and improvements in lip synchronization and identity preservation.

Critical strengths
  1. The reported results are competitive on both datasets, improving identity similarity and lip-motion metrics over MoDiTalker while maintaining similar FID.

  2. Maintaining audio guidance during both landmark refinement and video synthesis is well motivated. The complete 2×2 ablation provides useful evidence for the contributions of these two components.

To me, the most interesting potential contribution is using a frozen language backbone to generate continuous facial motion from multimodal inputs.

Critical weaknesses
  1. The contribution of the central MLLM planner is not sufficiently isolated. Table 3 only varies the two audio-guidance components, while the planner comparison in Fig. 3 is qualitative. Without a controlled planner replacement, it remains unclear how much of the improvement comes from the MLLM rather than the additional guidance or other training differences.

  2. Important implementation and evaluation details remain unclear, including autoregressive motion-token generation, frame-level alignment of ImageBind features, and the source of pose frames during inference. It is also unclear whether baselines use comparable inputs, training data, and the same test split. These omissions limit reproducibility and the interpretation of the reported improvements.

Questions for authors
  1. What improvement is attributable specifically to the MLLM planner? I suggest replacing only the planner with a conventional audio–identity-conditioned Transformer while keeping the remaining pipeline, training data, and supervision fixed.

  2. How are motion tokens generated and fed back autoregressively with frozen Vicuna, and how are ImageBind audio features aligned with video frames?

  3. Where do the identity and pose frames come from during evaluation, and are baseline inputs, training data, and test splits matched?

  4. What evidence distinguishes identity-dependent motion modeling in the planner from identity preservation provided by the video generator?

Reviewer_VgZmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PlanTalk, an audio-driven talking head generation framework that combines MLLM-based facial motion planning with hierarchical audio guidance. The method uses ImageBind and Vicuna-7B to generate identity-aware landmark representations, followed by audio-landmark fusion and audio-conditioned diffusion for video synthesis. Experiments on HDTF and CelebV-HQ show improvements in lip synchronization and identity preservation over several existing methods.

Critical strengths

The paper addresses the challenge of maintaining accurate audio-visual synchronization across multi-stage talking head generation. The idea of introducing audio guidance at both landmark prediction and video synthesis stages is reasonable. PlanTalk achieves strong results on HDTF and CelebV-HQ, particularly in LMD, LSE-D, and CSIM. The reported inference time is also competitive, and the paper includes comparisons with several representative baselines.

Critical weaknesses
  1. The main contribution is framed as MLLM-based motion planning, but the paper does not explain what reasoning or planning capabilities the language model actually provides. Since Vicuna-7B is frozen and the model directly predicts landmark representations, it is unclear whether this is fundamentally different from conventional multimodal sequence modeling.
  2. The audio-landmark fusion module uses standard joint self-attention, while the denoising-stage audio guidance relies on conventional cross-attention. Both are well-established techniques. The paper mainly combines existing architectures and attention mechanisms, without introducing a clearly new modeling approach.
  3. Table 3 only evaluates audio-landmark fusion and audio attention. It does not compare the proposed MLLM planner with simpler alternatives, such as Transformers or conventional audio-to-motion models, under the same training and rendering settings. Figure 3 provides only qualitative comparisons, which are insufficient to establish the benefits of MLLM-based planning.
  4. The evaluation uses only 98 HDTF videos and 50 CelebV-HQ videos, randomly selected from datasets also used for training. The paper does not clearly establish identity-disjoint evaluation or test generalization to unseen speakers. The paper also includes no cross-dataset experiments, user studies, or statistical significance analyses.
  5. Although PlanTalk performs well overall, its FID is slightly worse than MoDiTalker on both datasets (14.38 vs. 14.15 on HDTF and 19.00 vs. 18.93 on CelebV-HQ). The improvement in LSE-D on HDTF is also relatively small (9.02 vs. 9.15). Without repeated experiments or uncertainty estimates, it is difficult to assess whether these differences are meaningful.
  6. The paper does not clearly explain how the frozen LLM generates continuous landmark representations autoregressively, how audio tokens are temporally aligned with landmark tokens, or how the MLLM is supervised to learn speaker-specific articulation. These details are important for understanding and reproducing the proposed motion planner.
Questions for authors
  1. What specific capabilities of the MLLM make it better suited for facial motion planning than a standard Transformer? Can the authors provide a controlled comparison using the same input features and training data?
  2. How are landmark tokens generated autoregressively from Vicuna-7B? In particular, how are the continuous motion representations constructed, temporally aligned with audio, and supervised during training?
  3. Can the authors provide a direct ablation of the MLLM planner itself? The current ablations mainly validate the two audio-guidance modules rather than the claimed core contribution.
  4. Are the training and test sets strictly identity-disjoint? How does PlanTalk perform on unseen identities, cross-dataset evaluation, and longer speech sequences?
  5. Can the authors report results over multiple random seeds and provide a user study? Improvements over MoDiTalker are small, and the current automatic metrics alone do not fully establish perceptual superiority.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.