- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PlanTalk addresses audio-driven talking-head generation with a multi-stage pipeline. Instead of regressing landmarks from audio with identity as a conditioning signal, it formulates facial motion generation as identity-aware cross-modal planning: a frozen Vicuna-7B MLLM with frozen ImageBind encoders takes the identity frame and driving audio and autoregressively produces landmark hidden states. Hierarchical audio guidance then re-injects speech at two levels: audio–landmark MM-Attention fusion refining the planned motion, and denoising-stage audio cross-attention inside a MoDiTalker-style Motion-to-Video diffusion generator. Training uses landmark reconstruction/residual/multi-scale-velocity/identity losses plus standard diffusion loss. On HDTF and CelebV-HQ, PlanTalk reports the best LMD/LSE-D/PSNR/CSIM, competitive FID, and the fastest inference. Ablations show that both guidance levels contribute.
Critical strengths
- Persistent audio guidance is a real and well-validated idea. The denoising-stage audio attention lets speech cues correct errors that landmark prediction already baked in — addressing a genuine weakness of multi-stage pipelines. The ablation (Table 3) shows consistent, monotone improvements from each guidance level.
- Competitive efficiency despite an MLLM. Replacing iterative motion diffusion with a single autoregressive pass while retaining diffusion only for pixels is a sensible design point; 20.1 s/clip with the best lip-sync numbers is a practically attractive combination, if the timing protocol holds up.
Critical weaknesses
- No ablation study of the core component. The paper's title and first contribution is the MLLM motion planner, yet nothing isolates it: no comparison against a same-size non-LLM transformer planner, no test of whether the LM's linguistic knowledge matters at all (vs. ImageBind features alone), and the Fig. 3 landmark comparison is qualitative only. Given that Vicuna-7B and ImageBind are frozen and only projectors/fusion are trained, it is plausible the MLLM is just an expensive generic encoder. This directly undermines the paper's central claim.
- Evidence strength does not match the SOTA claim. Improvements over MoDiTalker are within noise on most metrics, FID is not better on CelebV-HQ, test sets are small (98/50 clips, in-domain, self-reconstruction), and there is no human evaluation of lip sync or identity. For a generation paper at this venue, the evaluation is thin.
Questions for authors
- Please provide an ablation replacing the MLLM planner with a parameter-matched non-pretrained transformer (and with ImageBind features + a small temporal module, no LLM). Does the planning advantage survive?
- How many tokens does the MLLM emit per clip, and what is the per-stage runtime breakdown (planner vs. fusion vs. diffusion) for the 20.1 s figure? Which GPU and batch size?
- Do results hold with variance across multiple sampling seeds, and on cross-identity audio?
- The authors are recommended to discuss in the future work section: can the MLLM planner accept textual emotion/style instructions?