- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper conditions a frozen talking-video diffusion model on multiple reference clips. A domain-adapted video encoder and a query-based aggregator compress these clips into a fixed set of identity tokens, which enter the generator through additional cross-attention. Training first uses the target clip as its own reference and then switches to non-overlapping reference clips of the same person. The intended benefit is to capture characteristic expressions and movements as well as appearance.
Critical strengths
Personalizing behavior is a useful goal, and the method improves expression and pose distribution distances over the image-reference baseline. The fixed number of conditioning tokens is a sensible design for multiple video references. The paper documents its architecture, preprocessing and evaluation in considerable detail, attributes its use of Q-Former and IP-Adapter, and includes relevant talking-avatar comparisons and a human study. It also clearly labels the preceding-ground-truth-clip setting as an oracle.
Critical weaknesses
1. The evaluation does not separate a person's characteristic behavior from the circumstances of a particular recording. Appendix B states that cross-clip references are other segments of the same longer recording. Removing temporal overlap does not remove shared mood, conversation, posture, clothing or camera setup. Better matching the target under this protocol is useful, but it does not establish that the representation captures behavioral identity that persists across sessions.
The metrics do not resolve this distinction. E-FID and P-FID compare distributions of individual frames, without measuring their temporal order. P-FID includes translation, and Mesh-FID includes facial geometry, so these scores can also respond to framing and appearance. The DINOv3 comparison changes the encoder and its pretraining, rather than isolating temporal information in the same encoder. The human study supports the quality of Non-Direct + Ours, but it does not test whether the behavior transfers across sessions or remains identifiable after appearance is controlled. I would like to see cross-session references, temporally shuffled or repeated-frame references with the same encoder, and a test of whether observers can identify a person's behavior across contexts. This matters because behavioral identity is the main claim distinguishing the paper from richer visual conditioning, including the already-cited TAVR [1].
2. Important ablations and metrics are not controlled well enough to support the attribution. Appendix C.1 trains the same-clip-only and cross-clip-only variants for 10,000 steps each, while the sequential variant receives 20,000 steps. The result therefore does not isolate the proposed curriculum from additional training. Both single-stage controls should receive the same total budget.
FVDtarget is also unusually fragile here: it fits a distribution of 400-dimensional I3D features separately for each subject, with only 581 clips across 119 subjects, or about five clips per subject on average. With so few videos, the covariance estimate has very low rank and the distance can be strongly affected by sampling. The paper provides neither uncertainty estimates nor a sample-count sensitivity analysis. The pooled FVDtest set is helpful, but it answers a different question and does not validate this subject-level estimate. Multiple generation seeds and subject-level confidence intervals would make the reported differences more convincing. Finally, encoding eight references once in 7.9 seconds does not give the total inference cost; the three guidance predictions per denoising step also need to be included.
I see a useful conditioning method, but the evidence for persistent, person-specific behavior is not yet strong enough for acceptance.
Questions for authors
- Can you evaluate reference and target clips from different recording sessions of the same person? How much of the improvement survives changes in context, framing and clothing?
- With the same video encoder, what happens when reference frames are shuffled or a single frame is repeated? Can behavior be identified independently of facial appearance?
- Do the curriculum gains remain when all variants receive 20,000 optimization steps?
- What are the per-subject clip counts and confidence intervals for FVDtarget? Please report sensitivity to sample count, multiple generation seeds, and total generation time/memory for the complete guidance procedure.
Reference
[1] Generate Your Talking Avatar from Video Reference (TAVR). https://arxiv.org/html/2604.27918
Thank you for recognizing the value of behavioral personalization, the improvements in expression and pose matching, the fixed-size conditioning design, and the detailed experimental documentation.
W1 / Q1: References across recordings and recording conditions
Reference clips are not restricted to segments of the target recording. They also come from different recordings of the same person, potentially wearing different clothes. However, lighting and scene remain very similar due to the nature of the dataset. We clarify it in Appendix B.
W1 / Q2: Temporal information and appearance-independent behavior
Following Reviewer GFRH's suggestion, we have added landmark-velocity Fréchet distance and a face-normalized variant that removes translation and scale changes. Our method achieves the lowest face-normalized velocity distance and an image-coordinate velocity distance close to the best result. Adding our conditioning to FramePack improves both velocity metrics while largely preserving appearance similarity.
With the same trained model and encoder, shuffling reference frames at inference leaves appearance similarity relatively unchanged but worsens behavioral matching below the image-only baseline. This supports reliance on temporal cues, although the distribution shift prevents interpreting it as proof of appearance-independent behavior. Our tokens jointly encode appearance and behavior. We will provide the cross-identity experiment requested by Reviewer Xpxa as an additional diagnostic.
The repeated-frame control and an appearance-independent behavioral-identification study are not included in the current results.
W2 / Q3: Training-budget controls
We selected the 10,000-step single-stage checkpoints because further training did not improve performance. Cross-clip-only training even began to degrade. These observations explain our checkpoint choice but do not constitute a completed equal-budget comparison at 20,000 steps.
W2 / Q4: Per-subject FVD and generation seeds
Thank you for highlighting the sample-size limitation of per-subject FVD. We have removed FVD_target and retain pooled FVD for dataset-level video assessment. We will also evaluate multiple generation seeds and report the variance, as suggested.
We hope the additional evidence and clarifications address your main concerns and would appreciate your reconsideration of the assessment.