- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper generates 3D head motion for both people in a conversation at the same time, without needing the partner's motion as input. It has two stages: first a deterministic model predicts rough anchor motions for both people using audio, who is speaking, and a style clip; then a flow model refines both motions together, with attention between the two people.
Critical strengths
I like the setting which is practical since it doesn't need the partner's motion. The idea of letting the deterministic part handle lip movement and the flow handle the less predictable parts (expressions, head reactions) makes sense. Besides, results on UniLS are good on most metrics, especially F-FID/P-FID, and there is a user study and extra results on DualTalk.
Critical weaknesses
- I think there may be ground-truth leakage in the evaluation. Test sequences are 400 frames and the style reference starts at min(L/2, L-100) = 200, so the reference is frames 200-300, which is exactly one of the four test windows. The model also gets the mean/std of this reference. This could be why even Stage 1 alone has about 5x better F-FID than UniLS. I'd like to see results with a reference from a different clip, or with no reference.
- The contribution is somewhat incremental. Deterministic prior + residual flow is from SubtleTalk, cross-person attention is from InterGen, and pyramid flows already exist.
- The system is very complicated: six training phases, more than 20 losses and lots of hand-set weights, with no sensitivity study. It's hard to know what actually brings the improvement.
Questions for authors
Please refer to weaknesses.