- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a method to achieve interactive talking-head avatar generation for dyadic conversations, that is to say live conversations between two participants. The approach consists in a motion generator, an audio encoder, a renderer, and a streamer. The results give good continuity and stability, in particular because of the motion representation, I feel. The method is compared quantiatively with respect to several competing works, and with various metrics designed for video quality evaluation. The quantitative results are compared with these approaches, but only in still frames; the video results only show the proposed methods' output.
Critical strengths
1/ The problem of avatar generation is timely and useful. The goal is clearly stated and important. 2/ The results are of good quality, and are shown over long rollouts. It is clear that the choice of using the motion representation presented in section 3.1 has a positive influence on the stability of the method. Indeed, other approaches often may have subtly unrealistic motions, quite characteristic of Generative AI. The proposed approach does not seem to exhibit this, even if this does result in a slight wooden quality of the faces (the expressions remain quite neutral).
Critical weaknesses
1/ My main criticism is the heavily "system" orientation of the paper, and the explanations of the technical details, which are very hard to understand. It does not seem like a novel algorithm is proposed, rather a system of existing algorithmic bricks. The authors did not convey what was new very well, I find. The difficulty in understanding the technical details is due in large part to the lack of formalism in the paper, which I go into further detail below. I do not feel that the diagrams are clear enough to understand everything from them. Formulae and/or pseudo-algorithms would have been useful here.
2/ I will go here into more specifics concerning the lack of formalism. Apart from in the appendix, there are no notation or any equations, apart from at line 231. This necessarily makes the paper difficult to read, and hard to pin down what is going on exactly. This is particularly problematic for a research paper, which should have much more formalism. At the moment, it is a long textual description of the approach, and all technical details are pushed into the appendix, which is problematic. Some examples:
- Line 107: "Each frame is represented by a 42-dimensional vector of head rotation and expression deformation derived from LivePortrait’s motion–appearance disentangled space (Guo et al., 2024). The renderer separately handles appearance. Further details are provided in Appendix". These details need to be in the main paper, especially the formalism and notation.
- Line 121: "The conditional flow-matching Transformer (Lipman et al., 2023) autoregressively generates five-frame motion chunk ... ". Again, this needs to be formalised, and more details given. What is "other-audio" ?
- What is a "Stack" exactly? I feel that this is a system terminology which is not super common in the present literature (autoregressive video models). This should be made clear.
- Line 140: "To solve this, we build a chunk-based encoder through self-distillation of the original model without changing its architecture." You put the details in the appendix, but I feel that it should be in the main paper, because of its importance.
- Line 150: "At initialization, it registers the source portrait and caches its LivePortrait appearance features and reference keypoints." Caches how? Where? With respect to what model?
- Rendering pipeline: At no point do you say what model is used for rendering, or how it works from a technical viewpoint. The details are, again, pushed into the Appendix, but even looking, I could not find them. Is it a diffusion model? You say that a diffusion model is used for generating the motion, but what about the image/video renderer? How does it work?
- Equation (no number, line 321): Why are and random variables? It seems logical, but you do not explain why, or say it explicitly.
3/ There are no visual comparisons with other methods in the supplementary material. Why is this ? It seems important to do this, and would potentially be in your favour. As it is, the only visual comparisons are in Figures 7-8, and these do not really seem to favour your method. In particular in Figure 7, it seems that the proposed approach does not move much with respect to the ground truth (speaker). Video comparisons would help dispel this concern.
4/ Section 3.3 I found particularly difficult to read. It is a long sequence of textual description, with no notation at all for any of the objects introduced. This whole section would need to be re-written with clear, formal notation, otherwise I feel that it is impossible to understand or implement.
Questions for authors
1/ Can you summarise what is different in your approach, compared to previous ones, in algorithmic terms ? Apart from the system aspect, what is algorithmically different ? What is your added value ?
2/ The name of the method "XXXX-1" seems a bit odd. Is this a mistake on the part of the authors ? (maybe a macro that was not set ?). I would suggest creating a slightly more memorable name, in the interest of the method.
3/ Section 3.4. I had great trouble understanding this section, but most of all, I had trouble understanding in what sense the latency model was "analytical". When we say "analytical", we usually mean a problem has a closed-form solution, ie a mathematical formula to solve the problem. Where is this the case? In the equation? In general, I could not really understand what was going on in this Section, could you explain it simply, and indicating what you mean by "analytical"?
Small comments:
4/ Although I see that the term "dyadic" has been used in other publications, maybe you could define it here? It may not be obvious to all readers.
We would like to thank reviewer tmh4 for the detailed critique of the formalism, which drove the revision.
W1. System orientation of the paper, novelty
W2. Difficult understanding of the paper
W3. Lack of visual comparisons with other methods
W4. Section 3.3. is difficult to read
We rewrote it with formal notation and a pseudo-algorithm (see above).
Q1: what is algorithmically different
Prior work reports model speed or generation-only latency. None of it covers the path from a voice agent's reply to playback, although it adds significant latency. Our contribution is this path: a window-formation algorithm whose rules give closed-form bounds on the added latency, validated live. Any chunk-based generator that meets the renderer interface can replace our model.
Q2: the name "XXXX-1"
It is the anonymized name for double-blind review, shared with the anonymized repository. A footnote now explains this.
Q3: in what sense the latency model is "analytical"
We meant closed form: each latency contribution is a sum of fixed and bounded random terms, so the bounds follow directly. We agree that "theoretically" overstated this, and the revision says "closed-form bounds".
Define "Dyadic"
Speech-driven talking head generation aims to generate facial and head movements conditioned on given audio. Dyadic systems extend this setting by conditioning motion generation on signals from both conversation participants.