- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for real-time, streaming, avatar generation, using a causal, flow-matching based, model. The method takes as input a bidirectional teacher, and converts it to an auto-regressive model, and then distills this model using Distribution Matching Distillation. Special care is taken with respect to the context put into the KV-cache, which allows the method to have flexible chunk sizes in inference. The method shows good quantitative results in comparison with other approaches.
Critical strengths
In terms of quantitative metrics, the proposed method presents good results, outperforming LiveAvatar in particular, a reference method for the avatar streaming setup. The consideration of noise level in the cache context seems like a good issue to highlight, and it is natural that it should have a strong impact on the results. The introduction and motivation of the paper are well-done and clear.
Critical weaknesses
1/ My main criticism of the paper is that it does not clarify the key novelty or important contributions very well. This is due, in part, to heavy use of jargon everywhere which makes the paper difficult to read. While I understand that this is a domain with many different moving parts and techniques in play, and thus a certain amount of jargon is necessary, I felt the paper did not do a good job of highlighting why their method is different and better than other approaches.
2/ Another major criticism of the paper is that there are no video results (ie in video format) that I could see! This is a major drawback, because it is almost impossible to evaluate video generation from still images. In particular, one cannot see the temporal consistency and smoothness, over short or long horizons. I did not see a supplementary material or a link which would show this.
3/ In this same theme, concerning results, it feels like you are over-selling them (Line 372). You state that "DeForcing provides the strongest overall streaming trade-off on both benchmarks: it leads LiveAvatar ...". While this is true, it ignores SoulX-FlashTalk, which has better metrics on 5 out of 10 metrics/situations (counting short and long videos) than DeForcing. Thus I feel that it is a bit disingenuous. The conclusions are not so clear, and since there are no video results to view, it is difficult to tell whether the method is indeed better or not. In terms of qualitative results (compared with other approaches, the only ones which exist are in Figure 6, as far as I can tell. This is far from enough to confirm that the results are better visually.
Smaller notes: 4/ Line 73: Define the acronym Causal SFT
5/ Line 163: "Appendix A.2 gives the full gradient derivation". I don't know why you didn't just include equation 9 in the main paper, since it helps to understand what is done better. And I feel that "derivation" is a bit of an exaggeration here; you just copy the equation written in DMD, which does not really derive the gradient, in the sense that you do not show how the equations are obtained. Maybe think about including equation 9 in the main paper, or replace the phrasing "derivation".
Questions for authors
1/ If I have understood correctly, the core of the work is based on a careful consideration of whether the "history", ie the context which will be put into the KV cache is noised or not. It is presented as a novelty, or it seems to me, but both approaches are done in previous methods (self-forcing, causvid, live-avatar). It was not clear to me what the exact novelty is, can you explain this more clearly ?
2/ Section 2, related work. The Audo-drive avatar generation makes it seem like all other methods are non auto-regressive, when in fact you have at least SoulX-FlashTalk which is indeed Auto-Regressive (if I have understood). Also, you don't mention this paper in the related work, why ? Finally, the related work is very short, as you have missed at least the previous method. Please redo it.
3/ I found the section 3.2.2 extremely difficult to follow. You have added a whole new layer of jargon (AR rollout, prefill inference, MTP inference) for not much gain it seems to me. The people reading the paper are not necessarily experts in LLMs, so why this added jargon. Furthermore, the section is named "Flexible Chunk-Wise Inference" (and indeed this is a core contribution), and it is not said at all how these chunks are defined. If the flexibility is so crucial, shouldn't you discuss it more ? It is barely discussed in this section, and indeed it is difficult to figure out what is important.
4/ Line 201: "Clean- and noisy-cache formulations admit different combinations of causal initialization and DMD teacher choice. Our final pipeline uses noise-aligned causal initialization (Causal SFT or Causal CFG distillation), followed by rolling-free DMD with a bidirectional fake score estimator and frozen bidirectional teacher". It seems that this is at the core of the method, and is what much of the paper is leading up to. But I did not understand why this was novel, or what you are proposing which is central to the success of the paper. You should re-write this section to be much clearer.
5/ Concerning the execution time (Table 2), do you achieve real-time as you state in the title ? This is not clear, and you do not discuss it much. Real-time is 30 frames per second, do you achieve this, and if so, in which configuration ? If this is not the configuration of the results in Table 1, then this needs to be stated, or the results in Table 1 need to be updated. Otherwise, this is disingenuous.
6/ I found the jargon "unit" to be confusing. What you mean here is a latent temporal frame right ? I feel this would be much clearer, even if I understand why you wanted to avoid "frame". I feel "temporal latent frame" or "latent frame" is better.
7/ Line 308: "By the prefill-rollout equivalence established in Section 3.5 ...". This was extremely unclear to me, and seems central to the paper. Section 3.5 is very short, and I am not sure how this "equivalence" is established. It seem more like you just say it is so.
8/ Line 359: "MiniMax H3 Ref2VA". It is not explained what this is in the main paper, there is no way of knowing, and this stops us knowing what DeForcing-H3 is. Please rectify