- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a method for jointly generating long-form audio and video of a talking avatar. In the proposed method, an existing joint audio-video generation model is distilled into a few-step causal model, which generates autoregressively while referring to fixed references and continuously updated memory, in addition to a KV cache that holds the most recent outputs. The fixed references are compressed representations of the video and audio from the initial block. The adaptive memory comprises two types of information. The first one is normalized content representations, obtained by normalizing and projecting previously generated blocks. The other one is channel-level and low-frequency appearance statistics. Both are updated with a moving average that mixes in the anchor, and for the video content and appearance statistics, the update rate is adaptively determined by their agreement with the anchor. During generation, the content representations are fed into the model through dedicated memory attention, and a part of the appearance statistics is injected through FiLM. In training, the model is distilled on contexts that include student-generated and re-noised history to reduce exposure bias. At inference, the memory is kept fixed within each window, and stage-parallel execution, which assigns each denoising stage to a separate GPU, accelerates generation. In the experiments, the method outperforms the baselines, particularly in long-term consistency.
Critical strengths
- The proposed method achieves the best results in the long-term consistency metrics and LipSync, and the high consistency can also be confirmed in the supplementary materials. However, it has not been shown that this improvement attributes to the proposed persistent memory (see Weaknesses).
Critical weaknesses
- The effectiveness of the persistent memory, which is the central contribution, is not clearly demonstrated.
- In Table 3, the variant without memory already reaches a Long Cons. of 0.9614. This is on par with LiveAvatar (0.9619) and far above OmniForcing (0.3404). Its Color Δ of 0.00380 also already outperforms the best baseline, LiveAvatar (0.00541).
- The improvement from the full model appears to be very limited (e.g., +0.0062 in Long Cons.). Moreover, since the variance is reported, I cannot judge whether these differences are significant.
- The performance of the proposed method differs between Table 1 and Table 3 (Long Cons. of 0.9755 and 0.9676, and Color Δ of 0.00260 and 0.00276, respectively), and no explanation is given for this discrepancy.
- Most of the gap to the baselines comes from components other than the memory, such as the distillation scheme or retaining the initial latent. However, no ablation on these points is provided.
- The evaluation only covers settings in which the subject, background, and camera are fixed. The paper states that the memory must be continuously updated to adapt to scene evolution, but this necessity is not verified.
- The evaluation does not sufficiently support the claims.
- The consistency metrics measure the stability of color and brightness, which means that outputs that are closer to static images tend to score higher. On the other hand, there is no evaluation of audio and video quality or of the amount of motion. In the supplementary videos, the quality appears lower than that of the baselines (the person looks blurry and the audio contains considerable noise). This raises the possibility that the gain in consistency comes at the cost of quality.
- The baseline values cited in Table 2 do not match the those reported in the original papers. In the cited papers, LiveAvatar reports 45.2 FPS on five H100 GPUs, and LongLive-2.0 reports 45.7 FPS (2 steps) on a single GB200 GPU with one additional GPU for VAE decoding. The main text claims that TaoMate exceeds the reported throughput of these methods by 1.52× and 2.19×, but this claim is not supported according to the values in the original papers.
- In Table 1, the throughput of OmniForcing (8.14 DiT FPS) is about one third of the roughly 25 FPS reported in the original paper. This baseline appears to be a reimplementation on LTX-2.3, but its detail and measurement conditions are not described.
- Although the paper claims real-time generation, latency is not reported.
Questions for authors
- In the ablation study, why does the variant without memory already perform on par with or better than the baselines?
- Why is there no evaluation of video and audio quality, even though such metrics are included in VABench?
- Where do the values for LiveAvatar (23.0 FPS) and LongLive-2.0 (16.0 FPS) in Table 2 come from? In addition, why is latency not reported? How was the OmniForcing baseline implemented and measured?