acceptodds
Under review as a conference paper at ICLR 2027

Alive-Avatar: Streaming Audio-Video Avatar Generation with Persistent Identity

Abstract

Audio-video avatar generation is moving toward real-time, interactive streaming, where the core challenge is to sustain fast response, interactive control, and a persistent identity together over long sessions. We present ALIVE-AVATAR, a streaming audio-video avatar generation model that meets these demands. We first adapt the pretrained ALIVE base model through prompt-to-shot teacher forcing to enable promptable shot streaming for interactive control. Then, we design a distillation curriculum, initialized with consistency distillation and refined by our PCM-regularized DMD2, that attains high-quality generation in four sampling steps. At inference, our streaming pipeline uses Context Multiplicity Modulation to regulate attention to recent generated context, mitigating identity drift while retaining bounded historical memory. Together, these designs yield a four-step, 4-NFE model that, on our long-sequence benchmark, outperforms strong avatar baselines in identity consistency, face quality, and speech correctness, renders 480p audio-video at 24 fps faster than playback, and preserves identity even in one-hour rollouts. Building on ALIVE-AVATAR, we further construct a live-streaming agent that delivers planned content and answers viewer comments, demonstrating ALIVE-AVATAR's practicality for real-world avatar live streaming.

Primary area:
generative models
Keywords:
Streaming Audio-Video Generation, Interactive Avatars, Diffusion Distillation, Long-Term Identity Preservation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_71n1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ALIVE-AVATAR, a real-time streaming audio-video generation framework designed for interactive avatar generation with persistent identity. Starting from a pretrained ALIVE model, the authors first introduce prompt-to-shot autoregressive training, followed by sCM initialization and DMD2 refinement with PCM regularization to achieve four-step generation. At inference, Context Multiplicity Modulation (CMM) reduces attention to recent generated context to mitigate identity drift. Experiments on a 23-case benchmark demonstrate improvements in identity consistency, facial quality, and speech correctness over several baselines, alongside qualitative hour-long generation results.

Critical strengths
  1. Practical and relevant problem setting. The paper addresses the challenging combination of real-time joint audio-video generation, interactive prompt control, and long-term identity preservation. The integrated streaming system and live-agent demonstration highlight its practical relevance.

  2. Promising empirical performance. The method achieves four-step, faster-than-playback generation while reporting competitive quality against substantially larger baselines. In particular, CMM provides a notable improvement in identity consistency, suggesting its practical effectiveness.

Critical weaknesses
  1. Unclear motivation and mismatch between shot-aware rollout and the actual inference setting. The proposed shot-aware rollout appears inconsistent with the paper's core motivation of prompt-to-shot generation. During inference, each prompt is intended to generate a complete shot, and subsequent shots may be conditioned on different prompts. However, the proposed training strategy randomly splits a single clip into a prefix and suffix, and both rollouts use the same prompt condition. The split does not necessarily correspond to a meaningful shot boundary or a prompt transition. As a result, the proposed procedure primarily simulates continuation from self-generated history, rather than the actual shot-to-shot, prompt-switching scenario that the paper aims to address. It is therefore unclear what makes this procedure specifically shot-aware, or why it should be more suitable than standard Self-Forcing. Although Table 2 reports improvements over a Self-Forcing baseline, the paper does not sufficiently explain or isolate the benefits of this particular rollout construction.

  2. Insufficient experimental evaluation. All main quantitative quality comparisons and ablation studies are conducted on a newly constructed benchmark containing only 23 cases. The paper does not report results on established public benchmarks or sufficiently justify why existing evaluation protocols are unsuitable. Given the relatively small evaluation set and the use of only one random seed per case, the reported results provide limited evidence of the method's generalizability and robustness. Moreover, persistent identity over hour-long generation is a central claim of the paper, yet the quantitative evaluations cover sequences of only 1.5–5 minutes. The one-hour results are primarily qualitative, without systematic measurements of identity drift over time. More comprehensive evaluations on established benchmarks, together with quantitative long-horizon analysis, are needed to substantiate the paper's broad performance claims.

  3. Limited methodological novelty and insufficient justification of key design choices. The overall framework appears relatively incremental, combining several established techniques rather than introducing a substantially new modeling or training paradigm. The distillation pipeline largely builds on existing sCM, DMD2, and PCM methods, while the distinct contribution of the proposed shot-aware rollout remains insufficiently justified, as discussed above. Similarly, Context Multiplicity Modulation (CMM) is essentially a simple attention-logit reweighting strategy that suppresses recent generated context. While its reported improvement in identity consistency is substantial, the mechanism itself is relatively straightforward, and the paper does not adequately justify its specific design choices. In particular, the choice of the fixed multiplicity factor of 0.6 lacks a sensitivity analysis, and no comparisons are provided against natural alternatives such as directly strengthening anchor attention or adjusting the anchor/recent context allocation. Consequently, it remains unclear whether the proposed CMM design offers distinct advantages over simpler attention or memory reweighting strategies.

Questions for authors
  1. What specifically makes the proposed training shot-aware? Since both training stages use random clip splits without explicit prompt transitions, how does this design address shot-to-shot prompt switching beyond standard Self-Forcing? Can the improvements be isolated from differences in rollout construction and supervision?

  2. Can the authors provide quantitative identity-consistency evaluations over substantially longer sequences, ideally up to one hour, together with time-resolved drift measurements and evaluation across more cases or benchmarks?

  3. How sensitive is CMM to the multiplicity factor of 0.6? How does suppressing recent-context attention compare with alternatives such as strengthening anchor attention or changing the anchor/recent context allocation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 71n1 (Part 1/3)

Thank you for highlighting the practical streaming setting, the four-step generation results, and the identity gain from CMM. We have added 77 benchmark cases and 30 complete one-hour runs of our model, including three-seed, same-checkpoint video-CMM comparisons. We present the additional results and clarify the rollout construction and the scope of the evidence below.

Responses to Questions and Critical Weaknesses

Q1: What makes the training shot-aware without explicit prompt transitions, how does it address prompt switching, and can rollout/supervision gains be isolated?

The key distinction is that the prediction endpoint is always aligned with the end of the shot associated with the current prompt. For a prompt-conditioned shot ending at T, we randomly sample the continuation start s and regenerate the entire remaining suffix [s,T), rather than a fixed-length next block. Thus, the start and prediction length vary, but the target always extends to the same prompt-associated endpoint (Sec. 3.1; Fig. 3).

In the fixed-block Self-Forcing-style baseline, block boundaries are determined by block length and need not coincide with prompt completion. Our construction instead trains completion of the current prompt's remaining shot, matching the inference interface in which one prompt controls one complete, variable-duration segment. The random split need not be a scene cut or a prompt transition: shot-awareness comes from the aligned endpoint, not from the split itself. Sharing a prompt across prefix and suffix is therefore consistent with this objective; a subsequent prompt controls the next shot. Both methods use self-generated history; our distinction is the prompt-end-aligned prediction unit.

AuthorsOpenReview ↗

Response to Reviewer 71n1 (Part 2/3)

Q2: Can you quantify identity consistency up to one hour, including time-resolved drift and evaluation across more cases or benchmarks?

Q2.1: Full-hour identity, quality, and temporal drift

We report 30 full-hour runs of our model: five synthetic identities x three seeds x two configurations, namely ours and video-CMM off. The seeds are 104729, 130363, and 155921. Each case has a 3,600-second, 601-segment script. The five identities were already used in the added cohort; these are new long-horizon scripts, not unseen-identity tests. Historical demonstrations and development clips are excluded from the formal sample.

We evaluate visual metrics at 1 Hz over the entire hour and compute scores for all 60 one-minute windows. The table below summarizes the full-hour scores and first/last five-minute endpoints. Full-video IDC is the median reference-to-frame face similarity, while Q-Face averages face-crop quality. We then average the three seeds within each identity and the five identity means. Intervals use 10,000 identity-cluster bootstrap draws; seeds and frames are not treated as independent identities.

ConfigurationFull-hour IDC [95% CI]Full-hour Q-Face [95% CI]First 5 min IDCLast 5 min IDC
Ours0.8751 [0.8690, 0.8845]0.8868 [0.8328, 0.9409]0.87990.8722
Video-CMM off0.8067 [0.7534, 0.8504]0.8817 [0.8241, 0.9393]0.83930.7930

The new evaluation measures temporal drift rather than just final-frame quality. Defining D_ID = IDC_first5min - IDC_last5min, ours declines by 0.0077 versus 0.0464 with video-CMM off. The paired reduction in decline is 0.0387 [0.0113, 0.0818]. The final-window IDC advantage is 0.0792 [0.0268, 0.1504]. This comparison uses the same checkpoint, reference, script, and seed, with video-CMM enabled or disabled; it evaluates the deployed video-CMM configuration as a whole.

Ours has first/last-window Q-Face 0.8864/0.8873. Its Q-Face decline is -0.0010 [-0.0020, 0.00004], consistent with little average change; the final-window on/off Q-Face difference has a CI crossing zero, so we do not claim a clear improvement on that endpoint. Both visual evaluators have valid faces at all 3,600 planned timestamps for every run.

Our strongest controlled conclusion is reduced long-horizon identity deterioration with video-CMM across these five identities and three seeds, not stability for arbitrary identities and durations.

Q2.2: More evaluation cases, benchmarks, and generalization

The new evaluation expands the input set from 23 to 100 cases, adding 77 cases to the original 23. The added cohort contains 40 English and 37 Chinese cases, 2,940 prompt-controlled segments, and 241 minutes of target content. Each new case lasts 90-300 seconds and uses an API-generated fictional-adult reference. It covers basic speech (11), head poses (20), gestures/partial occlusion (19), object interaction (10), and repeated control changes (17).

All five systems were evaluated on the added cohort. The following results use only the 77 new cases, avoiding an apparent improvement caused solely by recombining the original scores:

SystemIDCQ-FaceSync-DSpeech Corr.
Ours0.89100.89658.87840.9906
OmniForcing0.34010.496812.49920.4437
TaoMate0.10410.445514.11840.6657
LiveAvatar0.78370.673813.08100.9943
LongCat0.55750.72358.02460.9945

Every entry has 77 valid videos; each video has equal weight. IDC is the mean of per-video medians, matching Appendix B.5. On the combined 100 cases, ours has IDC 0.8675 and Q-Face 0.8686, also the highest system means. Prompt format and input distribution differ between the original and added cohorts, so higher scores on the added cohort are not a before/after model improvement.

Our protocol is designed to test jointly generated speech and video under successive prompt/action/duration controls over minutes. An externally supplied-audio evaluation alone does not measure this entire interface, although it remains useful for complementary comparisons. We have not completed an established external-benchmark evaluation and do not claim that such benchmarks are unsuitable. The expansion improves task coverage, while Q2.1 separately adds multi-seed long-horizon evidence; it does not constitute a multi-seed test of all 100 cases or establish open-domain generalization. Baseline interfaces, resolutions, and seeds remain method-specific, so the table is a system-level comparison rather than a fully matched causal test.

AuthorsOpenReview ↗

Response to Reviewer 71n1 (Part 3/3)

Q3: How sensitive is CMM to 0.6, and how does it compare with stronger anchor attention or different anchor/recent memory allocation?

Q3.1: Multiplicity selection and sensitivity

We selected 0.6 on validation cases because identity drift was already barely perceptible at this setting, and further suppression produced no additional visible benefit. Smaller multiplicities downweight recent context more strongly; the goal is sufficient drift control, not maximal suppression.

The following additional sweep provides quantitative support for this choice:

CMM multiplicityIDCSync-DQ-FaceSpeech Corr.
0.90.7589.2790.7900.977
0.80.7799.2630.7820.978
0.70.7859.2680.7770.976
0.6 (default)0.7899.3080.7740.979
0.50.7829.3290.7730.977
0.40.7869.3560.7700.977
0.30.7879.2920.7680.977
0.20.7849.3940.7660.981
0.10.7829.3420.7630.970

The default row reproduces Table 2 of the submission.

The reported IDC values stay within 0.779-0.789 across the tested settings from 0.1 to 0.8, indicating a relatively broad operating region rather than a sharply tuned optimum. Lowering the multiplicity below 0.6 provides no consistent further identity gain.

Q3.2: Anchor-attention and memory-allocation alternatives

Our goal is to reduce error feedback from recent generated context, so we directly downweight that context rather than compensate for it indirectly. CMM reduces recent-context attention while retaining the tokens needed for local continuity, without changing their values, positions, or memory allocation.

Strengthening anchors instead adjusts the competing reference signal, while reallocating memory changes which historical information is available. Both are possible alternatives, but neither is needed to implement the targeted intervention. We therefore choose recent-context suppression as the most direct control of the identified error-feedback pathway. The CMM ablation and the sensitivity results in Q3.1 support the effectiveness of this choice.

Q4: What is the methodological contribution beyond combining existing techniques?

Our contribution is the task-specific formulation of prompt-controlled, persistent-identity joint A/V streaming: prompt-end-aligned, variable-duration continuation, consistency-initialized refinement for four-step generation, and control of generated-history influence without expanding memory. The claim is not that sCM, DMD2, or PCM individually are new.

The design addresses distinct failure modes at the same inference budget. Removing sCM initialization reduces IDC from 0.789 to 0.710; removing refinement reduces Q-Face from 0.774 to 0.722; removing PCM reduces IDC to 0.749. Table 4 additionally tests PCM weight and phase coverage. CMM raises IDC from 0.611 to 0.789 under fixed-model inference. These ablations support the specific formulation, while CMM's simple implementation makes it a practical inference intervention rather than an additional training burden.

Reviewer_6bZqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents alive-avatar, a streaming joint audio-video avatar generator post-trained from the pretrained ALIVE model. The method combines prompt-to-shot autoregressive training for variable-length, prompt-controlled segments; a four-step distillation pipeline using consistency-model initialization followed by PCM-regularized DMD2 refinement; and Context Multiplicity Modulation (CMM) to downweight recent generated context during streaming. Compared against four baselines, the paper reports favorable performance for alive-avatar in identity consistency, face quality, and end-to-end script adherence.

Critical strengths

The paper addresses a relevant and challenging problem: combining interactive prompt control, low-step generation, and persistent identity in streaming audio-video avatars. The prompt-to-shot formulation is a coherent way to align the unit of generation with an utterance or action, and CMM is motivated by a concrete tension between retaining context for continuity and limiting error propagation.

The method is described with substantial technical detail, and the paper includes ablations and candid failure cases. The reported CMM ablation, for example, shows a sizeable change in IDC. These are useful components of a research contribution, but they do not resolve the concerns about independent verification and reproducibility.

Critical weaknesses

The central reproducibility issue is that the work is built upon ALIVE, yet the supplementary material provides no code sufficient to reproduce the training and evaluation pipelines. Consequently, I am concerned about whether the reported performance gains can truly be attributed to the proposed post-training method rather than the underlying proprietary pretrained model. Furthermore, the training corpus and preprocessing procedures are also directly inherited from ALIVE.

Regarding the empirical comparison, the experimental scale is too limited and the evaluation settings are inconsistent to support strong claims of superiority. The main table includes only four baselines, of which only two are joint audio-video generators, while the other two are audio-driven systems paired with edge-tts. Additionally, the compared systems differ in resolution, conditioning modalities, and temporal chunking protocols.

Questions for authors
  1. Could you provide public access to the complete ALIVE initialization code, the final ALIVE-AVATAR code, configuration files, and evaluation scripts? If public release is not feasible, could you provide experimental results trained on an open-source base model? What empirical evidence supports pairing a 2B AudioDiT with a 1.6B VideoDiT? I find this allocation puzzling, given that existing dual-tower models (such as OVI) are typically symmetric, whereas the LTX series features an audio tower smaller than the video tower.

  2. Are the experimental conditions fair? Could you provide a stronger set of joint audio-video baselines evaluated under matched resolution, conditioning modalities, speech sources, and temporal protocols?

  3. What are the training compute costs and the per-component inference latency across different GPUs (including VAE encoding/decoding and DiT speeds)? These details are necessary to evaluate the claim of faster-than-playback inference and to reproduce the efficiency results.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6bZq (Part 1/3)

Thank you for recognizing the relevance of the task, the prompt-to-shot formulation, and the CMM ablation. Our central contribution is to turn a pretrained joint A/V model into a prompt-controllable, four-step streaming avatar. We have expanded the evaluation to 100 cases and 30 complete one-hour runs of our model, including a same-checkpoint, multi-seed video-CMM ablation. We present the additional results below.

Responses to Questions and Critical Weaknesses

Q1: Can the exact initialization, code, configurations, and evaluation scripts be accessed, or independently verified if release is unavailable?

Public release of our model weights and code is currently undergoing internal approval. The training corpus is internal; its curation, filtering, captioning, and training recipe are documented in Secs. 3-4 of the ALIVE technical report. We use the same corpus and preprocessing, with our post-training and inference settings specified in Appendix B.1-B.3.

Our evaluation uses established methods and publicly available scoring components:

  • Q-Face: the Q-Align image-quality scorer, applied to detected face crops.
  • Sync-D: the SyncNet implementation for audio-visual synchronization distance.
  • Speech Correctness: Whisper large for ASR, followed by transcript-to-script comparison.
  • IDC: cosine similarity between face embeddings of the original reference and generated frames, with the video-level aggregation defined in Appendix B.5.

Appendix B.5 describes the metric computation and aggregation protocol.

Q2: What evidence supports the 1.6B VideoDiT / 2B AudioDiT allocation, and separates ALIVE pretraining from the proposed post-training gains?

Parameter count alone does not determine computation: the two branches process very different sequence lengths. At 864 x 480 and 24 fps, Wan2.1-VAE compresses video by 4 x 8 x 8 in time/height/width. With our 1 x 2 x 2 DiT patchification, this gives approximately (24/4) x (480/16) x (864/16) = 9,720 video tokens per second. The audio VAE produces 25 latent tokens per second (24,000 / 960). Thus, the video generation sequence is approximately 389 times longer:

Generated durationVideo tokens (approx.)Audio tokens (approx.)
3 s29,16075
5 s48,600125
8 s77,760200

These counts describe generated content before adding history/conditioning and rounding. The much shorter audio sequence makes a relatively large AudioDiT computationally affordable. We retain its 2B capacity to prioritize speech fidelity, timbre, and acoustic detail, while keeping the video backbone compact to control the much larger video-token workload. This is a capacity-versus-compute design rationale, not a claim that token-count ratios equal measured latency ratios.

Attribution to our components is tested with the ALIVE initialization held fixed. In Table 2, the implemented Self-Forcing-style rollout gives IDC 0.725 versus 0.789 for ours; removing sCM initialization gives 0.710; removing PCM gives 0.749; and removing DMD2 refinement reduces Q-Face from 0.774 to 0.722. More directly, the new one-hour experiment changes only the video-CMM settings at the same final checkpoint and shows a paired reduction in identity decline (Q4). These comparisons establish incremental benefits within the shared backbone, rather than attributing every cross-system difference to our method. We have not performed modality-size ablations, so we do not claim that the 1.6B/2B allocation is uniquely necessary or optimal.

AuthorsOpenReview ↗

Response to Reviewer 6bZq (Part 2/3)

Q3: Can you provide multi-seed uncertainty estimates and stronger joint A/V baselines under matched resolution, conditioning, speech source, and temporal protocol?

Q3.1: Expanding the test set for broader evaluation coverage

Rather than running additional seeds on the same 23 cases, we chose to expand the test set to 100 cases. This prioritizes broader evaluation coverage over repeated generation on the original inputs. We retained the original 23 cases and added 77 new cases, evaluating all five systems on the expanded benchmark.

The added cohort contains 40 English and 37 Chinese cases, 2,940 segments, and 241 minutes of target content, covering basic speech, head poses, gestures/partial occlusion, object interaction, and repeated control changes. This tests the method across a wider range of inputs rather than concentrating the additional evaluation on the original small set.

SystemIDC, all 100Q-Face, all 100Sync-D, all 100Speech Corr., all 100
Ours0.86750.86869.03490.9855
OmniForcing0.32410.482012.68980.3971
TaoMate0.13530.472513.17930.6221
LiveAvatar0.76490.652613.13810.9763
LongCat0.53570.69958.10540.9749

Q3.2: Baseline comparison and matching of evaluation conditions

The baselines cover two distinct groups: TaoMate and OmniForcing jointly generate A/V from reference images and text; LongCat and LiveAvatar are audio-driven systems. The original benchmark supplies the latter with edge-tts speech. Audio-driven systems are complementary system-level references, not comparisons of speech-generation capability. Accordingly, Speech Corr. measures end-to-end script adherence.

All systems receive the same task specification: identity, script, and target duration, while retaining their supported interfaces. Variable-duration prompt alignment is itself part of our proposed capability; forcing every system to use our shot construction would change the compared methods. We will separate the two groups and make resolution and chunk-allocation rules explicit. The fixed-duration comparison answers a budget-controlled streaming question, distinct from performance when a baseline is allowed extra time to finish its utterance.

Q4: What are the quantitative identity and quality results over complete one-hour rollouts and repeated runs?

All 30 runs of our model cover the full 3,600 seconds: five identities x three seeds (104729, 130363, and 155921) x two configurations (ours and video-CMM off), using 601 scripted segments per case; the earlier qualitative demonstrations and development clips are excluded. This gives 15 runs per configuration. The video-CMM on/off comparison uses the same checkpoint, reference, script, and seed. We sample visual metrics at 1 Hz throughout the hour and compute IDC from reference-to-frame face-embedding similarities and Q-Face from face crops. The evaluation covers all 60 one-minute windows; the table below summarizes the full-hour scores and first/last five-minute endpoints. Within each full video, IDC is a median and Q-Face is a mean.

We first average the three seeds within each identity and then average the five identity means. The 95% confidence intervals use 10,000 identity-cluster bootstrap draws, keeping each identity's seed results and on/off pairs together. The statistical units are five identities, not individual frames or 30 independent identities.

ConfigurationFull-hour IDC [95% CI]Full-hour Q-Face [95% CI]First 5 min IDCLast 5 min IDC
Ours0.8751 [0.8690, 0.8845]0.8868 [0.8328, 0.9409]0.87990.8722
Video-CMM off0.8067 [0.7534, 0.8504]0.8817 [0.8241, 0.9393]0.83930.7930

Ours has an identity decline of 0.0077, versus 0.0464 with video-CMM off. Its first/last Q-Face scores are 0.8864/0.8873. Both visual evaluators return valid faces at all 3,600 planned timestamps in every run, so the results do not omit low-coverage portions.

Defining D_ID = IDC_first5min - IDC_last5min, the paired reduction in identity decline with video-CMM is 0.0387 [0.0113, 0.0818]. The final-window IDC advantage is 0.0792 [0.0268, 0.1504].

Speech and synchronization were also evaluated over the whole hour. Speech Corr. is 0.9909/0.9902 and Sync-D is 9.0126/8.9880 for ours/video-CMM off; every run has 601 scored utterances and 60 valid Sync minute blocks. Here speech is duration-weighted and Sync-D averages valid tracks within each minute before averaging minutes, so these scores are not pooled with the differently aggregated 100-case results.

The five identities were already evaluated in the added-77 cohort, with newly written hour-long scripts: the evidence supports repeated hour-long generation on these identities, not unseen-identity or unlimited-duration generalization.

AuthorsOpenReview ↗

Response to Reviewer 6bZq (Part 3/3)

Q5: What are the hardware, training compute, and full-system latency, including VAE decoding and video encoding/writing?

Training hardware and schedule. Each post-training stage uses 32 NVIDIA H100 GPUs, with the schedule reported in Appendix B.1:

StageHardwareTraining schedule
Autoregressive training32 H100 GPUs10,000 optimization steps
sCM initialization32 H100 GPUs2,000 optimization steps
PCM-regularized DMD2 refinement32 H100 GPUs2,000 training iterations

These schedules describe post-training, not ALIVE pretraining compute. We do not have measured stage wall-clock totals to report, so we do not infer GPU-hours from iteration counts alone.

Inference hardware and latency. Our serving deployment uses 8 NVIDIA H100 GPUs. At 864 x 480, 24 fps, ctx_oldest=18, recent=1, and four NFEs, the reported DiT RTF is 0.458 (Table 5, SP=8), and the VAE decoding RTF is 0.080 (Sec. D.3, CP=8). For a prompt requesting a shot of duration D, the combined generation-and-decoding latency is estimated as:

L_generation(D) = D x (RTF_DiT + RTF_VAE)
                = D x (0.458 + 0.080) = 0.538 D.
Requested duration (s)Estimated generation + decoding latency (s)
31.614
42.152
52.690
63.228
73.766
84.304

For example, a 5-second shot requires approximately 2.69 seconds including VAE decoding, leaving time to prepare subsequent content while earlier content is played. This generation-and-decoding budget is faster than playback and supports our intended buffered live-streaming setting.

Agent response time is a separate quantity. The autonomous agent decides whether and when to incorporate a viewer comment and may defer a response to a later utterance. Comment-to-response time therefore includes policy-dependent decision and scheduling delays, not just generator execution. We report the generation-and-decoding component above rather than assign a fixed latency to the agent's autonomous behavior. These values are estimates from the reported component RTFs, not separately measured end-to-end latencies; asynchronous video encoding/writing is excluded, as stated in Sec. D.3.

Response to Ethics Concerns

E1: What are the provenance, permissions or consent, and retention practices for real-person references?

All portraits and avatar examples displayed in the submitted paper are AI-generated fictional identities. The original quantitative benchmark additionally uses livestream screenshots and public celebrity photographs; these real-person references and their generated videos are not displayed in the paper (Ethics Statement; Sec. B.4). Public accessibility describes provenance, not individual consent or a reuse license.

Retention. Videos generated from real-person references are used exclusively for metric computation and deleted immediately after the metrics were computed.

The 77 newly added cases use API-generated fictional adults, and the formal one-hour experiment reuses five of these synthetic identities. Thus, the new synthetic-only cohort and long-horizon evidence can be considered separately from the real-person subset in the original 23 cases.

I appreciate the authors’ response and find the additional experiments they conducted during the rebuttal encouraging. However, I still have strong concerns regarding the reproducibility of the proposed method, as it relies heavily on internal models and proprietary datasets rather than widely accessible open-source foundations (such as Wan 2.1/2.2, MiniMax H3, or LTX). Therefore, I still believe my initial overall rating reflects my assessment of the paper, and I would like to maintain my score. I encourage the authors to validate their approach on open-source base models to ensure broader community verification and look forward to seeing a strengthened version of this work at a future top venue.

Reviewer_UxoVOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tackles the problem of audio-visual generation for talking heads focusing on efficiency to guarantee real-time execution, as well as preserving identity of the speaker over long generation time. Starting from the existing model "ALIVE", the proposer adopts a three phase post-training strategy: Phase 1 adapts ALIVE to be autoregressive using prompt-to-shot training. Phase 2 distill ALIVE from 50 steps of diffusion to 4 steps using sCM. Phase 3 further refines the model using DMD2 distillation. Such distillation methods are common in this literature, but usually starts from a bidirectional teacher (i.e. without the Phase 1 underlined above)

Critical strengths
  • the paper introduces a new benchmark
  • the paper ablates each component of the proposed training strategy
  • the ppaer compares to recent video generation models on the proposed benchmark
Critical weaknesses
  • I did not understand the use of prompt-to-shot. In particular at inference: how can the model be truly real-time/interactive if it generate shot by shot, does this means that the avatar can never be interrupted ? This seems to defeat the purpose of an avatar. There is a paragraph about streaming inference (3.4) but I found it hard to understand and it ends in a strange way (no clear conclusions or insights)

  • I found the paper poorly written overall. I found it really hard to understand the context and follow through the logic of the text.

  • There is no clear latency results. Table 1 only contains a binary "real-time" row, but that's hardly a latency measurement. It is not even clear on which device thsi is measured, if it is running on a single GPU etc

  • If I understood correctly the proposed method is only evaluated on a single benchmark of 23 videos, introduced in the current paper. This seems very little

Questions for authors
  • How is the duration DiD_i associated to prompt PiP_i at inference ? this seems like a non trivial problem
  • From the ablation it looks like DMD2 refinement has a small effect: How costly was this phase and is it really worth the compute cost considering the small impact ?
3 comments
AuthorsOpenReview ↗

Response to Reviewer UxoV (Part 1/2)

Thank you for recognizing our benchmark, component-wise ablations, and comparisons with recent avatar models. We have expanded the evaluation from 23 to 100 cases and completed 30 quantitative one-hour runs of our model. We clarify the streaming interface and generation-latency estimates below, followed by the new evaluation and training-cost evidence.

Responses to Questions for Authors

Q1: How is the duration D_i associated with prompt P_i at inference?

Duration is an explicit generator input, not an unknown that must first be predicted by diffusion. The benchmark supplies a target duration for every prompt. The live agent estimates English utterance duration at two words per second (Sec. E.1); this sets the temporal extent of the audio-video segment.

Q2: How costly is DMD2 refinement, and is its gain worth the training cost?

Refinement primarily improves motion/detail quality after identity-preserving consistency initialization. Table 2 shows Q-Face increasing from 0.722 to 0.774, while IDC improves from 0.777 to 0.789; Fig. 6 illustrates reduced blur in moving hands. Both variants retain four-NFE inference, so this gain adds no denoising steps at deployment.

Section B.1 reports 2,000 refinement iterations. This is a one-time training investment for the higher-quality operating point used in our main results, rather than a recurring inference cost.

Responses to Critical Weaknesses

W1: How can shot-by-shot generation support real-time interaction and interruption?

A shot is a short, prompt-controlled segment, not an entire session. Each subsequent shot accepts a new prompt while retaining the avatar's audio-video history, so viewer feedback can change the next utterance or action without restarting generation.

Our target is continuous live streaming: the avatar presents content and responds to comments at utterance boundaries. The component-RTF estimates in W3 give generation-and-decoding times below playback duration for 3-8 second shots, allowing subsequent content to be prepared while earlier content is played. This supports the intended buffered streaming setting, but is distinct from interrupting an already-generated utterance at an arbitrary frame. We do not demonstrate arbitrary-frame barge-in; feedback changes a subsequent shot rather than retroactively changing a shot already generated or buffered for playback.

W2: Can the context, overall method, and streaming inference be explained more clearly?

The method addresses three requirements of one task: prompt control, efficient generation, and persistent identity. Prompt-to-shot AR training enables variable-duration continuation; consistency initialization and PCM-regularized DMD2 produce a four-NFE generator; CMM limits error feedback from generated history.

The inference loop is correspondingly simple: receive a prompt and duration, select bounded history, generate and decode the shot, then update history for the next prompt. The operational result is prompt-controllable streaming with bounded historical memory and unchanged four-step sampling. We will foreground this loop in Sec. 3.4 and separate the implementation flow from the detailed CMM derivation.

AuthorsOpenReview ↗

Response to Reviewer UxoV (Part 2/2)

W3: What is the generation latency, on which device, and with how many GPUs?

Table 1's baseline "real-time" labels come from the respective papers and their test environments. We additionally benchmarked each method on our 8-H100 system, using its original configuration and, where feasible, an eight-GPU configuration. We report the faster result and its actual GPU count below. Official source code was unchanged; OmniForcing and TaoMate used external parallel-execution adapters.

ModelGPUSNFEPlayback FPSchunk durationDiT latency (s)DiT RTF
LongCat-Video-Avatar-1.588+1253.2 / 209.5152.973
OmniForcing24+1241 / 31.2511.251
TaoMate33+0241 / 30.7530.250
LiveAvatar54+0250.48 / 36.3600.682

Our model's generation-and-decoding latency. Our reported serving deployment uses 8 H100 GPUs for inference. At 864 x 480, 24 fps, ctx_oldest=18, recent=1, CMM, and four NFEs, the reported RTFs are 0.458 for the DiT (Table 5) and 0.080 for VAE decoding (Sec. D.3). For a submitted prompt specifying D seconds of output, the estimated generation-and-decoding latency is:

L_generation(D) = D x (RTF_DiT + RTF_VAE)
                = D x (0.458 + 0.080) = 0.538 D.
Output duration (s)Estimated generation + decoding latency (s)
31.614
42.152
52.690
63.228
73.766
84.304

A 5-second shot therefore requires approximately 2.69 seconds including VAE decoding. This is faster than playback and provides the generation-and-decoding budget needed for buffered live streaming: subsequent content can be prepared while earlier content is played, without pre-generating the full session.

W4: Is evaluation on a single, self-constructed 23-case benchmark sufficient?

We have expanded the benchmark to 100 multi-prompt cases, retaining the original 23 and adding 77 synthetic-reference cases. The added cohort contains 40 English and 37 Chinese scripts, 2,940 segments, and 241 minutes of target content; each case lasts 90-300 seconds. It covers basic speech, head poses, gestures/partial occlusion, object interaction, and repeated control changes. All five systems were evaluated on the new cases.

SystemIDC, all 100Q-Face, all 100Sync-D, all 100Speech Corr., all 100
Ours0.86750.86869.03490.9855
OmniForcing0.32410.482012.68980.3971
TaoMate0.13530.472513.17930.6221
LiveAvatar0.76490.652613.13810.9763
LongCat0.53570.69958.10540.9749

We additionally report five identities x three seeds x two configurations = 30 full-hour runs of our model, with video-CMM enabled or disabled. Ours has full-hour IDC 0.8751 [0.8690, 0.8845] and Q-Face 0.8868 [0.8328, 0.9409], with 95% CIs from 10,000 identity-cluster bootstrap draws after averaging seeds within identity. Evaluation samples the full hour at 1 Hz and computes scores for all 60 minute windows. With the same checkpoint, script, and seed, disabling video-CMM increases first-to-last-five-minute IDC decline from 0.0077 to 0.0464; the paired difference is 0.0387 [0.0113, 0.0818].

These additions address case coverage and quantified long-horizon repetition, rather than relying only on the original 23 videos or selected hour-long frames.

Hello thank for your response,

Q1; I understand the duration are not predicted but that is the problem I was higlighting: If the duration have to be given by the dataset it seems like a constraint not always easy to meet in real life

Q2; I understand training is a one-time cost but I think the easiest a method is to train, the broader its application can be. I assume you mean Table 3, while Q-Face score increases it is also the only one that does so

W1; I understand that shot-by-shot can still be real-time in the outputs. However I was specifically talking about interaction and interruption. If the model has to complete an entire shot before being able to do anything else, this means the user can only interrupt/change it at the end of the shot; if the shots are not small enough, this causes a visible delay during execution

W2; thank you for the explanation

W3; Thank you for the additional results

I thank the authors for the additional results but i think most of the weaknesses I listed still exist and the experimental comparison is not robust/thorough enough to assess the performance of the proposed method. I will raise my score to 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.