- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ALIVE-AVATAR, a real-time streaming audio-video generation framework designed for interactive avatar generation with persistent identity. Starting from a pretrained ALIVE model, the authors first introduce prompt-to-shot autoregressive training, followed by sCM initialization and DMD2 refinement with PCM regularization to achieve four-step generation. At inference, Context Multiplicity Modulation (CMM) reduces attention to recent generated context to mitigate identity drift. Experiments on a 23-case benchmark demonstrate improvements in identity consistency, facial quality, and speech correctness over several baselines, alongside qualitative hour-long generation results.
Critical strengths
-
Practical and relevant problem setting. The paper addresses the challenging combination of real-time joint audio-video generation, interactive prompt control, and long-term identity preservation. The integrated streaming system and live-agent demonstration highlight its practical relevance.
-
Promising empirical performance. The method achieves four-step, faster-than-playback generation while reporting competitive quality against substantially larger baselines. In particular, CMM provides a notable improvement in identity consistency, suggesting its practical effectiveness.
Critical weaknesses
-
Unclear motivation and mismatch between shot-aware rollout and the actual inference setting. The proposed shot-aware rollout appears inconsistent with the paper's core motivation of prompt-to-shot generation. During inference, each prompt is intended to generate a complete shot, and subsequent shots may be conditioned on different prompts. However, the proposed training strategy randomly splits a single clip into a prefix and suffix, and both rollouts use the same prompt condition. The split does not necessarily correspond to a meaningful shot boundary or a prompt transition. As a result, the proposed procedure primarily simulates continuation from self-generated history, rather than the actual shot-to-shot, prompt-switching scenario that the paper aims to address. It is therefore unclear what makes this procedure specifically shot-aware, or why it should be more suitable than standard Self-Forcing. Although Table 2 reports improvements over a Self-Forcing baseline, the paper does not sufficiently explain or isolate the benefits of this particular rollout construction.
-
Insufficient experimental evaluation. All main quantitative quality comparisons and ablation studies are conducted on a newly constructed benchmark containing only 23 cases. The paper does not report results on established public benchmarks or sufficiently justify why existing evaluation protocols are unsuitable. Given the relatively small evaluation set and the use of only one random seed per case, the reported results provide limited evidence of the method's generalizability and robustness. Moreover, persistent identity over hour-long generation is a central claim of the paper, yet the quantitative evaluations cover sequences of only 1.5–5 minutes. The one-hour results are primarily qualitative, without systematic measurements of identity drift over time. More comprehensive evaluations on established benchmarks, together with quantitative long-horizon analysis, are needed to substantiate the paper's broad performance claims.
-
Limited methodological novelty and insufficient justification of key design choices. The overall framework appears relatively incremental, combining several established techniques rather than introducing a substantially new modeling or training paradigm. The distillation pipeline largely builds on existing sCM, DMD2, and PCM methods, while the distinct contribution of the proposed shot-aware rollout remains insufficiently justified, as discussed above. Similarly, Context Multiplicity Modulation (CMM) is essentially a simple attention-logit reweighting strategy that suppresses recent generated context. While its reported improvement in identity consistency is substantial, the mechanism itself is relatively straightforward, and the paper does not adequately justify its specific design choices. In particular, the choice of the fixed multiplicity factor of 0.6 lacks a sensitivity analysis, and no comparisons are provided against natural alternatives such as directly strengthening anchor attention or adjusting the anchor/recent context allocation. Consequently, it remains unclear whether the proposed CMM design offers distinct advantages over simpler attention or memory reweighting strategies.
Questions for authors
-
What specifically makes the proposed training shot-aware? Since both training stages use random clip splits without explicit prompt transitions, how does this design address shot-to-shot prompt switching beyond standard Self-Forcing? Can the improvements be isolated from differences in rollout construction and supervision?
-
Can the authors provide quantitative identity-consistency evaluations over substantially longer sequences, ideally up to one hour, together with time-resolved drift measurements and evaluation across more cases or benchmarks?
-
How sensitive is CMM to the multiplicity factor of 0.6? How does suppressing recent-context attention compare with alternatives such as strengthening anchor attention or changing the anchor/recent context allocation?
3 comments
Response to Reviewer 71n1 (Part 2/3)
Q2: Can you quantify identity consistency up to one hour, including time-resolved drift and evaluation across more cases or benchmarks?
Q2.1: Full-hour identity, quality, and temporal drift
We report 30 full-hour runs of our model: five synthetic identities x three seeds x two configurations, namely ours and video-CMM off. The seeds are 104729, 130363, and 155921. Each case has a 3,600-second, 601-segment script. The five identities were already used in the added cohort; these are new long-horizon scripts, not unseen-identity tests. Historical demonstrations and development clips are excluded from the formal sample.
We evaluate visual metrics at 1 Hz over the entire hour and compute scores for all 60 one-minute windows. The table below summarizes the full-hour scores and first/last five-minute endpoints. Full-video IDC is the median reference-to-frame face similarity, while Q-Face averages face-crop quality. We then average the three seeds within each identity and the five identity means. Intervals use 10,000 identity-cluster bootstrap draws; seeds and frames are not treated as independent identities.
| Configuration | Full-hour IDC [95% CI] | Full-hour Q-Face [95% CI] | First 5 min IDC | Last 5 min IDC |
|---|---|---|---|---|
| Ours | 0.8751 [0.8690, 0.8845] | 0.8868 [0.8328, 0.9409] | 0.8799 | 0.8722 |
| Video-CMM off | 0.8067 [0.7534, 0.8504] | 0.8817 [0.8241, 0.9393] | 0.8393 | 0.7930 |
The new evaluation measures temporal drift rather than just final-frame quality. Defining D_ID = IDC_first5min - IDC_last5min, ours declines by 0.0077 versus 0.0464 with video-CMM off. The paired reduction in decline is 0.0387 [0.0113, 0.0818]. The final-window IDC advantage is 0.0792 [0.0268, 0.1504]. This comparison uses the same checkpoint, reference, script, and seed, with video-CMM enabled or disabled; it evaluates the deployed video-CMM configuration as a whole.
Ours has first/last-window Q-Face 0.8864/0.8873. Its Q-Face decline is -0.0010 [-0.0020, 0.00004], consistent with little average change; the final-window on/off Q-Face difference has a CI crossing zero, so we do not claim a clear improvement on that endpoint. Both visual evaluators have valid faces at all 3,600 planned timestamps for every run.
Our strongest controlled conclusion is reduced long-horizon identity deterioration with video-CMM across these five identities and three seeds, not stability for arbitrary identities and durations.
Q2.2: More evaluation cases, benchmarks, and generalization
The new evaluation expands the input set from 23 to 100 cases, adding 77 cases to the original 23. The added cohort contains 40 English and 37 Chinese cases, 2,940 prompt-controlled segments, and 241 minutes of target content. Each new case lasts 90-300 seconds and uses an API-generated fictional-adult reference. It covers basic speech (11), head poses (20), gestures/partial occlusion (19), object interaction (10), and repeated control changes (17).
All five systems were evaluated on the added cohort. The following results use only the 77 new cases, avoiding an apparent improvement caused solely by recombining the original scores:
| System | IDC | Q-Face | Sync-D | Speech Corr. |
|---|---|---|---|---|
| Ours | 0.8910 | 0.8965 | 8.8784 | 0.9906 |
| OmniForcing | 0.3401 | 0.4968 | 12.4992 | 0.4437 |
| TaoMate | 0.1041 | 0.4455 | 14.1184 | 0.6657 |
| LiveAvatar | 0.7837 | 0.6738 | 13.0810 | 0.9943 |
| LongCat | 0.5575 | 0.7235 | 8.0246 | 0.9945 |
Every entry has 77 valid videos; each video has equal weight. IDC is the mean of per-video medians, matching Appendix B.5. On the combined 100 cases, ours has IDC 0.8675 and Q-Face 0.8686, also the highest system means. Prompt format and input distribution differ between the original and added cohorts, so higher scores on the added cohort are not a before/after model improvement.
Our protocol is designed to test jointly generated speech and video under successive prompt/action/duration controls over minutes. An externally supplied-audio evaluation alone does not measure this entire interface, although it remains useful for complementary comparisons. We have not completed an established external-benchmark evaluation and do not claim that such benchmarks are unsuitable. The expansion improves task coverage, while Q2.1 separately adds multi-seed long-horizon evidence; it does not constitute a multi-seed test of all 100 cases or establish open-domain generalization. Baseline interfaces, resolutions, and seeds remain method-specific, so the table is a system-level comparison rather than a fully matched causal test.
Response to Reviewer 71n1 (Part 3/3)
Q3: How sensitive is CMM to 0.6, and how does it compare with stronger anchor attention or different anchor/recent memory allocation?
Q3.1: Multiplicity selection and sensitivity
We selected 0.6 on validation cases because identity drift was already barely perceptible at this setting, and further suppression produced no additional visible benefit. Smaller multiplicities downweight recent context more strongly; the goal is sufficient drift control, not maximal suppression.
The following additional sweep provides quantitative support for this choice:
| CMM multiplicity | IDC | Sync-D | Q-Face | Speech Corr. |
|---|---|---|---|---|
| 0.9 | 0.758 | 9.279 | 0.790 | 0.977 |
| 0.8 | 0.779 | 9.263 | 0.782 | 0.978 |
| 0.7 | 0.785 | 9.268 | 0.777 | 0.976 |
| 0.6 (default) | 0.789 | 9.308 | 0.774 | 0.979 |
| 0.5 | 0.782 | 9.329 | 0.773 | 0.977 |
| 0.4 | 0.786 | 9.356 | 0.770 | 0.977 |
| 0.3 | 0.787 | 9.292 | 0.768 | 0.977 |
| 0.2 | 0.784 | 9.394 | 0.766 | 0.981 |
| 0.1 | 0.782 | 9.342 | 0.763 | 0.970 |
The default row reproduces Table 2 of the submission.
The reported IDC values stay within 0.779-0.789 across the tested settings from 0.1 to 0.8, indicating a relatively broad operating region rather than a sharply tuned optimum. Lowering the multiplicity below 0.6 provides no consistent further identity gain.
Q3.2: Anchor-attention and memory-allocation alternatives
Our goal is to reduce error feedback from recent generated context, so we directly downweight that context rather than compensate for it indirectly. CMM reduces recent-context attention while retaining the tokens needed for local continuity, without changing their values, positions, or memory allocation.
Strengthening anchors instead adjusts the competing reference signal, while reallocating memory changes which historical information is available. Both are possible alternatives, but neither is needed to implement the targeted intervention. We therefore choose recent-context suppression as the most direct control of the identified error-feedback pathway. The CMM ablation and the sensitivity results in Q3.1 support the effectiveness of this choice.
Q4: What is the methodological contribution beyond combining existing techniques?
Our contribution is the task-specific formulation of prompt-controlled, persistent-identity joint A/V streaming: prompt-end-aligned, variable-duration continuation, consistency-initialized refinement for four-step generation, and control of generated-history influence without expanding memory. The claim is not that sCM, DMD2, or PCM individually are new.
The design addresses distinct failure modes at the same inference budget. Removing sCM initialization reduces IDC from 0.789 to 0.710; removing refinement reduces Q-Face from 0.774 to 0.722; removing PCM reduces IDC to 0.749. Table 4 additionally tests PCM weight and phase coverage. CMM raises IDC from 0.611 to 0.789 under fixed-model inference. These ablations support the specific formulation, while CMM's simple implementation makes it a practical inference intervention rather than an additional training burden.
Response to Reviewer 71n1 (Part 1/3)
Thank you for highlighting the practical streaming setting, the four-step generation results, and the identity gain from CMM. We have added 77 benchmark cases and 30 complete one-hour runs of our model, including three-seed, same-checkpoint video-CMM comparisons. We present the additional results and clarify the rollout construction and the scope of the evidence below.
Responses to Questions and Critical Weaknesses
Q1: What makes the training shot-aware without explicit prompt transitions, how does it address prompt switching, and can rollout/supervision gains be isolated?
The key distinction is that the prediction endpoint is always aligned with the end of the shot associated with the current prompt. For a prompt-conditioned shot ending at
T, we randomly sample the continuation startsand regenerate the entire remaining suffix[s,T), rather than a fixed-length next block. Thus, the start and prediction length vary, but the target always extends to the same prompt-associated endpoint (Sec. 3.1; Fig. 3).In the fixed-block Self-Forcing-style baseline, block boundaries are determined by block length and need not coincide with prompt completion. Our construction instead trains completion of the current prompt's remaining shot, matching the inference interface in which one prompt controls one complete, variable-duration segment. The random split need not be a scene cut or a prompt transition: shot-awareness comes from the aligned endpoint, not from the split itself. Sharing a prompt across prefix and suffix is therefore consistent with this objective; a subsequent prompt controls the next shot. Both methods use self-generated history; our distinction is the prompt-end-aligned prediction unit.