acceptodds
Under review as a conference paper at ICLR 2027

Avatar-Forever: Decoupled Parallel Training for High-Quality Real-Time Infinite Avatars

Abstract

Existing streaming video systems often rely on sequential, distillation-centered training pipelines to enable few-step long-video generation. However, this paradigm suffers from two limitations. First, failures or distribution shifts introduced in earlier stages affect later optimization, complicating the training process to converge. Second, the distillation-centric objective favours short-term generation but is prone to quality degradation when autoregressive errors accumulate over long rollouts. We propose Avatar-Forever, a decoupled parallel training framework for high-quality real-time infinite interactive avatars. Instead of coupling generation efficiency and long-horizon robustness under a sequential distillation pipeline, we treat them as two independent capabilities that can be trained in parallel. One branch performs full-parameter distillation to train an efficient generator with high visual quality, while another trains a lightweight long-horizon adapter via Recovery-oriented Rollout Training (RRT), which improves generation robustness under long-horizon inference conditions. Our decoupled parallel training design simplifies the overall training process and avoids unnecessary objective conflicts between few-step generation and long-horizon adaptation. We further introduce ForeverCache, a chunk-wise feature caching mechanism to substantially reduce redundant history computation during streaming inference. Built upon a 22B video foundation model, Avatar-Forever supports unbounded audio-driven avatar generation while maintaining identity consistency, motion coherence, and visual fidelity, enabling an end-to-end throughput of high-resolution 768x512 videos at 27.2 FPS on a single H100 GPU and providing a practical path toward stable digital humans.

Primary area:
generative models
Keywords:
Avatar, Streaming video, diffusion model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3dqpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a voice-driven digital human video generation method called Avatar-Forever, aiming to achieve high quality, real-time performance, and long-term stability. The paper optimizes short-step generation efficiency and long-term generation stability by separating the parallel training framework. A recovery-oriented rolling training (RRT) method is proposed, utilizing DMD distillation to obtain an efficient four-step generator and simulating autoregressive error accumulation to train a long-term robust adapter. Furthermore, ForeverCache reduces inference computation by reusing historical features. Experiments on multiple datasets validate the superiority of this method in terms of generation quality, long-term stability, and inference efficiency, achieving a generation speed of 27.2 FPS at 768×512 resolution on a single H100 GPU.

Critical strengths
  • The proposed method is well-designed and practically applicable. Addressing the challenge of balancing efficiency and long-term stability in real-time video generation, this paper proposes a training framework that decouples these two optimization objectives. Specifically, Regression-Regression Training (RRT) simulates error accumulation through multi-step autoregression, which better meets the practical needs of long-video generation compared to training to recover from local perturbations.

  • Experimental results are substantial, demonstrating competitive system performance. The paper presents quantitative and qualitative comparisons on multiple digital human generation datasets and validates the method's effectiveness through ablation experiments and user studies. The proposed method achieves good results in terms of long-term generation quality and inference efficiency, especially its real-time single-card generation capability, showcasing strong practical application value.

Critical weaknesses
  • In the independent learning process, potential inconsistencies may exist between the training and inference distributions. While the RRT model is initially trained using 30 sampling steps, the final model is a 4-step generator extracted from the DMD. The autoregressive error distributions of these two models may differ significantly. This paper does not adequately analyze or experimentally verify this difference, nor does it sufficiently demonstrate that the learned long-term robustness can reliably transfer to the final generator.

  • The advantages of the independent learning mechanism and the rationality of the parameter combination have not been fully validated. Although the authors directly update the independently learned DMD parameters into the LoRA RRT, no perturbation analysis or parameter compatibility analysis was performed. Despite relevant ablation experiments, the necessity and superiority of the independent design itself cannot be fully demonstrated due to the lack of direct comparison with standard sequence fine-tuning or co-optimization strategies with similar learning overhead. I believe such related experiments are necessary.

Questions for authors
  • Difference between Training and Inference Distributions: RRT is trained using a 30-step expansion, but the final model employs a 4-step generation method. Does this difference affect the model's robustness? Can you provide corresponding experimental verification results?

  • Necessity of Independent Training: Compared to sequential training where RRT is fine-tuned after DMD distillation, does independent parallel training offer an overall advantage? Can you directly compare the computational costs of the two methods?

  • Importance of Parameter Combinations: Why can DMD parameter updates obtained through independent training be directly added to the RRT's LoRA? Have parameter perturbations or different weight combinations been observed to affect generation quality?

  • Long-Term Generation Stability: Current quantitative experiments primarily focus on time periods under 30 seconds. Can you provide quantitative data on identity consistency, visual quality, and audiovisual synchronization performance over longer periods (e.g., 1 minute, 5 minutes, 10 minutes) to support claims of unlimited generation capabilities?

6 comments

Thank you for recognizing the method's design, experimental coverage, and practical single-GPU value. We address W1/Q1 together through the four-step-history experiment and W2/Q2 through the training-quality and cost comparisons. Q3 and Q4 then cover merge-weight sensitivity and minute-scale generation.

W1 + Q1. Training–inference mismatch and four-step-history adaptation

Experiment. Starting from our final DMD + RRT model, we freeze the DMD branch and continue training the RRT branch for 1,000 updates, using four denoising steps per rollout chunk with K=4K=4. We then evaluate the original and fine-tuned models on 30-second generation.

Table D1. Additional RRT adaptation using four-step histories. Best values are bold.

LLM JudgeAutomatic Metrics
MethodAudio-Visual↑Visual↑Motion↑Overall↑IQA↑ASE↑Sync-C↑Sync-D↓FID↓FVD↓
DMD + RRT (original)4.424.504.084.324.883.736.857.9433.33858.06
DMD + RRT (+1k RRT)4.364.394.024.264.813.416.038.4444.33986.06

Results. Using four-step histories for additional adaptation does not improve the final generator: all ten metrics favor the original model. Visual/Motion scores decrease from 4.50/4.08 to 4.39/4.02; synchronization also deteriorates, with Sync-C falling from 6.85 to 6.03 and Sync-D rising from 7.94 to 8.44. FID/FVD increase by 33.0%/14.9%, alongside lower IQA and ASE. These results show that matching rollout and inference step counts alone is insufficient to improve perceptual quality or robustness under the tested fine-tuning recipe.

Why use 30-step rollouts? The rollout schedule primarily controls the quality of the historical video supplied as conditioning during RRT; it does not prescribe the final model's inference cost. We use 30 steps to obtain sufficiently high-quality history conditions for recovery learning. Simply reducing the original, non-distilled RRT branch to four sampling steps can produce lower-quality histories than four-step generation with the composed DMD-distilled model. Thus, matching the number of sampling steps alone need not match conditioning quality. This distinction motivates our 30-step rollout design; the fine-tuning experiment above provides a separate empirical test of using four-step histories in the final model.

W2 + Q2. Independent versus sequential training: quality, compatibility, and cost

We compare both sequential orders with Decoupled: DMD→RRT (D2RRT), as requested, and RRT→DMD (RRT2D). Each configuration receives 2,000 DMD and 2,000 RRT updates at batch size 8, totaling 4,000 updates. Table D2 reports the results.

Table D2. Decoupled-training ablation (revised manuscript Table 3). Perceptual and automatic metrics are reported together; best values are bold.

LLM JudgeAutomatic Metrics
Training configurationAudio-Visual↑Visual↑Motion↑Overall↑IQA↑ASE↑Sync-C↑Sync-D↓FID↓FVD↓
D2RRT3.9013.8973.7233.8474.8283.6016.2838.68647.7501474.360
RRT2D4.1754.1883.6544.0234.8363.6416.5768.87038.180944.260
Decoupled DMD + RRT4.2504.2673.7504.1054.8513.6846.6948.03234.978901.811

Does sequential application of both objectives reach the same quality? The decoupled model performs best on all ten metrics. Compared with the closest sequential variant, RRT2D, FID/FVD decrease by 8.4%/4.5%, and all four perceptual scores improve. D2RRT achieves IQA=4.828, close to the decoupled model's 4.851, but its FVD is 1474.360 versus 901.811 for the decoupled model, a 38.8% relative reduction. In the corresponding visual comparisons, RRT2D has relatively clear appearance but still exhibits long-horizon color drift, whereas D2RRT shows flickering and jitter. The decoupled model better maintains appearance and temporal consistency. FVD measures video-distribution quality, while flickering is assessed qualitatively in the comparisons on our anonymous website.

Training cost and parallel development time. Based on the branch runtimes in Table D3, the RRT→DMD (RRT2D) sequential reference would take approximately 57 hours. Our independent branches can instead run concurrently and finish in approximately 37 hours when 16 A100 GPUs are available simultaneously. With the same total of 4,000 updates, this saves about 20 hours of waiting time (an estimated 35.5%) and allows distillation and recovery training to be iterated independently. This reduces wall-clock waiting time, not total GPU-hours; the estimate applies to RRT2D.

Table D3. Branch training runtimes and compute costs. Each branch uses 8 A100 GPUs instead of H100 due to limited computational resources during the rebuttal period; GPU-hours equal its measured runtime in hours multiplied by 8.

BranchSettingHardwareBatch sizeUpdatesMeasured runtimeA100 GPU-hours
RRTPer chunk: 30-step rollout, K=48×A10082,00036 h 44 min293.87
DistillationFour-step generator distillation8×A10082,00020.25 h162

Q3. How are parameter updates combined, and do merge weights affect quality?

We have completed an inference-time RRT merge-weight ablation for the final model, fixing the final DMD-distilled model and RRT update while varying only the external weight β\beta over 0/0.5/0.75/1/1.25. The external coefficient of the DMD update remains 1. β\beta is distinct from LoRA's internal scaling; β=0\beta=0 omits the RRT update in this final configuration. This experiment changes inference-time composition weights, not training checkpoints, and is separate from the small-scale component-training ablation in Table 3.

Table D4. RRT merge-weight ablation with fixed final checkpoints. Perceptual and automatic metrics are reported together; best values are bold.

LLM JudgeAutomatic Metrics
RRT weight β\betaAudio-Visual↑Visual↑Motion↑Overall↑IQA↑ASE↑Sync-C↑Sync-D↓FID↓FVD↓
04.154.123.533.954.753.426.378.2941.331180.50
0.54.334.273.784.144.823.576.488.2240.151070.66
0.754.364.423.904.244.843.676.588.0937.49940.88
14.424.504.084.324.883.736.857.9433.33858.06
1.254.364.453.964.274.853.716.778.1135.33983.23

Empirical effectiveness of parameter composition. With the DMD checkpoint fixed, adding the independently learned RRT update at β=1\beta=1 improves all ten metrics over omitting RRT: Motion increases by 15.6%, Sync-C by 7.5%, and FID/FVD decrease by 19.4%/27.3%. These joint gains provide practical evidence that the recovery increment complements the distilled generator in motion, synchronization, and visual fidelity. The unit coefficient also retains its training-time strength. Performance falls at β=1.25\beta=1.25, however, indicating that the benefit depends on the update magnitude; the results support the tested composition rather than arbitrary parameter combinations being free of interference. We report this ablation in the revised appendix.

Q4. Identity consistency, visual quality, and audio–visual synchronization at 1/5/10 minutes

Evaluation at 1/5/10 minutes. We use 16 fixed pairs of EMTD first frames and ten-minute LibriVox audio clips. We compare Decoupled with DMD-only to assess how the complete framework performs over long horizons relative to distillation without recovery training. We include SoulX-FlashTalk as an external baseline for its strong overall generation quality and fast inference, which makes multi-input ten-minute evaluation practical given the substantially longer runtimes of other baselines. Each model generates one continuous ten-minute trajectory per input at 768×512768\times512, 25 fps. The shorter durations are prefixes of those trajectories. Scores average one-minute windows within each prefix and then input pairs equally. Dino-S is a DINOv2-based proxy for appearance consistency.

Table D5. Visual quality, appearance consistency, and synchronization over minute-scale prefixes. Best values within each duration are bold.

ModelPrefix (min)IQA↑ASE↑Sync-C↑Sync-D↓Dino-S↑
SoulX-FlashTalk14.59963.67386.88248.17640.91544
DMD-only14.78993.92515.59589.21310.86746
Decoupled14.85993.93136.94787.88010.92759
SoulX-FlashTalk54.61313.69046.79508.25760.91608
DMD-only54.78603.83235.48799.28560.81365
Decoupled54.85203.90576.84127.95510.92975
SoulX-FlashTalk104.61503.69136.75858.27450.91604
DMD-only104.77513.81295.30779.52000.80389
Decoupled104.85753.91066.78347.98800.92047

Quality, consistency, and synchronization. Decoupled achieves the best means on all five metrics at 1, 5, and 10 minutes. At ten minutes, its IQA/ASE and Dino-S remain 4.8575/3.9106 and 0.92047, compared with 4.7751/3.8129 and 0.80389 for DMD-only; Sync-C/D also favor Decoupled (6.7834/7.9880 versus 5.3077/9.5200). It likewise outperforms SoulX-FlashTalk on all five means, with stronger gains in image quality and aesthetics and smaller margins in appearance consistency and Sync-C. The combined evidence shows that our advantage extends beyond 30-second generation to the measured ten-minute horizon across multiple inputs. The DMD-only comparison uses different efficiency-branch initializations and therefore evaluates complete configurations rather than isolating RRT alone. We have added these results to the appendix.

Reviewer_KLj6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper trains a four-step DMD generator and a long-time robust LoRA separately, then adds their parameter updates together. RRT performs gradient-free autoregressive rollout starting from the degradation history, with flow matching supervision applied only at the end; ForeverCache reuses historical features across denoising steps. Based on the 22B LTX-2.3, the method reports audio-driven digital human generation at 27.2 FPS on a single H100.

Critical strengths
  • Compared with directly reconstructing the perturbed context, recursively generating and then supervising recovery by the model better approximates the scenario where historical errors propagate across multiple segments, while the objective remains standard flow matching.
Critical weaknesses
  • The historical input is fixed, which does not guarantee that its layer-wise representation is immune to the modulation of noisy tokens or timesteps.
  • Section 3.3 and Appendix B do not specify the corresponding masks and dependencies. Table 1 provides a comparison of cache switches; for instance, the 30-second FVD of HDTF rises from 593.92 to 686.96, yet the uncertainty of this discrepancy is not reported.
  • Table 3 supports that merging DMD and RRT updates is valid under the current configuration. However, the robustness branch uses a 30-step rollout, while deployment adopts a four-step model with optional caching, so the error distributions may not be consistent.
Questions for authors
  • Quantify the impact of K, LoRA merge coefficients and DMD checkpoints on quality. Using a model deployed in the actual four-step process to generate RRT history, can the results be further improved?
  • Does the historical token completely mask the current noisy token, and is its time-step modulation fixed?
11 comments

Thank you for recognizing the recovery-training mechanism and highlighting the cache dependencies and training–deployment differences. We address W1, W2, and Q2 together below, combining the implementation clarification, manuscript quality–efficiency results, and new long-form visual examples. W3 and Q1 address training–deployment differences and the additional ablations.

W1 + W2 + Q2. ForeverCache dependencies and practical trade-offs

Dependencies and feature reuse. ForeverCache computes historical features at the first denoising step of each chunk, reuses them for the remaining steps, and refreshes the cache for the next chunk, providing acceleration without retraining. Although historical latents remain unchanged and their within-modality timestep is fixed at 0, attention does not fully mask current noisy tokens for historical features: the history and current chunk still interact through self-attention. Historical features can therefore vary across denoising steps, making their reuse an approximation. Nevertheless, visual inspection of our cache-off/cache-on examples reveals no obvious degradation in visual quality. We explored masking these interactions to reduce this dependency, but the attention libraries supporting custom masks that we tested were less efficient than the optimized FlashAttention implementation and slowed inference in our configuration. We therefore retain FlashAttention with approximate feature reuse as a practical quality–efficiency trade-off.

Interpreting the HDTF FVD change. The cache-off/cache-on comparison in Table 1 makes this trade-off explicit: HDTF FVD increases from 593.92 to 686.96 (+15.7%). A possible contributor is variation in the generated motion trajectory. HDTF predominantly contains face-centered videos, while our avatars exhibit varied head and upper-body dynamics. Caching can alter these trajectories for the same input; because FVD compares generated and ground-truth video-feature distributions, such differences can affect the score even when motion remains visually plausible. We present this as a possible explanation, not an established cause, and do not claim statistical insignificance without a set-level confidence interval.

Practical quality–efficiency balance. Despite the increase relative to cache-off, the cached model still exceeds all competing baselines in 30-second LLM Overall on EMTD, HDTF, and TalkVid, and its HDTF FVD remains better than the strongest competing baseline (686.96 versus 793.74). ForeverCache reduces runtime from 38.85 to 26.71 seconds (31.2%), reaching approximately 4.7 times the throughput of the fastest competing baseline. We have also added cache-enabled ten-minute videos to our anonymous demonstration website, providing qualitative evidence of sustained appearance and temporal coherence beyond the quantitative evaluation horizon.

W3 + Q1 (four-step histories). Training–inference mismatch and additional adaptation

Condition quality and distribution mismatch. In our experiment, the 30-step schedule serves to construct coherent video conditions while allowing autoregressive errors to propagate, so RRT learns recovery from imperfect but usable histories. It does not set the final generator's inference cost. Because DMD explicitly optimizes few-step generation, four steps in the original, non-distilled RRT branch can yield lower-quality histories than four steps in the composed model. Equal step counts therefore do not guarantee comparable conditioning quality.

Check using deployment-generated histories. To test whether adapting to four-step histories helps, we start from the final DMD + RRT model, keep DMD fixed, and fine-tune only RRT for 1,000 updates using four-step rollouts with K=4K=4. Table K1 compares the original and adapted models on 30-second generation.

Table K1. Additional RRT adaptation using four-step histories. Best values are bold.

LLM JudgeAutomatic Metrics
MethodAudio-Visual↑Visual↑Motion↑Overall↑IQA↑ASE↑Sync-C↑Sync-D↓FID↓FVD↓
DMD + RRT (original)4.424.504.084.324.883.736.857.9433.33858.06
DMD + RRT (+1k RRT)4.364.394.024.264.813.416.038.4444.33986.06

Results and interpretation. Additional adaptation lowers all four LLM Judge scores, including Visual from 4.50 to 4.39 and Overall from 4.32 to 4.26, and worsens all six automatic metrics. The IQA change is modest (4.88 to 4.81), but ASE falls by 8.6% and Sync-C by 12.0%, with Sync-D increasing from 7.94 to 8.44. Thus, the quality loss extends beyond individual-frame fidelity to aesthetics and audio–visual alignment. FID/FVD also increase by 33.0%/14.9%, reinforcing the broader decline in image- and video-distribution quality. These consistent changes indicate that matching the history sampler to the deployment step count alone does not improve the final model's quality balance under this recipe. Continuing RRT training with four-step histories therefore does not necessarily yield further performance gains, even with an additional training budget.

Q1. Rollout horizon and RRT merge coefficient

We quantify the remaining training and composition variables below.

1. Rollout horizon K

We have added the K=0/1/2/4K=0/1/2/4 ablation to manuscript Table 4 and reproduce it in Table K2. KK counts autoregressive rollout chunks, not denoising steps per chunk; K=0K=0 retains history perturbation but applies recovery supervision immediately.

Table K2. Rollout horizon K (revised manuscript Table 4). Perceptual and automatic metrics are reported together; best values are bold.

LLM JudgeAutomatic Metrics
KAudio-Visual↑Visual↑Motion↑Overall↑IQA↑ASE↑Sync-C↑Sync-D↓FID↓FVD↓
04.0504.0803.5173.9004.8373.5696.5138.23437.590923.189
14.1644.1733.5093.9704.8373.5886.4448.08636.155922.670
24.1174.2173.6834.0214.8413.5916.4798.08235.919917.412
44.2504.2673.7504.1054.8513.6846.6948.03234.978901.811

Results. K=4K=4 achieves the best scores on all ten metrics. Relative to K=0K=0, Visual/Motion improve by 4.6%/6.6% and Sync-C by 2.8%, alongside lower Sync-D and FID/FVD reductions of 6.9%/2.3%. The visual comparisons also show fewer artifacts and stronger appearance, background, and color consistency. These joint gains support supervising recovery after model errors propagate through multiple chunks. The results favor K=4K=4 among the tested horizons, without implying that every metric improves monotonically with larger KK.

2. External RRT merge coefficient

To quantify the coefficient's effect independently of training progress, we hold the final DMD checkpoint and RRT update fixed and vary only the external weight β\beta over 0/0.5/0.75/1/1.25. The DMD update coefficient remains fixed, while β=0\beta=0 removes RRT. This external weight is distinct from LoRA's internal scaling.

Table K3. External RRT merge weight with fixed final checkpoints. Perceptual and automatic metrics are reported together; best values are bold.

LLM JudgeAutomatic Metrics
RRT weight β\betaAudio-Visual↑Visual↑Motion↑Overall↑IQA↑ASE↑Sync-C↑Sync-D↓FID↓FVD↓
04.154.123.533.954.753.426.378.2941.331180.50
0.54.334.273.784.144.823.576.488.2240.151070.66
0.754.364.423.904.244.843.676.588.0937.49940.88
14.424.504.084.324.883.736.857.9433.33858.06
1.254.364.453.964.274.853.716.778.1135.33983.23

Quantified coefficient sensitivity. Table K3 shows that the recovery benefit is not confined to one coefficient: both β=0.5\beta=0.5 and 0.75 improve all ten metrics over omitting RRT, with further gains at β=1\beta=1. At unit weight, Sync-C increases from 6.37 to 6.85 and FVD decreases from 1180.50 to 858.06, alongside improvements in all four perceptual scores and IQA/ASE. This optimum also retains the adapter strength used during training. Raising β\beta to 1.25 reverses the trend across all ten metrics: Overall falls from 4.32 to 4.27, Sync-D rises from 7.94 to 8.11, and FVD increases by 14.6%. The sweep therefore quantifies both the value of recovery and its sensitivity to rescaling, supporting the unit coefficient for the tested checkpoints without assuming that larger updates are always preferable.

Manuscript revisions. We have added the appendix section “Effect of the RRT Merge Weight” with all ten metrics for the five weights, the composition formula, and the analysis, and reproduce the complete results in Table K3.

Reviewer_gBwdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Avatar-Forever, a decoupled training framework for real-time, long-duration audio-driven avatar generation. Starting from the same pretrained video foundation model, it independently trains a four-step generator through DMD and a long-horizon LoRA adapter through Recovery-oriented Rollout Training (RRT), then merges their parameter updates at deployment. RRT perturbs the initial history, propagates errors through autoregressive rollouts without gradient tracking, and applies flow-matching supervision to a subsequent target chunk. Additionally, ForeverCache reuses historical features across denoising steps within each chunk to reduce redundant computation.

The framework is implemented using the 22B LTX-2.3 model and synthetic training data. Evaluation on 5-second and 30-second videos from EMTD, HDTF, and TalkVid combines automatic metrics, LLM-based scoring, human evaluation, and component ablations to assess generation quality, temporal consistency, and efficiency. The experiments report quality improvements over the selected baselines and an end-to-end generation throughput of 27.2 FPS at 768×512 resolution on a single H100 GPU.

Critical strengths
  1. The decoupled adaptation framework and recovery-oriented training constitute clear technical contributions. The paper separates few-step distillation and long-horizon robustness adaptation into independently trained branches whose parameter updates are combined at deployment. RRT perturbs early history, performs autoregressive rollouts without gradient tracking, and applies flow-matching supervision to a subsequent target chunk, explicitly incorporating histories affected by propagated model errors into training. This design separates robustness adaptation from the distillation objective and provides a concrete mechanism for supervised recovery training under accumulated errors.

  2. The experiments provide complementary evidence of generation quality and inference efficiency. Across the short- and long-video evaluations on three datasets, the full model achieves higher overall perceptual scores than the evaluated baselines. Human evaluation further supports its advantages in visual quality and motion naturalness. Comparisons with and without ForeverCache demonstrate a clear quality–efficiency trade-off: caching substantially reduces generation time while retaining most perceptual quality gains. Together with real-time end-to-end throughput on a single GPU, these results support the framework’s effectiveness and practical value under the evaluated settings.

Critical weaknesses
  1. The contribution and optimization benefits of decoupled parallel training are insufficiently established. The framework independently trains DMD and RRT-LoRA updates and merges them at deployment. Related ideas involving training on model-induced degraded histories and reusing distillation updates have been explored previously (e.g., MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model), so the distinct contribution requires clearer delineation. Moreover, Table 3 changes both the optimization objective and the training arrangement. Without a sequential control using the same DMD and RRT objectives under comparable training budgets, the results do not isolate the benefits of decoupling or substantiate the explanations concerning reduced objective interference and improved training stability.

  2. Evidence for long-horizon stability remains limited. Systematic quantitative evaluation covers generation up to 30 seconds, while the single example exceeding 11 minutes does not establish robustness across identities and audio inputs. Prior streaming avatar work includes quantitative evaluation on multiple sequences exceeding five minutes (e.g., Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length), whereas comparable evidence is missing here. A small-scale evaluation on multiple minute-scale sequences, with temporal breakdowns and comparisons against the model without RRT and representative baselines, would help distinguish higher initial generation quality from slower error accumulation and more directly validate RRT’s contribution to long-horizon stability.

Questions for authors
  1. Prior work has considered composing distillation and adaptation updates, so the framework’s distinct contribution requires clarification. What motivates full-parameter DMD distillation paired with LoRA-based RRT, rather than reversing these choices or using the same parameterization for both branches? Could a focused comparison or analysis support this choice and assess how merging the updates affects few-step generation quality and long-horizon robustness, including potential interference?

  2. Could the authors provide a sequential control that first performs DMD and then trains an RRT-LoRA adapter on the distilled generator, using the same training data and objectives under a comparable budget? This would help isolate the benefits of decoupling from those of introducing flow-matching supervision and LoRA adaptation.

  3. Could the authors evaluate a small set of sequences lasting several minutes, covering multiple identities and audio inputs, and compare the full model, the efficiency branch without RRT, and one representative baseline? Reporting visual quality, identity consistency, and audio–visual synchronization across temporal segments would help distinguish improvements in initial quality from reductions in long-term degradation.tinguish improvements in initial quality from reductions in long-term degradation and directly assess RRT’s contribution.

6 comments

Thank you for recognizing our decoupled training framework and its quality–efficiency benefits. We address the overlapping comments together below, reporting the new sequential-training comparisons, merge-weight ablation, and multi-input ten-minute evaluation alongside the corresponding manuscript revisions.

W1 + Q2. Distinction from prior work and sequential-training controls

Distinction from prior work. MaineCoon is a concurrent work whose source code and model weights are not publicly available, so a direct quantitative comparison under our evaluation protocol is currently infeasible. Its self-resampling applies single-step denoising to perturbed real chunks. RRT instead recursively generates subsequent chunks from noise after perturbing the initial history, then supervises recovery after model errors propagate. We additionally train DMD and RRT independently and compose their updates at deployment, enabling parallel training and separate branch iteration.

Ablations on training budget and sequential training. To distinguish decoupling gains from additional training, we extend DMD-only and FM-only from 2,000 to 4,000 updates at batch size 8: the original ablation used 2,000 steps per branch, but Decoupled's two branches required more GPU-hours. We also address your requested sequential baseline with D2RRT, which trains an RRT adapter on the distilled generator. Both D2RRT and Decoupled receive 2,000 DMD and 2,000 RRT updates.

Table G1. Ablation results for extended training and sequential training (revised manuscript Table 3). “+2k” denotes 2,000 additional updates, for 4,000 in total. All ten metrics are reported together; best values are bold.

LLM JudgeAutomatic Metrics
Training configurationAudio-Visual↑Visual↑Motion↑Overall↑IQA↑ASE↑Sync-C↑Sync-D↓FID↓FVD↓
DMD-only (2k)4.0834.1673.5833.9624.7933.6716.4278.08539.5151080.230
DMD-only (+2k)4.1324.1653.4913.9514.8013.6736.4248.04438.9221077.053
FM-only (2k)2.7501.9172.2502.3083.4092.5902.03110.76076.5771071.200
FM-only (+2k)2.7681.9872.3252.3613.4352.6722.11310.76776.5811069.613
D2RRT3.9013.8973.7233.8474.8283.6016.2838.68647.7501474.360
Decoupled4.2504.2673.7504.1054.8513.6846.6948.03234.978901.811

According to the table, Decoupled achieves the best results on all ten metrics. Compared with extended DMD-only, it improves Motion/Sync-C by 7.4%/4.2% and reduces FID/FVD by 10.1%/16.3%. Continued DMD-only training lowers Motion from 3.583 to 3.491 with essentially unchanged Overall; FM-only remains weak in appearance and synchronization, supporting the complementary benefits of combining both updates. Against D2RRT, it improves all perceptual and synchronization scores and reduces FVD by 38.8%, despite similar frame-level IQA. The videos show D2RRT's flickering and jitter, whereas Decoupled better preserves appearance and temporal consistency. These results support independently learning and composing the complementary capabilities over either longer single-objective training or sequential fine-tuning in the tested setting.

Manuscript revisions. We have clarified the distinction from MaineCoon in Related Work and updated Table 3 and Figure 4 with the additional controls and corresponding quantitative and visual analyses. We have also added the video comparisons corresponding to Figure 4 to our anonymous website.

Q1. Parameterization and merge-weight sensitivity

Why full-parameter DMD and LoRA-RRT? In early engineering experiments, full-parameter DMD produced better few-step generation than LoRA distillation, motivating its use for the efficiency branch. LoRA represents recovery as a lightweight, independently updatable increment, making it convenient to combine with the distilled model and iterate each branch separately.

RRT LoRA merge-weight ablation. We fix the final four-step DMD model and trained RRT LoRA adapter, and vary only the adapter's inference-time merge weight β\beta over 0/0.5/0.75/1/1.25. The DMD contribution remains unchanged, and no variant is retrained. Here, β=0\beta=0 omits the RRT update, while β=1\beta=1 retains its original learned strength. Table G2 compares four LLM Judge scores and six automatic metrics under the same evaluation protocol.

Table G2. RRT merge-weight ablation with fixed final checkpoints. All ten metrics are reported together; best values are bold.

LLM JudgeAutomatic Metrics
RRT weight β\betaAudio-Visual↑Visual↑Motion↑Overall↑IQA↑ASE↑Sync-C↑Sync-D↓FID↓FVD↓
04.154.123.533.954.753.426.378.2941.331180.50
0.54.334.273.784.144.823.576.488.2240.151070.66
0.754.364.423.904.244.843.676.588.0937.49940.88
14.424.504.084.324.883.736.857.9433.33858.06
1.254.364.453.964.274.853.716.778.1135.33983.23

According to the table, β=1\beta=1 performs best on all ten metrics: relative to omitting RRT, Motion/Sync-C improve by 15.6%/7.5% and FID/FVD decrease by 19.4%/27.3%, alongside higher Visual, IQA, and ASE scores. The unit weight retains the adapter strength used during training, supporting its default use. Increasing β\beta to 1.25 worsens all ten metrics, including a 14.6% increase in FVD, showing that stronger recovery updates do not necessarily improve quality. We have added these results and the composition formula to the appendix section “Effect of the RRT Merge Weight.”

W2 + Q3. Multi-input minute-scale generation

Evaluation. We constructed 16 fixed input pairs, each combining a first frame from EMTD with a ten-minute audio clip from LibriVox. We compare Decoupled and DMD-only on these inputs. Considering both inference time and overall generation quality, we additionally select SoulX-FlashTalk as a representative external baseline, since generating ten-minute videos with the remaining baselines would require substantially longer runtimes. Each model generates continuous ten-minute videos at 768×512768\times512, 25 fps, and we evaluate their one-, five-, and ten-minute prefixes. We first average scores over one-minute windows within each prefix, then average across the 16 input pairs with equal weight. Alongside IQA/ASE and Sync-C/D, we report Dino-S, which measures DINOv2 feature similarity to the reference first frame as a proxy for appearance consistency.

Table G3. Mean scores for 1/5/10-minute prefixes. Best values within each duration are bold.

ModelPrefix (min)IQA↑ASE↑Sync-C↑Sync-D↓Dino-S↑
SoulX-FlashTalk14.59963.67386.88248.17640.91544
DMD-only14.78993.92515.59589.21310.86746
Decoupled14.85993.93136.94787.88010.92759
SoulX-FlashTalk54.61313.69046.79508.25760.91608
DMD-only54.78603.83235.48799.28560.81365
Decoupled54.85203.90576.84127.95510.92975
SoulX-FlashTalk104.61503.69136.75858.27450.91604
DMD-only104.77513.81295.30779.52000.80389
Decoupled104.85753.91066.78347.98800.92047

Multi-input long-horizon evidence. Decoupled leads on all five mean scores at each duration, retaining IQA around 4.85 and ASE around 3.91 across the prefixes. At ten minutes, its Dino-S is 0.92047 versus 0.80389 for DMD-only, alongside better synchronization (Sync-C/D: 6.7834/7.9880 versus 5.3077/9.5200). Against SoulX-FlashTalk, it improves ten-minute IQA/ASE by 5.3%/5.9% and reduces Sync-D by 3.5%, with modest advantages in Dino-S and Sync-C. These comparisons extend the evidence beyond a single long example to multiple inputs and two reference configurations, supporting sustained visual quality, appearance consistency, and audio–visual alignment.

Manuscript revision. We have added this table, the evaluation protocol, and core analysis to the appendix section “Minute-Scale Long-Horizon Evaluation,” complementing our visual example exceeding 11 minutes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.