acceptodds
Under review as a conference paper at ICLR 2027

TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation

Abstract

Real-time long-form digital-human generation relies on causal models to extend audio-visual content while preserving subject appearance and audio-video synchronization across successive segments. A bounded cache retains local motion and phonetic context but discards older evidence, whereas attending to the complete generated history is computationally expensive and can propagate accumulated errors. We present TaoMate, an anchor-guided persistent-memory framework for few-step joint audio-video generation. The framework preserves an immutable visual anchor, compresses completed video and audio blocks into fixed-capacity dynamic states, and retrieves those states through modality-specific residual attention without extending the active cache. A reference-aware modulation method additionally conditions video features on dynamic and anchor appearance statistics. Anchor-preserving causal-context distillation varies rollout horizon, prefix provenance, and cache-history reliability while keeping the immutable visual anchor unperturbed. By separating persistent memory from stage-local denoising dependencies, \method further admits stage-parallel execution across blocks, accelerating autoregressive inference without pipeline-specific retraining. We evaluate long-form video continuations with appearance, temporal, synchronization, facial, and speech diagnostics. Results show that \method preserves stable appearance across prompt-conditioned segments and strong audio-visual synchronization under autoregressive generation.

Primary area:
generative models
Keywords:
Real-time Video Generation, Digital Human Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xWGrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for jointly generating long-form audio and video of a talking avatar. In the proposed method, an existing joint audio-video generation model is distilled into a few-step causal model, which generates autoregressively while referring to fixed references and continuously updated memory, in addition to a KV cache that holds the most recent outputs. The fixed references are compressed representations of the video and audio from the initial block. The adaptive memory comprises two types of information. The first one is normalized content representations, obtained by normalizing and projecting previously generated blocks. The other one is channel-level and low-frequency appearance statistics. Both are updated with a moving average that mixes in the anchor, and for the video content and appearance statistics, the update rate is adaptively determined by their agreement with the anchor. During generation, the content representations are fed into the model through dedicated memory attention, and a part of the appearance statistics is injected through FiLM. In training, the model is distilled on contexts that include student-generated and re-noised history to reduce exposure bias. At inference, the memory is kept fixed within each window, and stage-parallel execution, which assigns each denoising stage to a separate GPU, accelerates generation. In the experiments, the method outperforms the baselines, particularly in long-term consistency.

Critical strengths
  • The proposed method achieves the best results in the long-term consistency metrics and LipSync, and the high consistency can also be confirmed in the supplementary materials. However, it has not been shown that this improvement attributes to the proposed persistent memory (see Weaknesses).
Critical weaknesses
  • The effectiveness of the persistent memory, which is the central contribution, is not clearly demonstrated.
    • In Table 3, the variant without memory already reaches a Long Cons. of 0.9614. This is on par with LiveAvatar (0.9619) and far above OmniForcing (0.3404). Its Color Δ of 0.00380 also already outperforms the best baseline, LiveAvatar (0.00541).
    • The improvement from the full model appears to be very limited (e.g., +0.0062 in Long Cons.). Moreover, since the variance is reported, I cannot judge whether these differences are significant.
    • The performance of the proposed method differs between Table 1 and Table 3 (Long Cons. of 0.9755 and 0.9676, and Color Δ of 0.00260 and 0.00276, respectively), and no explanation is given for this discrepancy.
    • Most of the gap to the baselines comes from components other than the memory, such as the distillation scheme or retaining the initial latent. However, no ablation on these points is provided.
    • The evaluation only covers settings in which the subject, background, and camera are fixed. The paper states that the memory must be continuously updated to adapt to scene evolution, but this necessity is not verified.
  • The evaluation does not sufficiently support the claims.
    • The consistency metrics measure the stability of color and brightness, which means that outputs that are closer to static images tend to score higher. On the other hand, there is no evaluation of audio and video quality or of the amount of motion. In the supplementary videos, the quality appears lower than that of the baselines (the person looks blurry and the audio contains considerable noise). This raises the possibility that the gain in consistency comes at the cost of quality.
    • The baseline values cited in Table 2 do not match the those reported in the original papers. In the cited papers, LiveAvatar reports 45.2 FPS on five H100 GPUs, and LongLive-2.0 reports 45.7 FPS (2 steps) on a single GB200 GPU with one additional GPU for VAE decoding. The main text claims that TaoMate exceeds the reported throughput of these methods by 1.52× and 2.19×, but this claim is not supported according to the values in the original papers.
    • In Table 1, the throughput of OmniForcing (8.14 DiT FPS) is about one third of the roughly 25 FPS reported in the original paper. This baseline appears to be a reimplementation on LTX-2.3, but its detail and measurement conditions are not described.
    • Although the paper claims real-time generation, latency is not reported.
Questions for authors
  • In the ablation study, why does the variant without memory already perform on par with or better than the baselines?
  • Why is there no evaluation of video and audio quality, even though such metrics are included in VABench?
  • Where do the values for LiveAvatar (23.0 FPS) and LongLive-2.0 (16.0 FPS) in Table 2 come from? In addition, why is latency not reported? How was the OmniForcing baseline implemented and measured?
Reviewer_nKbYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses appearance and identity drift in long-form digital human generation. TaoMate combines immutable references with adaptive audiovisual memory and appearance modulation to stabilize recursive generation. Causal-context distillation and stage-parallel inference further support continuous, real-time audio-video generation.

Critical strengths
  1. Practical problem and challenging task. Accumulated drift severely limits the usability of long-form digital humans, making this a valuable problem to address.
  2. Novelty. Separating immutable anchors, evolving content, and appearance statistics provides an interesting memory design tailored to this problem.
  3. Most of all, the results look promising. The visual comparisons suggest improved appearance stability over extended generation.
Critical weaknesses
  1. Limited experiments. The ablations do not sufficiently cover the complexity of the method, particularly the contributions of audio memory and the proposed training strategy.
  2. The evaluation omits direct identity-preservation measurements commonly used in portrait animation, such as face-recognition embedding cosine similarity. Color and brightness statistics, together with normalized facial geometry, are insufficient to establish that the generated person retains the same identity.
  3. The limitations are explicitly acknowledged. However, the lack of substantially longer evaluations remains consequential for the central claim of mitigating long-horizon drift. Results showing how identity and visual quality evolve over time would be more convincing than aggregate scores and sparse snapshots.
Questions for authors
  1. Within the single-presenter setting, have you observed a trade-off between drift suppression and responsiveness to prompted actions or expressions? How does anchor-agreement gating distinguish corrupted history from legitimate changes caused by motion?
  2. Beyond correcting color and brightness drift, what evidence most directly supports improved facial identity preservation? Have you observed cases where appearance statistics remain stable but identity changes, and does TaoMate improve those cases?
Reviewer_wUKiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TaoMate, a framework for long-form autoregressive audio-video generation of digital humans. The method separates temporal conditioning into a bounded context for recent motion and phonetic continuity, fixed reference anchors, and dynamically updated persistent memory. Memory attention retrieves compressed historical information, while an anchor-agreement gate regulates how strongly new visual observations update the memory. Reference-aware FiLM additionally conditions video features on dynamic and anchor appearance statistics.

For visual content memory, the method temporally averages video latents and normalizes each channel across spatial dimensions. Channel-level and low-frequency appearance statistics are maintained separately and used through FiLM and an additional latent-calibration pathway. Audio content representations are normalized along the temporal dimension. The paper also introduces stage-parallel inference, assigning denoising stages to different GPUs with independent stage-local KV caches to overlap computation across blocks.

Critical strengths
  • A clear separation of temporal roles.
  • A principled memory-update mechanism.
  • A useful distinction between content and appearance conditioning.
  • Practical inference acceleration. The stage-local cache design enables concurrent processing of different blocks at different denoising stages without pipeline-specific retraining.
Critical weaknesses

The proposed separation of content and appearance is insufficiently validated.

The method relies on the assumption that normalized latent patterns retain useful structural information, while channel means, scales, and low-frequency statistics capture appearance attributes such as color and brightness. Spatial normalization does remove channel-wise offsets and scales, but this does not establish a clean semantic separation between structure and appearance in a learned latent space.

The paper does not directly test this assumption. At minimum, an ablation comparing normalized and unnormalized memory representations would help establish whether normalization is beneficial. Additional analyses of how the representations respond to changes in identity, pose, illumination, and color would make the motivation more convincing.

The specific contribution of reference-aware FiLM remains unclear.

FiLM is a general conditioning mechanism. In this method, its connection to appearance preservation comes from conditioning on anchor and dynamic appearance statistics; the operation itself does not explicitly enforce appearance consistency or correct deviations from the reference.

The design appears intended to reintroduce appearance information through channel-wise modulation after separating it from normalized content memory. However, the paper does not establish whether this separation and subsequent conditioning work as intended.

The reported ablation shows that adding FiLM without gating degrades performance, whereas the full combination improves it. Since a memory-plus-gate configuration without FiLM is missing, the incremental benefit of FiLM under reliable memory updates remains unresolved.

The choice of FiLM over alternative appearance-conditioning mechanisms is not adequately justified.

Channel-wise modulation is a plausible way to apply coherent adjustments across video tokens. However, it is unclear whether this is preferable to supplying appearance information through attention or another conditioning pathway. A controlled comparison would help distinguish the benefit of the appearance information itself from the benefit of the chosen modulation mechanism.

The batch-averaged reliability gate is questionable.

As written, the method averages anchor similarity across samples in a mini-batch before computing a shared gate. Several high-similarity samples could therefore mask a severely inconsistent sample, allowing its unreliable observation to update memory at a relatively high rate.

This also makes a sample’s update depend on the other samples in its batch. The paper should clarify whether the implementation follows this formulation and explain why a batch-level gate is preferable to a per-sample gate. Reporting gate distributions and representative failure cases would also be helpful.

The evaluation does not sufficiently establish identity preservation.

Long Consistency is primarily based on color drift, brightness variation, and local color discontinuity. These measures are useful for evaluating appearance stability, but they cannot by themselves establish that the same person is preserved over time. The restricted benchmark, which maintains consistent subject and scene descriptions, also leaves open how broadly the reported stability generalizes.

Similarly, Face Consistency measures normalized facial geometry using MediaPipe. This is meaningful as a shape-stability diagnostic, but it is not a biometric identity metric. A direct evaluation of visual identity preservation would better support the paper’s identity-related claims.

The audio evaluation is too limited for the proposed multimodal memory framework.

The reported results emphasize audio-video synchronization and broader alignment or expressiveness scores. They do not directly establish transcript accuracy, long-term speaker consistency, or speech quality. WER/CER, speaker-similarity measurements, and speech-quality evaluations would provide substantially stronger evidence.

The ablation study also reports visual metrics without separately isolating the effects of audio and video memory. Consequently, the contribution of persistent audio memory remains unclear.

Comparisons across different generation settings require more careful interpretation.

The main results include both audio-conditioned video generators and joint audio-video generators. Although these methods provide useful reference points, they operate under different input conditions and generation requirements. Comparisons and headline conclusions should more clearly distinguish these settings so that improvements can be attributed to the relevant capabilities.

Questions for authors

Does the method preserve speaker-specific speaking style and prosodic characteristics over long sequences, while allowing appropriate utterance-level changes in intonation?

What happens when audio memory is removed while video memory is retained? In particular, how do transcript accuracy, speaker consistency, and synchronization change?

What happens when video memory is removed while audio memory is retained? Can the benefits of the two memory pathways be established independently?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.