acceptodds
Withdrawn from ICLR 2027

SteadyTalk: Remembering Every Speaker for Long-Horizon Audio-Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Digital humans often appear in long videos with several speakers, and each speaker must keep the same voice and face throughout. Joint autoregressive generators drift in timbre and color over such long videos and forget a speaker’s voice and face after a long silence. In audio-driven generators, the given audio dominates the output and limits control over the scene. To address these issues, we propose SteadyTalk, a long audio-video generation framework for single- and multi-person scenes, built upon three modules. First, Audio-First Asynchronous Generation drives the joint model with its own audio. A frozen audio branch generates the audio of each chunk first, and the video branch follows both this audio and the text prompt. Second, Multi-Person Audio-Visual Memory stores key audio segments and frames for each participant and recalls them with the current prompt, so a returning speaker keeps the same voice and face. Third, Reference-Anchored Drift Correction aligns the color statistics of each chunk to the opening frames and its audio latent statistics to the speaker’s first utterance. On one-minute single- and two-person conversations, SteadyTalk outperforms existing methods.

Primary area:
generative models
Keywords:
Talking Avatar, Audio-Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_EDzFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SteadyTalk, a long audio-video generation framework for scenes with one or two speakers. The core ideas cosist of three parts: Audio-First Asynchronous Generation to balance text control and stable video output with clean audio, Multi-Person Audio-Visual Memory for identity preservation tailored to multi-person scenarios, and Reference-Anchored Drift Correction to suppress color and timbre drift in a long generation. The paper also proposes a new audio-video benchmark to evaluate these scenarios. Experiments show that the proposed method achieves better audiovisual synchronization, speech quality, identity preservation, and color/brightness stability than baselines.

Critical strengths

The paper is well motivated and proposes a comprehensive pipeline to address the challenges of multi-speaker long-video generation. The paper also introduces a new benchmark for this task, and the experimental results demonstrate the proposed method's effectiveness on the target tasks.

Critical weaknesses
  • The empirical support for the paper’s claims is insufficient. The paper claims that Audio-First Asynchronous Generation is a general inference design to maintain text control and stable audiovisual alignment. However, the evaluations are conducted on only a single base model (MiniMax-H3). Also, Multi-Person Audio-Visual Memory is tested with at most two speakers, making it difficult to assess the scalability and computational cost of the proposed method as the number of speakers increases. Additionally, the main table omits the performance of the base MiniMax-H3, which prevents isolating the proposed method's effectiveness from performance improvements due to the different base model.
  • The reviewer is concerned that the improvements reported in Table 1 may be largely attributable to drift correction. All models are evaluated in a long-video setting, and metrics other than stability also appear to be affected by video and audio statistics (as in Table 4). Because only SteadyTalk explicitly corrects color and brightness in video and timbre in audio, the baselines may accumulate drift in later segments, leading to lower scores across multiple metrics. This makes it difficult to distinguish the contributions of the generation framework and memory design from those of drift correction. Because the proposed correction is inherently orthogonal to the generation framework and memory design, its effectiveness should be evaluated independently to clarify the contribution of each component.
Questions for authors
  • Have the authors tested the proposed Audio-First Asynchronous Generation with LTX-2.3, Ovi, and MOVA?
  • Can the authors provide quantitative results for baselines with the proposed drift correction?
Reviewer_uxhvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SteadyTalk, a framework for long-horizon audio-video generation in single- and two-person conversations, built on MiniMax-H3. It introduces three components: audio-first generation, where a frozen audio branch generates speech before a LoRA-adapted, distilled video branch produces the corresponding video; participant-specific memory that retrieves historical appearance and voice references; and reference-anchored correction that aligns video color and audio latent statistics to initial references to reduce drift. The authors also introduce SteadyBench, comprising 150 one-minute interview-style scenarios. Comparisons and ablations report improvements in lip synchronization, speech intelligibility, identity consistency, and audiovisual stability.

Critical strengths

Well-motivated audio-first generation. Separating audio generation from video denoising provides stable audio conditioning while preserving the undistilled audio branch. The equal-step comparison supports benefits beyond simply increasing the number of audio sampling steps.

Explicit handling of speaker-specific identity. The participant-partitioned memory directly addresses confusion between alternating speakers by associating historical voice and appearance references with individual identities. The two-person ablations support the usefulness of this separation, particularly compared with injecting both speakers’ audio into a shared context.

Useful component-level validation. The paper includes a joint-generation baseline trained with the same backbone and data, separate memory and drift-correction ablations, and measurements of degradation over time. The bilingual benchmark and human preference study provide additional evidence of practical effectiveness within the evaluated one-minute conversation setting.

Critical weaknesses

The benefit of clean-audio conditioning is insufficiently isolated. Table 2 changes both the audio-conditioning strategy and whether the audio branch is distilled. It therefore does not isolate the advantage of fixed clean-audio conditioning over TaoMate-H3-style trajectory-based audio guidance. This distinction is important for establishing the specific contribution of the proposed audio-first design.

Speaker resumption is insufficiently validated. The supplementary material provides limited multi-person demonstrations, and some examples exhibit lip-sync failures when speakers resume after prolonged silence. Aggregate synchronization scores and zero identity swaps do not establish reliable articulation during these transitions.

Aggregate and category-level results appear inconsistent. Under the stated scenario-averaging protocol, weighting Table 6 by category sizes yields approximately 0.0693 Desync and 1.047% WER, whereas Table 1 reports 0.057 and 0.85%. These discrepancies exceed rounding error and require clarification.

Cross-chunk continuity is insufficiently evaluated. Although Color Δ measures changes between adjacent intervals, dedicated measurements of audio, motion, and local synchronization at actual generation boundaries are missing. Given per-chunk audio correction and segment-level cache resets, reduced global drift does not necessarily imply smooth local

Questions for authors
  1. How does fixed clean-audio conditioning compare with TaoMate-H3-style staged audio guidance? Could you provide a controlled comparison using the same frozen audio generator, matched video sampling and training budgets, and video models appropriately adapted to each conditioning strategy?
  2. Could you evaluate synchronization specifically around speech resumption, grouped by the preceding silence duration? Please also discuss the resumption failures in the supplementary examples and whether participant-specific memory improves local lip synchronization beyond preserving identity.
  3. Were Tables 1 and 6 computed using the same checkpoint, samples, and aggregation procedure? Please explain why the category-weighted Desync and WER values do not reproduce the aggregate results.
  4. Do the intervals used for Color Δ align with actual generation chunk boundaries? Could you report audio and visual continuity around ordinary chunk boundaries and segment-reset boundaries separately, with and without correction, while accounting for intended speaker or scene changes?
Reviewer_fGDYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SteadyTalk, a framework aimed at improving audio-visual generation. The authors propose three modules, audio-first asynchronous generation, multi-person audio-visual memory and reference-anchored drift correction to improve the consistency and quality of generation. Experimental results provided by the authors demonstrate the effectiveness of the methods.

Critical strengths
  1. The authors focus on the important problems of generation, including inconsistency of the visual features of characters in long-form generation and accumulated drift. These issues are crucial in modern audio-visual generation frameworks.
  2. The methods proposed are theoretically reasonable and effective according to the experimental results. The authors designed three techniques to address the three issues respectively.
Critical weaknesses
  1. The motivations and mechanisms underlying the three components appear to be relatively independent. For example, the asynchronous generation and the drift correction focus on different aspects of audio-video generation. Moreover, the techniques seem to be not really novel, the memory mechanisim is a commonly used method, while aligning feature statistics to reduce unintended drift is also proposed before.
  2. The asynchronous audio-video generation makes the complex joint generation task easier, which is reasonable for more efficient generation. However, this design may impose a limitation on the achievable generation quality, as the two modalities rely on each other in the real world. It would be more clear if the authors provide qualitative or quantitative explanation.
  3. The memory design seems to have a defect that it only compares with the previous memory of a specific person. If the person frequently performs a small set of actions (e.g. lowering head to read the transcript and raising head when giving a speech), the size of the set may continue to grow. Moreover, retrieval is based on prompt similarity, which may not be able to keep reflecting the latest visual features of the person. I would appreciate it if the authors provide a clearer explanation of the memory design and discuss possible improvements to address these issues.
Questions for authors

Please see the Weakness section for more details. The questions are about the overall structure of the method, the potential limitation of the asynchronous audio-video generation, the design issues in the multi-person audio-visual memory. It would be clearer if the authors provide experimental evidence or explanations regarding these concerns.

Reviewer_zVszOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies long audio-video generation in which one or two on-screen participants must retain their voice and appearance across speaking turns. SteadyTalk builds on MiniMax-H3 and combines three components: a frozen audio branch that generates each chunk before a distilled video branch conditions on the completed audio; a participant-partitioned reservoir that retrieves earlier speech and visual crops; and reference-based alignment of video color statistics and audio latent moments. The video branch is adapted with LoRA and distribution matching distillation on causal rollouts. Evaluation uses SteadyBench, a collection of 150 scripted, one-minute interview scenarios in Mandarin and English, with supplied off-screen interviewer speech. The reported comparisons and ablations suggest improvements in speech intelligibility, speaker retention, and appearance stability, although inconsistencies between the main and category-level results currently prevent a reliable assessment of the precise gains.

Critical strengths

A concrete and practically relevant formulation of participant retention. Sections 3.1 and 3.3 distinguish recent context, which supports local motion and phonetic continuity, from older participant-specific evidence needed after a speaker has been silent. The explicit speaker tags, separate audio histories, and visual crops give the system a clear mechanism for maintaining identity. The paper also states its two-participant limit rather than claiming unrestricted group conversation support. Useful controlled ablations of the proposed components. Table 2 includes a joint four-step baseline and an audio-first four-step variant on the same backbone. The latter improves reported WER from 5.93% to 3.91% and LipSync from 5.23 to 6.12 before increasing the audio sampler to ten steps. Table 3 separates visual memory, audio memory, and participant partitioning; removing all memory gives a reported swap rate of 0.25, compared with zero for the full system. Table 4 independently enables the video and audio corrections. These experiments are more informative about the design than comparisons between unrelated systems alone. The appendix provides meaningful implementation and evaluation detail. Appendices A.1–A.4 describe the scripted conversation structure, training data volume, optimization settings, memory sizes, context policy, and definitions of the speech and identity metrics. Evaluating speaker similarity with WavLM rather than the CAM++ encoder used for memory insertion is a sensible separation. Appendix A.8 also correctly distinguishes a bounded generator working memory from an uncapped reservoir and duration-dependent retrieval cost.

Critical weaknesses

The main results are inconsistent with the category-level results under the stated aggregation protocol. Appendix A.4 states that metrics are computed per scenario and then averaged over scenarios. Sections 4.1 and A.1 specify 70 single-presenter, 20 complex-action, and 60 two-person scenarios. Therefore, each overall metric should equal the category means in Table 6 weighted by 70/150, 20/150, and 60/150. Several reported values do not satisfy this requirement: Metric Overall implied by Table 6 Reported in Table 1 LipSync 6.775 6.960 Desync 0.0693 0.0570 Listener (%) 80.77 78.50 WER (%) 1.047 0.850 Spk-Sim 0.5789 0.5960 ID-Sim 0.6847 0.6960 These differences exceed the displayed rounding precision. In contrast, the same calculation reproduces Long Consistency as 0.971 and ColorΔ as approximately 0.00149, making a uniformly different aggregation rule an insufficient explanation without further detail. The all-scenario results in Tables 2 and 4 repeat several Table 1 values. The authors should reconcile the tables against one set of per-scenario outputs and state whether different subsets, runs, or checkpoints were used. The comparative evidence does not adequately separate the proposed framework from backbone strength and evaluation conditions. MiniMax-H3 supplies the frozen audio generator, teacher, synthetic training trajectories, and initialization, yet its native generation performance is not included in Table 1. This leaves the incremental benefit relative to the source model unclear. Appendix A.4 also evaluates some methods for 30 seconds and others for 60 seconds, restricts single-subject audio-driven methods to a subset of categories, and retains native resolutions and frame rates. Further, SteadyTalk receives synthesized interviewer audio, whereas methods without audio input generate those segments from transcripts. These are understandable capability differences, but a single table does not provide a fully matched estimate of relative performance. The user study appropriately truncates paired videos for the 30-second baselines; analogous matched-duration and matched-category quantitative comparisons are missing. A native-backbone reference and matched subsets of the existing evaluations would make the claims substantially more convincing. The principal long-term memory claim is not sufficiently distinguished from simpler speaker references. The benchmark contains four answers per minute, each participant answers only twice in two-person scenarios, and speaker identity, appearance, voice, and scene descriptions are supplied repeatedly in the prompts (Appendix A.1). Appendix A.7 includes examples of a speaker returning after 20 seconds of silence, which is relevant evidence, but the quantitative evaluation does not stratify results by absence duration or number of intervening turns. Moreover, Table 3 does not compare the full reservoir and retrieval policy against a fixed first-utterance/first-crop reference for each participant. The visual-only and audio-only tagged variants both already achieve zero swaps. Thus, the results support the value of participant-specific conditioning, but do not establish the added necessity of gated storage and prompt-guided retrieval. A simple fixed-reference comparison and a breakdown of existing cases by speaker absence would directly address this gap. Preserving normalized audio latent patterns does not establish preservation of decoded content and timing. Section 3.4 argues that Eq. (4) changes channel moments while keeping normalized temporal structure, and therefore moves timbre toward the reference “without altering what is said or when.” The algebra establishes an affine transformation of latent channels; it does not establish that the MiniMax-H3 audio decoder disentangles speaker identity from phonetic content, prosody, or timing in those moments. The cited instance-normalization motivation is not itself a validation of this particular audio representation. Table 4 provides encouraging average WER and LipSync results, but does not measure paired changes in transcripts or timing before and after correcting the same generated latent. Also, ColorΔ and Long Consistency explicitly reward color-statistical stability, which is directly targeted by Eq. (3); their gains alone cannot distinguish removal of unwanted drift from suppression of intended visual changes. The authors should narrow the invariance claim and provide a focused paired check of audio preservation, along with evidence that intended changes such as removing a jacket remain intact. Important details needed to reproduce the mechanism and interpret its cost are missing. Section 3.3 does not operationally specify how participant crops are detected and matched to appearance descriptions, or identify the aesthetic scorer, blur check, and quality threshold in Eq. (2). Appendix A.3 omits the color blending coefficient α. The LoRA configuration, exact initial checkpoint, sampler settings, and synthetic trajectory filtering procedure are also not sufficiently specified. Section 3.2 describes ground-truth clean audio during training, whereas Appendix A.2 describes generated audio within student rollouts; the relationship between these training conditions needs clarification. Finally, bounded context tokens do not imply bounded total storage or constant retrieval cost, as Appendix A.8 itself explains. Measured chunk latency, peak memory, reservoir growth, and retrieval overhead are not reported. This matters because the proposed schedule adds a ten-step audio stage before a four-step video stage, even though no real-time performance claim should be inferred solely from few-step video sampling.

Questions for authors

Can the main and category-level evaluation results be reconciled? Please identify the exact scenarios, checkpoints, random seeds, and aggregation rules used for Tables 1, 2, 4, and 6. Under the stated per-scenario averaging, why do the category counts and Table 6 imply WER ≈ 1.05%, Desync ≈ 0.0693, and Spk-Sim ≈ 0.579 rather than the reported 0.85%, 0.057, and 0.596? A corrected table derived from consistent per-scenario records is important. What improvement remains relative to the native backbone and on matched comparison subsets? Please report MiniMax-H3 under the same continuation protocol and separate comparisons on a common first-30-second interval and common scenario categories. Please also clarify how external interviewer audio versus generated interviewer speech affects the listener and later-answer evaluations. These results would help distinguish framework gains from inherited model quality and input differences. How much does the reservoir contribute beyond fixed participant anchors? Please compare the full memory with one fixed early audio reference and visual reference per participant while retaining the same recent context. Can the existing test cases also be grouped by absence duration and answer order, especially alternating speakers versus the R,R,L,L pattern in Table 8? The answer would determine whether the evidence supports the full retrieval mechanism or primarily participant-tagged reference conditioning. What evidence supports content and timing preservation by audio correction, and how are the training conditions implemented? For identical sampled audio latents, please compare decoded speech before and after Eq. (4), including transcript errors and word or phoneme timing changes. Please explain which training stages use ground-truth versus generated audio and supply the missing crop-assignment, quality-gate, color-blending, and LoRA settings. These clarifications would improve both technical confidence and reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.