acceptodds
Under review as a conference paper at ICLR 2027

Edge-Avatar: Pose-Controlled Real-Time Speech-to-Video on a Single Consumer GPU

Abstract

Audio-driven human animation reaches high fidelity, but its many-step guided sampling is far too slow for an interactive avatar. We present Edge-Avatar, a speech-to-video model for controllable audio-driven human animation that runs in real time at 25 FPS on a single consumer GPU under pose control. Starting from a bidirectional 2B diffusion transformer that draws text and audio conditioning from a single Qwen2.5-Omni model, two training stages convert it into a chunk-autoregressive generator, built on recent streaming works and extended with an area-adaptive face loss and memory noising. Pose control is trained rather than imposed at inference, through a pose track that may begin or end partway through a chunk, so a gesture is entered and left smoothly. An optional training-free reset returns the avatar to the reference image mid-stream and clears the carried memory. Deployment pairs the few-step sampler with FP8 linear layers, block-sparse attention, compiled execution, and a distilled decoder, so that denoising and decoding fit inside the playback interval of each chunk at SD resolution on a single NVIDIA RTX 5090. Two optional pieces are independent of each other, a cheaper opening denoising step carried in the frequency domain and a further-upscaling decoder, and together they deliver HD inside that same budget. On common speech-to-video benchmarks the result leads the open-source real-time avatar generation systems it is measured against, including ones an order of magnitude larger and ones that reach real time only on multi-GPU nodes, and a side-by-side study against the three closest of them comes out in its favour.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech-to-video generation, audio-driven human animation, real-time video generation, diffusion transformer, diffusion distillation, pose-controlled generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5EtxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Edge Avatar addresses real time speech driven avatar generation with pose control. It converts a two billion parameter video model into a three step generator, combining an adaptive face loss, memory noising, and partial span pose conditioning to support continuous generation on a single consumer GPU.

Critical strengths

The model generates video faster than playback on a single RTX 5090 and achieves competitive identity preservation and lip synchronization across multiple benchmarks. The paper provides detailed measurements of memory usage, component latency, and quality changes introduced by deployment optimizations.

The evaluation includes two minute and five minute videos, alongside human comparisons with three strong baselines. The memory noising ablation shows improved identity preservation, supporting the usefulness of this component.

Critical weaknesses

Main Weaknesses

Table 15 compares different pose entry times using the same model, without a control trained without partial span pose conditioning. It therefore does not isolate the contribution of this training design. The evaluation measures only changes between frames, without pose tracking error or gesture completion rates, leaving open whether smoother transitions come at the cost of reduced motion amplitude or control accuracy.

Table 2 measures only denoising and decoding, excluding conditioning encoding performed once per utterance. Each chunk represents 1.28 seconds of video, but the paper does not report audio buffering requirements, first frame latency, or gesture response time when inputs arrive incrementally. The results support real time generation throughput, while the latency of online interaction remains insufficiently characterized.

The face loss ablation in Table 14 is conducted only at the first autoregressive stage, and all variants inherit a base model already trained with the adaptive loss. There is no comparison after distillation or breakdown by face size. Its contribution to lip synchronization for small faces and to preventing tighter framing therefore remains unclear.

Minor Weaknesses

Samples with failed metric computation are excluded, but the effective sample counts for each model are not reported. Confidence intervals are also missing for the main metrics and human preference results.

Questions for authors

Please compare models trained with and without partial span pose conditioning, reporting pose tracking accuracy, gesture completion, and transition smoothness.

Please report first frame latency including conditioning encoding, specify the required future audio context, and measure response time to gesture requests during generation.

Please evaluate the adaptive face loss after distillation and report lip synchronization and framing changes across different face sizes.

2 comments
AuthorsOpenReview ↗

(part 1/2)

(part, 1/2)

We thank the reviewer for the positive assessment and for the constructive questions on pose control, latency and the face loss. Table, figure and section numbers in this answer refer to the revised version.

Please compare models trained with and without partial span pose conditioning, reporting pose tracking accuracy, gesture completion, and transition smoothness.

Control. We trained a control without partial spans: in all training stages, the pose always covers the whole chunk. Everything else matches the deployed model. We evaluate both models on EMTD-30s. Each clip is driven by its own pose track, and we extract keypoints from the generated videos with the same ViTPose-H. No clip is excluded.

Transition smoothness. When the pose is switched on inside the chunk, only our model stays smooth at the frame where it is switched on (Table R3).

Table R3: Smoothness of switching the pose on inside the chunk, EMTD-30s, 25 clips, measured at the switch frame.

ModelSpike ↓Bend ↓Window ↓
Without partial spans1.863.62.11
With partial spans (ours)1.161.51.41

To check that this is a jerk and not regular motion, we compare the switch frame with the same frame of the same clip when the pose is always on. For the control, spike and bend rise clearly. For our model they change only within noise (Table R4).

Table R4: Spike and bend at the frame where the pose is switched on inside the chunk ("pose switched on"), compared with the same frame of the same clip in a video where the pose is fed from the first frame ("pose always on"). Δ is the difference. EMTD-30s, 25 clips. Our model's values come from a re-run, so they differ slightly from Table R3.

ModelSpike ↓: pose switched onSpike ↓: pose always onΔBend ↓: pose switched onBend ↓: pose always onΔ
Without partial spans1.861.00+0.863.61.7+1.9
With partial spans (ours)1.131.07+0.061.51.3+0.2

Tracking accuracy, gesture completion and amplitude. With the pose on from the first frame, both models follow it equally well and move the hands as widely as the track. When the pose is switched on later, our model reaches the hand positions of the track in more gestures: 96.9% against 83.3% (Table R5).

Table R5: How well the model follows the pose, EMTD-30s, 50 clips; last column: the pose is switched on inside a chunk during a gesture, 30 clips. Distances are measured in the person's shoulder width, so they do not depend on frame size or distance to the camera. PCK: share of keypoints within the threshold of the pose track. Amplitude: how wide the hand moves compared with the track (1 means the same). Completion: share of gestures in the track that the hand reaches. No pose: the same model without a pose track.

Model[email protected] body ↑[email protected] hands ↑Amplitude →1Completion ↑, pose always onCompletion ↑, pose enters mid-clip
No pose0.530.200.46––
Without partial spans0.9450.9121.0097.0%83.3%
With partial spans (ours)0.9410.9121.0297.3%96.9%

Lip sync and quality are preserved: CSIM 0.814 vs 0.810, IQA 4.82 for both. Sync-C is 0.08 lower (7.64 vs 7.72, about 1%).

The revision states that the original pose-entry table (now Table 21) measures at the chunk boundary, and adds Tables R3 to R5 and the lip-sync numbers to Appendix G.2 (Tables 22 to 24), with the main results in Table 4.

Please report first frame latency including conditioning encoding, specify the required future audio context, and measure response time to gesture requests during generation.

Latency of online interaction. Table 1 reports throughput, not live latency. We measured all three on the same RTX 5090 (configuration of Table 1). The released code encodes the whole utterance first, so live sessions with real-time speech apply the audio tower incrementally, per chunk or per 2 s block.

First-frame latency. Encoding is cheap (Table R1). From the speech of the first chunk (1.16 s, 29 speech tokens) being available to its first frame: 0.94 s in SD, the same compute for either incremental encoding. In live sessions with real-time speech the first frame appears 2.11 s after utterance start with per-chunk encoding (1.16 s of speech plus the compute) and 4.15 s with block-aligned encoding (2.0 s of speech, the compute, a 1.2 s display pre-roll against later block waits); about 1 s with the whole utterance from a TTS. This is the usual case for a conversational agent, whose TTS runs ahead of playback, and the 2.11 s and 4.15 s apply only to speech arriving strictly in real time.

AuthorsOpenReview ↗

(part 2/2)

(part, 2/2)

Table R1: First-frame latency on the RTX 5090, ms, medians of 10 utterances. Per-chunk: the first chunk of speech encoded alone. Released: one Thinker pass over the prompt and a 30 s utterance.

ConfigurationPer-chunk: AudioPer-chunk: DiTPer-chunk: DecodePer-chunk: First frameReleased: Joint pass, 30 sReleased: First frame
SD (512x768)77182179421241059
SD + SPD7529217754123871
HD (768x1152)771731710411221155
HD + SPD7532330868126987

Future audio context. The DiT sees only its chunk's speech tokens, but the audio encoder attends within 2 s blocks, so the released features need speech to the end of the current block: 0 to 2 s beyond the chunk, 1 s on average, 2.0 s of speech for the first chunk. Block-aligned encoding reproduces these features (cosine 0.988). Per-chunk encoding needs no look-ahead but changes them (cosine 0.606), and we have not evaluated video generated from them, so we do not claim zero look-ahead.

Gesture response. With requests at random times during real-time speech (Table R2), the first frame generated under the pose appears about 2.6 s after the request with either encoding under the released rule (pose on the last two latents of the next chunk, generated one chunk ahead of the display). Entry on the first latent gives 1.6 s at the price of the seam of Table 21. These are now in Section 5.4.1 and Appendix F.2. With a whole TTS utterance and up to three chunks queued, the response grows to 4.75 s unless queued chunks are regenerated with the pose (rewind, 2.95 s).

Table R2: Response to a gesture request, seconds from the request, mean (min to max), 3 to 5 sessions of 30 s. First posed frame: first displayed frame generated under the pose. Visible onset: first frame that differs from a rollout without the pose. First frame: from the utterance start.

ScenarioAudio encodingRequestsFirst frame, sFirst posed frame, sVisible onset, s
Real-time speech, entry on the last 2 latents (released)per chunk72.112.60 (1.97–3.16)2.06 (1.37–2.72)
block-aligned74.152.63 (2.31–3.06)2.10 (1.81–2.42)
Real-time speech, entry on the first latentper chunk52.111.59 (1.01–1.98)1.81 (1.05–2.22)
block-aligned54.151.58 (1.35–1.89)1.80 (1.49–2.17)
Utterance at once, queue (up to 3 chunks)per chunk50.984.75 (3.40–5.61)4.22 (2.88–5.13)
Utterance at once, rewind of queued chunksper chunk50.992.95 (2.25–3.40)2.39 (1.65–2.88)

Please evaluate the adaptive face loss after distillation and report lip synchronization and framing changes across different face sizes.

Why stage 1. Stage 2 (DMD) uses no face term, and its student, teacher and critic all start from the stage-1 EMA. The face loss reaches the deployed model only through these weights, so we ablate it at stage 1.

Base model and face size. We compare two base models trained with memory, pose and silent clips that differ only in the face-loss weight, adaptive or fixed. We evaluate them on EMTD-30s and on HDTF-30s, whose faces are much larger, and split HDTF into two halves by face area. gen/ref is the face area in the generated video over that in the reference, so 1 means the framing is kept.

Table R6: Face loss in the base model.

DatasetFace area, % of frameRunCSIM ↑Sync-C ↑Sync-D ↓IQA ↑ASE ↑gen/ref →1
EMTD2.1 to 5.6, mean 3.6 (n = 50)Adaptive weight0.7507.67.914.773.571.038
Without the adaptive weight0.7397.567.944.763.561.097
HDTF12.7 to 20.6, mean 16.9 (n = 25)Adaptive weight0.8457.927.784.373.391.011
Without the adaptive weight0.8357.568.024.363.381.052
21.1 to 37.3, mean 27.7 (n = 25)Adaptive weight0.8637.927.804.663.661.011
Without the adaptive weight0.8517.877.824.663.671.031

The adaptive weight gives higher CSIM in every group and equal or better lip sync. The Sync-C gain is largest on the smaller HDTF faces.

As for framing, with the fixed weight the generated face grows relative to the reference by 9.7% on EMTD and by 5.2% and 3.1% on HDTF, against 3.8%, 1.1% and 1.1% with the adaptive weight. They are now in Table 20.

Minor Weaknesses

In our runs the effective number of clips matched the full set for every model and metric, with no clip dropped, as Appendix E now states. The revision adds 95% confidence intervals for all metrics and the side-by-side preferences (Appendix E).

We thank the reviewer again for the suggestions.

Reviewer_V4uNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Task Description

  • This paper addresses real-time audio-driven human animation: streaming a talking-avatar video from a reference image, speech, a text prompt, and an optional pose track at 25 FPS on a single consumer GPU.

Research Motivation

  • Diffusion-based speech-to-video models achieve high fidelity, but their many-step guided sampling (~100 NFEs per chunk) is too slow for interactive avatars, and existing real-time systems are much larger or need multi-GPU nodes.

Main Contributions

  • Edge-Avatar turns a bidirectional 2B DiT into a 3-step chunk-autoregressive generator via teacher forcing and self-forcing DMD distillation, adding an area-adaptive face loss, memory noising, and trained partial-span pose control, deployed on one RTX 5090.
Critical strengths
  • S1. The paper addresses a practically relevant problem: real-time, controllable audio-driven avatar generation on a single consumer GPU, a setting of clear interest for interactive applications.
  • S2. The proposed approach is conceptually simple: it requires no new architectural modules and converts the base model into a real-time generator through a two-stage training recipe, which should make it straightforward to adopt.
  • S3. The method is deployment-friendly: it runs faster than real time on a single RTX 5090, in contrast to baselines that reportedly require multi-GPU nodes.
Critical weaknesses
  • W1. Presentation and organization fall short of the venue’s clarity standard. The main text omits a dedicated Related Work section (related work is reduced to two inline paragraphs, with the substantive discussion deferred to Appendix A) and contains no Limitations section; Most of the evidence needed to assess the claims (all ablations, per-benchmark tables, two of the three human studies, and any qualitative frame-level comparison) lives in the appendix, so the 9-page paper does not stand on its own. The method is described largely in prose with few, unnumbered equations, making the training objective (DMD term, regression term, background term, area-adaptive face weight) hard to reconstruct precisely, while descriptions on implementation (thread locks, decoder threads) occupy main-text space.

  • W2. Limited technical novelty and unclear transferable insight. By the paper’s own attribution, the two-stage recipe follows SoulX-FlashTalk/CausVid/Self-Forcing, the regression term follows SoulX-FlashHead, the background term follows the public FlashTalk training code, full intra-chunk attention follows SoulX-FlashTalk, and memory noising closely resembles Live Avatar’s noisy KV cache. What remains new is a clipped inverse-area face weight, bone-group pose augmentation, and a training-free reset. Combining known components can be valuable, but the most of things in this paper seem more close to engineering works. For instance, Contribution 3 is a deployment stack of off-the-shelf tools (FP8, SpargeAttention/SageAttention, torch.compile, Turbo-VAED), and Contribution 4 is a result rather than a contribution.

Questions for authors

I would ask the authors

  • Q1. to move a concise Related Work, contain the core ablations, a qualitative comparison figure, and a Limitations paragraph into the main paper, formalize the full loss with numbered equations, and relocate deployment engineering details to the appendix.

  • Q2. to clarify what generalizable knowledge and insightful information a reader should take away beyond “this configuration runs in real time on an RTX 5090.”

4 comments

We thank the reviewer for the thorough review and the concrete suggestions on the organization of the paper.

(W1) Presentation and organization fall short of the venue's clarity standard … and Q1

We agree and have reorganized the paper in the revision, using its additional page. Concretely:

  • A Related Work section in the main text (Section 2), condensed from Appendix A.
  • A consolidated ablation table (Table 3), the three side-by-side studies (Figure 2) and a qualitative comparison figure (Figure 3) in the main text.
  • A Limitations section (Section 6).
  • Numbered equations, with the full student objective in one place, including the DMD gradient (Section 5.2).
  • Implementation details (threading, FP8 coverage, encoder loading) moved to Appendix F.1.

(W2) Limited technical novelty and unclear transferable insight… and Q2

We agree that the two-stage recipe and the distillation objective are adopted from prior work, as the paper states. The paper is submitted to the applications track, where a working system with carefully measured quality can itself be a contribution. Contribution 4 now lists the transferable findings below and what they give together: a 2B model in real time on a single RTX 5090 that is, in aggregate, ahead of open-source systems up to 11 times larger.

Memory noising. This differs from the noisy KV cache of Live Avatar. There the cache is noisy because it is computed at the timestep of the block being denoised, so its noise level follows the sampler step. Our history is two latent frames, corrupted as a separate, deliberate input: a small level independent of the sampler step, s ~ U(0, 0.1) in training and s = 0.05, its mean, at inference, given to the model through time-embedding modulation of the memory tokens. The new ablation (Table 25) shows that s = 0.05 with noise in training is the best setting:

RunCSIM ↑Sync-C ↑Sync-D ↓IQA ↑ASE ↑
EMTD-30s
Train noise, s = 0.05 (deployed)0.85947.658.024.853.73
Train noise, s = 0.10.83667.638.024.843.71
No train noise, s = 00.82557.598.074.853.74
Train noise, s = 00.80707.558.054.843.71
Cross-driven, 2 min
Train noise, s = 0.05 (deployed)0.82867.997.844.803.71
Train noise, s = 0.10.80017.987.884.803.70
No train noise, s = 00.74477.907.994.793.70
Train noise, s = 00.63127.468.224.783.69

Partial-span pose control. Beyond the bone-group augmentation, pose is trained on spans that start or stop inside a chunk, in the base model and in both autoregressive stages. This lets a gesture enter between chunks: entering inside the chunk rather than on its boundary lowers the motion spike from 1.98 to 1.24 and the bend from 12.4 to 3.1 (Table 21).

To check that this comes from training, we trained a control without partial spans. When the pose is switched on inside the chunk, the control jerks at that frame. Our model does not: compared with the same frame when the pose is always on, its spike and bend change only within noise (Table 23). This matters for live use: gesture requests arrive between chunks, and partial-span training lets them start inside the chunk without a visible jump. We see it as one of the paper's important contributions.

EMTD-30s, 25 clips, pose switched on inside the chunk.

ModelSpike ↓: pose switched onSpike ↓: pose always onΔBend ↓: pose switched onBend ↓: pose always onΔ
Without partial spans1.861.00+0.863.61.7+1.9
With partial spans (ours)1.131.07+0.061.51.3+0.2

Deployment. Our own component is Turbo-VAED-x1.5, the distilled decoder rebuilt with a spatial upscale of 12 and fine-tuned, which delivers HD from the same latents at a small decoding cost and clearly above interpolation (Tables 1, 28). We also apply spectral progressive diffusion (Xiao et al., 2026) on the opening step of the 3-step sampler, which takes about a quarter off the denoiser. Together they deliver HD faster than the default SD path (Table 1).

Transferable insight. The ablations support findings for chunk-autoregressive generators in general:

  • Noise the carried history in training and at inference (table above).
  • Do not switch a control signal on at the latent where the memory is replaced (Table 21).
  • Weight a region loss against the area it covers (Tables 19, 20).
  • Move upscaling into the decoder rather than interpolating its output (Table 28).
  • A few-step distilled sampler tolerates a reduced-resolution opening step (Table 27).

We hope these clarifications and results address the reviewer's concerns, and would be grateful if the reviewer would consider revisiting the score.

I thank the authors for the detailed responses and the revised paper. The added Related Work and Limitations sections and the numbered objective address the organizational part of W1. However, after carefully reading the rebuttal, I find that most of my major concerns remain unresolved.

Regarding memory noising (W2), I disagree with the claimed distinction from Live Avatar. Its noisy history is not a sampler by-product: it deliberately removes the clean-cache refresh pass (history corrupt) to suppress error accumulation, the same rationale given for memory noising, and ablates it separately. The remaining difference, a fixed noise level independent of the sampler step, is the very baseline Live Avatar already found inferior: in its experiments, fixed noise improves over a clean cache, but only step-matched noise removes flicker and best preserves identity. Conditioning on lightly noised history with the level given to the model also follows Diffusion Forcing and GameNGen. Therefore, memory noising is not a new concept.

Regarding the evidence added for W1(and Q1), first, Table 3 is not a component ablation: it mostly varies inference-time settings (noise level, kernels, decoders), its face-loss rows are stage-1 runs not comparable with the rest, and key training components such as the regression and background terms are never ablated. Second, Figure 3 is unconvincing: it shows a single subject in four frames with a two-sentence description, leaving unclear what improves over existing methods and which component accounts for it. Third, Figure 2 does not support the claim of superiority across criteria: against the strongest baseline, SoulX-FlashTalk, ties dominate every category. Finally, although temporal evidence and observation is critical for this application, the paper shows only frames at a few samples, with neither side-by-side video comparisons nor a temporal-consistency metric against the baselines.

For these reasons, I maintain my score(2: Weak rejection).

AuthorsOpenReview ↗

(part 1/2)

We thank the reviewer for the follow-up. We have uploaded an updated revision, and table, figure and section numbers in this answer refer to it.

W2. Memory noising is not a new concept compared to Live Avatar.

Thank you for this point. We have clarified the description of Live Avatar in Section 2 to state that its cache is kept noisy by design (History Corrupt). We do not claim noisy history as new. It goes back to Diffusion Forcing, and Live Avatar applies it to the rolling key-value cache of past blocks at the level of the current sampler step. Our memory noise applies it in a different regime. Only the two latent frames carried between chunks are corrupted, at a small level drawn from U(0, 0.1) in training and fixed at inference to s = 0.05, the mean of the training range. The level stays the same across the denoising steps rather than following them, and the model receives it through the time embedding of the memory tokens.

The reviewer notes that Live Avatar found step-matched noise better than a fixed level. That comparison does not carry over to our setting. With our 3-step sampler (levels 1.0, 0.839 and 0.617, Section 5.2), step-matched noise would keep the two carried frames at level 0.617 or higher on every step, more than six times the largest level we train with. Unlike Live Avatar, our model keeps no cache of past blocks, so these two frames are the only state that carries motion across the boundary. Among the levels we tested, s = 0.05 is best (Table 3): CSIM 0.859, against 0.837 at s = 0.1, 0.807 at s = 0 and 0.825 without noise in training. The gap grows with the length of the video. On the two-minute cross-driven test (Table 25), the deployed model reaches CSIM 0.829, against 0.745 for the model trained with a clean memory, a margin of 0.084 against 0.034 on EMTD-30s.

Table 3 is not a component ablation.

We respectfully disagree that Table 3 is not a component ablation. Table 3, together with Table 4 that ablates partial pose spans, covers the components we list as contributions in Section 1: the area-adaptive face loss, memory noise, partial pose spans and the x1.5 decoder. Three of these ablations change training, not inference. The face-loss rows compare three stage-1 models trained with the area-adaptive weight, with a fixed weight, and without the face term. The memory-noise block runs the noise-trained model at inference levels s = 0, 0.05 and 0.1, and adds a model trained with a clean memory. Table 4 compares the deployed model with a control trained without partial pose spans. The face-loss rows are stage-1 runs because the face term is not used in stage-2 training, and the stage-1 model enters stage 2 as the teacher and the initialization of the student (Section 5.2).

We take the regression term from SoulX-FlashHead and the background term from the public training code of SoulX-FlashTalk (Section 5.2), and they are not among our contributions. Following the reviewer's request, we have also ablated them, training the student without each term on the deployed recipe (EMTD-30s, 95% confidence intervals). Besides the five metrics, we report background drift, the mean absolute difference from the first generated frame over static background pixels, in 8-bit units, at the end of the clip:

RunCSIM ↑Sync-C ↑Sync-D ↓IQA ↑ASE ↑Drift ↓
Full objective (deployed)0.859±0.0117.65±0.248.02±0.144.85±0.043.73±0.071.04±0.09
w/o background loss0.842±0.0117.62±0.238.07±0.144.85±0.053.73±0.071.75±0.47
w/o latent regression0.841±0.0117.62±0.238.08±0.134.84±0.043.72±0.071.04±0.08

Without either term, CSIM drops by 0.017 and 0.018, beyond its confidence interval of ±0.011. Without the background term, background drift also rises from 1.04 to 1.75, while the regression term leaves it unchanged. The other metrics stay within their intervals. We have added these results as a separate table in Appendix G.7 (Table 29), referenced in Section 5.5.

AuthorsOpenReview ↗

(part 2/2)

Figure 2: ties dominate against SoulX-FlashTalk.

We do not claim that Edge-Avatar wins every category. Section 5.5 and Appendix E.6 state that most pairs are ties, and that in none of the 21 categories across the three studies does the baseline receive more votes than Edge-Avatar. Against SoulX-FlashTalk, in five of the seven categories our share is higher with non-overlapping 95% intervals (Table 15): lip synchronization 32% [24, 42] against 2% [1, 7], motion naturalness 28% [20, 37] against 10% [6, 17], teeth and mouth 22% [15, 31] against 5% [2, 11], identity preservation 15% [9, 23] against 0% [0, 4], and background stability 12% [7, 20] against 0% [0, 4]. In visual quality the intervals overlap, and in camera stability all judgements are ties.

These ties should be read against the cost of the two systems. SoulX-FlashTalk is a 14B model that needs a node of 8×H800 to reach real time, as its authors report. Edge-Avatar has 2B parameters and reaches real time on a single consumer RTX 5090 (Table 2). Edge-Avatar ties this seven-times larger model on eight data-center GPUs in most pairs and is preferred significantly in five of the seven categories.

Figure 3 shows one subject, and there are no side-by-side videos or temporal-consistency metric.

Figure 3 shows a CelebV-HQ clip, and Appendix E.7 complements it with Figures 7 to 10, three EMTD-30s clips and one HDTF-30s clip generated by the same four systems. We have added the videos behind these figures to the anonymous repository, and the caption of Figure 3 now points to them. For each of the five clips shown in Figures 3 and 7 to 10, the repository holds the reference image and the videos generated by Edge-Avatar, SoulX-FlashTalk, SoulX-LiveAct and Avatar-Forever from the same reference and speech, so the generations can be viewed and compared directly. It also holds the same four-way comparison on 30 further clips from CelebV-HQ, HDTF-30s and the cross-driven test, where an EMTD reference image is driven by HDTF speech, the longest of them up to five minutes.

For temporal behaviour we rely on the side-by-side studies, where annotators watch whole videos and judge visual quality, motion naturalness, camera stability and background stability, and on the videos now in the repository. VBench-style consistency scores separate systems of this quality poorly. In Table 1 of the SoulX-FlashTalk report, Motion-S lies between 99.25 and 99.76 and Temporal-F between 98.86 and 99.89 for all systems, and Ditto, which warps the reference face rather than generating it, scores highest on both.

Reviewer_LbBVOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Edge-Avatar, a real-time audio-driven human video generation model trained from Kandinsky 5.0 Video foundation models with 2B parameters. The model is trained with a short-chunk teacher-forcing stage followed by distribution-matching distillation with multi-chunk self-forcing to a 3-step sampler. A two-frame rolling memory with injected noise, an area-adaptive face loss, and pose conditioning trained on partial prefix/suffix spans so a gesture can begin or end mid-chunk. Measured on one RTX 5090, the loop costs 764 ms of compute per second of video (real-time factor 1.31x, 0.98 s per 1.28-s chunk, 18.4 GB peak; Table 2). On frozen 50-clip splits of HDTF-30s, EMTD-30s, CelebV-HQ, and a 2-minute cross-driven stress test, the proposed method beats all eight open-source streaming avatar systems (1.3B-22B).

Critical strengths
  • Honest, well-instrumented engineering accounting. Table 2 gives a full latency budget, and every deployment choice is quantified by an ablation: FP8 + sparse attention + distilled decoder together cost only 0.005 CSIM and 0.1 Sync-C (Table 17); SPD costs 0.029 CSIM and is left off by default for exactly that reason (Table 18); the trained x1.5 decoder beats naive interpolation by +0.17 IQA and +0.37 ASE (Table 19).

  • The paper is transparent about what "real time" excludes, in the text itself (conditioning encoder outside the loop; ASR/LLM/TTS/compositing/transport acknowledged, p. 8), and about what is not released (training code, training corpus).

Critical weaknesses
  • The qualitative results are highly unconvincing. No visual results are presented in the main paper, and Appendix Figure 8 shows only a single female subject. Furthermore, the anonymous repository contains only the same example, raising serious concerns about the robustness and generalization capability of the proposed method. The limited qualitative evidence suggests that the method may be overfitted to this specific case, and its effectiveness across diverse subjects and scenarios remains unsubstantiated. Given the lack of convincing evidence demonstrating generalization, I have significant concern about the paper's contributions and lean toward rejection.

  • The claimed aggregate lead is thinner, and less universal, than the text asserts. Against the nearest competitor SoulX-FlashTalk (14B, 8x H800), the pooled identity margin is 0.865 vs 0.857 CSIM (0.008), IQA 4.60 vs 4.50, ASE 3.55 vs 3.43 (Table 3). The main text says Edge-Avatar is "first in every benchmark taken separately" on identity (p. 9), but Table 9 shows SoulX-FlashHead Pro at 0.918 CSIM vs 0.908 on HDTF-30s, and Table 12 shows FlashTalk and FlashHead Pro both at 0.833 vs 0.829 on the cross-driven test; Sync-D is lost to SoulX-LiveAct overall (7.99 vs 7.94) and on two of four benchmarks (Tables 9, 12); on HDTF, IQA is led by Live Avatar (4.62 vs 4.57) and LiveTalk (4.61).

Questions for authors

None

1 comment

We thank the reviewer for the valuable feedback and for the helpful comments on our quantitative and qualitative results. Table, figure and section numbers in this answer refer to the revised version.

The limited qualitative evidence suggests that the method may be overfitted to this specific case, and its effectiveness across diverse subjects and scenarios remains unsubstantiated…

We agree that the paper shows too few visual examples. However, the model is not overfitted to one subject. The single subject in Figure 11 and the repository was chosen to demonstrate the released code, while all metrics are measured on many different people: 50 clips per benchmark from HDTF, EMTD and CelebV-HQ plus a two- and five-minute cross-driven test, with benchmark speakers excluded from training by source video and by ArcFace identity (Appendix B.6). The three side-by-side studies against SoulX-FlashTalk, SoulX-LiveAct and Avatar-Forever also run on EMTD clips. We have added diverse generated videos and qualitative comparison videos from the paper to the anonymous repository. The revision adds a qualitative comparison with the three closest baselines to the main text (Figure 3) and four more on EMTD and HDTF clips to Appendix E.7 (Figures 7 to 10).

The claimed aggregate lead is thinner, and less universal, than the text asserts. Against the nearest competitor SoulX-FlashTalk (14B, 8x H800), the pooled identity margin is 0.865 vs 0.857 CSIM (0.008), IQA 4.60 vs 4.50, ASE 3.55 vs 3.43 (Table 3)...

The clause on p. 9 of the original version referred to lip-sync confidence, on which Edge-Avatar is first in each of Tables 10 to 14. The lead on identity is on the average over the four benchmarks (Table 2, 0.865 vs 0.857), not in each one. Edge-Avatar is also first on both aggregates in each of Tables 10 to 14. We have rephrased the sentence in Section 5.5. The per-benchmark numbers the reviewer cites are correct, and Section 5.5 now states these exceptions explicitly.

We agree that the margin over SoulX-FlashTalk is small. SoulX-FlashTalk is seven times larger (14B against 2B) and reaches real time on 8x H800, as its authors report, while Edge-Avatar runs in real time on a single RTX 5090. Among the baselines that reach real time on a single consumer GPU, LiveTalk and SoulX-FlashHead Lite, the gap is large: averaged over the four benchmarks, CSIM is 0.865 against 0.677 and 0.767, and Sync-C is 7.68 against 3.95 and 6.01 (Table 2).

We hope the added visual examples and the clarifications above address the reviewer's concerns, and we would be grateful if the reviewer would consider revisiting the score.

Comments on the paper

3 comments
Area_Chair_KCBiOpenReview ↗

Please read the response from authors and discuss

Dear reviewer fellows,

The author rebuttal has been posted. The rebuttal period is a critical part of our review process, allowing authors to clarify misunderstandings and address specific concerns.

Please take a moment to read the author response, engage in the discussion by replying to the authors' responses, and update your review if the rebuttal changes your assessment.

Your active participation is vital to ensuring a fair and thorough evaluation of the submissions. A constructive discussion helps us make the most informed decisions.

Thank you for your time and valuable contributions to the review process.

Best regards,

Your AC

Dear Area Chair and Reviewers,

Thank you again for your thoughtful reviews. We have uploaded a revised version of the paper that incorporates all the changes promised in our responses and addresses the reviewers' comments. In particular, it adds Related Work and Limitations sections, qualitative comparisons, the consolidated ablations and side-by-side studies in the main text, the pose-control, latency, memory-noise and face-loss experiments, and confidence intervals, rephrases Contribution 4, and moves deployment details to the appendix. Additionally, we have added diverse generated videos and the qualitative comparison videos from the paper to the anonymous repository.

We hope the revision addresses the reviewers' concerns and would be grateful if it could be taken into account in the final assessment.

Dear Area Chair and Reviewers,

We have uploaded an updated revision following the second round of discussion. It adds an ablation of the regression and background terms of the student objective (Appendix G.7, referenced in Section 5.5), clarifies the description of Live Avatar in Section 2, and links the Figure 3 caption to the qualitative comparison videos in the anonymous repository, which now also include 30 further comparison clips.

We hope the update addresses the remaining concerns and would be grateful if it could be taken into account in the assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.