- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Edge Avatar addresses real time speech driven avatar generation with pose control. It converts a two billion parameter video model into a three step generator, combining an adaptive face loss, memory noising, and partial span pose conditioning to support continuous generation on a single consumer GPU.
Critical strengths
The model generates video faster than playback on a single RTX 5090 and achieves competitive identity preservation and lip synchronization across multiple benchmarks. The paper provides detailed measurements of memory usage, component latency, and quality changes introduced by deployment optimizations.
The evaluation includes two minute and five minute videos, alongside human comparisons with three strong baselines. The memory noising ablation shows improved identity preservation, supporting the usefulness of this component.
Critical weaknesses
Main Weaknesses
Table 15 compares different pose entry times using the same model, without a control trained without partial span pose conditioning. It therefore does not isolate the contribution of this training design. The evaluation measures only changes between frames, without pose tracking error or gesture completion rates, leaving open whether smoother transitions come at the cost of reduced motion amplitude or control accuracy.
Table 2 measures only denoising and decoding, excluding conditioning encoding performed once per utterance. Each chunk represents 1.28 seconds of video, but the paper does not report audio buffering requirements, first frame latency, or gesture response time when inputs arrive incrementally. The results support real time generation throughput, while the latency of online interaction remains insufficiently characterized.
The face loss ablation in Table 14 is conducted only at the first autoregressive stage, and all variants inherit a base model already trained with the adaptive loss. There is no comparison after distillation or breakdown by face size. Its contribution to lip synchronization for small faces and to preventing tighter framing therefore remains unclear.
Minor Weaknesses
Samples with failed metric computation are excluded, but the effective sample counts for each model are not reported. Confidence intervals are also missing for the main metrics and human preference results.
Questions for authors
Please compare models trained with and without partial span pose conditioning, reporting pose tracking accuracy, gesture completion, and transition smoothness.
Please report first frame latency including conditioning encoding, specify the required future audio context, and measure response time to gesture requests during generation.
Please evaluate the adaptive face loss after distillation and report lip synchronization and framing changes across different face sizes.
2 comments
(part 2/2)
(part, 2/2)
Table R1: First-frame latency on the RTX 5090, ms, medians of 10 utterances. Per-chunk: the first chunk of speech encoded alone. Released: one Thinker pass over the prompt and a 30 s utterance.
| Configuration | Per-chunk: Audio | Per-chunk: DiT | Per-chunk: Decode | Per-chunk: First frame | Released: Joint pass, 30 s | Released: First frame |
|---|---|---|---|---|---|---|
| SD (512x768) | 7 | 718 | 217 | 942 | 124 | 1059 |
| SD + SPD | 7 | 529 | 217 | 754 | 123 | 871 |
| HD (768x1152) | 7 | 717 | 317 | 1041 | 122 | 1155 |
| HD + SPD | 7 | 532 | 330 | 868 | 126 | 987 |
Future audio context. The DiT sees only its chunk's speech tokens, but the audio encoder attends within 2 s blocks, so the released features need speech to the end of the current block: 0 to 2 s beyond the chunk, 1 s on average, 2.0 s of speech for the first chunk. Block-aligned encoding reproduces these features (cosine 0.988). Per-chunk encoding needs no look-ahead but changes them (cosine 0.606), and we have not evaluated video generated from them, so we do not claim zero look-ahead.
Gesture response. With requests at random times during real-time speech (Table R2), the first frame generated under the pose appears about 2.6 s after the request with either encoding under the released rule (pose on the last two latents of the next chunk, generated one chunk ahead of the display). Entry on the first latent gives 1.6 s at the price of the seam of Table 21. These are now in Section 5.4.1 and Appendix F.2. With a whole TTS utterance and up to three chunks queued, the response grows to 4.75 s unless queued chunks are regenerated with the pose (rewind, 2.95 s).
Table R2: Response to a gesture request, seconds from the request, mean (min to max), 3 to 5 sessions of 30 s. First posed frame: first displayed frame generated under the pose. Visible onset: first frame that differs from a rollout without the pose. First frame: from the utterance start.
| Scenario | Audio encoding | Requests | First frame, s | First posed frame, s | Visible onset, s |
|---|---|---|---|---|---|
| Real-time speech, entry on the last 2 latents (released) | per chunk | 7 | 2.11 | 2.60 (1.97–3.16) | 2.06 (1.37–2.72) |
| block-aligned | 7 | 4.15 | 2.63 (2.31–3.06) | 2.10 (1.81–2.42) | |
| Real-time speech, entry on the first latent | per chunk | 5 | 2.11 | 1.59 (1.01–1.98) | 1.81 (1.05–2.22) |
| block-aligned | 5 | 4.15 | 1.58 (1.35–1.89) | 1.80 (1.49–2.17) | |
| Utterance at once, queue (up to 3 chunks) | per chunk | 5 | 0.98 | 4.75 (3.40–5.61) | 4.22 (2.88–5.13) |
| Utterance at once, rewind of queued chunks | per chunk | 5 | 0.99 | 2.95 (2.25–3.40) | 2.39 (1.65–2.88) |
Please evaluate the adaptive face loss after distillation and report lip synchronization and framing changes across different face sizes.
Why stage 1. Stage 2 (DMD) uses no face term, and its student, teacher and critic all start from the stage-1 EMA. The face loss reaches the deployed model only through these weights, so we ablate it at stage 1.
Base model and face size. We compare two base models trained with memory, pose and silent clips that differ only in the face-loss weight, adaptive or fixed. We evaluate them on EMTD-30s and on HDTF-30s, whose faces are much larger, and split HDTF into two halves by face area. gen/ref is the face area in the generated video over that in the reference, so 1 means the framing is kept.
Table R6: Face loss in the base model.
| Dataset | Face area, % of frame | Run | CSIM ↑ | Sync-C ↑ | Sync-D ↓ | IQA ↑ | ASE ↑ | gen/ref →1 |
|---|---|---|---|---|---|---|---|---|
| EMTD | 2.1 to 5.6, mean 3.6 (n = 50) | Adaptive weight | 0.750 | 7.6 | 7.91 | 4.77 | 3.57 | 1.038 |
| Without the adaptive weight | 0.739 | 7.56 | 7.94 | 4.76 | 3.56 | 1.097 | ||
| HDTF | 12.7 to 20.6, mean 16.9 (n = 25) | Adaptive weight | 0.845 | 7.92 | 7.78 | 4.37 | 3.39 | 1.011 |
| Without the adaptive weight | 0.835 | 7.56 | 8.02 | 4.36 | 3.38 | 1.052 | ||
| 21.1 to 37.3, mean 27.7 (n = 25) | Adaptive weight | 0.863 | 7.92 | 7.80 | 4.66 | 3.66 | 1.011 | |
| Without the adaptive weight | 0.851 | 7.87 | 7.82 | 4.66 | 3.67 | 1.031 |
The adaptive weight gives higher CSIM in every group and equal or better lip sync. The Sync-C gain is largest on the smaller HDTF faces.
As for framing, with the fixed weight the generated face grows relative to the reference by 9.7% on EMTD and by 5.2% and 3.1% on HDTF, against 3.8%, 1.1% and 1.1% with the adaptive weight. They are now in Table 20.
Minor Weaknesses
In our runs the effective number of clips matched the full set for every model and metric, with no clip dropped, as Appendix E now states. The revision adds 95% confidence intervals for all metrics and the side-by-side preferences (Appendix E).
We thank the reviewer again for the suggestions.
(part 1/2)
(part, 1/2)
We thank the reviewer for the positive assessment and for the constructive questions on pose control, latency and the face loss. Table, figure and section numbers in this answer refer to the revised version.
Control. We trained a control without partial spans: in all training stages, the pose always covers the whole chunk. Everything else matches the deployed model. We evaluate both models on EMTD-30s. Each clip is driven by its own pose track, and we extract keypoints from the generated videos with the same ViTPose-H. No clip is excluded.
Transition smoothness. When the pose is switched on inside the chunk, only our model stays smooth at the frame where it is switched on (Table R3).
Table R3: Smoothness of switching the pose on inside the chunk, EMTD-30s, 25 clips, measured at the switch frame.
To check that this is a jerk and not regular motion, we compare the switch frame with the same frame of the same clip when the pose is always on. For the control, spike and bend rise clearly. For our model they change only within noise (Table R4).
Table R4: Spike and bend at the frame where the pose is switched on inside the chunk ("pose switched on"), compared with the same frame of the same clip in a video where the pose is fed from the first frame ("pose always on"). Δ is the difference. EMTD-30s, 25 clips. Our model's values come from a re-run, so they differ slightly from Table R3.
Tracking accuracy, gesture completion and amplitude. With the pose on from the first frame, both models follow it equally well and move the hands as widely as the track. When the pose is switched on later, our model reaches the hand positions of the track in more gestures: 96.9% against 83.3% (Table R5).
Table R5: How well the model follows the pose, EMTD-30s, 50 clips; last column: the pose is switched on inside a chunk during a gesture, 30 clips. Distances are measured in the person's shoulder width, so they do not depend on frame size or distance to the camera. PCK: share of keypoints within the threshold of the pose track. Amplitude: how wide the hand moves compared with the track (1 means the same). Completion: share of gestures in the track that the hand reaches. No pose: the same model without a pose track.
Lip sync and quality are preserved: CSIM 0.814 vs 0.810, IQA 4.82 for both. Sync-C is 0.08 lower (7.64 vs 7.72, about 1%).
The revision states that the original pose-entry table (now Table 21) measures at the chunk boundary, and adds Tables R3 to R5 and the lip-sync numbers to Appendix G.2 (Tables 22 to 24), with the main results in Table 4.
Latency of online interaction. Table 1 reports throughput, not live latency. We measured all three on the same RTX 5090 (configuration of Table 1). The released code encodes the whole utterance first, so live sessions with real-time speech apply the audio tower incrementally, per chunk or per 2 s block.
First-frame latency. Encoding is cheap (Table R1). From the speech of the first chunk (1.16 s, 29 speech tokens) being available to its first frame: 0.94 s in SD, the same compute for either incremental encoding. In live sessions with real-time speech the first frame appears 2.11 s after utterance start with per-chunk encoding (1.16 s of speech plus the compute) and 4.15 s with block-aligned encoding (2.0 s of speech, the compute, a 1.2 s display pre-roll against later block waits); about 1 s with the whole utterance from a TTS. This is the usual case for a conversational agent, whose TTS runs ahead of playback, and the 2.11 s and 4.15 s apply only to speech arriving strictly in real time.