- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents FaceGAN, a single-pass GAN for real-time audio-driven facial animation. It combines causal audio conditioning with acausal noise shaping to improve temporal smoothness without additional audio latency. Experiments show competitive generation quality and substantially faster inference than the evaluated baselines.
Critical strengths
-
Separating what is acausal (synthetic noise, whose future can be sampled now) from what is causal (observed audio) is a clever design, and it is exactly what real-time deployment needs.
-
The ablations are thorough: causal vs. acausal noise, window size, lookahead, discriminators, and each loss term are all checked.
-
Evaluation goes beyond generation quality and measures latency, lip-sync, motion variance, and sample diversity as well.
Critical weaknesses
-
The theory covers a much narrower setting than FaceGAN's: scalar, stationary, unconditional, time-invariant, and a strict perfect stopband. FaceGAN is 137-D and conditioned on non-stationary audio, and facial motion needs no strict spectral null, so the theorem supports an intuition rather than establishing the jitter–diversity trade-off. Also, reading Eq. (3) as "how much input noise variance survives" still needs a scale/gain derivation.
-
The 3.3 ms covers only the generator on an H200 — no Mimi encoder, no renderer, no end-to-end latency — and "generates indefinitely without drift" is asserted architecturally but never verified on long continuous runs.
-
Sec. 5.5 says one frame per call, but App. D says one 5-frame block per forward pass. This inconsistency directly affects the frame-level streaming claim (Table 1) and how latency should be computed.
Questions for authors
-
Could you either narrow the theoretical claim to the setting it actually proves, or extend it to the vector-valued, audio-conditioned case? And could you justify the scale interpretation of Eq. (3)?
-
Which is correct — one frame per call (Sec. 5.5) or one 5-frame block per forward pass (App. D)? What is the true frame-level latency?
-
What is the end-to-end latency including the Mimi encoder and the renderer, ideally also on a consumer GPU?
-
Is there any empirical evidence for "without drift" on long streams, e.g. minutes-long continuous generation?
6 comments
Reply for Q2 & Q3
Q2 Thank you for highlighting this issue. The latency numbers have been reported with one frame per call NOT one forward pass per 5 frame block, that is in error, and Table 8 (App. D) needs to be updated.
Q3 The Mimi encoder and renderer are not part of the scope of this paper, we are however using a CUDA graph compiled streaming implementation of the encoder in inference mode for this, and are able to achieve 1.1ms/1.9ms latency on H200/A100 respectively per call (corresponding to 80ms of input audio). Since the Mimi encoder acts on 80ms of input audio, that causes the effective end-to-end latency to be ~84ms for our case (40ms for the audio input chunk used by our network, and another 40ms for the lookahead, ~1.1ms for Mimi encoding, ~3ms model forward pass). We have not performed any ablations on the renderer latency as that is again out of scope of our analysis as it would ideally be equivalent across all the baselines we are comparing against, depending on the choice of the renderer. We do not currently have access to consumer GPUs and a means to report those numbers immediately, but we can report the per call latency for our model on an older generation, widely available A100 GPUs to be 6.4ms.
Reply for Question 4
Q4 We thank reviewer for raising this question. To assess drift in predictions, we ran a looped audio track (of length 10 seconds) 20 times (with known GT motion codes) repeatedly, and compared the predictions from the first and last loop as seen in the tables below:
| FaceGAN (streaming) | First 5 loops | Last 5 loops |
|---|---|---|
| Sync Score ↑ | 0.931 | 0.927 |
Sync Score of FaceGAN (higher is better) over a continuous 200 s stream of one 10 s validation clip repeated 20 times; ground-truth motion scores 0.926. This compares the Sync Score of the prediction against the audio between the first 5 loops and last 5 loops of the audio track.
Reply for Q1
First of all, we would like to thank the reviewer for the careful reading, thoughtful feedback, and constructive suggestions.
Q1 We thank the reviewer for the concrete suggestion. We extend (3), which already needs no exact stopband, to FaceGAN's setting: finite-length clips conditioned on the audio, and each of the 137 output channels or any fixed linear combination of them. For fixed audio a, FaceGAN is an unconditional generator driven by i.i.d. noise, so the audio-conditioned model is a family of unconditional generators, one per audio input. Noise is the only source of diversity in each (without it the model regresses to the conditional mean), so we apply (3) to the diversity y−E[y∣a]. Unlike generator (1), each is time-varying, since its audio input changes every frame, so we state (3) for a finite clip. For a clip and a channel, let Σa=Cov(y1:n∣a) over noise draws, and let Pa(ωk)=fk∗Σafk, with fk the unit-norm DFT vectors, be the draw-averaged periodogram of y−E[y∣a] (estimated from our 64 draws per clip). With et2(a) the best linear one-step prediction error of yt given y1:t−1 and a, Cholesky and Hadamard's inequality in the DFT basis give ∏tet2(a)=detΣa≤∏kPa(ωk)≤(εδmaxkPa(ωk))n whenever Pa≤εmaxPa on a fraction δ of frequencies. Since et2(a) varies from frame to frame, the bound is on its geometric mean over the clip: (∏tet2(a))1/n≤εδmaxkPa(ωk). This is the finite-clip analogue of Eq. (3) for each audio input: by (2), σ∞2 is itself the geometric mean of the spectrum, and for generator (1), which is stationary (App. A), the clip's geometric-mean innovation tends to σ∞2 as n→∞ (Szegő), recovering (2)–(3). In a generator causal in its noise, et2(a) also bounds what the newest noise frame adds, on average over past noise and given the audio. For smooth motion, the newest frame therefore adds little in geometric mean, and diversity must come mostly from past noise. After relabelling zt′=zt+128, this applies to ours as well, so the architecture comparison rests on Table 4 (see our reply to 8hVx). On scale, the reviewer is right that "input noise variance" was the wrong unit: (3) is relative to the spectral peak. In the paper's example (ε=10−4, δ=0.9), the geometric-mean innovation is at most 0.025% of the peak, or 0.25% of the mean variance across draws for a flat passband. The revised Sec. 3 will present this bound and the corrected scale.