acceptodds
Under review as a conference paper at ICLR 2027

No Distillation Needed: Single-Pass Real-Time Talking Heads via Acausal Noise Shaping

Abstract

Audio-driven facial animation underpins real-time avatars, telepresence, and embodied virtual agents. And it must run online: each frame emitted from audio observed up to the current time, at interactive rates. Recent progress is dominated by diffusion models, which need many network evaluations per sample and are therefore a poor fit for streaming. We argue the cost is unnecessary in this domain. Audio-conditioned facial motion occupies a comparatively low-dimensional manifold, a regime where a single-pass GAN suffices. The obstacle is not capacity but stochastic structure. We show that a causal, time-invariant generator driven by i.i.d. noise cannot suppress its output spectrum over a band without collapsing its per-step innovation. We proposed FaceGAN, which dissolved the limitation by shaping the noise pathway acausally. Because the driving noise is synthetic, its future can be sampled now, so the audio-to-expression path stays causal, and the model supports fully causal operation. FaceGAN emits expression and head pose in a single forward pass per frame and matches or outperforms state-of-art approaches in generation quality. Being feed-forward with bounded attention windows, it generates indefinitely without drift.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-driven facial animation, generative models, GANs, real-time streaming inference, spectral analysis of generative processes

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T712OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents FaceGAN, a single-pass GAN for real-time audio-driven facial animation. It combines causal audio conditioning with acausal noise shaping to improve temporal smoothness without additional audio latency. Experiments show competitive generation quality and substantially faster inference than the evaluated baselines.

Critical strengths
  1. Separating what is acausal (synthetic noise, whose future can be sampled now) from what is causal (observed audio) is a clever design, and it is exactly what real-time deployment needs.

  2. The ablations are thorough: causal vs. acausal noise, window size, lookahead, discriminators, and each loss term are all checked.

  3. Evaluation goes beyond generation quality and measures latency, lip-sync, motion variance, and sample diversity as well.

Critical weaknesses
  1. The theory covers a much narrower setting than FaceGAN's: scalar, stationary, unconditional, time-invariant, and a strict perfect stopband. FaceGAN is 137-D and conditioned on non-stationary audio, and facial motion needs no strict spectral null, so the theorem supports an intuition rather than establishing the jitter–diversity trade-off. Also, reading Eq. (3) as "how much input noise variance survives" still needs a scale/gain derivation.

  2. The 3.3 ms covers only the generator on an H200 — no Mimi encoder, no renderer, no end-to-end latency — and "generates indefinitely without drift" is asserted architecturally but never verified on long continuous runs.

  3. Sec. 5.5 says one frame per call, but App. D says one 5-frame block per forward pass. This inconsistency directly affects the frame-level streaming claim (Table 1) and how latency should be computed.

Questions for authors
  1. Could you either narrow the theoretical claim to the setting it actually proves, or extend it to the vector-valued, audio-conditioned case? And could you justify the scale interpretation of Eq. (3)?

  2. Which is correct — one frame per call (Sec. 5.5) or one 5-frame block per forward pass (App. D)? What is the true frame-level latency?

  3. What is the end-to-end latency including the Mimi encoder and the renderer, ideally also on a consumer GPU?

  4. Is there any empirical evidence for "without drift" on long streams, e.g. minutes-long continuous generation?

6 comments
AuthorsOpenReview ↗

Reply for Q1

First of all, we would like to thank the reviewer for the careful reading, thoughtful feedback, and constructive suggestions.

Q1 We thank the reviewer for the concrete suggestion. We extend (3), which already needs no exact stopband, to FaceGAN's setting: finite-length clips conditioned on the audio, and each of the 137 output channels or any fixed linear combination of them. For fixed audio aa, FaceGAN is an unconditional generator driven by i.i.d. noise, so the audio-conditioned model is a family of unconditional generators, one per audio input. Noise is the only source of diversity in each (without it the model regresses to the conditional mean), so we apply (3) to the diversity y−E[y∣a]y - \mathbb{E}[y \mid a]. Unlike generator (1), each is time-varying, since its audio input changes every frame, so we state (3) for a finite clip. For a clip and a channel, let Σa=Cov(y1:n∣a)\Sigma_a = \mathrm{Cov}(y_{1:n} \mid a) over noise draws, and let Pa(ωk)=fk∗ΣafkP_a(\omega_k) = f_k^* \Sigma_a f_k, with fkf_k the unit-norm DFT vectors, be the draw-averaged periodogram of y−E[y∣a]y - \mathbb{E}[y \mid a] (estimated from our 64 draws per clip). With et2(a)e_t^2(a) the best linear one-step prediction error of yty_t given y1:t−1y_{1:t-1} and aa, Cholesky and Hadamard's inequality in the DFT basis give ∏tet2(a)=det⁡Σa≤∏kPa(ωk)≤(εδmax⁡kPa(ωk))n\prod_t e_t^2(a) = \det \Sigma_a \le \prod_k P_a(\omega_k) \le \big(\varepsilon^\delta \max_k P_a(\omega_k)\big)^n whenever Pa≤εmax⁡PaP_a \le \varepsilon \max P_a on a fraction δ\delta of frequencies. Since et2(a)e_t^2(a) varies from frame to frame, the bound is on its geometric mean over the clip: (∏tet2(a))1/n≤εδmax⁡kPa(ωk)\big(\prod_t e_t^2(a)\big)^{1/n} \le \varepsilon^\delta \max_k P_a(\omega_k). This is the finite-clip analogue of Eq. (3) for each audio input: by (2), σ∞2\sigma^2_\infty is itself the geometric mean of the spectrum, and for generator (1), which is stationary (App. A), the clip's geometric-mean innovation tends to σ∞2\sigma^2_\infty as n→∞n \to \infty (Szegő), recovering (2)–(3). In a generator causal in its noise, et2(a)e_t^2(a) also bounds what the newest noise frame adds, on average over past noise and given the audio. For smooth motion, the newest frame therefore adds little in geometric mean, and diversity must come mostly from past noise. After relabelling zt′=zt+128z_t' = z_{t+128}, this applies to ours as well, so the architecture comparison rests on Table 4 (see our reply to 8hVx). On scale, the reviewer is right that "input noise variance" was the wrong unit: (3) is relative to the spectral peak. In the paper's example (ε=10−4\varepsilon = 10^{-4}, δ=0.9\delta = 0.9), the geometric-mean innovation is at most 0.025% of the peak, or 0.25% of the mean variance across draws for a flat passband. The revised Sec. 3 will present this bound and the corrected scale.

AuthorsOpenReview ↗

Reply for Q2 & Q3

Q2 Thank you for highlighting this issue. The latency numbers have been reported with one frame per call NOT one forward pass per 5 frame block, that is in error, and Table 8 (App. D) needs to be updated.

Q3 The Mimi encoder and renderer are not part of the scope of this paper, we are however using a CUDA graph compiled streaming implementation of the encoder in inference mode for this, and are able to achieve 1.1ms/1.9ms latency on H200/A100 respectively per call (corresponding to 80ms of input audio). Since the Mimi encoder acts on 80ms of input audio, that causes the effective end-to-end latency to be ~84ms for our case (40ms for the audio input chunk used by our network, and another 40ms for the lookahead, ~1.1ms for Mimi encoding, ~3ms model forward pass). We have not performed any ablations on the renderer latency as that is again out of scope of our analysis as it would ideally be equivalent across all the baselines we are comparing against, depending on the choice of the renderer. We do not currently have access to consumer GPUs and a means to report those numbers immediately, but we can report the per call latency for our model on an older generation, widely available A100 GPUs to be 6.4ms.

AuthorsOpenReview ↗

Reply for Question 4

Q4 We thank reviewer for raising this question. To assess drift in predictions, we ran a looped audio track (of length 10 seconds) 20 times (with known GT motion codes) repeatedly, and compared the predictions from the first and last loop as seen in the tables below:

FaceGAN (streaming)First 5 loopsLast 5 loops
Sync Score ↑0.9310.927

Sync Score of FaceGAN (higher is better) over a continuous 200 s stream of one 10 s validation clip repeated 20 times; ground-truth motion scores 0.926. This compares the Sync Score of the prediction against the audio between the first 5 loops and last 5 loops of the audio track.

Reviewer_x6NEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FaceGAN, a single-pass, real-time generator of facial expression and head pose from streaming audio. It combines a bidirectional noise-processing stack with a causal audio-conditioning stack, motivated by a spectral limitation of causal generators driven by i.i.d. noise. The method is evaluated on one large dataset against four baselines, with a user study, ablations, and latency measurements. FaceGAN achieves competitive quality at high inference speed.

Critical strengths
  1. A well-motivated design for streaming generation. The two-stack design separates bidirectional noise processing from causal audio conditioning, adding temporal structure to the noise without requiring additional audio lookahead. At matched attention width, the acausal noise stack achieves higher sample diversity than the causal version.

  2. A good balance between quality and efficiency. FaceGAN is competitive with the baselines in Table 2, with a clearly higher Sync Score, and its generator forward pass takes 3.3 ms on an H200.

  3. The ablations show that the conditional discriminator is essential for lip synchronisation and that the diversity loss is essential for sample diversity.

Critical weaknesses
  1. The title and introduction argue that diffusion distillation is unnecessary, and Sec. 5.2 attributes the quality loss of distilled models to mode collapse. However, the paper does not compare against a few-step diffusion sampler or a distilled one-step diffusion model trained on the same data. DiffPoseTalk is run with 500 steps only. The quality–latency trade-off against accelerated or distilled diffusion models therefore remains unclear.

  2. The main text describes frame-level streaming inference, with one frame per call and 40 ms of lookahead (Sec. 5.5). However, Table 8 in Appendix D states that FaceGAN emits a 5-frame block per forward pass.

  3. The efficiency comparison is not sufficient to show an advantage over efficient alternatives. Runtime and latency are reported only for FaceGAN. The baselines are compared only by step counts in Table 1, not by measured latency or throughput on the same hardware.

  4. The results in Table 5 show substantial run-to-run variability, with standard deviations of 0.52 for FED and 0.32 for FPD, comparable to some of the gaps between methods in Table 2. It would be helpful to report the variance in Table 2.

Questions for authors

Clarification:

  1. Section 5.5 describes one frame per call, whereas Table 8 counts one forward pass per 5-frame block. Which mode is used in deployment, and how do the step counts in Tables 1 and 8 relate to the reported latency?
  2. FaceGAN achieves a substantially higher Sync Score, but the user study shows no clear lip-sync preference over Fallingwater or DiffPoseTalk. How do the authors interpret this difference between the automatic metric and human judgments?

Suggestions:

  1. Could the authors compare against reduced-step DiffPoseTalk or a distilled diffusion model using the same data and representation, reporting both quality and latency?
  2. The paper links acausal noise shaping to reduced jitter, but provides no quantitative jitter evaluation. Could the authors report jitter or spectral measures for the causal and acausal variants under the same training objective? For head pose, does acausal processing improve smoothness beyond the existing first- and third-order difference penalties?
4 comments
AuthorsOpenReview ↗

Reply for the suggestions

First of all, we would like to thank the reviewer for the careful reading, thoughtful feedback, and constructive suggestions.

Because Lee et al. (2025) do not release code, we instantiate their baseline using the same AudioUA model architecture while following the diffusion and distillation losses and training settings described in their paper as closely as possible; the main difference is that their original method conditions on eye images, whereas our comparison uses audio-only conditioning because eye-image inputs are not available in our setting. We additionally report two jitter-related temporal metrics computed on the full 137-D motion latent (128 expression + 9 headpose): HF deviation measures the mismatch in high-frequency motion energy above 4 Hz, and jerk deviation measures the mismatch in RMS third-order temporal derivatives; both are reported as |log(Pred/GT)| with lower values indicating closer agreement to ground-truth temporal dynamics. We observe that adding an acausal noise layer substantially improves temporal smoothness and audiovisual alignment relative to the causal-noise variant even for single-step distilled models.

MethodFED ↓FPD ↓Similarity ↑Sync Score ↑Expr Var → 1Pose Var → 1HF Dev. ↓Jerk Dev. ↓
Fallingwater12.48241.60700.09760.69640.98031.18630.50540.2167
DiffPoseTalk11.35461.47200.10160.57680.75910.53541.20100.7068
MemoryTalker20.70473.76930.13280.53460.21710.08322.42041.3894
ARTalk14.54051.27270.29110.23421.39981.05390.87180.5130
AudioUA w our model architecture, causal noise17.45722.86680.11880.34650.73970.60960.67530.5169
AudioUA w our model architecture, acausal noise34.92193.28880.11810.71190.61990.30590.47260.2212
FaceGAN (ours)12.14831.33350.15420.87670.96470.99210.39110.1798
AuthorsOpenReview ↗

Reply for clarifications

Q1 We thank the reviewer for pointing this out. The latency numbers have been reported with one frame per call NOT one forward pass per 5 frame block, that is in error, and Table 8 needs to be updated. The table below shows the per-forward calls for each of these models, but note that the primary contributor to latency is from the input chunk sizes required by the model, causing higher end-to-end latency.

MethodEmission unitSequential passesTime per forward pass (H200 / A100)Compute per 40 ms frame (H200 / A100)First-frame latency¶ (H200 / A100)Notes
MemoryTalkerwhole utterance1 / utterance14.6 / 26.3 ms†0.02 / 0.04 ms*full utterancenon-causal, whole utterance
DiffPoseTalk4 s chunk (100 fr)500 / chunk4.4 / 7.2 ms22.2 / 36.0 ms*6.40 / 7.83 s500 denoising steps / chunk
ARTalk4 s chunk (100 fr)5 / chunk7.0 / 12.0 ms0.35 / 0.60 ms*4.20 / 4.23 s5 AR scales / chunk
Fallingwater1 frame (40 ms)1–4 / frame9.9 / 16.2 ms21.6 / 34.6 ms#102 / 141 ms1–4 AR steps / frame; incl. 80 ms encoder frame
AudioUA w our model architecture, causal noise1 frame (40 ms)1 / frame3.4 / 5.7 ms9.6 / 15.6 ms§50 / 56 mscausal noise layer; no look-ahead
FaceGAN (ours)1 frame (40 ms)1 / frame3.3 / 6.4 ms3.8 / 7.3 ms‡84 / 87 ms40 ms audio look-ahead

* Amortized: chunk (or utterance) compute divided by its number of output frames. No frame is emitted before its whole chunk (or utterance) is generated, so this reflects throughput, not latency.

† One pass over a 30 s utterance; grows with utterance length.

‡ Generator forward plus streaming audio encoding (0.5 / 0.95 ms per frame).

§ Exact streaming update: recomputes the model's 6 s receptive field (151 frames; the causal noise layer adds 2 s), including re-encoding that window's audio (6.2 / 9.9 ms of the total).

# Motion tokens at four time scales (spanning 4 s, 160 ms, 80 ms and 40 ms) are each generated when their first frame's audio arrives, so a 40 ms input triggers 1–4 sequential AR steps (1.76 on average); per-frame compute is the average.

¶ Chunk-based (DiffPoseTalk, ARTalk): 4.00 s of audio buffering + one chunk of compute + 0.16 s smoothing look-ahead. Frame-based: time to receive the audio the causal audio encoder needs (40 ms; 80 ms for Fallingwater) + model look-ahead (40 ms for FaceGAN, none for AudioUA and Fallingwater) + per-frame compute. Per-pass and per-frame times are batch 1, CUDA-synchronized; baseline timings include their audio encoders.

Q2 Improvements in the automatic metric may be subtle to produce a consistent perceptual preference, particularly when both methods already provide plausible synchronization. Human judgments can also vary across participants, who may differ in their sensitivity to timing errors and how they assess lip-sync quality, making small differences harder to resolve through paired comparisons. Overall, the two evaluations are consistent with strong lip-sync performance.

Reviewer_x6NEOpenReview ↗

Thank you for the response

First, I thank the authors for the detailed response and the additional experiments. Most of my concerns have been addressed in the rebuttal. The latency comparison is now much more complete, and the streaming inconsistency has been clarified. However, after reading the other reviewers’ comments and the authors' response, I still have some reservations. I therefore keep my original score.

AuthorsOpenReview ↗

PDF update

We thank the reviewer for the constructive suggestions and discussion, and we have uploaded a revised PDF that incorporates the corrections and additional results from our rebuttal.

Reviewer_8hVxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper argues that audio-driven facial motion, being low-dimensional, does not need iterative diffusion sampling, and presents FaceGAN: a single-pass generator with a 2-layer bidirectional "noise stack" over sampled noise (past and future), followed by a 6-layer causal conditioning stack over frozen Mimi audio features with one frame (40 ms) of lookahead. Two multi-scale discriminators (audio-conditioned and unconditional) provide the adversarial signal, with feature matching and a mode-seeking term. The theoretical motivation (Sec. 3) uses the Kolmogorov–Szegő formula to argue that a causal time-invariant generator driven by i.i.d. noise cannot place a spectral stopband without collapsing its per-step innovation, and that acausally shaping the noise removes this limitation at no audio latency. On the Fallingwater dataset (64 test clips), FaceGAN obtains the best Sync Score (0.877 vs. 0.696) and pose variance ratio, is second on FED/FPD, is at parity on lip sync and preferred on naturalness in a 19-person user study, and runs at 43 ms latency per stream on one H200.

Critical strengths
  1. Strong practical results for a simple model. A single forward pass reaches the best Sync Score, user-study parity or preference against diffusion and multi-scale AR baselines, and real time up to 4,096 concurrent streams. This is a useful data point against the default of distilling diffusion models for this task.
  2. Careful ablation reporting. The full model is trained three times and differences inside the 2σ band are marked (Tables 4–5). The discriminator, feature-matching, and mode-seeking ablations show clearly which component controls which property.
Critical weaknesses

Update after rebuttal (17 Nov 2026). W1 is resolved by concession: the authors agree that Eq. (3) also binds the acausally shaped model and will reframe the gain as an inductive-bias effect, so I raise soundness from 1 to 2. This changes the paper's central motivation, and no revised PDF has been posted yet. W2 is partly resolved: residual spectra and a synthetic test were added, but the HF/jerk comparison lacks the causal-noise FaceGAN row, and the re-implemented distilled baseline looks under-tuned (acausal FED 34.92 vs. 17.46 causal). My condition for raising to 3 is therefore only partly met, and I keep my rating at 2. Details are in my comment below.

  1. The theory does not appear to support the proposed fix. Eq. (2) and the trade-off in Eq. (3) depend only on the spectral density of the output process y. They therefore apply to any stationary output with the same stopband, whether it is produced by a causal generator with i.i.d. noise or with acausally shaped noise. Acausal shaping does break condition (4), so the no-stopband theorem no longer applies, but Eq. (3) does not depend on (4): it still bounds the innovation of any output with that stopband. The statement in Sec. 3.2 that the downstream generator "is no longer bound by (3)" therefore seems incorrect. Furthermore, by the Wold decomposition, any regular stationary process (σ²∞ > 0, which includes real facial motion without exact spectral nulls) can be written as a causal filter of white (uncorrelated) noise, which is i.i.d. in the Gaussian case, so the theorem does not identify a class of natural motion that causal generators cannot represent. In the linear-Gaussian case, acausally filtered noise can be re-expressed as minimum-phase causally filtered noise with the same distribution. Finally, the paper equates small innovation σ²∞ with "muted" motion, but smooth large-amplitude motion also has small σ²∞; amplitude and innovation are different quantities. The theorem itself (no exact stopband) is a direct corollary of Kolmogorov–Szegő. If the intended statement concerns an approximate stopband or finite-length sequences, please state the quantitative version; as written, Eq. (3) applies equally to the acausally shaped model. The empirical gain in Table 4 may still be real, but it would then be an optimization or inductive-bias effect, not the escape from an expressivity limit that Sec. 3 claims.
  2. The central spectral claim is not measured, and the title's comparison is missing. No output power spectra or jitter metrics are reported for causal vs. acausal noise. The traces in Fig. 1 are labeled as illustrative schematics, Fig. 3 shows single frames (which cannot show a temporal effect such as jitter), and Table 4 shows lower variance and diversity rather than jitter, while Sync and FED are essentially unchanged with causal noise (0.8902 vs. 0.8810; 11.64 vs. 11.76). The title claims distillation is unnecessary, but there is no comparison with a distilled real-time model (e.g., Lee et al., SIGGRAPH Asia 2025, which the paper cites) or with the same architecture trained as a one-step flow or diffusion model.
Questions for authors
  1. (Clarification) Eq. (3) is a property of the output spectrum alone. Why does it not apply to the output of the acausally shaped model? For outputs with the same stopband, what is σ²∞ in each case?
  2. (Decision-relevant) Can you show power spectra (or a jitter metric) of generated vs. real motion for causal-noise and acausal-noise models? A synthetic test with a known target spectrum (e.g., a band-limited Gaussian process) would check the theory directly: train both generators on it and compare their output spectra and per-step innovation with the known values.
  3. (Decision-relevant) Can you compare with a distilled or one-step real-time baseline, such as Lee et al. 2025 or the same architecture trained with a one-step flow objective?
  4. (Suggestion, not required) Temporally structured noise inputs have prior work that is not cited: correlated noise priors (PYoCo, ICCV 2023; "How I Warped Your Noise", ICLR 2024) and future-indexed noise in causal or streaming generation (Rolling Diffusion, ICML 2024; Diffusion Forcing, NeurIPS 2024). How does acausal noise shaping differ from these, and how does it compare with a causal correlated-noise baseline such as PYoCo's progressive noise?
  5. (Clarification) How does the method differ from earlier GAN-style audio-driven talking-head generation such as StyleTalker (Min et al., 2022)? GANs tend to mode-collapse: diversity collapses without the mode-seeking term (Table 5), and motion similarity to the ground truth is low (Table 2: 0.154 vs. 0.291 for ARTalk). How well does the model cover the domain beyond this benchmark?
  6. (Clarification) Sec. 5.5 describes one frame per call, while Table 8 says one forward pass per 5-frame block. Which is used for the latency numbers?
  7. (Clarification) Related work describes Fallingwater as strictly causal with zero lookahead, but Table 1 marks it as chunk-level and not real-time. Which is correct?

If the spectra in Q2 show the claimed effect and the method matches a distilled or one-step baseline (Q3), I would raise my rating to 3.

9 comments
AuthorsOpenReview ↗

Reply for Q1

First of all, we would like to thank the reviewer for the careful reading, thoughtful feedback, and constructive suggestions.

W1 & Q1

We thank the reviewer for the careful reading. The reviewer is right that (3) depends only on the output spectrum, so it binds our model too: the same stopband gives the same bound, and equal spectra give equal σ∞2\sigma^2_\infty. We will remove "no longer bound by (3)", which was wrong, and revise the abstract, Fig. 1, Secs. 1–3 and 5.4, and the conclusion to match; our reply to T712 gives the finite-length, audio-conditioned version of (3).

By "muted" we meant the regression regime: a generator that does not use its noise collapses toward the conditional mean, which is perceptually muted. We agree that σ∞2\sigma^2_\infty measures innovation, not amplitude, and will reword Sec. 3.2 to make "muted" unambiguous.

What (3) does constrain is how a generator can use its noise. In a generator causal in its noise, the newest noise frame can add, on average over past noise and given the audio, at most the audio-conditioned innovation et2(a)e_t^2(a), whose geometric mean (3) caps (see our reply to T712): for smooth motion, diversity must come mostly from past noise. As the reviewer's Wold argument suggests, our noise look-ahead adds no representational power: our noise stack reads 128 frames of future noise, but relabelling zt′=zt+128z_t' = z_{t+128} makes our model formally causal in its noise, with the same output distribution. What differs from a causal stack is the residual path: a causal stack puts the newest frame there, which smooth motion can barely use, while ours puts a frame that previous outputs already depend on, so using it need not roughen the motion.

Table 4 isolates this: same audio path and attended width, and after relabelling only the residual frame differs (up to sequence edges and an absorbable RoPE offset). The causal stack has 14% and 22% lower expression and pose diversity (both >3σ>3\sigma) at similar Sync and FED: a partial regression toward the conditional mean, which is what "muted" referred to. In Table 4, Expr Var is 0.92–0.93 at every causal width and 1.01–1.03 at every acausal one. As the reviewer suggests, this is an inductive-bias (or optimization) effect, and the revised Sec. 3 will say so.

AuthorsOpenReview ↗

reply for Q2

Q2 (Decision-relevant)

We thank the reviewer for the concrete suggestion. We provide the requested power spectra of generated versus real motion for expression, head rotation, and translation. The comparison uses three independent training runs per noise condition, 64 clips, and 64 stochastic draws per clip, with both noise stacks attending 129 frames per layer. We evaluate EMA checkpoints near 200k steps under the same protocol, remove the first second, and measure raw motion without smoothing.

To isolate the injected noise from the shared audio-driven motion, we additionally analyze the stochastic residual: each prediction minus the mean across noise draws for the same audio. The acausal architecture preserves 11.2% greater expression diversity and 21.6% greater pose diversity. Its expression residual has 25.9% lower power share above 4 Hz, while RMS jerk normalized by residual temporal standard deviation is 22.6%, 23.1%, and 22.5% lower for expression, head rotation, and translation, respectively. Paired training-seed/subject bootstrap intervals support these diversity and normalized-roughness improvements. This quantifies the benefit of acausal noise shaping for producing stronger, more temporally coherent stochastic variation.

For the synthetic experiment, we use a unit-variance Gaussian process at 25 Hz, with a passband below 3 Hz, a transition between 3 and 5 Hz, and a power-spectrum floor of 10^-6 above 5 Hz, corresponding to approximately −60 dB attenuation. Each small model contains one learned noise-attention block followed by one causal generator block. Both have 21,025 parameters, hidden dimension 32, eight input-noise channels, two attention heads, and a 65-frame noise-attention window. The only architectural difference is the noise-attention mask: the causal layer reads current and past noise, whereas the acausal layer reads a symmetric past-and-future window.

We train both models from scratch using five new seeds. Within each seed pair, the models start with identical parameter values and use the same underlying Gaussian-noise stream, optimizer, objective, and training budget. There is no GAN discriminator. Training directly matches the known target spectrum: we derive its covariance and expected finite-Hann-window periodogram, then minimize log-spectrum discrepancy together with mean and variance penalties. We use Adam with learning rate 0.001, batch size 32, and 256 measured output frames after a 128-frame burn-in.

We evaluate every 100 updates on 256 fixed held-out noise draws shared between the paired models. The comparison budget of 2,000 updates was fixed before running these five seeds, and we report the complete learning curves.

MetricCausalAcausal
Held-out target-fit loss1.4471.159
Log-spectrum MSE1.4201.126
Spectral RMSE (dB)5.174.60
Output std. (target = 1)0.9830.987

At the shared checkpoint, the acausal model reduces held-out target-fit loss by 19.9% and log-spectrum MSE by 20.7%. It has lower endpoint loss in all five seed pairs. Its average objective over the complete learning curve is also 4.5% lower, with an improvement in all five pairs. The near-unit output standard deviations show that the lower spectral error is not obtained by shrinking the generated signal.

Together, these results provide controlled evidence of improved finite-budget learning in the small models and a better measured coherence–diversity tradeoff in the full animation system. Since future synthetic noise is available internally, the acausal noise pathway adds no audio lookahead.

AuthorsOpenReview ↗

Reply for Q3 and Q4

Q3 (Decision-relevant)

We thank the reviewer for the concrete suggestion. Because Lee et al. (2025) do not release code, we instantiate their baseline using the same AudioUA model architecture while following the diffusion and distillation losses and training settings described in their paper as closely as possible; the main difference is that their original method conditions on eye images, whereas our comparison uses audio-only conditioning because eye-image inputs are not available in our setting.

We additionally report two jitter-related temporal metrics computed on the full 137-D motion latent (128 expression + 9 headpose): HF deviation measures the mismatch in high-frequency motion energy above 4 Hz, and jerk deviation measures the mismatch in RMS third-order temporal derivatives; both are reported as |log(Pred/GT)| with lower values indicating closer agreement to ground-truth temporal dynamics.

We observe that adding an acausal noise layer substantially improves temporal smoothness and audiovisual alignment relative to the causal-noise variant even for single-step distilled models.

MethodFED ↓FPD ↓Similarity ↑Sync Score ↑Expr Var → 1Pose Var → 1HF Dev. ↓Jerk Dev. ↓
Fallingwater12.48241.60700.09760.69640.98031.18630.50540.2167
DiffPoseTalk11.35461.47200.10160.57680.75910.53541.20100.7068
MemoryTalker20.70473.76930.13280.53460.21710.08322.42041.3894
ARTalk14.54051.27270.29110.23421.39981.05390.87180.5130
AudioUA w our model architecture, causal noise17.45722.86680.11880.34650.73970.60960.67530.5169
AudioUA w our model architecture, acausal noise34.92193.28880.11810.71190.61990.30590.47260.2212
FaceGAN (ours)12.14831.33350.15420.87670.96470.99210.39110.1798

Q4

We thank the reviewer for highlighting these relevant precedents. We will cite them and clarify that our contribution is a learned acausal noise pathway for single-pass streaming facial animation.

PYoCo uses prescribed Gaussian correlations, including a causal progressive prior. "How I Warped Your Noise" transports noise using motion fields while preserving its distribution. Our bounded noise-attention stack learns temporal structure in noise-latent space jointly with the generator, before audio fusion and without optical-flow warping. Rolling Diffusion and Diffusion Forcing use per-frame noise levels and iterative denoising. Our method shapes synthetic noise before a single-pass motion prediction. We plan to include a direct comparison with PYoCo's progressive noise prior in the next version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.