- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
HiddenFace predicts facial-motion codes every 160 ms from the audio and text hidden states of a frozen large audio-language model. A causal RVQ motion tokenizer, a short previous-segment memory and a depth-wise code predictor support streaming generation. The paper also develops geometric surrogate objectives and uses synthetic self-play conversations for training. It evaluates real conversations, generated conversations, human preferences and runtime.
Critical strengths
The motion head is small and fast, and attaching it to a frozen conversational model is useful. The paper gives substantial implementation detail, compares hidden states with audio codes, studies auxiliary losses and memory, and reports limitations and metric trade-offs. The human study uses blinded presentation and rater bootstrap intervals. The authors also discuss closely related systems, including Ex-Omni, FacePlex and DyaPlex, rather than presenting speech–motion generation as a new problem.
Critical weaknesses
1. Causality in the audio-language model's tokens is not sufficient to establish causality in the input waveform. Sec. 4.2 describes the streaming comparison as using no audio beyond the current 160 ms block. However, Appendix C says each real recording is encoded once with the non-streaming Voila tokenizer, and prefix re-encoding shows that a 40 ms code frame can depend on up to four later frames, with a median of three. A causal transformer operating on these codes can therefore inherit future waveform information.
The key missing detail is whether the real-conversation test states in Table 1 are extracted using this same full-recording procedure. If so, the claimed matched causal comparison is not established: the inputs to HiddenFace may contain future information even when its motion predictor is causal. Please rerun with prefix-only audio tokenization, or measure the actual waveform lookahead and give the baselines the same allowance. Appendix F's past-only re-encoding experiment shows one possible live implementation, but does not establish that the main quantitative comparison used the same input boundary. This is a question about the evaluation protocol; the appendix alone does not prove that every reported result used future audio.
2. The practical and comparative evidence is incomplete for the claimed streaming interface. The 5.8 ms number measures the motion component with CUDA graphs. Appendix F gives a more informative system experiment, but it uses separate LLM/codec GPUs, excludes text-generation stalls of up to 1.6 seconds from the real-time subset, and reports its steady-state table after decoder warm-up. These choices are described, but the resulting timing should not stand in for the latency a user experiences. Please report all-block end-to-end latency, including startup and speech/text stalls, with tail percentiles, missed-deadline rates and interruption examples.
The main motion baselines are also adapted offline systems. The paper already cites closer conversational systems, but there is no quantitative comparison with a streaming speech–motion architecture. In particular, DyaPlex [1] also connects a trainable motion pathway to hidden states of a frozen speech model, and FacePlex [2] is designed for online joint speech/facial motion. Their backbones and tasks differ, so an unadapted score would not be a fair comparison; a matched-backbone implementation or a carefully controlled component comparison would be more useful. The audio-code ablation is informative, but does not replace that comparison. The human study does not establish a significant advantage over DualTalk, as the authors themselves acknowledge.
The system is promising. I currently lean toward rejection because the main causal comparison needs to be resolved, and the deployment evidence does not yet cover the whole streaming system.
Questions for authors
Questions For Authors
- Were Table 1's real-test hidden states extracted from full-recording audio codes? Please specify the waveform context available to every code/state in the current block.
- What happens to quality under prefix-only tokenization, with any lookahead charged to the latency budget? Can all methods use the same waveform boundary?
- Please report end-to-end startup and steady-state p50/p95/p99 latency across all blocks, including text stalls, plus deadline misses and interruption behavior.
- Can you compare against a closely related streaming speech–motion architecture under a matched backbone/data budget, separating the benefit of hidden-state input from the predictor design?
References
[1] DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interaction. Secs. 3–4 describe a frozen speech tower and a trainable motion tower attending to its hidden states. https://arxiv.org/html/2606.03874v1
[2] FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars. https://arxiv.org/html/2606.30145v1
7 comments
Response to Reviewer X1Ah (2/6): W1, Q1, Q2, continued
We followed the first of your two suggestions. We applied prefix-only tokenization in two settings and re-extracted the Voila states of both the training and test sets. In the partner-prefix setting, only the partner's audio is encoded from prefixes, and the agent's codes remain encoded once as the closer proxy for the codes Voila generates. In the both-prefix setting, both speakers are encoded from prefixes, so that every input reflects only the audio up to the end of the current block. In each setting, we evaluated the submitted checkpoint as is ("submitted") and a model retrained with the same code, hyperparameters, seed, and training schedule ("retrained").
| Tokenization (partner / agent) | Model | S-LVE↓ | S-MHD↓ | MOD↓ | Lip-W↓ | S-FDD↓ | L-FDD↓ | AVC↑ | MTM↓ | MSE↓ | FD↓ | P-FD↓ | rPCC↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UniLS [4] (released), 160 ms streaming (Table 1) | – | 8.604 | 2.484 | 3.725 | 3.940 | 24.58 | 22.99 | 0.305 | 58.4 | 0.241 | 15.32 | 16.44 | 0.135 |
| Prefix-only / prefix-only | submitted | 8.426 | 2.290 | 3.797 | 3.553 | 25.33 | 27.09 | 0.266 | 81.7 | 0.185 | 11.76 | 12.75 | 0.114 |
| Prefix-only / prefix-only | retrained | 8.732 | 2.338 | 3.929 | 3.474 | 23.29 | 26.50 | 0.327 | 74.6 | 0.188 | 11.67 | 12.66 | 0.116 |
| Encoded once / encoded once | submitted (Table 1) | 8.229 | 2.247 | 3.730 | 3.452 | 23.33 | 26.88 | 0.374 | 63.9 | 0.178 | 11.00 | 11.96 | 0.113 |
| Prefix-only / encoded once | submitted | 8.198 | 2.242 | 3.751 | 3.501 | 23.18 | 26.64 | 0.377 | 62.4 | 0.178 | 10.95 | 11.92 | 0.112 |
| Prefix-only / encoded once | retrained (Table 1) | 8.189 | 2.244 | 3.711 | 3.459 | 23.32 | 26.52 | 0.388 | 61.3 | 0.179 | 10.94 | 11.91 | 0.112 |
Table B. Results under the three tokenization settings (partner / agent).
For each of the 12 metrics, we compare HiddenFace with each of the seven released baseline rows of Table 1. For each comparison, we compute the difference between HiddenFace and the baseline on the same test items and obtain its 95% confidence interval by a paired bootstrap over the 1,554 continuous runs of the test split. The comparison counts as not different if this interval includes 0 and as better or worse otherwise.
In the partner-prefix setting, every comparison gives the same result as in Table 1 with or without retraining. For the submitted checkpoint without retraining, every metric changes by at most 2.4%.
In the both-prefix setting, every input ends at the end of the block (last row of Table A). This is the same boundary as the streaming input of the baselines (Appendix E: UniLS and ARTalk set the audio not yet received to zero, and DualTalk is rerun on the conversation up to each step). No look-ahead beyond the 160 ms block therefore remains, and charging the look-ahead to the latency budget adds no delay.
Table B lists the both-prefix setting for the submitted checkpoint without retraining and for a model retrained in it. The retrained both-prefix model remains the best among the released streaming methods on eight of the 12 metrics, but it is less accurate than streaming UniLS in S-LVE and MOD. Four of its advantages over offline baselines are also no longer significant. The other comparisons give the same result as in Table 1. Because the agent's codes in this setting differ greatly from those Voila generates, we regard it as a waveform-causal replay rather than as the condition of real-time operation.
The live run of Appendix F indeed does not establish the input boundary of Table 1. Table B therefore does not rely on Appendix F. It repeats the quantitative comparison itself with prefix-only inputs.
The remaining HiddenFace results on real conversations (Table 1 except its partner-prefix rows, and the ablations in Tables 3, 4, and 7) use states extracted from codes encoded once. The baseline rows and the latency columns of Table 1 do not pass through the Voila tokenizer and are unaffected. The self-play conversations of the human study (Table 2) do not pass through it either, because the two Voila instances exchange the codes they generate. Only the audio that opens a conversation is encoded once.
Sec. 4.1 of the revision now states that the hidden states of Table 1 can reflect audio beyond the block, up to 120 ms in our measurement. It also reports the partner-prefix result. The both-prefix result is in Sec. 4.2 and Appendix H, and the waveform context of each setting is in Appendix H. Table 1 has new partner-prefix rows, so the partner-prefix result can be read from the main table.
We also corrected the wording you pointed out. Where the paper called the comparison "matched" or "same condition" (abstract, Sec. 1, the sub-header of Table 1, Sec. 4.2, and Sec. 5), we removed these words and now write only "block-causal".
Response to Reviewer X1Ah (3/6): W2 (first part), Q3
W2 (first part), Q3 (latency over all blocks and interruptions)
"Please report end-to-end startup and steady-state p50/p95/p99 latency across all blocks, including text stalls, plus deadline misses and interruption behavior."
We address the three choices you list in turn (the separate LLM/codec GPUs, the excluded text-generation stalls, and the steady state reported after decoder warm-up). We use two GPUs because the system did not run in real time on one. In a single-GPU run, the computation of Voila's speech path alone took 1.06 times real time, and the facial branch added another 0.22. In the two-GPU setup, Voila's LLM and audio tokenizer run on separate GPUs, and the facial branch adds 6.3 ms per block on the tokenizer GPU. We have no run of Voila without the facial branch, so these measurements do not show how much sharing a GPU with it slows Voila itself.
For the other two choices, we now analyze every block. This includes the intervals in which Voila generates text tokens in succession and its audio pauses, the intervals in which it then processes the queued input, and the interval right after the run starts. We leave out only the first block, which contains Voila's initialization, and analyze the remaining 999 blocks.
| Blocks | N | Face step, med / max (ms) | Block-end-to-emission p50 / p95 / p99 (ms) | Late > 160 ms (from waiting for face) | Delayed by face (max) |
|---|---|---|---|---|---|
| Steady state | 796 | 6.3 / 11.4 | 9.4 / 19.3 / 19.6 | 0 (0) | 1 (2.1 ms) |
| + startup | 849 | 6.3 / 28.1 | 9.4 / 20.3 / 30.8 | 0 (0) | 35 (16.9 ms) |
| + text stalls, catch-up (all) | 999 | 6.3 / 28.1 | 9.5 / 881 / 1300 | 135 (0) | 35 (16.9 ms) |
Table C. Latency over the 999 blocks after the first block of the live run.
Step in Table 1 (5.8 ms) is the time the facial branch takes to compute the motion of one block, measured with the facial branch alone. Face step in Table C is the same computation measured while running with Voila. Block-end-to-emission, in contrast, is not a computation time. It is the latency of Voila and the facial branch together, from the end of a block's fourth row interval until both its audio and face are ready and the block is emitted. In the steady state, the face is ready 5.0 ms after the block ends and the fourth row's audio becomes playable at 9.4 ms, so the 9.4 ms in Table C is the time Voila takes to prepare the audio. Because audio and motion are emitted per block, the first row of a block is emitted about 130 ms after its own interval ends. This wait comes from emitting motion in 160 ms blocks and not from waiting for future context.
We distinguish the blocks that waited for the face from those whose audio was already late. Of the 999 blocks, 135 (13.5%) were emitted more than 160 ms after they ended. All of them fall in the intervals in which Voila generates text tokens in succession (1.59, 1.13, and 0.88 s) and then processes the queued partner input. In these blocks Voila's own audio is late, and no block missed the 160 ms deadline from waiting for the face. This is also why the 95th and 99th percentiles over all blocks rise to 0.9–1.3 s.
Separately, the face is usually ready before the audio and waits for it. The face was ready after the audio in only 35 of the 999 blocks, so that the block waited for the face (Delayed by face in Table C). The wait was at most 16.9 ms, and these 35 blocks were all emitted within 160 ms. Of them, 34 fall in the first 53 blocks. During these blocks the motion decoder's input window has not yet filled and CUDA graphs cannot be applied, so the face step takes 22 ms instead of 6.3 ms.
Adding the startup changes the latency only slightly (99th percentile: 30.8 ms). The first motion is emitted 0.21 s after the run starts (0.16 s to receive the input of the first block, 0.02 s for Voila to process its last row, and 0.03 s for the first face computation).
We added this table and analysis to Appendix F, and Sec. 4.2 now reports its main numbers.
For interruptions, we collected all cases in which the partner starts speaking while the agent is speaking in three live runs and found 26, 26, and 23 cases (Appendix F). Voila decides whether the agent then stops or continues, and the face follows Voila's hidden states. We therefore examined whether mouth motion also subsides once Voila stops speaking. Because the partner's speech is a recording prepared in advance and does not react to the agent, these runs do not test interaction, and we do not claim that the system handles interruptions.
In these cases Voila mostly keeps speaking (it stops within 2 s in 15–42% of them). When it stops, mouth motion subsides within one to two blocks. The speed at which the mouth opens and closes averages 34–48 mm/s in the 0.4 s before the stop, 8–13 mm/s in the first 0.2 s after it, and 2–3 mm/s in the next 0.2 s. In one case of the Appendix F run, the partner starts speaking at 28.52 s, Voila stops 0.16 s later, and the speed falls from 19.9 to 2.1 mm/s within 0.2 s of the stop.
Response to Reviewer X1Ah (1/6): W1, Q1, Q2
Thank you for the specific questions about the evaluation protocol. We have uploaded a revised paper with the changes marked in blue. The sections, tables, and appendices cited below refer to this revision, and Appendices H, I, and J are new.
We first summarize our answers to the two reasons you gave for leaning toward rejection.
First, the states in Table 1 were extracted from codes obtained by encoding each recording once. As you suggested, we repeated training and evaluation with prefix-only tokenization. When only the partner's audio is encoded from prefixes, the comparison with each released baseline of Table 1 gives the same result as before on all 12 metrics. We explain under W1 why the agent's audio stays encoded once in this setting. When both speakers are encoded from prefixes, so that all methods use the same waveform boundary, the model retrained in this setting remains better than streaming UniLS on eight of the 12 metrics but is less accurate in S-LVE and MOD (W1, Q1, Q2).
Second, we now report every block of the live run, including the startup and the text-generation stalls. Of the 999 blocks after the first, 135 (13.5%) were emitted more than 160 ms after they ended, and the 99th-percentile latency is 1.30 s. No block missed the 160 ms deadline from waiting for the face (W2, Q3).
For the comparison with related architectures, we change one core design choice at a time on the same backbone and data (W2, Q4).
Yes. The test states in Table 1 were extracted from codes obtained by encoding each recording once ("encoded once" in the tables), in the same way as the training states. Each state is computed only from the tokens before its row and the predictor reads states only up to the current block, so HiddenFace is causal at the token level (Appendix C). As you point out, however, the released Voila [3] tokenizer is not a streaming codec, and a 40 ms code frame depends on up to four later audio frames (three at the median). Table A lists how much later audio each input of Voila and the resulting states can contain under three tokenization settings.
Table A. Waveform context of the inputs of Voila and of the resulting states at row t (partner / agent). * Measured dependence, excluding the rare isolated code changes noted in Appendix C. These values are not verified worst-case bounds.
In our measurement, a code frame depends on up to 160 ms of later audio. Because the states At and Tt are computed from the codes up to row t−1, the last states of a block then reflect up to 120 ms of audio beyond the block.
Whether this look-ahead departs from real-time operation depends on the speaker. In real-time operation, the partner's audio is encoded from only the audio received so far. Partner codes encoded once thus contain audio beyond the end of the block that a real-time system cannot know. The agent's codes are instead generated by Voila and not encoded from a waveform, so they involve no tokenizer look-ahead and no waiting that would be charged to the latency budget.
Real conversations, however, contain no codes generated by Voila. Proxy codes encoded from the recorded audio are therefore needed, and they should be close to the codes Voila generates. Prefix-only tokenization encodes each code frame from the waveform up to the end of that frame. When speech generated by Voila is re-encoded and compared with the generated codes, all four codebooks match in 24.5% of the speech frames (86.9% for the first codebook) if the audio is encoded once but in only 3.0% (57.1%) with prefix-only tokenization. By this measure, encoding once is the closer of the two proxies for the agent.