- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
REALM is a coarse-to-fine framework for audio-driven listener facial motion generation. It combines history-aware temporal alignment with stochastic expression refinement to produce coherent, varied responses. Experiments on ViCo and L2L show improvements over evaluated baselines, while Ameca robot deployment and a user study demonstrate its practical applicability.
Critical strengths
- This paper further reduces expression over-smoothing over UniLS. Audio-conditioned stochastic refinement adds facial variation while preserving head pose, improving expression dynamics and blink realism.
- This paper balances response timing and motion continuity. Delay-aware attention and adaptive gating combine earlier speech cues with motion history, though gains in pointwise accuracy are modest.
- This paper demonstrates applicability to physical embodiment (Ameca Robot), this is not common in other similar papers. This deployment shows promising visual results, but offline generation and video-based evaluation leave real-time human–robot interaction unverified.
Critical weaknesses
- Contextual appropriateness remains insufficiently validated. The reported motion metrics assess reconstruction accuracy and distributional similarity, but do not directly establish whether expressions and listener responses occur at contextually appropriate moments. Beyond the user study, could LLMs or VLMs help quantify contextual appropriateness?
- The gap between robot deployment and real-world interaction needs clarification. The paper explicitly considers offline window-conditioned generation. Could the authors clarify whether the robot demonstrations use precomputed trajectories and report the latency and duration of the evaluated pipeline?
- The contribution of stochasticity remains unclear. The refinement pathway combines additional network capacity, audio conditioning, noise injection, and adversarial training. While the component ablations and unmodulated-noise comparison support the complete design, they do not isolate the benefit of stochastic sampling. An audio-conditioned deterministic refinement variant, retaining the same architecture and training objectives while removing the injected noise, would help establish the contribution of stochasticity. (Audio itself is also inevitably to contain some noise, whether from the background or the equipment.)
- The validation of the eight-frame delay prior is limited. Table 4 supports an 8-frame attention-bias center among the tested settings, but reconstruction and distribution metrics do not establish whether this delay produces natural responses; it may simply reflect dataset-specific patterns. (For instance, the delay involved in discussing a difficult question is not the same as that in a simple conversation.)
Questions for authors
- Beyond the user study, could LLMs or VLMs help quantify contextual appropriateness?
- Could the authors clarify whether the robot demonstrations use precomputed trajectories and report the latency and duration of the evaluated pipeline?
- Could the authors evaluate an audio-conditioned deterministic refinement variant?
- Could we find a better way to evaluate the frames we delay?
2 comments
Response to Reviewer yVmN (Part 2/2)
[W3 & Q3] Contribution of Stochastic Sampling
Reviewer (W3 & Q3): "An audio-conditioned deterministic refinement variant, retaining the same architecture and training objectives while removing the injected noise, would help establish the contribution of stochasticity."
Response. We trained an audio-conditioned deterministic refinement variant that retains the same architecture, audio-dependent shift , and Stage II training objectives (L1 + adversarial), but sets during training and inference. This allows us to isolate the contribution of stochastic sampling from refinement capacity and audio conditioning.
Deterministic vs. Stochastic Refinement on ViCo Test Set
| Method | (Exp.) ↓ | (Exp.) ↓ | Blink Rate Error (BPM) ↓ | Blink Duration Error (ms) ↓ |
|---|---|---|---|---|
| Without RM | 11.85 | 12.49 | 10.50 | 74.65 |
| Unmodulated noise | 12.64 | 14.82 | 10.39 | 341.32 |
| Deterministic RM () | 11.63 | 6.84 | 9.12 | 46.20 |
| Stochastic RM (Ours) | 11.58 | 3.91 | 7.40 | 18.45 |
Deterministic refinement improves expression from 12.49 to 6.84, while stochastic refinement further reduces it to 3.91. Similar improvements are observed in blink dynamics, with duration error decreasing from 46.20 to 18.45 ms. In contrast, unmodulated noise degrades motion quality, highlighting the importance of audio-conditioned stochastic refinement.
Both refinement variants receive identical speaker audio, so their performance differences cannot be attributed to variations in the audio input. We will incorporate this ablation into Table 3 and Appendix D.
[W4 & Q4] Validating the Reaction Delay
Reviewer (W4 & Q4): "...it may simply reflect dataset-specific patterns. (For instance, the delay involved in discussing a difficult question is not the same as that in a simple conversation.) Could we find a better way to evaluate the frames we delay?"
Response. We provide an additional Speaker-Cue-to-Listener-Reaction Timing Evaluation to directly assess when generated listener reactions occur relative to speaker cues. We also outline future directions for modeling and evaluating reaction timing across different conversational contexts.
(1) Direct evaluation of reaction delay. Following prior work on prosodic cues and listener backchannels (Ward and Tsukahara, 2000), we extract speaker acoustic cues from ViCo audio, including energy peaks and pause onsets, and match them with subsequent listener motion onsets. We measure Median Lag (cue-to-reaction delay) and Lag MAE (generated versus ground-truth delay error). We also assess Timing Naturalness through human ratings of listener videos presented alongside synchronized speaker audio.
Speaker-Cue-to-Listener-Reaction Timing Evaluation
| Delay Prior (frames) | Median Lag (frames) | Lag MAE (frames) ↓ | Timing Naturalness (1–5) ↑ |
|---|---|---|---|
| 0 | 3.0 | 3.42 | 3.6 |
| 4 | 5.0 | 2.35 | 3.9 |
| 8 (Ours) | 8.0 | 1.48 | 4.3 |
| 12 | 10.5 | 2.84 | 3.8 |
Among the tested settings, achieves the lowest reaction-delay error (1.48 frames) and highest timing-naturalness rating (4.3). This provides more direct evidence for the selected delay prior than the motion-quality metrics in Table 4.
(2) Context-dependent reaction timing. We plan to investigate context-dependent timing by (i) expanding training and evaluation data to cover more diverse conversational content and complexity levels, (ii) incorporating a learned context-dependent adjustment into the current delay prior, and (iii) validating reaction timing across different conversational domains and datasets.
References
- Ward, N., & Tsukahara, W. (2000). Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics.
Response to Reviewer yVmN (Part 1/2)
We sincerely thank the reviewer for the thoughtful and constructive feedback. We particularly appreciate the reviewer's recognition of REALM's improvements in facial dynamics, temporal modeling, and physical embodiment. The comments have helped us identify several aspects of our evaluation that deserve further clarification and analysis. Below, we address these concerns through additional evaluations of contextual appropriateness, pipeline latency, stochastic refinement, and reaction timing.
[W1 & Q1] Contextual Appropriateness Beyond Motion Metrics
Response. We further evaluate contextual appropriateness using two complementary approaches: event-level temporal metrics and video-based human ratings. While video-capable VLMs offer a possible alternative, their reliability in assessing subtle facial reactions and reaction timing remains uncertain. We therefore focus on directly measuring reaction-event alignment and evaluating the contextual suitability of generated listener behaviors through human judgments.
Event-level evaluation. We identify listener reaction events from motion trajectories and evaluate their agreement with ground-truth reactions using F1 and temporal IoU (tIoU). We additionally evaluate smile events to examine whether generated expressions occur at appropriate moments.
Human evaluation. To assess contextual appropriateness (App.), we render the generated 3D facial motions as listener videos using PIRenderer (Ren et al., 2021), paired with the corresponding speaker audio. Ten human evaluators rate the contextual appropriateness of each listener video on a five-point Likert scale, considering both the suitability and timing of facial reactions. App. is reported as the mean opinion score (MOS).
Contextual Appropriateness Evaluation on ViCo
REALM achieves the highest scores across all four metrics, including Reaction F1 (0.72), Smile F1 (0.69), and Reaction tIoU (0.58), indicating better agreement with the timing of observed listener reactions. Human evaluators also rate REALM highest in contextual appropriateness (4.1/5), suggesting that its generated expressions and responses better match the conversational context.
We will clarify this evaluation protocol and include the additional results in the revised manuscript.
[W2 & Q2] Offline Robot Demonstrations and Pipeline Latency
Response. Yes. As noted in the limitations (Sec. 5), REALM currently operates on offline conversational windows, and all robot demonstrations use precomputed trajectories. Motion is generated offline, retargeted, and then played back on Ameca at 30 FPS. The demonstrations establish the feasibility of physical embodiment, not real-time interaction, and we will state this explicitly in Sec. 4.2.
Demonstration Duration. As described in Appendix E.1, the Ameca user study uses conversational sequences sampled from the ViCo test set, each approximately 15–30 seconds long. The generated facial motions are retargeted and executed on the physical Ameca robot, and its actual facial movements are recorded on video. Twenty-five evaluators each viewed five randomly selected sequences, comparing recordings of REALM and five baselines under the same robotic setup.
Compute time per 16-frame window (mean over 1,000 windows after warm-up; RTX 5090 for the model stages, host CPU for post-processing)
These are compute times, not end-to-end response latency. As an audio-driven framework, REALM does not require image or video inputs during inference, avoiding the computational overhead of visual feature extraction. Its lightweight architecture also makes real-time deployment computationally promising.
References