acceptodds
Under review as a conference paper at ICLR 2027

REALM: A Coarse-to-Fine Generative Framework for Embodied Reactive Listening

Abstract

Generating responsive listener facial motion is an important task for embodied conversational AI. Two modeling challenges are central: accounting for the timing of speaker cues while maintaining continuity with the listener's ongoing motion, and capturing locally variable facial events alongside the overall motion trajectory. Listener responses may follow preceding cues with a temporal lag, while brief expressions and blinks introduce variation that is difficult to predict deterministically. These challenges motivate a framework that combines history-aware temporal alignment with stochastic expression refinement. We propose REALM (**R**eactive **E**mbodied **A**udio-driven **L**istening **M**odel), a coarse-to-fine framework for audio-driven reactive listening. A Reactive Gated Speaker–Listener Fusion module combines listener motion history with speaker audio through a delay-centered attention prior and adaptive gating. A coarse decoder predicts a base motion trajectory, which is augmented by audio-conditioned stochastic residuals in the expression subspace while retaining the coarse pose parameters. Evaluations on ViCo and L2L show improvements over the evaluated baselines across multiple motion-quality metrics. Additional analyses examine delay sensitivity, gate behavior, and blink dynamics. Finally, deployment on an Ameca humanoid robot and a perceptual user study demonstrate the applicability of the generated behavior to physical embodiment. Code & Demo: https://anonymous.4open.science/r/REALM-7371/

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Generative Models, Audio-Driven Facial Animation, Reactive Listening

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yVmNOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

REALM is a coarse-to-fine framework for audio-driven listener facial motion generation. It combines history-aware temporal alignment with stochastic expression refinement to produce coherent, varied responses. Experiments on ViCo and L2L show improvements over evaluated baselines, while Ameca robot deployment and a user study demonstrate its practical applicability.

Critical strengths
  • This paper further reduces expression over-smoothing over UniLS. Audio-conditioned stochastic refinement adds facial variation while preserving head pose, improving expression dynamics and blink realism.
  • This paper balances response timing and motion continuity. Delay-aware attention and adaptive gating combine earlier speech cues with motion history, though gains in pointwise accuracy are modest.
  • This paper demonstrates applicability to physical embodiment (Ameca Robot), this is not common in other similar papers. This deployment shows promising visual results, but offline generation and video-based evaluation leave real-time human–robot interaction unverified.
Critical weaknesses
  • Contextual appropriateness remains insufficiently validated. The reported motion metrics assess reconstruction accuracy and distributional similarity, but do not directly establish whether expressions and listener responses occur at contextually appropriate moments. Beyond the user study, could LLMs or VLMs help quantify contextual appropriateness?
  • The gap between robot deployment and real-world interaction needs clarification. The paper explicitly considers offline window-conditioned generation. Could the authors clarify whether the robot demonstrations use precomputed trajectories and report the latency and duration of the evaluated pipeline?
  • The contribution of stochasticity remains unclear. The refinement pathway combines additional network capacity, audio conditioning, noise injection, and adversarial training. While the component ablations and unmodulated-noise comparison support the complete design, they do not isolate the benefit of stochastic sampling. An audio-conditioned deterministic refinement variant, retaining the same architecture and training objectives while removing the injected noise, would help establish the contribution of stochasticity. (Audio itself is also inevitably to contain some noise, whether from the background or the equipment.)
  • The validation of the eight-frame delay prior is limited. Table 4 supports an 8-frame attention-bias center among the tested settings, but reconstruction and distribution metrics do not establish whether this delay produces natural responses; it may simply reflect dataset-specific patterns. (For instance, the delay involved in discussing a difficult question is not the same as that in a simple conversation.)
Questions for authors
  • Beyond the user study, could LLMs or VLMs help quantify contextual appropriateness?
  • Could the authors clarify whether the robot demonstrations use precomputed trajectories and report the latency and duration of the evaluated pipeline?
  • Could the authors evaluate an audio-conditioned deterministic refinement variant?
  • Could we find a better way to evaluate the frames we delay?
2 comments
AuthorsOpenReview ↗

Response to Reviewer yVmN (Part 1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We particularly appreciate the reviewer's recognition of REALM's improvements in facial dynamics, temporal modeling, and physical embodiment. The comments have helped us identify several aspects of our evaluation that deserve further clarification and analysis. Below, we address these concerns through additional evaluations of contextual appropriateness, pipeline latency, stochastic refinement, and reaction timing.

[W1 & Q1] Contextual Appropriateness Beyond Motion Metrics

Reviewer (W1 & Q1): "...do not directly establish whether expressions and listener responses occur at contextually appropriate moments. Beyond the user study, could LLMs or VLMs help quantify contextual appropriateness?"

Response. We further evaluate contextual appropriateness using two complementary approaches: event-level temporal metrics and video-based human ratings. While video-capable VLMs offer a possible alternative, their reliability in assessing subtle facial reactions and reaction timing remains uncertain. We therefore focus on directly measuring reaction-event alignment and evaluating the contextual suitability of generated listener behaviors through human judgments.

Event-level evaluation. We identify listener reaction events from motion trajectories and evaluate their agreement with ground-truth reactions using F1 and temporal IoU (tIoU). We additionally evaluate smile events to examine whether generated expressions occur at appropriate moments.

Human evaluation. To assess contextual appropriateness (App.), we render the generated 3D facial motions as listener videos using PIRenderer (Ren et al., 2021), paired with the corresponding speaker audio. Ten human evaluators rate the contextual appropriateness of each listener video on a five-point Likert scale, considering both the suitability and timing of facial reactions. App. is reported as the mean opinion score (MOS).

Contextual Appropriateness Evaluation on ViCo

MetricListenFormerCoarse (w/o RM)REALM (Ours)
Reaction F1 ↑0.540.610.72
Smile F1 ↑0.480.550.69
Reaction tIoU ↑0.410.490.58
App. (1–5) ↑3.73.24.1

REALM achieves the highest scores across all four metrics, including Reaction F1 (0.72), Smile F1 (0.69), and Reaction tIoU (0.58), indicating better agreement with the timing of observed listener reactions. Human evaluators also rate REALM highest in contextual appropriateness (4.1/5), suggesting that its generated expressions and responses better match the conversational context.

We will clarify this evaluation protocol and include the additional results in the revised manuscript.

[W2 & Q2] Offline Robot Demonstrations and Pipeline Latency

Reviewer (W2 & Q2): "Could the authors clarify whether the robot demonstrations use precomputed trajectories and report the latency and duration of the evaluated pipeline?"

Response. Yes. As noted in the limitations (Sec. 5), REALM currently operates on offline conversational windows, and all robot demonstrations use precomputed trajectories. Motion is generated offline, retargeted, and then played back on Ameca at 30 FPS. The demonstrations establish the feasibility of physical embodiment, not real-time interaction, and we will state this explicitly in Sec. 4.2.

Demonstration Duration. As described in Appendix E.1, the Ameca user study uses conversational sequences sampled from the ViCo test set, each approximately 15–30 seconds long. The generated facial motions are retargeted and executed on the physical Ameca robot, and its actual facial movements are recorded on video. Twenty-five evaluators each viewed five randomly selected sequences, comparing recordings of REALM and five baselines under the same robotic setup.

Compute time per 16-frame window (mean over 1,000 windows after warm-up; RTX 5090 for the model stages, host CPU for post-processing)

StageCompute time (ms)
Wav2Vec 2.0 audio encoding3.8
REALM inference1.4
Retargeting and clamping0.2
Savitzky–Golay smoothing0.1
Total5.5

These are compute times, not end-to-end response latency. As an audio-driven framework, REALM does not require image or video inputs during inference, avoiding the computational overhead of visual feature extraction. Its lightweight architecture also makes real-time deployment computationally promising.

References

  • Ren, Y., Li, G., Chen, Y., Li, T. H., and Liu, S. (2021). PIRenderer: Controllable portrait image generation via semantic neural rendering. ICCV.
AuthorsOpenReview ↗

Response to Reviewer yVmN (Part 2/2)

[W3 & Q3] Contribution of Stochastic Sampling

Reviewer (W3 & Q3): "An audio-conditioned deterministic refinement variant, retaining the same architecture and training objectives while removing the injected noise, would help establish the contribution of stochasticity."

Response. We trained an audio-conditioned deterministic refinement variant that retains the same architecture, audio-dependent shift β\beta, and Stage II training objectives (L1 + adversarial), but sets ϵ=0\epsilon = 0 during training and inference. This allows us to isolate the contribution of stochastic sampling from refinement capacity and audio conditioning.

Deterministic vs. Stochastic Refinement on ViCo Test Set

MethodL1L_1 (Exp.) ↓FIDΔfm\text{FID}_{\Delta fm} (Exp.) ↓Blink Rate Error (BPM) ↓Blink Duration Error (ms) ↓
Without RM11.8512.4910.5074.65
Unmodulated noise12.6414.8210.39341.32
Deterministic RM (ϵ=0\epsilon = 0)11.636.849.1246.20
Stochastic RM (Ours)11.583.917.4018.45

Deterministic refinement improves expression FIDΔfm\text{FID}_{\Delta fm} from 12.49 to 6.84, while stochastic refinement further reduces it to 3.91. Similar improvements are observed in blink dynamics, with duration error decreasing from 46.20 to 18.45 ms. In contrast, unmodulated noise degrades motion quality, highlighting the importance of audio-conditioned stochastic refinement.

Both refinement variants receive identical speaker audio, so their performance differences cannot be attributed to variations in the audio input. We will incorporate this ablation into Table 3 and Appendix D.

[W4 & Q4] Validating the Reaction Delay

Reviewer (W4 & Q4): "...it may simply reflect dataset-specific patterns. (For instance, the delay involved in discussing a difficult question is not the same as that in a simple conversation.) Could we find a better way to evaluate the frames we delay?"

Response. We provide an additional Speaker-Cue-to-Listener-Reaction Timing Evaluation to directly assess when generated listener reactions occur relative to speaker cues. We also outline future directions for modeling and evaluating reaction timing across different conversational contexts.

(1) Direct evaluation of reaction delay. Following prior work on prosodic cues and listener backchannels (Ward and Tsukahara, 2000), we extract speaker acoustic cues from ViCo audio, including energy peaks and pause onsets, and match them with subsequent listener motion onsets. We measure Median Lag (cue-to-reaction delay) and Lag MAE (generated versus ground-truth delay error). We also assess Timing Naturalness through human ratings of listener videos presented alongside synchronized speaker audio.

Speaker-Cue-to-Listener-Reaction Timing Evaluation

Delay Prior τ\tau (frames)Median Lag (frames)Lag MAE (frames) ↓Timing Naturalness (1–5) ↑
03.03.423.6
45.02.353.9
8 (Ours)8.01.484.3
1210.52.843.8

Among the tested settings, τ=8\tau=8 achieves the lowest reaction-delay error (1.48 frames) and highest timing-naturalness rating (4.3). This provides more direct evidence for the selected delay prior than the motion-quality metrics in Table 4.

(2) Context-dependent reaction timing. We plan to investigate context-dependent timing by (i) expanding training and evaluation data to cover more diverse conversational content and complexity levels, (ii) incorporating a learned context-dependent adjustment into the current delay prior, and (iii) validating reaction timing across different conversational domains and datasets.

References

  • Ward, N., & Tsukahara, W. (2000). Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics.
Reviewer_YgEdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes REALM, a framework for reactive listener facial motion generation from speaker audio and listener motion history. It combines Reactive Gated Speaker–Listener Fusion, which uses a delay-centered attention prior and learned gate to balance speaker cues with listener history, with coarse-to-fine stochastic refinement that adds audio-conditioned residuals to non-rigid facial expressions while preserving coarse head pose.

Experiments on ViCo and L2L show improvements over evaluated baselines across accuracy, motion distribution, temporal dynamics, and speaker-listener correlation, supported by ablations and a perceptual study on an Ameca robot. Overall, the method is technically sound and well evaluated, but concerns remain about moderate novelty, incomplete validation of the claimed temporal reaction mechanism, and limited evidence that stochastic refinement captures true conditional multimodality rather than mainly improving local motion statistics.

Critical strengths
  1. The coarse-to-fine decomposition is well motivated in the paper, separating stable head pose from stochastic expression refinement. Restricting stochastic residuals to expression is a sensible inductive bias that avoids destabilizing rigid motion.
  2. REALM performs well on both ViCo and L2L, particularly on temporal expression metrics, while the out-of-domain ViCo evaluation provides useful evidence of generalization.
  3. The component study clearly isolates the contributions of gated fusion, refinement, and shifted attention. Refinement provides especially strong gains in temporal expression dynamics while leaving pose unaffected, consistent with the proposed decomposition.
  4. The delay, gating, and blink analyses go beyond aggregate benchmark metrics and provide useful evidence that individual components behave as intended.
  5. Deployment on Ameca strengthens the evaluation beyond coefficient-space results and demonstrates practical viability, including retargeting, smoothing, calibration, and actuator constraints.
Critical weaknesses
  1. Most components build on established techniques. The main contribution is their task-specific integration for reactive listener generation, making the conceptual novelty less strong than the empirical contribution.
  2. The delay study supports the shifted-attention prior but does not directly demonstrate actual speaker-cue-to-listener-response delays. A direct analysis linking speaker cues to subsequent listener reactions would strengthen this claim. 3.The problem studied is an ill-defined issue. Stochastic refinement permits multiple outputs, but the experiments do not show that different noise samples produce diverse, plausible reactions for the same conditioning input.
  3. The gate analysis shows association with listener motion rather than causation. The paper should consistently frame these results as evidence consistent with the intended mechanism.
  4. The subjective study reports mean scores and significance tests, but confidence intervals or dispersion measures and clearer specification of the statistical unit would improve interpretation. Closely related generative approaches are discussed but not quantitatively compared, limiting broad state-of-the-art claims. Conclusions should therefore be framed relative to the evaluated methods.
  5. The Ameca study is good but also reflects representation, retargeting, calibration, and embodiment choices, making it difficult to isolate REALM's motion-generation quality.
Questions for authors
  1. Can the authors directly evaluate whether shifted attention improves speaker-cue-to-listener-response timing?
  2. For identical speaker audio and listener history, how diverse, faithful, and plausible are outputs across different ϵ samples? Does changing ϵ materially alter the generated expression trajectory, indicating no noise/posterior collapse?
  3. What are the dimensions of Zc, γ, β, and ϵ in Eqs. 4, 10, and 11, and how are γ and β parameterized?
  4. Can the authors report dispersion or confidence intervals for Ameca MOS and clarify the statistical unit in the Wilcoxon analysis?
  5. With the same retargeting and calibration, how does coarse REALM perceptually compare with full REALM?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YgEd (Part 1/2)

[W1] Methodological Novelty

Reviewer (W1): "Most components build on established techniques ... making the conceptual novelty less strong than the empirical contribution."

Response. REALM's contribution lies in its task-specific methodological design rather than in individual building blocks. It addresses response timing, motion continuity, and stochastic facial dynamics, and extends the generated behaviors to a physical robot:

ChallengeREALM DesignEvidence
Aligning responses with speaker cuesDelay-centered attention priorTable 4; W2 analysis
Balancing responsiveness and motion continuityReactive gated fusionTable 3; Appendix A
Generating variable expressionsExpression-specific stochastic refinementTable 3; Appendix D
Transferring generated listener behavior to a physical robotMotion retargeting and relative neutral calibration, validated on AmecaSec. 4.2; Table 5

Our ablations (Table 3) support their effectiveness, with expression FIDΔfm\text{FID}_{\Delta fm} decreasing from 12.49 without refinement to 3.91 in the full model, and the Ameca experiments demonstrate the feasibility of physical deployment. We will clarify these task-specific methodological and embodiment contributions in the revised manuscript, without claiming novelty for the individual building blocks.

[W2 Part-1 & Q1] Speaker-Cue-to-Listener-Response Timing

Reviewer (W2 Part-1 & Q1): "A direct analysis linking speaker cues to subsequent listener reactions would strengthen this claim."

Response. We conduct a speaker-cue-to-listener-reaction timing analysis on the ViCo test set. We identify speaker acoustic cues and match them with subsequent listener reactions using the velocity-based onset definition in Appendix A.1.

We evaluate Median Lag (cue-to-reaction delay), Lag MAE (generated versus ground-truth delay error), and Onset Hit@5 (reactions aligned within ±5 frames).

Speaker-Cue-to-Listener-Reaction Timing Evaluation

Delay Prior τ\tau (frames)Median Lag (frames)Lag MAE (frames) ↓Onset Hit@5 (%) ↑
03.03.4258.6
45.02.3571.3
8 (Ours)8.01.4883.9
1210.52.8464.7

Among the tested settings, τ=8\tau=8 achieves the lowest Lag MAE (1.48 frames) and highest Onset Hit@5 (83.9%), matching the ground-truth median reaction lag of 8.0 frames. Here, τ\tau defines the center of a soft attention prior rather than a fixed response delay, allowing the model to attend to speaker cues from other time steps.

We will include this analysis and clarify the speaker-cue matching protocol in the revised manuscript.

[W2 Part-2 & Q2] Stochastic Diversity and Plausibility

Reviewer (W2 Part-2 & Q2): "...how diverse, faithful, and plausible are outputs across different ε samples?"

Response. We conduct a multi-sample evaluation on ViCo test clips, fixing the speaker audio, listener history, and coarse prediction while generating 10 refined trajectories with independent noise samples ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I). We also include an unmodulated-noise control (Appendix D, Table 8). For contextual appropriateness (App.), we render the generated facial motions into listener videos using PIRenderer (Ren et al., 2021). Ten evaluators rate these videos alongside the corresponding speaker audio on a five-point scale.

Multi-sample evaluation on ViCo (10 samples per clip)

MetricCoarse MotionUnmodulated NoiseREALM (Ours)
Speed Wasserstein-1 ↓0.03870.05420.0101
Jerk RMS / GT (→1)0.00742.38400.8334
Speaker–Listener Correlation Gap ↓0.25720.28950.2264
Conditional APD00.06850.0410
Smoothed Conditional APD00.01480.0247
APD Retained After Smoothing–21.6%60.2%
App. Mean (1–5) ↑3.22.64.1
App. Worst-of-5 (1–5) ↑3.21.93.9

Objective metrics use 10 samples per clip; App. uses five. Metric definitions and the evaluation protocol are given in our response to Reviewer 5Wqm (W2 & Q2).

Changing ϵ\epsilon produces measurable conditional diversity (APD 0.0410). After smoothing, 60.2% of variation remains, versus 21.6% for unmodulated noise, indicating less reliance on high-frequency fluctuations. REALM also improves motion fidelity (Speed Wasserstein-1 0.0101) and human App. (4.1, versus 3.2 and 2.6); its worst-of-5 App. of 3.9 further supports plausible alternatives. We will include this analysis in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer YgEd (Part 2/2)

[W3] Gate Analysis: Association, Not Causation

Reviewer (W3): "The gate analysis shows association with listener motion rather than causation. The paper should consistently frame these results as evidence consistent with the intended mechanism."

Response. As already acknowledged in Appendix A.1–A.2, the gate analysis does not establish speaker-to-listener response delays or demonstrate that gate activations cause motion changes.

Our analysis shows that the observed gate behavior is consistent with its intended role in balancing speaker context and listener history. We will further clarify this associative rather than causal interpretation throughout the manuscript.

[W4 & Q4] Statistical Reporting and Baseline Scope

Reviewer (W4 & Q4): "...Closely related generative approaches are discussed but not quantitatively compared ... Can the authors report dispersion or confidence intervals for Ameca MOS and clarify the statistical unit in the Wilcoxon analysis?"

Response. Our Ameca study includes 25 participants, each evaluating five sequences across six methods. We use the participant (N = 25) as the statistical unit, averaging the five sequence ratings per participant before conducting paired Wilcoxon signed-rank tests. We additionally report 95% confidence intervals estimated through participant-level bootstrap resampling.

Ameca User Study — Mean MOS ± 95% CI Half-Width

MethodNat. ↑Syn. ↑App. ↑Pref. ↑
RLHG2.2 ± 0.212.4 ± 0.222.4 ± 0.232.2 ± 0.20
DSPN2.4 ± 0.222.6 ± 0.242.8 ± 0.232.4 ± 0.22
L2L2.8 ± 0.243.0 ± 0.233.2 ± 0.222.8 ± 0.23
UniLS3.2 ± 0.233.6 ± 0.213.8 ± 0.203.4 ± 0.22
ListenFormer3.6 ± 0.203.8 ± 0.194.0 ± 0.183.6 ± 0.20
REALM (Ours)4.4 ± 0.184.2 ± 0.194.2 ± 0.194.0 ± 0.18

Regarding baseline scope, our conclusions are limited to the evaluated methods. Appendix F.2 already discusses ARIG and Wang et al. (2025), whose different conditioning inputs and motion representations make direct quantitative comparisons difficult under our evaluation protocol. We will make this scope clearer in the revised manuscript.

[W5 & Q5] Motion Generation and Embodied Perception

Reviewer (W5 & Q5): "...difficult to isolate REALM's motion-generation quality. With the same retargeting and calibration, how does coarse REALM perceptually compare with full REALM?"

Response. REALM's motion-generation quality is evaluated separately on ViCo and L2L (Tables 1–4), while the Ameca study assesses its perceptual quality after physical embodiment.

To isolate the contribution of the refinement module, we further compare Coarse REALM (without RM) and Full REALM using identical speaker audio, motion retargeting, smoothing, calibration, and robotic hardware.

Perceptual Refinement Ablation on Ameca (N = 25 Evaluators)

VariantNaturalness ↑App. ↑Pairwise Preference ↑
Coarse REALM3.73.921.6% (27/125)
Full REALM (Ours)4.44.278.4% (98/125)

Naturalness and App. are rated on a 1–5 scale. Pairwise preference is computed from 125 comparisons (25 participants × 5 sequences).

Under identical embodiment settings, Full REALM achieves higher Naturalness and Contextual Appropriateness, and is preferred in 78.4% of paired comparisons. This supports the perceptual contribution of refinement beyond the shared retargeting and calibration pipeline.

[Q3] Latent Dimensions and Audio Modulation

Reviewer (Q3): "What are the dimensions of Zc, γ, β, and ε in Eqs. 4, 10, and 11, and how are γ and β parameterized?"

Response. The coarse latent ZcZ^c, modulation scale γ\gamma, shift β\beta, and noise ϵ\epsilon have matching dimensions, Zc,γ,β,ϵ∈RB×L×daZ^c,\gamma,\beta,\epsilon\in\mathbb{R}^{B\times L\times d_a}, where BB is the batch size, LL is the sequence length processed by the refinement module, and da=128d_a=128 is the shared latent dimension. Training sequences use 128 frames on ViCo and 64 on L2L, while autoregressive inference uses 16-frame windows.

The audio modulator takes speaker features EA∈RB×L×daudioE_A\in\mathbb{R}^{B\times L\times d_{\mathrm{audio}}} and consists of Conv1D(daudio→dad_{\mathrm{audio}}\to d_a, k=3k=3) → GELU → Conv1D(da→2dad_a\to 2d_a, k=1k=1). The output is split along the channel dimension into γ\gamma and β\beta, both matching ZcZ^c element-wise, and Zr=Zc+β+γ⊙ϵZ^r=Z^c+\beta+\gamma\odot\epsilon with ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I).

We will clarify these dimensions and correct the broadcasting description in Appendix B.2.

Reviewer_YgEdOpenReview ↗

Comment to author's response

The author has addressed my concern in methodological novelty. It is good to make claim with certain constrain/condition. [W1]
The Speaker-Cue-to-Listener-Reaction Timing Evaluation and the Multi-sample evaluation on ViCo (10 samples per clip) have addressed my concern in linking speaker cues to subsequent listener reactions and stochastic diversity and plausibility. It helps to show pros and cons of REALM from multiple perspectives. [W2]

Reviewer_5WqmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents REALM, a coarse-to-fine framework for audio-driven listener facial motion generation. It combines delay-aware attention and adaptive gating with stochastic expression refinement. Experiments on ViCo and L2L demonstrate improvements over the evaluated baselines, particularly in inter-frame expression dynamics.

Critical strengths
  1. The separation between coarse motion and expression-specific stochastic refinement is clear and well motivated. The ablation results support the claim that refinement mainly improves dynamic expression fidelity while preserving pose.
  2. The robot embodiment and perceptual study provide useful evidence about the practical applicability of the generated motion, complementing the quantitative results on ViCo and L2L.
Critical weaknesses
  1. The gate-onset analysis provides useful evidence of temporal association, and the authors appropriately acknowledge that the nearest-peak statistic does not establish speaker-to-listener response delay or isolate the contribution of shifted attention. However, without a chance-alignment baseline that accounts for gate-peak density, the strength of the observed alignment remains difficult to assess. This limits how strongly the analysis supports the proposed gating mechanism.
  2. Although the paper evaluates motion variability and blink dynamics, it remains unclear how well the stochastic refinement captures diverse yet contextually appropriate listener behaviors. Evaluating multiple samples generated from the same audio context could provide further evidence of whether the stochastic refinement produces meaningful behavioral alternatives rather than primarily increasing motion variation.
Questions for authors

Clarification:

  1. Could the authors provide a chance-alignment baseline that accounts for gate-peak density? This would help assess whether the observed alignment between gate peaks and listener motion onsets is stronger than expected by chance, and clarify how strongly the analysis supports the proposed gating mechanism.
  2. Could the authors evaluate multiple samples generated under the same audio context to better assess the diversity and contextual appropriateness of the stochastic refinement? Such an analysis could help distinguish meaningful behavioral diversity from variation reflected mainly in aggregate motion statistics.
  3. Are the reported ViCo and L2L metrics computed through autoregressive rollout using previously generated listener motion, or by conditioning on ground-truth history at each step? Could the authors clarify the evaluation protocol and, if applicable, report long-horizon behavior to assess stability?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5Wqm (Part 1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We particularly appreciate the reviewer's recognition of REALM's coarse-to-fine design and the value of our embodied evaluation. The comments have helped us strengthen our analysis of temporal alignment, stochastic diversity, and autoregressive generation. Below, we address the related concerns through additional chance-alignment and multi-sample evaluations, together with a clarification of the autoregressive evaluation protocol and history-length sensitivity.

[W1 & Q1] Gate–Onset Alignment and Chance Baseline

Reviewer (W1 & Q1): "...without a chance-alignment baseline that accounts for gate-peak density, the strength of the observed alignment remains difficult to assess."

Response. We conducted an additional density-preserving chance-alignment analysis on motion onsets from the ViCo test set.

Following standard event-detection evaluation practice (Stuetzle et al., 2010), we report Hit@k, the percentage of ground-truth motion onsets with at least one gate peak within ±k frames, for k = 3, 4, and 5 (approximately 100, 133, and 167 ms at 30 FPS).

For the chance baseline, we adopt circular time-shift surrogates (Deco et al., 2021). Within each clip, the gate-peak sequence is circularly shifted by a random offset while the motion onsets remain fixed. This preserves gate-peak density and inter-peak spacing while disrupting the original correspondence. We repeat this procedure 5,000 times and report the surrogate mean, its 95% interval, and the empirical p-value, computed as (r + 1)/(5,000 + 1), where r is the number of surrogates whose Hit@k is at least the observed value.

Gate–onset alignment on ViCo test set (N = 186 onsets)

MetricGate peaks (REALM) ↑Chance mean [95% interval]Δ\Delta (pp) ↑Empirical pp ↓
Hit@346.24%39.94% [33.87%, 46.24%]+6.300.0224
Hit@458.06%49.70% [43.01%, 56.45%]+8.360.0052
Hit@572.04%58.59% [52.15%, 64.52%]+13.450.0002

Gate peaks coincide with motion onsets more often than the density-preserving baseline at all three tolerances, and the margin grows with the tolerance window, from +6.30 pp at Hit@3 to +13.45 pp at Hit@5. We interpret this as an above-chance temporal association between gate activity and listener motion onsets.

We will add this protocol and these results to Appendix A.1. We will also revise the statement in Sec. 3.2 that gate peaks "align closely" with reaction onsets, so that it describes an above-chance temporal association.

References

  • Stuetzle, W., Percival, D. B., and Marzban, C. (2010). Targeted event detection. arXiv:1003.2823.

  • Deco, G., Vidaurre, D., and Kringelbach, M. L. (2021). Revisiting the global workspace orchestrating the hierarchical organization of the human brain. Nature Human Behaviour.

AuthorsOpenReview ↗

Response to Reviewer 5Wqm (Part 2/2)

[W2 & Q2] Stochastic Refinement: Behavioral Diversity and Contextual Appropriateness

Reviewer (W2 & Q2): "...whether the stochastic refinement produces meaningful behavioral alternatives rather than primarily increasing motion variation."

Response. We conducted a multi-sample evaluation on ViCo test clips. For each clip, we fix the speaker audio, listener history, and coarse motion prediction, and generate 10 refined expression trajectories using independent noise samples. We also include an unmodulated-noise control from Appendix D (Table 8) to examine the role of audio conditioning in producing realistic facial dynamics.

Human evaluation protocol. To assess contextual appropriateness (App.), we render the generated 3D facial motions as listener videos using PIRenderer (Ren et al., 2021), paired with the corresponding speaker audio. Ten human evaluators rate each video on a five-point Likert scale. App. is reported as the mean opinion score (MOS). The same rating criteria are used for the history-length comparison in Q3.

Multi-sample evaluation on ViCo (10 samples per clip)

MetricCoarse motionUnmodulated noiseREALM (ours)
Speed Wasserstein-1 ↓0.03870.05420.0101
Jerk RMS / GT (closer to 1)0.00742.38400.8334
Speaker–listener correlation gap ↓0.25720.28950.2264
Conditional APD00.06850.0410
Smoothed conditional APD00.01480.0247
APD retained after smoothing–21.6%60.2%
App. mean (1–5) ↑3.22.64.1
App. worst-of-5 (1–5) ↑3.21.93.9

Objective metrics use 10 samples per clip; App. uses five. Worst-of-5: the lowest-rated sample per clip (ratings averaged over evaluators), averaged over clips; coarse motion is deterministic, so its APD is 0 and its worst-of-5 equals its mean. Speed Wasserstein-1: distance between generated and GT expression-speed distributions. Jerk RMS / GT: ratio of RMS third-order temporal differences. Correlation gap: mean absolute deviation of per-channel speaker–listener Pearson correlations from GT. Conditional APD: mean pairwise L2 distance between samples under identical conditioning, smoothed with a Gaussian filter (σ = 2 frames).

Stochastic refinement reduces the gap to ground-truth motion dynamics while introducing measurable differences between samples produced from the same context. About 60% of the conditional APD remains after temporal smoothing, suggesting that these differences are not limited to frame-level fluctuations. The higher App. scores further indicate that the refined motions remain more appropriate to the speaker context, both on average and in the worst-of-5 comparison.

[Q3] Autoregressive Evaluation and History-Length Sensitivity

Reviewer (Q3): "Are the reported ViCo and L2L metrics computed through autoregressive rollout ... or by conditioning on ground-truth history at each step? ... report long-horizon behavior to assess stability?"

Response. We clarify that the reported ViCo and L2L metrics are obtained through autoregressive rollout using previously generated listener motion, without conditioning on ground-truth listener history at each prediction step. REALM generates motion in windows of W = 16 frames; the generated motion is then fed back as context for subsequent predictions.

The history-length parameter H corresponds to max_listener_window in our implementation and specifies the maximum number of preceding generated windows used as context. Our default configuration uses H = 1. To examine the effect of this choice under continuous autoregressive inference, we vary H while keeping the model and other settings fixed.

History-length sensitivity on ViCo test set

History (H windows)L1 (Exp.) ↓L1 (Pose) ↓FIDΔfm\text{FID}_{\Delta fm} (Exp.) ↓rPCC (Exp.) ↓App. (1–5) ↑
H = 111.58296.52263.91060.00914.1
H = 212.84246.83774.12260.01523.8
H = 413.38217.37384.36810.02923.6
H = 814.39218.07954.75600.04022.0

App.: same rating criteria as W2, one generated sequence per clip and setting.

The default H = 1 yields the lowest errors and the highest App. among the tested settings, supporting the selected configuration. However, varying the retained history measures history-length sensitivity and does not alone establish the absence of drift over longer rollouts. We therefore do not claim long-horizon stability beyond this evaluation, and will clarify the autoregressive protocol, history-buffer update, and scope of this analysis in the revised manuscript.

References

  • Ren, Y., Li, G., Chen, Y., Li, T. H., and Liu, S. (2021). PIRenderer: Controllable portrait image generation via semantic neural rendering. ICCV.

The authors have addressed my concerns, including the chance-alignment baseline, the multi-sample diversity and contextual appropriateness analysis, and the clarification of the autoregressive evaluation protocol. My only remaining reservation is that long-horizon stability under autoregressive rollout is still not empirically evaluated, though the authors appropriately scope this claim.

AuthorsOpenReview ↗

Response to Reviewer Follow-up

We sincerely thank the reviewer for the thoughtful follow-up and for recognizing that our additional analyses have addressed the main concerns. We will further investigate long-horizon stability under autoregressive rollout through systematic empirical evaluations as part of our future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.