acceptodds
Under review as a conference paper at ICLR 2027

Semantic-REACT: Temporally Grounded Linguistic Conditioning for Multiple Appropriate Facial Reaction Generation

Abstract

In dyadic human-human interactions, linguistic semantics help human listeners interpret the speaker's verbal behaviour and determine when to express appropriate facial reactions (AFRs). Existing automatic Multiple Appropriate Facial Reaction Generation (MAFRG) methods typically generate listener facial reactions conditioned on speaker audio-visual behaviour, where encoded speech representations primarily capture acoustic cues, without explicitly modelling word-level semantics or when those semantics are relevant to the listener’s response. Consequently, the generated AFRs are usually appear plausible in motion while remaining semantically mismatched or temporally misaligned with the interaction context. To address this limitation, we propose Semantic-REACT, the first MAFRG solution that specifically encodes speaker's verbal transcripts and aligns the resulting linguistic features with the generated listener facial motion timeline. A gated adapter then injects these aligned linguistic features into the diffusion denoiser. This linguistic conditioning provides temporally aligned guidance while preserving the original speaker audio-visual conditions and the diversity of the generated listener AFRs. Experiments on the official REACT 2025 MARS benchmark show that Semantic-REACT improved the appropriateness of the generated listener facial reaction compared to Trans-VAE, PerFRDiff, and REGNN while preserving decent diversity. Matched control experiments confirmed the importance of temporal alignment for linguistic conditioning. Qualitative analysis and human evaluations further indicate that the generated listener facial reactions exhibit temporal coherence, semantic appropriateness, and favourable perceptual quality.

Primary area:
generative models
Keywords:
Multiple Appropriate Facial Reaction Generation, Linguistic Conditioning, Temporal Alignment, Diffusion Models, Dyadic Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vy3aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how a speaker’s words can be used to generate appropriate facial reactions. Semantic-REACT extracts words and timestamps from speech, converts them into textual features, and aligns these features with the corresponding video frames. An adapter and a gating mechanism then integrate these features into an existing audio-visual diffusion model. The paper also introduces a separate branch that predicts response opportunities and reaction categories.

The method is evaluated on MARS using the REACT 2025 offline protocol. Across five runs, its FRC is 0.7058, compared to 0.6705 without text conditioning and 0.6594 when the text features are temporally shifted. A user study compares the generated reactions with recorded listener reactions, with the latter receiving higher preference and ratings.

Critical strengths

None

Critical weaknesses
  1. The semantic reaction prediction branch is not clearly explained. It is unclear how it is trained, how it operates during inference, and whether its outputs influence the main generator. Additionally, its contribution relative to existing response-timing and turn-taking methods, including those used in full-duplex dialogue models, is not well established.

  2. The temporal linguistic alignment module repeats each word embedding across its speech interval. However, a word may influence a reaction even after it has ended, and its influence does not necessarily remain constant during its pronunciation. The paper demonstrates that this alignment strategy improves performance in the evaluated setting but provides limited theoretical justification or targeted experimental analysis to explain why it works or whether it accurately captures the temporal relationship between spoken words and listener reactions.

  3. The motivation for including speaker transcripts is not sufficiently clear. The model already receives frame-level audio features that may contain linguistic information. Additionally, the paper does not discuss the latency introduced by transcription. Although the evaluation is conducted offline, these issues are relevant to the paper’s motivation of generating timely listener responses.

Questions for authors

The paper still has several significant issues, as outlined in the Critical Weaknesses section, and I do not believe it currently meets the standards for acceptance at ICLR. A substantial revision will be needed before resubmission.

The figures require clearer explanations. Fig. 1 and 2 both provide an overview, but their distinct purposes are unclear. Meanwhile, Fig. 3 displays individual components without clearly explaining how they interrelate.

2 comments
AuthorsOpenReview ↗

Response to Reviewer vy3a on Word Timing and the Role of the Semantic Branch

Thank you for pointing out that a listener may react after a word has ended. We repeat the embedding across the spoken interval to mark where that word occurred. Motion frames can give the same token different attention weights, and a later frame can still attend to it after the word ends (§3.2, p. 6). As you note, the audio features can carry lexical information. The transcript gives us an explicit word identity and interval to place on the motion timeline.

In the checkpoint-matched comparison, mean FRC was 0.7058 with the original alignment and 0.6594 after shifting the conditioning sequence. The paired gain was 0.0464 (95% CI [0.0384, 0.0544], §4.4, p. 8). For the offline evaluation, Whisper-tiny processes the full speaker clip to produce the transcript and timestamps before generation (§4.1, p. 7). An end-to-end latency figure for streaming would include transcription time.

The response-opportunity scores and reaction categories next to the motion in Fig. 5 come from the separate semantic branch. We use them to interpret the examples (§3, p. 4). The diffusion generator takes the speaker’s audio-visual inputs and aligned word features. Its denoising loss trains the adapter, gate, and cross-attention modules in the last two decoder layers (§3.3). Turn-taking and full-duplex work are concerned with the timing of speech. Here we generate continuous facial feedback, some of which occurs while the speaker still has the floor.

Fig. 1 puts an example interaction on a timeline. Fig. 2 follows the inputs through the model. The revised Fig. 3 caption follows the aligned features in panel (b) into the adapter and gate in panel (c) (pp. 4–5).

Reviewer_vy3aOpenReview ↗

Official Comment by Reviewer vy3a

I thank the authors for their response and apologize for my delayed reply. The revision clarifies how the semantic predictions are used in fig.5. However, it remains unclear how the predictor was trained and how it generated the response-opportunity scores and reaction categories shown in the figure. This concern remains unresolved, and I maintain my original score.

Separately, the main paper pdf contains a one-page rebuttal, while the supplementary material contains the revised manuscript. I'm not sure if this violates the formatting regulations.

Reviewer_njcLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Semantic-REACT uses word-level linguistic features aligned to listener motion to condition an audio-visual diffusion model for facial reaction generation. Experiments on MARS show gains over unconditioned and temporally shifted controls, achieving the highest FRC among listed baselines. A user study evaluates semantic and perceptual quality.

Critical strengths
  1. The matched controls are well designed. Comparing the aligned condition with both a no-language model and a checkpoint-matched temporal shift directly tests whether timing, rather than merely adding text features, matters.
  2. The method is lightweight and preserves the underlying stochastic generator. The five-run paired results consistently favor aligned linguistic conditioning across the reported metrics.
Critical weaknesses
  1. The semantic branch is evaluated mainly through qualitative examples, without quantitative validation against annotated response opportunities or reaction categories. Moreover, REGNN outperforms Semantic-REACT on FRD and TLCC, while generated reactions are preferred in only 42.7% of the user-study comparisons with recorded reactions.
  2. Although the linguistic features are aligned to listener-motion frames before being appended to the cross-attention memory, it remains unclear how the denoiser exploits this temporal correspondence. Further explanation of the temporal information available to the denoiser would strengthen the method’s technical clarity.
Questions for authors

Clarification:

  1. What transcripts and word-level timestamps are used during evaluation, and how robust is the method to ASR and alignment errors? Could performance be evaluated with perturbed transcripts or timestamps?
  2. How does the denoiser exploit the temporal correspondence of the appended linguistic tokens, and how exactly is the temporally shifted control implemented?

Suggestion for future work: Quantitatively evaluate the semantic branch against annotated response opportunities and reaction categories.

3 comments
AuthorsOpenReview ↗

Response to Reviewer njcL on Temporal Alignment and the Matched Shift Control

Thanks for asking how we applied the shift. We use the same checkpoint for both conditions in a pair and move the conditioning sequence along the frame axis at inference. The feature values, audio-visual inputs, and sampling settings stay fixed (§4.4, p. 8 and Supp. §S2). Mean FRC went from 0.7058 with the original alignment to 0.6594 after the shift. Across the five checkpoint pairs, that gives an advantage of 0.0464 for alignment (95% CI [0.0384, 0.0544]).

Disabling the linguistic path gave a mean FRC of 0.6705. The shift is the comparison we use to test temporal placement, since the generator sees the same feature values at different times. Word recognition errors can change the features themselves, so they pose a different question from the timing perturbation in this control.

At test time, frozen Whisper-tiny transcribes the speaker audio and supplies word timestamps (§4.1, p. 7 and Supp. §S1.1). In the configuration described in §S1.1, the loader reads the 64 word-feature channels from an 80-channel file. We use each word’s interval to place its features on the 30 fps motion timeline. After the adapter and gate, the resulting tokens sit alongside the audio-visual tokens in the denoiser’s shared cross-attention memory. A motion query can attend to a word token assigned to an earlier frame (§3.2, p. 6).

REGNN’s lower FRD indicates closer matches to reference trajectories under that measure. Its lower TLCC reflects a smaller measured lag. In the preference study, raters chose the generated reaction over the recording in 160 of 375 comparisons (42.7%, §4.5, p. 8). In Fig. 5, we use the semantic branch’s response-opportunity and category predictions to interpret the example motions qualitatively.

I thank the authors for clarifying the temporal-shift control. However, quantitative validation of the semantic branch against annotated response opportunities and reaction categories remains unaddressed; the semantic predictions are used only qualitatively in Fig. 5. The weaker FRD/TLCC and the 42.7% preference relative to recorded reactions are restated but not explained.

AuthorsOpenReview ↗

Follow-up to Reviewer njcL on Semantic Predictions and Perceptual Evaluation

Thank you for following up. We use the semantic predictions in Fig. 5 to help interpret the generated examples. The generator itself uses the speaker’s audio-visual inputs and frame-aligned word features (§3). How accurately the semantic branch predicts annotated response opportunities and reaction categories remains open.

REGNN’s lower FRD and TLCC indicate closer matches to reference motion after time warping and a smaller lag at peak cross-correlation. In the checkpoint-matched comparison, keeping the original alignment lowered FRD by 14.0650 and TLCC by 0.2597 relative to shifting the conditioning sequence (Supplementary Table S4). This shows that temporal placement affects both metrics within the same generator. The comparison with REGNN involves two complete systems.

The 42.7% is the share of choices for generated reactions when paired with a real recording under the same speaker context. A generated reaction can look plausible on its own and still be less preferred in a forced choice. In the separate rating task, generated reactions averaged 7.45–7.52 out of 10 across the four dimensions, while recorded reactions averaged 8.73–8.81 (Table 3). These ratings put the preference result in context.

Reviewer_dtFcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Semantic-REACT, a framework for Multiple Appropriate Facial Reaction Generation (MAFRG) that temporally grounds the speaker's linguistic semantics onto the listener's facial motion timeline. By combining Whisper-tiny-based word timestamp extraction, frame-aligned linguistic tokens, and zero-initialized gated adapters, the proposed method enhances reaction appropriateness and temporal alignment while preserving the motion diversity of existing audio-visual diffusion generators.

Critical strengths

The authors formalize the problem of temporal grounding between speaker word-level timestamps and listener facial motion timelines, proposing a gated adapter architecture to stably integrate linguistic conditions into a diffusion denoiser. In particular, through a shifted control experiment where linguistic sequences were artificially displaced, the authors convincingly demonstrate that the improvement in reaction appropriateness stems directly from precise temporal alignment rather than the mere presence of linguistic features.

Critical weaknesses

While the paper asserts the utility of the proposed method, the experimental and logical support for the authors' claims remains somewhat insufficient overall, with several limitations observed in experimental setup and metric interpretation. The specific weaknesses are as follows:

  • Direct qualitative analysis and user evaluations against existing baseline models are missing. The authors present qualitative visualization results in Figure 5 and conduct a user survey in Table 3, but both are strictly one-on-one comparisons between the Recorded Listener (GT) and the proposed model. Considering the numerical limitations of automated evaluation metrics such as FRC, a 1-on-1 qualitative comparison and preference test against recent SOTA models such as PerFRDiff or REGNN should have been conducted to properly validate the qualitative superiority of the proposed model.

  • Reference values for GT motion across FRC, FRD, and FRVar are not included in Table 1 as upper bounds to gauge human-level performance.

  • REGNN achieves superior performance in S-MSE (0.0010) and FRD (155.49), excelling at closely reconstructing the Ground-truth motion, but suffers from low diversity (FRVar 0.0048). Conversely, Semantic-REACT achieves high diversity (FRVar 0.0775) and appropriateness (FRC 0.7058) at the expense of deterministic reconstruction metrics such as S-MSE. The authors dismiss this merely as a trade-off, but they should have provided an in-depth discussion on why diverse, stochastic reaction generation (Semantic-REACT) is a more valid approach for MAFRG than close deterministic prediction (REGNN), as well as on the intrinsic nature of these metrics' behaviors.

Questions for authors
  • In Figure 5, the Ground-truth motion renderings exhibit geometric distortion, with the mesh appearing crushed. Is this due to fitting artifacts during the 3DMM (e.g., FLAME) parameter estimation process from 2D videos?

  • While this paper adopts the fixed metrics and dataset provided by the new REACT challenge, evaluating the model alongside the datasets (NoXI and RECOLA) and metrics used in prior literature would significantly aid in gaining a multi-faceted analysis and deep understanding of the model. Given that baseline models (PerFRDiff, REGNN, etc.) have reported scores on NoXI and RECOLA under prior challenge protocols, could the authors evaluate Semantic-REACT on these datasets as well and provide a direct comparison against the reported baseline results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer dtFc on Evaluation and Ground-Truth References

Thank you for highlighting the shift control. Across five checkpoint-matched pairs, FRC was 0.0464 higher with the original alignment than after the shift (95% CI [0.0384, 0.0544], §4.4).

Two listeners can react appropriately to the same utterance with different expressions or at slightly different moments. Sampling gives the generator room to express that variation. FRC scores agreement with eligible references. FRD measures the distance to a reference after dynamic time warping. REGNN also samples reactions, and its lower FRD means closer matches on that measure. FRVar tracks how much a single trajectory changes over time. In the released evaluator, S-MSE averages squared differences between pairs of candidates for the same input (Song et al., 2025, §5, and the REACT 2025 baseline evaluator, S_MSE.py).

The benchmark gives FRCorr of 10, FRDist of 0.00, and FRVar of 0.0669 for the recorded motions. When a recording is included in its own reference set, it can match itself on FRCorr and FRDist. The FRVar value is an observed level of motion variation. A trajectory with larger changes over time can exceed it (Song et al., 2025, Table 3).

In Track 1, the raters compared a generated and a recorded reaction for the same speaker context. They chose the generated reaction 160 times out of 375 (42.7%). We added that count in §4.5 (p. 8). Table 1 compares PerFRDiff and REGNN with our method using benchmark scores. As for the mesh distortion you noticed, it appears in the separate ground-truth rendering in Fig. 5. We revised the caption to distinguish that panel from the original video and the generated motion (p. 9). We would be cautious about attributing the deformation to FLAME fitting from the rendered result alone.

Our runs use the MARS split and reference sets from the REACT 2025 offline protocol. For NoXI or RECOLA, the published comparisons use the splits and scoring rules of the earlier protocol.

I would like to thank the authors for their detailed responses, clarifying the metric definitions including S-MSE and FRVar, and adding the ground-truth reference values. While these clarifications addressed several initial concerns, key issues regarding direct qualitative and quantitative validation against existing baselines remain insufficiently resolved. Without head-to-head qualitative visualizations or user preference studies against recent baseline models, the proposed model's perceptual superiority remains unverified beyond automated metrics. Additionally, while following the new challenge protocol is understandable, the lack of broader comparisons across established standard benchmarks leaves the generalizability of the quantitative claims somewhat limited. Because the empirical validation against existing baselines is not fully convincing both qualitatively and quantitatively, I maintain my original score.

AuthorsOpenReview ↗

Follow-up to Reviewer dtFc on Baseline Comparisons

Thank you for following up. Table 1 compares our method with PerFRDiff and REGNN on the REACT 2025 offline MARS benchmark. In the user study, raters compared our generated reactions with recorded reactions to the same speaker context. A perceptual ranking of the models would require raters to compare their outputs directly.

The result we put most weight on is the paired shift in §4.4. We held the checkpoint and feature values fixed and moved the conditioning sequence along the frame axis. Mean FRC fell from 0.7058 to 0.6594, a paired advantage of 0.0464 for the original alignment (95% CI [0.0384, 0.0544]). That is the evidence for our claim about timing on MARS. A matched comparison on NoXI or RECOLA would test how well that finding carries over.

The alignment step maps timestamped speaker words onto the motion frame grid. The same rule can be applied to other datasets with timed speaker audio and a motion timeline. On MARS, our checkpoint-matched comparisons show that temporal placement affects FRC under the REACT 2025 offline protocol.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.