- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how a speaker’s words can be used to generate appropriate facial reactions. Semantic-REACT extracts words and timestamps from speech, converts them into textual features, and aligns these features with the corresponding video frames. An adapter and a gating mechanism then integrate these features into an existing audio-visual diffusion model. The paper also introduces a separate branch that predicts response opportunities and reaction categories.
The method is evaluated on MARS using the REACT 2025 offline protocol. Across five runs, its FRC is 0.7058, compared to 0.6705 without text conditioning and 0.6594 when the text features are temporally shifted. A user study compares the generated reactions with recorded listener reactions, with the latter receiving higher preference and ratings.
Critical strengths
None
Critical weaknesses
-
The semantic reaction prediction branch is not clearly explained. It is unclear how it is trained, how it operates during inference, and whether its outputs influence the main generator. Additionally, its contribution relative to existing response-timing and turn-taking methods, including those used in full-duplex dialogue models, is not well established.
-
The temporal linguistic alignment module repeats each word embedding across its speech interval. However, a word may influence a reaction even after it has ended, and its influence does not necessarily remain constant during its pronunciation. The paper demonstrates that this alignment strategy improves performance in the evaluated setting but provides limited theoretical justification or targeted experimental analysis to explain why it works or whether it accurately captures the temporal relationship between spoken words and listener reactions.
-
The motivation for including speaker transcripts is not sufficiently clear. The model already receives frame-level audio features that may contain linguistic information. Additionally, the paper does not discuss the latency introduced by transcription. Although the evaluation is conducted offline, these issues are relevant to the paper’s motivation of generating timely listener responses.
Questions for authors
The paper still has several significant issues, as outlined in the Critical Weaknesses section, and I do not believe it currently meets the standards for acceptance at ICLR. A substantial revision will be needed before resubmission.
The figures require clearer explanations. Fig. 1 and 2 both provide an overview, but their distinct purposes are unclear. Meanwhile, Fig. 3 displays individual components without clearly explaining how they interrelate.
2 comments
Official Comment by Reviewer vy3a
I thank the authors for their response and apologize for my delayed reply. The revision clarifies how the semantic predictions are used in fig.5. However, it remains unclear how the predictor was trained and how it generated the response-opportunity scores and reaction categories shown in the figure. This concern remains unresolved, and I maintain my original score.
Separately, the main paper pdf contains a one-page rebuttal, while the supplementary material contains the revised manuscript. I'm not sure if this violates the formatting regulations.
Response to Reviewer vy3a on Word Timing and the Role of the Semantic Branch
Thank you for pointing out that a listener may react after a word has ended. We repeat the embedding across the spoken interval to mark where that word occurred. Motion frames can give the same token different attention weights, and a later frame can still attend to it after the word ends (§3.2, p. 6). As you note, the audio features can carry lexical information. The transcript gives us an explicit word identity and interval to place on the motion timeline.
In the checkpoint-matched comparison, mean FRC was 0.7058 with the original alignment and 0.6594 after shifting the conditioning sequence. The paired gain was 0.0464 (95% CI [0.0384, 0.0544], §4.4, p. 8). For the offline evaluation, Whisper-tiny processes the full speaker clip to produce the transcript and timestamps before generation (§4.1, p. 7). An end-to-end latency figure for streaming would include transcription time.
The response-opportunity scores and reaction categories next to the motion in Fig. 5 come from the separate semantic branch. We use them to interpret the examples (§3, p. 4). The diffusion generator takes the speaker’s audio-visual inputs and aligned word features. Its denoising loss trains the adapter, gate, and cross-attention modules in the last two decoder layers (§3.3). Turn-taking and full-duplex work are concerned with the timing of speech. Here we generate continuous facial feedback, some of which occurs while the speaker still has the floor.
Fig. 1 puts an example interaction on a timeline. Fig. 2 follows the inputs through the model. The revised Fig. 3 caption follows the aligned features in panel (b) into the adapter and gate in panel (c) (pp. 4–5).