acceptodds
Under review as a conference paper at ICLR 2027

ReactMotion: Generating Reactive Listener Motions from Speaker Utterance

Abstract

When a friend says I finally got the job, we lean in, clap, or raise a hand; when the news is bad, we respond differently. A listener's body answers what is said, and we study how to generate this answer from the speaker's audio, transcript, and emotion. Two obstacles stand in the way: a recording shows only one of the many reactions an utterance can elicit, and a generic nod looks acceptable next to almost any utterance, so evaluation must measure whether a motion depends on its input. We address both. ReactMotionNet builds one-to-many supervision at scale: it curates an inventory of listener reactions, writes utterances that would elicit each one, cross-pairs conditions with candidate motions, and grades every pair with three language-model scorers calibrated on human judgments, yielding 85,142 condition–motion pairs, 71,286 of them supportive or socially appropriate. ReactMotion is a multimodal language model that predicts body-and-hand motion tokens directly from the speaker condition, without an intermediate motion caption, trained on the supportive pairs and refined with Direct Preference Optimization. Gain scores how much better a motion fits its own utterance than unrelated ones, so an input-blind generator scores zero. On 358 held-out utterances, ReactMotion reaches 0.3392 Gain after supervised fine-tuning and 0.3790 after preference optimization, against at most 0.1157 for twelve caption-mediated pipelines, and human raters prefer it to retrieval and cascade baselines.

Primary area:
datasets and benchmarks
Keywords:
human motion generation, generative model, multimodality, human-computer interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RYNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a synthetic LLM-graded dataset for listener motion generation named ReactMotionNet, together with the ReactMotion model and the Gain metric. The problem is well motivated, and the metric design is thoughtful. However, the evaluation is circular, since training labels, DPO preferences, and the judge all come from the same LLM scoring, and the baselines are not trained on the task data.

Critical strengths
  1. The motivation of the paper is reasonable. the listener motion generation is a challenge task since it surfer from the one-to-many nature of reactions and the risk that generic motions look plausible for any input. this paper builds dataset and evaluation metrics facing these challenges.
  2. The Gain metric subtracts the compatibility score on distractor utterances, so a generator that ignores its input scores zero. Shuffled, constant, and random-motion controls confirm that the metric works as intended.
  3. The paper includes cue ablations, evaluation on recorded conversations, failure analysis, and human studies.
  4. the authors commit to releasing the data, code, and models.
Critical weaknesses
  1. I have concern about the volumn of the dataset constructed for this task, there are only 657 reactions, and every test reaction already appears in training. It makes the task more like to retrieval a suitable clip rather than learn to generate a new reaction.
  2. Training labels, DPO preferences, and the judge all come from the same LLM scoring. DPO improves Gain, but human raters do not prefer it over SFT.
  3. None of the baselines is trained on the task data, so the large gap does not show that direct generation beats caption mediation.
Questions for authors
  1. How does the model perform when test reactions are fully excluded from training?
  2. How does ReactMotion compare to a retrieval or classification baseline trained on the same data?
  3. Why does DPO improve Gain but not human preference?
Reviewer_6mULOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper generates a listener’s reaction motion generation from what the speaker says (audio, transcript, and an optional emotion label). The authors build a dataset, ReactMotionNet, which they describe as the first benchmark with graded one-to-many supervision for this setting. They collect reaction clips, have GPT-4o write utterances that would trigger each reaction, pair utterances with candidate motions, and let three LLMs score each pair. A small human study sets the score thresholds. The model, ReactMotion, fine-tunes Qwen2.5-Omni-3B to output motion tokens, first with SFT and then with DPO. They also propose a metric, Gain: how much better a motion fits its own utterance than unrelated ones. On 358 held-out utterances, ReactMotion clearly beats twelve caption-based baselines and is preferred by human raters over retrieval and cascade baselines. The appendix adds ablations and a test on recorded conversations.

Critical strengths
  1. Meaningful problem and useful dataset. One utterance can elicit many valid reactions, but recordings show only one. Building graded alternatives is a sensible fix, and the released data and pipeline could help future work.
  2. Careful metric design. Gain targets a real failure mode: generic motions that look fine but ignore the input. Shuffled, constant, and random controls all give Gain near zero, and the paper adds human judgments.
Critical weaknesses
  1. The baselines don’t isolate the benefit of direct generation. None of the twelve baselines is trained on ReactMotionNet, and every test reaction already appears in training (all 593 test identities are among the 657 training ones). Trained baselines are missing: nearest-neighbor retrieval over training utterances, an utterance-to-reaction classifier, or a fine-tuned captioner. Without them, it’s unclear whether the gain comes from the model or just from the supervision.
  2. Evaluation relies on the authors’ own judge. The same LLM scoring produces the training labels, the DPO preferences, and the judge’s labels, and most metrics use that judge. Humans don’t prefer DPO over SFT (44.7% vs 46.0%), yet the paper says DPO helps “at no perceived cost”. The human calibration is thin: 4 of 11 annotators kept, and the “incorrect” tier is still rated plausible 38.7% of the time. The audio is TTS of the transcript, and removing it doesn’t hurt, so the “speech-driven” framing is overstated.
  3. Poor motion quality. The supplementary video shows unnatural motions, the reaction motion isn’t very reasonable, and around 0:18 the right SMPL body has a visibly distorted spine.
  4. Minor (presentation): Figure 2 looks AI-generated.
  5. Missing references, all relevant to this paper:

Listener / dyadic behavior generation (the closest prior work):

• Yang, Metallinou & Narayanan, “Toward Body Language Generation in Dyadic Interaction Settings from Interlocutor Multimodal Cues”, ICASSP 2013: listener body motion from the speaker’s speech and gestures.

• Ng et al., “Can Language Models Learn to Listen?”, ICCV 2023: listener motion from the speaker’s transcript, using a language model to predict VQ motion tokens. This is close to ReactMotion’s design, but for the face.

• Jonell et al., “Let’s Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings”, IVA 2020.

• Tran et al., “DIM: Dyadic Interaction Modeling for Social Behavior Generation”, ECCV 2024.

• Sun et al., “Beyond Talking: Generating Holistic 3D Human Dyadic Motion for Communication”, arXiv 2024: speaker and listener holistic 3D motion.

• Zhang et al., “ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body”, CVPR 2026.

Questions for authors
  1. Trained baselines. Can you report baselines trained on ReactMotionNet: nearest-neighbor retrieval over training utterances returning their gold motion, an utterance-to-reaction classifier, and a fine-tuned captioner followed by the cascade? If these come close to ReactMotion, the main contribution is the dataset, not the model.
  2. Copying. All test reactions appear in training. How often is a generated motion a near-copy of a training clip?
  3. Role of audio. The audio is TTS of the transcript, and removing it doesn’t hurt. Does the model respond to prosody at all, e.g., the same text spoken with different emotion?
  4. Motion quality. Please report physical-plausibility metrics (jitter, foot sliding, penetration).
  5. Can you discuss and compare against dyadic listener models such as Audio2Photoreal and Seamless Interaction, at least on the recorded-conversation split?
Reviewer_qa7GOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies generating a listener’s body-and-hand reactions from a speaker’s audio, transcript, and emotion. It introduces ReactMotionNet, a dataset containing multiple graded reactions per utterance, and ReactMotion, a model that directly predicts motion tokens through supervised learning and preference optimization. The proposed Gain metric measures whether a motion fits its intended utterance better than unrelated ones. ReactMotion outperforms twelve caption-mediated pipelines on automatic metrics, and human raters prefer it to the evaluated retrieval and cascade baselines.

Critical strengths
  1. The evaluation explicitly tests whether generated reactions depend on the utterance. Gain is supported by input-independent controls and complemented by human judgments of appropriateness and naturalness.
  2. The experiments include twelve caption-mediated alternatives, modality and initialization ablations, preference-optimization sensitivity, and additional testing on recorded conversations, providing useful evidence about the system’s behavior.
Critical weaknesses
  1. All test motion identities also appear in training. Although the authors remove selected identities from reaction SFT and DPO in a controlled experiment, those assets remain in tokenizer training and caption–motion initialization, leaving generalization to entirely unseen motions unestablished.

  2. DPO improves automatic specificity scores but does not demonstrate a corresponding improvement in human preference over SFT. It also reduces within-condition motion diversity, so its practical benefit remains less clear than the automatic Gain improvement suggests.

Questions for authors
  1. Were the test condition–motion pairs and the reference motion selected for each utterance individually verified by humans? Please clarify how many test pairs were covered by the human calibration study and how the references were selected.
  2. Can ReactMotion generalize to motions excluded from all training stages, including tokenizer training and caption–motion initialization? How much additional data and fine-tuning are needed to incorporate new reaction types?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.