- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a synthetic LLM-graded dataset for listener motion generation named ReactMotionNet, together with the ReactMotion model and the Gain metric. The problem is well motivated, and the metric design is thoughtful. However, the evaluation is circular, since training labels, DPO preferences, and the judge all come from the same LLM scoring, and the baselines are not trained on the task data.
Critical strengths
- The motivation of the paper is reasonable. the listener motion generation is a challenge task since it surfer from the one-to-many nature of reactions and the risk that generic motions look plausible for any input. this paper builds dataset and evaluation metrics facing these challenges.
- The Gain metric subtracts the compatibility score on distractor utterances, so a generator that ignores its input scores zero. Shuffled, constant, and random-motion controls confirm that the metric works as intended.
- The paper includes cue ablations, evaluation on recorded conversations, failure analysis, and human studies.
- the authors commit to releasing the data, code, and models.
Critical weaknesses
- I have concern about the volumn of the dataset constructed for this task, there are only 657 reactions, and every test reaction already appears in training. It makes the task more like to retrieval a suitable clip rather than learn to generate a new reaction.
- Training labels, DPO preferences, and the judge all come from the same LLM scoring. DPO improves Gain, but human raters do not prefer it over SFT.
- None of the baselines is trained on the task data, so the large gap does not show that direct generation beats caption mediation.
Questions for authors
- How does the model perform when test reactions are fully excluded from training?
- How does ReactMotion compare to a retrieval or classification baseline trained on the same data?
- Why does DPO improve Gain but not human preference?