- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies human motion generation from binaural audio produced by moving sound sources. It introduces DySAM, with 2,336 synchronized audio-motion sequences and source trajectories, and DySAL, an auxiliary dataset for source localization. The proposed method, MOMO, conditions a motion diffusion model on predicted head-centric sound directions, frame-wise CLAP features, and a reaction-style label. It also introduces frame-wise motion modulation with an adaptive correction strength. Experiments include baseline comparisons, component ablations, localization evaluation, and a user study.
Critical strengths
1.The dataset contribution is useful. Recording binaural audio together with body motion and moving source trajectories allows the relationship between sound direction and human response to be studied directly. This adds a meaningful dimension to existing audio-conditioned motion datasets. DySAL also provides spatial supervision that may be useful beyond the motion generation task.
2.The experiments provide encouraging evidence for the proposed approach. In Table 3, removing spatial conditioning substantially reduces retrieval accuracy, while removing frame-wise correction worsens motion quality. These results support the design choices. The user study in Appendix H also favors MOMO on both motion quality and spatial audio alignment, providing evidence beyond the learned evaluator.
Critical weaknesses
1.My main concern is the evaluation protocol. Appendix C states that the evaluator is trained on all DySAM sequences, including those subsequently used for testing. The test motions and their paired conditions therefore contribute to learning the representation used for FID and R-precision. This makes the scores harder to interpret as an independent evaluation, particularly for the learned motion-condition correspondence. It does not show that the generator uses test data, but the effect on the reported results needs to be checked. The dataset split also needs more detail: the paper gives an 8:1:1 ratio without specifying whether actors, original sound recordings, or capture sessions overlap. Since each source recording is associated with multiple motion sequences, this distinction matters for what the benchmark measures.
2.The main baseline comparison changes the input representation as well as the generation method. The baselines use traditional acoustic features, whereas MOMO uses frame-wise CLAP features and predicted DoA. Table 3 shows that replacing CLAP with traditional features increases MOMO’s FID from 0.857 to 2.343, so this difference accounts for a substantial part of the improvement. MOSPA receives ground-truth source positions, but these are a different representation from head-centric DoA. The component ablations are helpful, yet a comparison using the same audio and spatial features would make the architectural contribution much clearer.
Questions for authors
1.Does “all sequences of DySAM” in Appendix C include the validation and test sets? If so, do the main results and method rankings hold when the evaluator is trained only on the training split?
2.How are the dataset splits constructed? Please clarify overlap in actor identities, original source recordings, and capture sessions, as well as any overlap relevant to joint localization training on DySAM and DySAL. Do the results remain consistent when actors or source recordings are held out?
3.How does MOSPA, or another strong baseline, perform when given the same frame-wise CLAP features and predicted DoA as MOMO? This comparison would help establish the gain from the proposed conditioning modules.