acceptodds
Withdrawn from ICLR 2027

MOMO: Human Motion from Moving Sounds

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Human responses to sound depend not only on its semantic content, but also on the evolving spatial relationship between source and listener, conveyed through spatial audio and its temporal dynamics. Modeling human reactions to such dynamic spatial audio is important for character animation, human behavior analysis, and robotics. As of yet, existing audio-driven motion generation methods primarily focus on non-spatial or static acoustic conditions, leaving this important problem largely underexplored. To bridge this gap, we formulate the novel task of dynamic spatial-audio-driven human motion generation and introduce two motion-captured datasets. We introduce the Dynamic Spatial Audio and Motion dataset (DySAM) as the first dataset and benchmark for this task, providing synchronized binaural audio, 3D human motions, and moving sound source trajectories. Recognizing the importance of dynamic spatial cues of sound sources for this task, we further provide the Dynamic Spatial Audio Localization dataset (DySAL), an auxiliary binaural localization dataset that pairs diverse acoustic events with listener head poses and moving source trajectories to support spatial audio perception. For benchmarking, we propose MOMO for human Motion generation from Moving Sounds, a diffusion-based framework guided by three coupled principles: localization, understanding, and generation. MOMO performs localization by estimating head-centric spatial direction cues of the moving sound source directly from raw binaural audio, avoiding reliance on ground-truth source coordinates. It performs understanding by extracting frame-wise audio-content representations, and performs generation by conditioning motion diffusion on both semantic audio features and dynamic spatial cues. Moreover, MOMO incorporates frame-wise motion correction during denoising, enabling temporally adaptive motion synthesis under evolving audio-spatial cues. Experiments show that MOMO outperforms existing baselines in motion fidelity, audio-motion alignment, and physical plausibility, demonstrating the potential of dynamic spatial audio for embodied motion generation.

Primary area:
datasets and benchmarks
Keywords:
dynamic spatial audio, human motion generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Gg2QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies human motion generation from binaural audio produced by moving sound sources. It introduces DySAM, with 2,336 synchronized audio-motion sequences and source trajectories, and DySAL, an auxiliary dataset for source localization. The proposed method, MOMO, conditions a motion diffusion model on predicted head-centric sound directions, frame-wise CLAP features, and a reaction-style label. It also introduces frame-wise motion modulation with an adaptive correction strength. Experiments include baseline comparisons, component ablations, localization evaluation, and a user study.

Critical strengths

1.The dataset contribution is useful. Recording binaural audio together with body motion and moving source trajectories allows the relationship between sound direction and human response to be studied directly. This adds a meaningful dimension to existing audio-conditioned motion datasets. DySAL also provides spatial supervision that may be useful beyond the motion generation task.

2.The experiments provide encouraging evidence for the proposed approach. In Table 3, removing spatial conditioning substantially reduces retrieval accuracy, while removing frame-wise correction worsens motion quality. These results support the design choices. The user study in Appendix H also favors MOMO on both motion quality and spatial audio alignment, providing evidence beyond the learned evaluator.

Critical weaknesses

1.My main concern is the evaluation protocol. Appendix C states that the evaluator is trained on all DySAM sequences, including those subsequently used for testing. The test motions and their paired conditions therefore contribute to learning the representation used for FID and R-precision. This makes the scores harder to interpret as an independent evaluation, particularly for the learned motion-condition correspondence. It does not show that the generator uses test data, but the effect on the reported results needs to be checked. The dataset split also needs more detail: the paper gives an 8:1:1 ratio without specifying whether actors, original sound recordings, or capture sessions overlap. Since each source recording is associated with multiple motion sequences, this distinction matters for what the benchmark measures.

2.The main baseline comparison changes the input representation as well as the generation method. The baselines use traditional acoustic features, whereas MOMO uses frame-wise CLAP features and predicted DoA. Table 3 shows that replacing CLAP with traditional features increases MOMO’s FID from 0.857 to 2.343, so this difference accounts for a substantial part of the improvement. MOSPA receives ground-truth source positions, but these are a different representation from head-centric DoA. The component ablations are helpful, yet a comparison using the same audio and spatial features would make the architectural contribution much clearer.

Questions for authors

1.Does “all sequences of DySAM” in Appendix C include the validation and test sets? If so, do the main results and method rankings hold when the evaluator is trained only on the training split?

2.How are the dataset splits constructed? Please clarify overlap in actor identities, original source recordings, and capture sessions, as well as any overlap relevant to joint localization training on DySAM and DySAL. Do the results remain consistent when actors or source recordings are held out?

3.How does MOSPA, or another strong baseline, perform when given the same frame-wise CLAP features and predicted DoA as MOMO? This comparison would help establish the gain from the proposed conditioning modules.

Reviewer_d4r9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies human motion generation in response to moving sound sources. It introduces DySAM, containing 6.5 hours of synchronized binaural audio, motion capture, and source trajectories, alongside an auxiliary localization dataset.. MOMO combines estimated sound direction and audio features with adaptive frame-wise conditioning. Experiments include baseline comparisons, ablations, user study and preliminary robot demonstrations. The reported results show improved motion quality and audio-motion alignment.

Critical strengths
  • DySAM provides a useful combination of real motion capture, binaural audio, and moving sound-source trajectories. Its coverage of 42 sound types and two reaction styles makes it a meaningful resource for studying spatially conditioned motion.
  • Estimating sound direction directly from binaural audio removes the need for ground-truth source coordinates at inference. This is a practical improvement for settings where source positions are unavailable.
  • The reported gains are substantial. MOMO achieves an FID of 0.857 compared with 3.909 for the strongest baseline on this metric, and improves top-1 retrieval over MOSPA from 0.585 to 0.623.
  • The ablations provide useful support for the conditioning design. Removing frame-wise correction increases FID drastically, while removing the adaptive strength predictor increases it slightly.
  • The user study provides evidence beyond the learned metrics: participants prefer MOMO most often for both motion quality and spatial-audio alignment.
Critical weaknesses
  • Appendix Section C (Line 944) states that the evaluator is trained on all DySAM sequences, which includes the test set. Since this evaluator defines FID and R-precision, the main results are measured using representations fitted to the evaluation examples. This raises concerns about memorization and requires a rerun with an evaluator trained only on the training split.
  • The baseline comparisons do not isolate the model’s contribution. MOMO uses pretrained frame-wise CLAP features and predicted directions, while the baselines use handcrafted acoustic features. Table 3 confirms that feature choice substantially affects performance. The paper’s own ablations help, but a comparison with MOSPA using the same audio and spatial features is still needed.
  • It is unclear whether actors, original audio clips or recording sessions overlap across train-val-test splits. With eight actors in one recording environment, the current results provide limited evidence of generalization to unfamiliar listeners or sounds.
  • The revised skating metric can miss a sliding foot whenever the other foot remains planted. It also excludes airborne and penetrating frames, but the promised separate statistics are not reported. A low score therefore provides incomplete support for the physical-plausibility claims.
  • The collection protocol does not clearly explain whether actors could see the moving speaker or how their reactions were instructed. These details matter because visually guided or rehearsed responses may not be recoverable from audio alone, which affects how the dataset and task should be interpreted.
Questions for authors
  • Does “trained on all sequences of DySAM” include the test split? If so, please report the results only on the train split for correct evaluation.
  • How does MOMO compare with MOSPA when both use the same frame-wise CLAP features and predicted head-relative directions?
  • What is the unit of splitting: individual sequences, actors, source audio clips, or recording sessions? Were test recordings excluded from training both the motion generator and the localization network?
  • Could you report the original skating metric alongside the proposed version, together with the floating and penetration ratios mentioned in Appendix D?
  • Could actors see the speaker during capture, and were their responses spontaneous, demonstrated, or scripted? How were the sensitive and insensitive styles communicated?
Reviewer_WjkHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how to generate human motion in response to moving sounds. It introduces two datasets: DySAM, which records binaural audio, human motion, and sound-source trajectories together, and DySAL, which adds data for sound localization. The proposed model, MOMO, combines predicted sound directions, frame-wise CLAP audio features, and a diffusion model. It also uses a frame-wise correction module that predicts both how to adjust the motion features and how strongly to apply the adjustment. The experiments include baseline comparisons, ablations, localization tests, a user study, and robot demonstrations.

Critical strengths
  1. The task and datasets are useful. Moving sound sources add an interesting aspect to audio-driven motion generation. Recording binaural audio, human motion, and source trajectories together makes DySAM a useful resource for studying this problem. Table 4 also shows that adding DySAL helps localization on its validation set. I see the task and data as a main contribution of this paper.

  2. The method fits the task and is supported by several experiments. Predicting sound direction from audio avoids needing ground-truth source positions during generation. Separating the motion correction from its strength also makes sense, and the ablations in Table 3 suggest that both are useful within the current model. The user study adds perceptual evidence beyond the learned metrics, while the robot demos show an initial example of transferring the generated motion to a robot.

Critical weaknesses
  1. The evaluator appears to have seen the test sequences during training. Appendix C (p.18, line 944) says that the evaluator is trained on all DySAM sequences. Appendix D then uses this evaluator to compute FID and R-precision on the test set. This is my main concern, since these metrics support the main performance claims and several ablations. It is unclear how much this affects the scores or whether it changes the ranking. I would like to see the main results recomputed with an evaluator trained only on the training split.

  2. It is still hard to tell where the gains come from and what the model generalizes to. MOMO uses frame-wise CLAP features, while the main baselines use traditional audio features, and the spatial inputs also differ. The ablations are useful: MOMO with traditional features still has a better FID than the baselines, so CLAP does not explain all of the improvement. Still, comparing against a close baseline with the same audio and spatial inputs would help clarify what the correction module adds. The description of the 8:1:1 split also needs to explain whether performers, original sound recordings, and recording sessions overlap. Finally, removing direction information shows that spatial cues help, but does not by itself show how the model uses the timing of source movement.

Questions for authors
  1. When you say the evaluator is trained on “all sequences of DySAM,” does that include the generation validation and test sets? If so, could you recompute the main results and key ablations using an evaluator trained only on the training split? Do the improvements and rankings stay similar?

  2. How was the 8:1:1 split made? Do the same performers, original sound recordings, or recording sessions appear in more than one split? Clarifying this would help explain what kind of generalization the results show.

  3. Could you compare MOMO with a close baseline using the same frame-wise CLAP features, predicted DoA, genre labels, and a comparable training budget? A standard conditioning module with these same inputs would also help show what the proposed correction mechanism adds.

  4. Beyond removing the direction input, do you have evidence that the motion changes in response to the timing of source movement? A small controlled comparison using the same sound content with different trajectories would help. The audio and direction inputs should remain consistent, so the comparison measures the response to source movement without introducing conflicting conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.