- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Dyna2Music, a latent flow-matching framework for dance-to-music generation. It separates joint velocities into slow and fast components across five body parts, then hierarchically combines part-wise energy with pretrained MotionBERT features. An auxiliary latent dynamics consistency (LDC) objective matches adjacent-frame change magnitudes between a predicted clean music latent and the paired reference. Experiments on AIST++ and TikTok report improved onset-level synchrony and distributional audio quality, complemented by rhythm analyses, conditioning ablations, and a user study.
Critical strengths
-
Well-motivated structured motion conditioning. Modeling body parts and temporal frequency bands separately provides a coherent approach to retaining complementary rhythmic cues that whole-body aggregation may obscure.
-
A simple and efficient dynamics-supervision strategy. LDC complements motion conditioning by explicitly supervising local music-latent variation, without requiring additional networks, waveform decoding during training, or inference computation.
-
Multifaceted evaluation with transparent interpretation. Experiments cover two datasets, multiple audio representations, rhythm controls, and human evaluation. The paper also explicitly distinguishes reference-rhythm agreement from local motion-dependent phase tracking.
Critical weaknesses
-
Limited motion-dependent phase tracking. The conditioning-shift analysis shows that shifting motion conditions by approximately a quarter or half beat leaves generated onset peaks near their original phase. This limits the evidence that the soundtrack follows local changes in movement timing. Quantitative measurements of onset shifts and same-track motion-mismatch controls would better establish the model’s responsiveness to dance conditions.
-
Confounded motion-conditioning ablations. Appendix C.1 explicitly acknowledges differing training budgets across the part/band variants. Consequently, the reported improvements cannot be cleanly attributed to spatial partitioning and frequency separation. Comparisons with matched optimization budgets, initialization, and checkpoint-selection procedures are needed to isolate these contributions.
-
Limited positioning within broader music–motion research. The discussion should extend beyond D2M to related advances in music-to-dance modeling. Directly relevant examples include GACA-DiT for multiscale rhythm extraction and MoMu-Diffusion for rhythm-aligned bidirectional generation. Broader precedents cover shared style–rhythm embeddings and beat-aware retrieval (ChoreoMaster, BeatDance); structured body representations and genre-aware modeling (FineDance, TokenDance, MEGADance); and diffusion/flow-based synthesis with temporal or physical constraints (EDGE, LODGE, FlowerDance). Reference-based local control and repair in CustomDance provide further context for controllable generation. Discussing transferable mechanisms and task differences would better clarify what SSMD and LDC contribute beyond these established directions.
Questions for authors
See Weaknesses. I will raise my score if my concerns are adequately addressed.
Responses to Reviewer ZGkv
W1: Onset shifts and motion-mismatch controls.
AIST++ test-set controls confirm clip-start-anchored onset phase and tempo-mediated motion conditioning; we will narrow our claims accordingly.
1. Motion shifts. We shifted all motion streams by ±¼ and ±½ beat (0.244 and 0.492 after latent-grid rounding) and measured onset-envelope cross-correlation lags against unshifted outputs (180 clips/condition). Mean lags: −0.003 to +0.009 beats, all CIs within ±0.021; 96–100% of clips retain onsets within ⅛ beat, and only 0–2% follow the shift. AIST++ clips start on a beat (t ≈ 0.035 s); training crops start at frame 0, providing a phase cue independent of motion.
2. Same-track motion mismatch. We scored each fixed output against motion beats of its own and three mismatched dances (beat-hit F1 ×100, τ = 70 ms, 10-track bootstrap; Δ = own − mismatched).
At fixed tempo (rows 1–2), dance-specific dependence is at most small; we do not claim dancer- or choreography-specific timing. Real music also has no detectable own-dance advantage at the same BPM, indicating mainly tempo-level beat correspondence in paired AIST++ supervision. Across tempos (row 3), conditioning-dance dependence is clear: ours recovers 59% of the real-music gap; eight baselines reach at most +4.0.
3. Tempo follows motion. Warping motion by 0.8, 0.9, 1.1 and 1.2× monotonically changes generated tempo to 0.86, 0.92, 1.05 and 1.11× (0.8× CI: [0.83, 0.91]; log-log slope 0.63; 1 = full following). Tempo determines synchrony: beat F1 is 88.6 with correct generated tempo versus 29.4 otherwise, near chance (33.4). The clip-start prior cannot explain this. Full conditioning (5 body parts × 2 frequency bands, 5×2) was compared with whole-body energy in one (1×1) or two (1×2) bands, all trained for 100k steps identically on the same uncropped clips, sharing the clip-start prior.
W2: Confounded motion-conditioning ablations.
Appendix C.1’s unequal budgets confound attribution to spatial partitioning or frequency separation. We replaced the table with a controlled rerun.
Table R1. Step-matched AIST++ ablation (180 test clips; one training run/variant). Onset metrics follow Table 1 (±50 ms); Beat F1 uses mir_eval (±70 ms); Tempo acc.: share with estimated tempo within 4% of nominal BPM. Parentheses: paired differences to 5p2b.
With matched steps, hyperparameters, initialization scheme, capacity and checkpoint rule, 5p2b significantly outperforms every ablation (onset F1 +6.7–10.7; beat F1 +15.2–25.7); the direction holds at matched samples. FAD is unchanged, indicating alignment gains, not audio-quality or capacity gains.
W3: Music–motion research.
We thank the reviewer for the references. Our Related Work covers only D2M; task differences limit M2D-to-D2M transfer.
(i) Rhythm source. M2D can extract beats from conditioning audio; FineDance’s body/hand split and TokenDance’s upper/lower-body tokens support plausible motion generation. D2M lacks reliable motion beat tracking: cues span limbs and time scales. SSMD partitions input by body part to infer tempo.
(ii) Constraint targets. EDGE’s velocity/foot-contact losses, LODGE’s foot refinement and FlowerDance’s physical-consistency constraints target verifiable physical properties of generated motion. Music has no direct physical counterpart; supervision targets temporal variation.
(iii) Many-to-many correspondence. Multiple tracks fit one dance and vice versa. ChoreoMaster and BeatDance use shared style/rhythm embeddings and retrieval; FineDance and MEGADance use genre-aware modules. Dyna2Music uses neither shared embeddings nor genre information.