acceptodds
Under review as a conference paper at ICLR 2027

Where the Body Keeps the Beat: Structured Motion Conditioning and Music Dynamics Supervision for Dance-to-Music Generation

Abstract

Dance-to-music (D2M) generation aims to synthesize musically plausible soundtracks whose temporal structure aligns with a given dance performance. Representative D2M methods do not explicitly distinguish motion cues across body parts and frequency bands, while standard flow matching lacks a dedicated objective for supervising local music-latent dynamics. To address these issues, we propose {Dyna2Music, a latent flow-matching framework that combines structured motion conditioning with explicit supervision of music-latent dynamics. An empirical analysis on AIST++ quantifies how spatial partitioning and frequency separation affect raw music-to-kinematic beat alignment and motion-reference density, informing the conditioning design. Accordingly, Dyna2Music decomposes joint velocities into slow and fast components and hierarchically fuses the resulting part-wise motion energy with pretrained joint features to condition music generation. To complement this representation, we introduce latent dynamics consistency (LDC), an auxiliary objective that matches adjacent-frame change magnitudes between a single-step clean-latent estimate and the paired reference. LDC makes local music-latent variation an explicit training target without adding trainable parameters or inference computation. Dyna2Music supports variable-length music generation, and experiments on AIST++ and TikTok demonstrate improved rhythmic alignment and audio quality over representative D2M baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dance to Music Generation, Structured Motion Conditioning, Music-Latent Dynamics Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZGkvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dyna2Music, a latent flow-matching framework for dance-to-music generation. It separates joint velocities into slow and fast components across five body parts, then hierarchically combines part-wise energy with pretrained MotionBERT features. An auxiliary latent dynamics consistency (LDC) objective matches adjacent-frame change magnitudes between a predicted clean music latent and the paired reference. Experiments on AIST++ and TikTok report improved onset-level synchrony and distributional audio quality, complemented by rhythm analyses, conditioning ablations, and a user study.

Critical strengths
  1. Well-motivated structured motion conditioning. Modeling body parts and temporal frequency bands separately provides a coherent approach to retaining complementary rhythmic cues that whole-body aggregation may obscure.

  2. A simple and efficient dynamics-supervision strategy. LDC complements motion conditioning by explicitly supervising local music-latent variation, without requiring additional networks, waveform decoding during training, or inference computation.

  3. Multifaceted evaluation with transparent interpretation. Experiments cover two datasets, multiple audio representations, rhythm controls, and human evaluation. The paper also explicitly distinguishes reference-rhythm agreement from local motion-dependent phase tracking.

Critical weaknesses
  1. Limited motion-dependent phase tracking. The conditioning-shift analysis shows that shifting motion conditions by approximately a quarter or half beat leaves generated onset peaks near their original phase. This limits the evidence that the soundtrack follows local changes in movement timing. Quantitative measurements of onset shifts and same-track motion-mismatch controls would better establish the model’s responsiveness to dance conditions.

  2. Confounded motion-conditioning ablations. Appendix C.1 explicitly acknowledges differing training budgets across the part/band variants. Consequently, the reported improvements cannot be cleanly attributed to spatial partitioning and frequency separation. Comparisons with matched optimization budgets, initialization, and checkpoint-selection procedures are needed to isolate these contributions.

  3. Limited positioning within broader music–motion research. The discussion should extend beyond D2M to related advances in music-to-dance modeling. Directly relevant examples include GACA-DiT for multiscale rhythm extraction and MoMu-Diffusion for rhythm-aligned bidirectional generation. Broader precedents cover shared style–rhythm embeddings and beat-aware retrieval (ChoreoMaster, BeatDance); structured body representations and genre-aware modeling (FineDance, TokenDance, MEGADance); and diffusion/flow-based synthesis with temporal or physical constraints (EDGE, LODGE, FlowerDance). Reference-based local control and repair in CustomDance provide further context for controllable generation. Discussing transferable mechanisms and task differences would better clarify what SSMD and LDC contribute beyond these established directions.

Questions for authors

See Weaknesses. I will raise my score if my concerns are adequately addressed.

1 comment
AuthorsOpenReview ↗

Responses to Reviewer ZGkv

W1: Onset shifts and motion-mismatch controls.

AIST++ test-set controls confirm clip-start-anchored onset phase and tempo-mediated motion conditioning; we will narrow our claims accordingly.

1. Motion shifts. We shifted all motion streams by ±¼ and ±½ beat (0.244 and 0.492 after latent-grid rounding) and measured onset-envelope cross-correlation lags against unshifted outputs (180 clips/condition). Mean lags: −0.003 to +0.009 beats, all CIs within ±0.021; 96–100% of clips retain onsets within ⅛ beat, and only 0–2% follow the shift. AIST++ clips start on a beat (t ≈ 0.035 s); training crops start at frame 0, providing a phase cue independent of motion.

2. Same-track motion mismatch. We scored each fixed output against motion beats of its own and three mismatched dances (beat-hit F1 ×100, τ = 70 ms, 10-track bootstrap; Δ = own − mismatched).

Mismatched danceReal music ΔOurs Δ
Other dancer, same track0.0 (same recording)+1.3 [0.1, 2.8]
Other track, same BPM+0.8 [−11.7, 12.6]+0.3 [−10.9, 11.6]
Other track, different BPM+14.5 [7.1, 22.4]+8.5 [3.5, 12.8]

At fixed tempo (rows 1–2), dance-specific dependence is at most small; we do not claim dancer- or choreography-specific timing. Real music also has no detectable own-dance advantage at the same BPM, indicating mainly tempo-level beat correspondence in paired AIST++ supervision. Across tempos (row 3), conditioning-dance dependence is clear: ours recovers 59% of the real-music gap; eight baselines reach at most +4.0.

3. Tempo follows motion. Warping motion by 0.8, 0.9, 1.1 and 1.2× monotonically changes generated tempo to 0.86, 0.92, 1.05 and 1.11× (0.8× CI: [0.83, 0.91]; log-log slope 0.63; 1 = full following). Tempo determines synchrony: beat F1 is 88.6 with correct generated tempo versus 29.4 otherwise, near chance (33.4). The clip-start prior cannot explain this. Full conditioning (5 body parts × 2 frequency bands, 5×2) was compared with whole-body energy in one (1×1) or two (1×2) bands, all trained for 100k steps identically on the same uncropped clips, sharing the clip-start prior.

W2: Confounded motion-conditioning ablations.

Appendix C.1’s unequal budgets confound attribution to spatial partitioning or frequency separation. We replaced the table with a controlled rerun.

  • Budget/hyperparameters: All AIST++ variants use the same training script, learning rate, warmup, sequence length and λ_LDC = 1, evaluated at 100k steps; only conditioning inputs differ.
  • Initialization: All runs start randomly with the same seed setting; none reuses another run’s weights.
  • Capacity: Only part-embedding table size varies: ≤1,024 parameters out of 1.246B (<0.0001%).
  • Checkpoint: Preselected 100k EMA weights; no test metrics used.

Table R1. Step-matched AIST++ ablation (180 test clips; one training run/variant). Onset metrics follow Table 1 (±50 ms); Beat F1 uses mir_eval (±70 ms); Tempo acc.: share with estimated tempo within 4% of nominal BPM. Parentheses: paired differences to 5p2b.

VariantStepsBCS ↑BHS ↑Onset F1 ↑Beat F1 ↑Tempo acc. ↑FAD_v ↓
5p2b (full)100k62.0659.6154.3071.1971.12.854
5p1b100k50.4356.0947.38 (−6.9)52.44 (−18.8)44.42.951
2p2b100k45.9252.3443.58 (−10.7)45.45 (−25.7)36.12.917
1p1b100k53.2052.8747.49 (−6.8)55.95 (−15.2)50.02.864
2p1b100k47.1458.4347.66 (−6.7)45.57 (−25.6)34.42.931

With matched steps, hyperparameters, initialization scheme, capacity and checkpoint rule, 5p2b significantly outperforms every ablation (onset F1 +6.7–10.7; beat F1 +15.2–25.7); the direction holds at matched samples. FAD is unchanged, indicating alignment gains, not audio-quality or capacity gains.

W3: Music–motion research.

We thank the reviewer for the references. Our Related Work covers only D2M; task differences limit M2D-to-D2M transfer.

(i) Rhythm source. M2D can extract beats from conditioning audio; FineDance’s body/hand split and TokenDance’s upper/lower-body tokens support plausible motion generation. D2M lacks reliable motion beat tracking: cues span limbs and time scales. SSMD partitions input by body part to infer tempo.

(ii) Constraint targets. EDGE’s velocity/foot-contact losses, LODGE’s foot refinement and FlowerDance’s physical-consistency constraints target verifiable physical properties of generated motion. Music has no direct physical counterpart; supervision targets temporal variation.

(iii) Many-to-many correspondence. Multiple tracks fit one dance and vice versa. ChoreoMaster and BeatDance use shared style/rhythm embeddings and retrieval; FineDance and MEGADance use genre-aware modules. Dyna2Music uses neither shared embeddings nor genre information.

Reviewer_XBJ9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dyna2Music, a latent flow-matching model for dance-to-music generation. The motion representation separates joint velocities into slow and fast components across five body parts, then combines their energy signals with frozen MotionBERT features through attention and gating. The model also uses a latent dynamics consistency loss that matches adjacent-frame change magnitudes between an estimated clean music latent and the paired reference. Experiments on AIST++ and TikTok report improved onset synchrony and lower FAD over the evaluated baselines.

Critical strengths
  • The overall results are good across two datasets. Onset F1 improves over the baseline on AIST++/TikTok. The method also obtains the lowestFAD with all three feature extractors on both datasets.

  • LDC is efficient and no additional inference cost. The loss reuses the existing velocity prediction and requires no waveform decoding during training nor extra model parameters.

Critical weaknesses
  • The main ablation is not a controlled comparison. The variants use different training budgets, which makes the large advantage of the model difficult to attribute to the proposed representation.

  • The other comparisons do not show a consistent benefit from either refinement alone: separating the bands reduces F1 in the two-part setting, while increasing the number of parts gives no consistent gain with merged bands.

  • It is still unclear whether the model follows changes in local motion timing. The matched-vs-mismatched track analysis shows correspondence with the reference rhythm, but it does not establish how much comes from local motion cues versus global tempo and clip-start priors. The experiments supports improved reference-onset synchrony more than the detailed motion conditioning.

Questions for authors
  • Do the five-part, two-band gains remain under matched training budgets?

  • How was the motion-shift experiment in Section 4.5 implemented? Were both the energy signals and MotionBERT features shifted, with the sampling noise held fixed?

  • Was the proposed transformer initialized from DiffRhythm weights or trained from scratch? Were methods asked to generate the same duration?

  • Are the AIST++ partitions disjoint by music track and choreography, with all camera views of a performance kept in the same partition.

1 comment
AuthorsOpenReview ↗

Responses to Reviewer XBJ9

W1. The ablation is not a controlled comparison.

Sorry for the confusion. All conditioning variants and the full model use the same training recipe: batch size 2, learning rate 7.5e-5, λLDC=1\lambda_{\mathrm{LDC}} = 1, seed 42, a 1200-frame sequence cap, the same DiT and HSF-Encoder code, and identical inference (32 ODE steps, no CFG, fixed seeds). They differ only in the conditioning, i.e., how motion energy is pooled over body parts and frequency bands.

W2: no consistent benefit from either refinement alone.

We thank the reviewer for the careful reading. We agree with this observation, which is consistent with our own analysis in Appendix C.1 (L922–925). Our claim is that the two are complementary, and we will revise the text to state this explicitly. Table A1 indicates an interaction rather than two additive effects: separating slow/fast bands changes F1 by −4.08 (AIST++) and −0.22 (TikTok) at two-part resolution, but by +8.38 and +9.54 at five-part resolution. A plausible explanation is that coarse parts pool joints with heterogeneous dynamics (e.g., torso and wrists within the upper body), so the per-part fast-energy channel is dominated by the most active limb and the slow/fast distinction carries little additional information while doubling the conditioning channels. With five kinematically more homogeneous parts, band separation provides part-specific rhythmic cues that the model can exploit.

W3: Local motion cues vs. global tempo and clip-start priors.

Thanks for your comment. We therefore complement reference-onset evaluation with direct music-to-kinematic alignment measurements, an oracle tempo-and-start control, and motion-timing interventions.

Direct music-to-motion alignment. On the AIST++ test set (180 clips), we evaluate four beat sequences against five motion references: (i) count-matched random beats; (ii) paired real music; and (iii) our generated music.

Motion referenceChancePaired real musicOurs
1 part × 1 band0.2670.3960.344
2 parts × 1 band0.3620.4980.447
2 parts × 2 bands0.4250.4890.465
5 parts × 1 band0.4980.5950.567
5 parts × 2 bands0.6130.6630.662

Our generated music exceeds the random baseline under all five references, with mean BAS gains of 0.040–0.085. Thus, the evaluation provides direct evidence of music–motion rhythmic compatibility in addition to reference-audio synchrony. Under the full five-part, two-band reference, generated and paired real music obtain BAS scores of 0.662 and 0.663, respectively. Their gains over chance are 0.049 and 0.050, showing that their mean scores remain close after subtracting the reference-specific random baseline.

Q1: Are the gains preserved under matched training budgets?

Yes. All variants were trained with the same number of training steps, batch size, and training hyperparameters to ensure a fair comparison. The gains from the five-part, two-band design were therefore obtained under matched training budgets.

Q2: How were the motion conditions shifted, and was the sampling noise fixed?

We jointly shifted all three conditioning streams: slow-band energy, fast-band energy, and MotionBERT semantic features. We used a fixed seed for each sample across the original and shifted conditions. For different generation lengths, we verified bitwise equality of the shared CUDA randn noise prefixes under the same seed in all tested cases. In delay mode, the sequence length remains unchanged, and the entire noise tensor is bitwise identical to that used for the original generation.

Q3: How was the transformer initialized, and how was generation duration determined?

Our diffusion transformer (DiT) was trained from scratch, without initialization from DiffRhythm weights. The duration of the generated music is determined by the input motion duration. Our method supports flexible-length generation, whereas baselines such as PN_Diffusion and CDCD generate music of a fixed length.

Q4: Are the data partitions disjoint, and are all camera views kept together?

Yes. We use the official AIST++ cross-modal split, with 50 music tracks for training and 10 for testing. There is no overlap in music tracks, performances, or choreographies between the training and test partitions. All nine camera views of each performance are kept in the same partition. We have explicitly verified these properties in our experimental data.

Reviewer_CScbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Dyna2Music, a flow-matching-based dance-to-music generator built on the DiffRhythm model. Motion conditioning comes from a Spatio-Spectral Motion Decomposition: joint velocities from H36M-17 poses are Butterworth-filtered into slow and fast bands, and squared speed is averaged within five body parts. These signals are combined with frozen MotionBERT features by a hierarchical encoder. The paper also adds an auxiliary Latent Dynamics Consistency (LDC) loss, an L1 match between the per-frame RMS change magnitudes.

Critical strengths
  1. Latent Dynamics Consistency (LDC) loss adds no parameters and no inference cost, is computed entirely in latent space, which is compute-friendly and easy to reproduce.
  2. The comparison spans 15 baselines across general V2M and specific D2M families, with FAD embeddings, onset precision/recall/F1 across cross-clip SDs, BAS under five motion references, beat-relative profiles, and cross-dance diversity. Few D2M papers evaluate this widely.
Critical weaknesses
  1. Limited dataset. The experiments are conducted on AIST++ and TikTok sources, totally about 10,000 video clips. Especially in AIST ( a small set of music tracks), the training set may share tracks and multi-view duplicates with the 180 test clips. A large generator may recognize the genre and largely reproduce the training track, inflating both metrics.

  2. General Video2Music models were not trained on AIST++ dataset, so beating them on in-domain onset F1/FAD is expected. It is also unclear whether Dyna2Music's backbone is initialized from pretrained DiffRhythm weights. If so, the large-scale music prior may explain much of the FAD advantage on the test set.

  3. Limited novelty relative to existing components. The method combines established parts: band-pass motion energy, part-wise pooling, standard attention and gating, and a DiffRhythm backbone.

Questions for authors
  1. Are the AIST++ test music tracks disjoint from the training tracks? Please report the number of shared tracks and a nearest-neighbour similarity between generated test outputs and training music, and compare onset F1 against the paired GT with F1 against a different training clip of the same track.
  2. How were the baseline outputs obtained: retrained on your splits, fine-tuned, or zero-shot from released checkpoints?
  3. Is the DiT backbone initialized from pretrained DiffRhythm weights? If so, it's better to apply the same initialization to a simple conditioning baseline (e.g., MotionBERT tokens only) or even directly finetuning, so that the gains can be attributed to the proposed methods.
1 comment
AuthorsOpenReview ↗

Responses to Reviewer CScb

W1. Limited dataset and possible train–test overlap on AIST++

Thank you for raising this.

No overlap. We use the official AIST++ cross-modal split: 50 training vs 10 test tracks (0 shared), all 9 camera views of a performance in the same partition, and no test choreography in training (dancers overlap, as in prior work). The 180 test clips are 20 performances × 9 views, so all 95% confidence intervals are cluster bootstraps over the 20 performances. On TikTok we use the D2M-GAN split unchanged.

Scale. These ~9.4k clips are the standard D2M benchmark, used with the same splits by existing representative works D2M-GAN, CDCD, PN-Diffusion and GACA-DiT. Larger synchronized dance–music corpora are not public, largely due to music copyright.

Reproduction. The genre-retrieval part of this hypothesis does not hold, for two reasons.

  • The selected track matches the dance's tempo, not its genre. Only 0.6% of copied tracks share the test track's genre (chance 10%), while 78% share its tempo (chance 14%).
  • Tempo is causally driven by the motion. When we time-warp the input dance by 0.8–1.2×, the generated tempo moves with it (log–log slope 0.63; 1 = fully follows, 0 = ignores). Time-warping does not change the genre, so genre retrieval would predict a slope of 0. Across four seeds per clip, the input dance explains 70–88% of the variance in generated tempo.

W2. Zero-shot baselines and possible pretrained initialization.

(1) In-domain comparison. We agree that beating zero-shot V2M models in-domain is expected. We therefore include seven dance-to-music baselines trained on AIST++: D2M-GAN, CDCD, LORIS, PN_Diffusion, Dance2MusicTI, AudioLDM-ControlNet and UniMuMo.

(2) Initialization. Our DiT is trained from scratch, with no DiffRhythm DiT weights. We reuse only DiffRhythm’s frozen VAE for audio encoding/decoding; it sets decoding fidelity but does not model what music to generate.

(3) Music priors. Two in-domain baselines use large-scale pretrained music priors: Dance2MusicTI (frozen MusicGen) and AudioLDM-ControlNet (pretrained AudioLDM). Both are trained on AIST++, yet achieve FAD_v 8.97 and 5.63, versus 2.69 for ours. Such priors alone therefore do not explain the FAD gap.

W3. Novelty beyond existing components.

(1) Our contribution addresses a specific gap: jointly modeling body parts and temporal frequency bands in motion-derived rhythmic conditioning. LORIS derives visual rhythm from aggregated joint motion; PN-Diffusion encodes skeletal motion with positive and negative conditioning. Neither explicitly retains part-specific slow and fast components as distinct conditioning streams.

(2) Our hypothesis is that body parts and frequency bands provide complementary rhythmic cues that may become less distinguishable after early aggregation. Using established operators, we preserve these distinctions during contextual integration, then combine them through within-band part aggregation and cross-band fusion. This structured conditioning design is our main methodological contribution.

Q1. Track disjointness, nearest-neighbour similarity and reference-swap control.

(1) Shared tracks. None: the official AIST++ cross-modal split has 50 training and 10 test tracks, with 0 shared tracks.

(2) Nearest-neighbour similarity. For each generated test output, we slide its full length (7–12 s) over every training track and take the best-aligned frame-wise log-mel cosine similarity.

AIST++ test (n = 180)Full-length NN similarity to training music
Real test music0.30
13 baselines0.25–0.70
Ours0.39

Our mean similarity (0.39) is close to held-out real music (0.30), within the baseline range (0.25–0.70), consistent with reflecting real test music characteristics rather than reproducing specific training tracks.

(3) Reference-swap control. The requested same-track training reference is unavailable: training and test tracks are disjoint, so no training clip contains the paired test reference’s track. We report this comparison as N/A.

Q2. Baseline protocol.

General video-to-music baselines use publicly released pretrained checkpoints for zero-shot inference, without fine-tuning. Dance-to-music baselines are retrained with the authors’ released code and our training/test splits, following each method’s original preprocessing and clip duration. For example, PN-Diffusion uses 5-second clips, so we segment data accordingly while preserving the original partition. Splits are thus consistent across retrained methods; segmentation follows each implementation.

Q3. Initialization.

No. The DiT is trained from scratch; no DiffRhythm DiT weights are loaded. We reuse only DiffRhythm’s frozen VAE to encode and decode music.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.