acceptodds
Under review as a conference paper at ICLR 2027

RAMP: Rhythm-Adaptive Modeling with Progressive Teacher Regularization for Dance-to-Music Generation

Abstract

Dance-to-music (D2M) generation requires fine-grained motion-rhythm control while maintaining coherent and semantically controllable music. We introduce RAMP, a framework for incorporating structured dance rhythm into a pretrained text-to-music (T2M) model. Its core component, the Style-Adaptive Rhythm Tokenizer (SART), performs global-to-local routing over shared rhythm-biased experts: a style-conditioned routing prototype captures clip-level rhythmic preferences, while motion-conditioned residual routing follows frame-level variation. To learn from these new conditions, RAMP uses teacher-regularized multimodal adaptation: it first adapts to the target text and lyric distribution, then introduces dance-aware conditioning while regularizing text-only velocity predictions toward a frozen Stage-1 teacher. The backbone's native text and lyric pathways remain explicit controls for high-level musical intent. We further construct Dance100K and a fixed 3K evaluation benchmark. Experiments show strong rhythmic alignment together with competitive semantic consistency and perceptual quality, supporting structured rhythm routing and teacher-regularized adaptation as complementary designs for controllable D2M generation. Anonymous project page:https://anonymous.4open.science/w/RAMP-Project.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dance-to-Music Generation, Rhythmic Modeling, Style-Adaptive Modeling, Prior-Preserving Adaptation, Multimodal Conditioning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_s5yGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RAMP, a dance-to-music generation framework that adapts a pretrained music model using dance, text, and lyric conditions. Its Style-Adaptive Rhythm Tokenizer (SART) combines style-conditioned expert routing with frame-level motion adaptation to represent dance rhythm. A two-stage training strategy first performs text/lyric adaptation and then introduces motion conditions, with a frozen Stage-1 teacher regularizing the student’s text-conditioned predictions. The paper also introduces Dance100K, a large-scale dataset of paired dance–music clips. Quantitative comparisons, ablations, prompt interventions, and a user study report improvements in rhythmic alignment, musical quality, and semantic consistency on five-second samples.

Critical strengths
  1. Well-motivated hierarchical rhythm modeling. SART combines global style preferences with frame-level motion adaptation, providing a coherent way to capture both style-dependent rhythmic patterns and local movement variations.

  2. A practically valuable adaptation framework. The method jointly considers dance–music alignment, musical quality, and semantic control. Staged adaptation and teacher regularization provide a sensible approach to incorporating motion conditions into a pretrained music model.

  3. Substantial data and multifaceted evaluation. Dance100K provides a sizable real-world resource, while controlled ablations, prompt interventions, and a user study offer complementary evidence for the proposed design.

Critical weaknesses
  1. Limited evaluation beyond short clips. Experiments focus on five-second segments, which primarily demonstrate local rhythmic alignment and audio quality. Although acknowledged, this limitation leaves musical development, phrase transitions, and sustained correspondence with longer choreography unverified.

  2. Unclear song-level independence of the data split. Source-video disjointness prevents overlapping segments from the same video, but does not exclude different videos containing the same song or recording. Could the authors report an audio-overlap audit and, ideally, results under a song-disjoint split?

  3. Limited methodological novelty. The combination of style-conditioned expert routing, staged adaptation, and teacher regularization appears incremental. The paper should more clearly distinguish SART from existing genre-adaptive rhythm modeling, such as GACA-DiT, and explain the conceptual advance beyond combining these components.

  4. Incomplete positioning within music–motion research. Relevant context includes cross-modal representation and alignment methods such as MoMu-Diffusion, BeatDance, TokenDance, and ChoreoMaster, alongside controllable or flow-based dance generation methods such as EDGE, CustomDance, and FlowerDance. Discussing transferable mechanisms while distinguishing their task settings would better clarify the contribution of SART.

Questions for authors

See weakness. I will raise my score if my concerns are adequately addressed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer s5yG — W1: Long-Form Evaluation

We thank the reviewer for highlighting this limitation. Accordingly, we have added a long-form evaluation on 81 continuous choreographies: 72 25-second AIST clips and 9 approximately 30-second crawler clips. Importantly, RAMP is trained only on five-second clips; these results therefore evaluate zero-shot temporal extrapolation. Each soundtrack is generated in a single diffusion pass, without long-duration fine-tuning, segment-wise generation, or audio stitching.

To assess musical development and phrase transitions, we derive structural boundaries independently from choreography and audio novelty curves and report objective cross-modal proxies. For both Gen. and Ref., Boundary F1 and offset compare audio-derived boundaries against the same choreography-derived boundaries, while novelty correlation compares the corresponding audio and choreography novelty curves. “Reference” denotes the ground-truth soundtrack processed by the same automatic protocol.

DatasetBoundary F1 (Gen./Ref.) ↑Novelty corr. (Gen./Ref.) ↑Boundary offset (Gen./Ref., ms) ↓
AIST-Long0.352 / 0.3800.627 / 0.659482 / 533
Crawler-Long0.380 / 0.3880.692 / 0.670675 / 526
Overall0.356 / 0.3810.634 / 0.660505 / 532

Boundary offset is averaged over successfully matched boundary pairs.

  1. Musical development and phrase transitions. The generated and reference music contain 3.83 and 3.84 detected boundaries per clip, respectively; generated music retains 93.3% of the reference Boundary F1 and 96.0% of its motion--music novelty correlation. We treat these measurements as objective structural proxies rather than human-annotated phrase labels.
  2. Sustained dance--music correspondence. Across 414 post-hoc five-second evaluation windows, Beat F1 and Dance-Onset F1 decrease by only 0.010 and 0.022 from the first to the final window, while Beat MAE increases by 1.81 ms. The final window retains 98.3% of the initial Beat F1 and 95.9% of the initial Dance-Onset F1.
  3. Full-clip quality. On AIST-Long/Crawler-Long, RAMP achieves Beat F1 scores of 0.518/0.544, Beat MAEs of 29.94/28.73 ms, and SongEval coherence scores of 1.61/1.66, with near-zero silence.

Despite being trained only on short clips, these automatic proxies provide evidence of structural continuity and sustained fine-grained dance--music correspondence over 25--30 seconds, with only modest temporal degradation. We have added the protocol, results, and this short-clip-training limitation to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer s5yG — W2: Song-Level Independence and Audio-Overlap Audit

We thank the reviewer for identifying that source-video disjointness does not guarantee song-level independence.

1. Audio-overlap audit. Accordingly, we have added a cross-split audit using exact AIST music IDs where available and CLAP nearest-neighbor retrieval for sources without song IDs. We removed samples with a matched AIST music ID or cosine similarity ≥0.95\geq 0.95. This conservative union removed 1,726/3,000 clips and retained 1,274. Because some sources lack canonical song IDs, we refer to this set precisely as a conservatively overlap-filtered subset, rather than claiming a strictly song-disjoint split.

2. Overlap-filtered evaluation. For the retained 1,274 IDs, we re-aggregate MDAphys, Beat F1, AudioBox PQ, and musicality from the corresponding Full-3K per-sample evaluation outputs, without retraining or regenerating audio. Beat MAE was not reported in the paper's main table; both Beat MAE values are computed under the same matched overlap-audit protocol.

Evaluation setMDAphys ↑Beat F1 ↑Beat MAE (ms) ↓AudioBox PQ ↑Musicality ↑
Full-3K0.29780.622436.797.66862.9804
Overlap-filtered (1,274 clips)0.29370.598036.987.65372.9471

The model retains comparable absolute performance after overlap filtering, with modest decreases in rhythm and audio-quality scores and nearly unchanged Beat MAE. The protocol and results are included in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer s5yG — W3: Methodological Novelty beyond Component Combination

We thank the reviewer for raising this concern. RAMP retains the pretrained music-generation backbone by design; its methodological contribution addresses what dance-rhythm information should be represented and how it can be added while preserving the pretrained T2M behavior.

1. SART introduces global-to-local rhythm routing. GACA-DiT combines multi-scale spatiotemporal motion features, adaptive joint weighting, and query-based temporal alignment. SART instead constructs kinematically biased rhythm candidates and explicitly factorizes their routing into a clip-level style prior and a bounded frame-level motion residual. This global-to-local factorization provides persistent style preferences together with frame-level motion adaptation.

2. Teacher-constrained modality expansion preserves pretrained behavior. In Stage 2, a full-condition student learns dance--rhythm--text control, while its text/lyric-only prediction is constrained by a frozen Stage-1 teacher under matched text inputs, noisy latents, and flow timesteps. Because the teacher receives text/lyrics without motion, it provides a modality-isolated reference for the pretrained text-conditioned velocity field and is removed after training.

3. Accordingly, we have added two ablation experiments to verify these distinct roles. First, we isolate the two motion-conditioning branches:

ConditioningABHS ↑MDAphys ↑Beat F1 ↑PQ ↑CLAP ↑
General Dance only0.10840.27580.58937.26700.3071
SART Rhythm only0.11030.28390.59287.20180.3006
Full0.11170.29780.62247.66860.2923

SART-only improves all three rhythm metrics over General-only, and Full further improves MDAphys (0.2978), Beat F1 (0.6224), and PQ (7.6686), supporting complementary roles. Second, we have added an auxiliary text-only ground-truth FM baseline. Under the same Full-3K evaluation protocol, the paper model (λSD=0.15\lambda_{\rm SD}=0.15) also outperforms the auxiliary text-only ground-truth FM baseline (λaux=0.20\lambda_{\rm aux}=0.20): PQ improves from 6.1928 to 7.6686, CLAP from 0.2203 to 0.2923, coherence from 2.7453 to 2.9947, and musicality from 2.6922 to 2.9804. Thus, SART supplies structured rhythm control, while teacher-constrained adaptation protects the pretrained musical and semantic behavior; the two mechanisms therefore serve complementary functions. We have made this distinction explicit in the revised method description and Figure 1.

AuthorsOpenReview ↗

Response to Reviewer s5yG — W4: Music–Motion Related Work

We thank the reviewer for identifying this missing context. We have added a dedicated “Music--Motion Cross-Modal Modeling” subsection covering all seven suggested studies and clarifying their relationship to RAMP:

  • Cross-modal representation and alignment: MoMu-Diffusion and BeatDance contribute aligned latent spaces and beat-aware contrastive alignment, but focus on joint generation or retrieval rather than dance-conditioned waveform generation.
  • Music-to-dance generation: TokenDance, ChoreoMaster, EDGE, CustomDance, and FlowerDance contribute token mapping, phrase structure, editability, and efficient diffusion/flow, but solve the reverse task and output motion rather than music.

The prior studies offer useful principles for alignment, tokenization, long-range synchronization, and controllability. RAMP addresses the complementary question of how dance kinematics can condition a pretrained text-native music generator. SART converts dance into a global-to-local mixture of kinematic rhythm views, while teacher-constrained adaptation introduces these controls while retaining the backbone's text/lyric interface. This positions RAMP as a task-specific bridge from dance kinematics to controllable music generation.

References

[1] You F, Fang M, Tang L, et al. MoMu-Diffusion: On learning long-term motion--music synchronization and correspondence[C]//Advances in Neural Information Processing Systems. 2024, 37.

[2] Yang K, Zhou X, Tang X, et al. BeatDance: A beat-based model-agnostic contrastive learning framework for music-dance retrieval[C]//Proceedings of the 2024 International Conference on Multimedia Retrieval. ACM, 2024: 11--19.

[3] Yang Z, Yang K, Tang X, et al. TokenDance: Token-to-token music-to-dance generation with bidirectional Mamba[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2026: 5520--5529.

[4] Chen K, Tan Z, Lei J, et al. ChoreoMaster: Choreography-oriented music-driven dance synthesis[J]. ACM Transactions on Graphics, 2021, 40(4): 1--13.

[5] Tseng J, Castellon R, Liu C K. EDGE: Editable dance generation from music[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 448--458.

[6] Tang X, et al. CustomDance: Customized 3D dance generation with coarse-to-fine human-centered interactive control[J]. arXiv preprint arXiv:2608.06722, 2026.

[7] Yang K, Tang X, Peng Z, et al. FlowerDance: MeanFlow for efficient and refined 3D dance generation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026: 56--74.

Reviewer_NdBoOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces RAMP, a dance-to-music generation framework that aims to capture fine-grained motion rhythm while maintaining musical quality and semantic control. The key idea is to incorporate structured rhythm representations from dance videos into a pretrained text-to-music model. A Style-Adaptive Rhythm Tokenizer (SART) is proposed to extract rhythmic features from human poses using four experts, along with a global dance-style prototype and local motion routing to combine the expert features into rhythm token representation. These tokens are then integrated with a general dance representation, text and lyrics to condition the pretrained ACE-Step text-to-music model. A two-stage adaptation strategy is used to first fine-tune the model on text and lyrics, and then incorporate dance information using a self-distillation objective to preserve the foundation text-to-music behavior. The work also collects Dance100K, a dataset containing approximately 100K dance–music clips. Experiments on this benchmark show improvements in rhythmic alignment, music quality, and semantic consistency over existing baselines.

Critical strengths
  • The paper addresses a meaningful challenge in dance-to-music generation, which is how to capture fine-grained rhythmic patterns from dance movements while retaining musical and semantic consistency. This is crucial for generating music that follows the dance performance while allowing user preferences over musical styles and lyrics content.
  • The proposed SART provides an intuitive way to model both global dance-style characteristics and local motion variations through adaptive expert routing. This design is potentially useful for capturing diverse rhythmic patterns across different dance styles, rather than relying on a single generic motion representation.
Critical weaknesses
  • Overall, I think several key architectural choices in SART are not well-motivated or justified. For example, why are the rhythm experts hardcoded to 4 with manually crafted features, and are they sufficient to capture different aspects of dance rhythms? Is this based on established music-dance rhythm analysis principles? Important mechanisms such as the expert view in Eq.2, kinematic scores rkr_k in Eq.3, or the bounded residual δt\delta_t in Eq.4 are also not clearly presented or explained in the main text, which makes it hard to comprehend the framework. In addition, the number of 512 dance tokens and 128 rhythm tokens seems arbitrary, despite both being extracted from the same video frames and pose sequence. The current ablations do not adequately investigate these design choices, which makes it difficult to evaluate the necessity of the proposed architecture. The distinctions and advantages of the proposed method over existing approaches are also not clearly demonstrated, such as the motion-rhythm representation or multi-expert design are explored in GACA-DiT or MEGADance.

  • The two-stage adaptation strategy seems complicated, while based on common techniques such as distillation. In particular, the adaptation requires two copies of a large foundation backbone (ACE-step 1.5), which introduces considerable training cost and may be less practical. Although the ablations show improved performance, it remains unclear whether comparable results could be achieved through simpler joint training or auxiliary text-only supervision. Despite those training complexity, the improvements over the baselines in Table 1 are not clear enough, especially in the Rhythmic Alignment metrics. Moreover, Table 6 shows quite similar text-only generation quality with and without self-distillation, which makes the claimed benefit of preserving the original generation capability less convincing.

  • The motivation and necessity of the new Dance100K dataset are not established. The paper does not clearly discuss which limitations of existing dance–music datasets (e.g., AIST++ or FineDance) are addressed by the proposed dataset, beyond the increase in number of training samples and additional annotations. It would be useful to demonstrate whether the Dance100k provides tangible advantages over existing datasets.

Questions for authors
  • Could the authors clarify the key technical distinction between SART and existing genre-aware rhythm representations or multi-expert approaches, such as GACA-DiT and MEGADance? Could the authors also provide justification or motivation for several important design choices?

  • Could the authors clarify the efficiency of the two-stage teacher-regularized adaptation? Given the comparable text-only generation performance with and without self-distillation in Table 6, what specific capabilities are being preserved?

  • Could the authors clarify which limitations of existing dance–music datasets motivate the new Dance100K, and whether it provides tangible benefits?

6 comments
AuthorsOpenReview ↗

Response to Reviewer NdBo — W1 (Part 1/2): SART Design Motivation, Architectural Choices, and Novelty

We thank the reviewer for requesting clearer motivation and controlled validation of the key SART design choices. We address each concern below.

Motivation for the four experts. Dance-rhythm studies characterize movement through accents, periodic groove, pauses, and temporal phrasing (Wilson et al., 2022; Fitch, 2016; Ling et al., 2025). We operationalize these cues as four fixed, style-agnostic views: sharp hit, groove, freeze/hold, and smooth phrase, while learning the expert encoders and routing weights. These four views provide a theory-guided operationalization of complementary temporal regimes, with one expert assigned to each view. The ablation below tests whether this granularity should be collapsed or expanded.

Accordingly, we have added an expert-count ablation. Across K∈{1,2,4,8}K\in\{1,2,4,8\}, we preserve the same four cue families: K=1K=1 averages them, K=2K=2 merges them into event and flow/phrase groups, K=4K=4 assigns one expert per view, and K=8K=8 duplicates each view with independent parameters. Thus, K=8K=8 tests additional capacity rather than introducing new handcrafted cues.

KKABHS ↑MDAphys ↑Beat F1 ↑PQ ↑CLAP ↑
10.10580.30040.61946.85970.2413
20.10440.28700.62527.75580.2503
4 (default)0.11170.29780.62247.66860.2923
80.09410.25260.60506.39920.2365

K=2K=2 is competitive, while K=8K=8 does not improve over K=4K=4 under this protocol. We retain K=4K=4 because it gives the most balanced result: the best ABHS and CLAP, while remaining close to the best MDAphys, Beat F1, and PQ.

Clarification of Eqs. 2–4.

  • Eq. 2 (expert candidates). Five frame-level channels---motion energy, acceleration, peak indicators, and sine/cosine peak-relative phase---are transformed into the four fixed kinematic views described above. An independently learned temporal encoder processes each view to produce an expert candidate.
  • Eq. 3 (fixed global prior). We thank the reviewer for identifying an implementation--description mismatch in the submitted equation. We verified the training code and checkpoint: all reported results use the fixed, uniformly smoothed prior p~sproto=(1−ϵ)ws+ϵ1/K\widetilde{\mathbf p}^{\mathrm{proto}}_s=(1-\epsilon)\mathbf w_s+\epsilon\mathbf 1/K, not a prior estimated from per-clip kinematic scores. The four rhythm regimes are theory-guided, whereas the numerical style-to-view weights ws\mathbf w_s are a transparent modeling prior rather than theoretically derived quantities. This prior supplies only a coarse clip-level preference; the expert encoders, expert bias, and frame-level residual router remain learned. We corrected the equation and appendix and disclose the mapping and fallback rule; no checkpoint or reported result changes.
  • Eq. 4 (bounded local correction). A temporal router predicts δt=δmax⁡tanh⁡(⋅)\boldsymbol{\delta}_t=\delta_{\max}\tanh(\cdot), yielding qt=Softmax⁡((λstylelog⁡p~sproto+b+αrouteδt)/τ)\mathbf q_t=\operatorname{Softmax}((\lambda_{\mathrm{style}}\log\widetilde{\mathbf p}^{\mathrm{proto}}_s+\mathbf b+\alpha_{\mathrm{route}}\boldsymbol{\delta}_t)/\tau). Thus, the fixed prior provides a coarse clip-level bias, while the learned bias and bounded residual provide frame-level adaptation. The bound limits local corrections but does not guarantee that the prior always dominates. Accordingly, λstyle\lambda_{\mathrm{style}} and αroute\alpha_{\mathrm{route}} independently control the global-prior and local-residual strengths; we do not interpret boundedness as a guarantee of prior dominance.
AuthorsOpenReview ↗

Response to Reviewer NdBo — W1 (Part 2/2): SART Design Motivation, Architectural Choices, and Novelty

Token-count ablation.

Accordingly, we have added a controlled token-count ablation. The two branches encode different information: General Dance preserves broad frame-level motion context, whereas SART represents a compact explicit rhythm trajectory. Their token counts therefore need not match the number of input frames or each other.

General / Rhythm tokensABHS ↑Beat F1 ↑PQ ↑CLAP ↑
256 / 1280.10990.59407.22780.2991
512 / 640.10940.59597.27360.2958
512 / 128 (default)0.11170.62247.66860.2923
512 / 2560.10890.59307.30180.2975

The default 512/128512/128 setting achieves the best ABHS, Beat F1, and PQ. The other token allocations obtain slightly higher CLAP, but increasing the rhythm budget to 256 does not improve rhythmic alignment or perceptual quality. We therefore retain 512/128512/128 as the representation-budget choice that prioritizes rhythmic alignment and PQ.

Distinction from GACA-DiT and MEGADance.

GACA-DiT performs genre-adaptive motion feature extraction and context-aware alignment, whereas SART softly composes shared, explicit kinematic expert candidates through a fixed clip-level style prior and a bounded frame-level residual. MEGADance instead applies a genre-aware MoE to the reverse music-to-3D-dance task.

The architectural comparison above clarifies the technical distinctions from these methods. Separately, we have added a conditioning-branch ablation to evaluate SART's contribution within RAMP. SART-only outperforms General-only on all three rhythm metrics, and combining the two branches further improves all three. These results support SART's effectiveness and complementarity with general dance conditioning for rhythmic alignment.

AuthorsOpenReview ↗

Response to Reviewer NdBo — W2: Teacher-Constrained Modality Expansion

We thank the reviewer for raising these concerns. We address them separately below.

1. Actual training and inference cost. Our implementation does not maintain two copies of the ACE-Step backbone. It uses one decoder with a trainable student LoRA and a frozen Stage-1 teacher LoRA; the teacher prediction is obtained by switching adapters under no_grad. The teacher is used only during training and is removed at inference. Accordingly, we have added an efficiency profile over 100 Stage-2 optimizer steps after 20 warm-up steps under identical settings (8 GPUs, BF16, global batch size 256):

Training objectiveStep time (s) ↓Peak memory/GPU (GiB) ↓
Without self-distillation2.6948.44
With self-distillation5.3018.97

Self-distillation increases training step time by 96.7%, but peak allocated memory by only 6.3%; it does not duplicate backbone memory and introduces no inference-time cost. We have reported this trade-off explicitly in the revised manuscript.

2. Comparison with simpler alternatives. The frozen teacher never receives dance/rhythm conditions; it constrains the student's text-conditioned velocity field under matched text/lyric inputs, noisy latents, and flow timesteps, while the full-condition branch learns the new controls. Accordingly, we evaluate two simpler training alternatives.

First, we compare single-stage joint training with two-stage training without distillation under the same Full-3K evaluation protocol:

Training strategyMDAphys ↑Beat F1 ↑PQ ↑CLAP ↑
Single-stage joint training0.26270.57155.79810.1598
Two-stage, no distillation0.29830.59556.84180.2567

This Full-3K comparison supports separating text/lyrics adaptation from motion conditioning.

Second, we replace the frozen-teacher constraint with a text/lyric-only ground-truth flow-matching loss of weight 0.20. We compare this auxiliary-supervision baseline with the default teacher-constrained model reported in the paper under the same Full-3K evaluation protocol:

Stage-2 supervisionMDAphys ↑Beat F1 ↑PQ ↑CLAP ↑Musicality ↑
Auxiliary text-only FM0.28860.59166.19280.22032.6922
Teacher-constrained model0.29780.62247.66860.29232.9804

The auxiliary-FM baseline is consistently worse across rhythmic alignment, production quality, text--music correspondence, and musicality. This provides supporting evidence that matching the frozen teacher's behavior is more effective than auxiliary text-only supervision during modality expansion.

3. Rhythmic-alignment gains in Table 1. Against the strongest prior result for each metric, RAMP improves MDAphys from 0.2840 (CMA-OT) to 0.2978 (+4.9%) and Beat F1 from 0.6049 (MotionComposer) to 0.6224 (+2.9%). Its ABHS of 0.1117 is slightly below LORIS (0.1169), while outperforming LORIS on MDAphys (0.2978 vs. 0.2467) and Beat F1 (0.6224 vs. 0.5101). These results support stronger multi-criterion rhythmic alignment.

4. What is preserved despite the similar Table 6 metrics. The similar aggregate text-only metrics motivate a direct analysis of the preserved quantity: the pretrained model's text-conditioned velocity field. Accordingly, we have added a function-drift analysis across 2,999 successfully evaluated benchmark examples: self-distillation reduces text-only prediction MSE to the frozen teacher from 0.1676 to 0.0446 (73.4% relative reduction; paired 95% CI: [-0.1268, -0.1193]), with lower discrepancy on 99.97% of examples. Thus, the regularizer directly constrains function drift during multimodal adaptation, even when coarse generation metrics are similar.

Reviewer_3iaAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To model persistent dance-style rhythmic preferences together with frame-level motion variations, this paper proposes RAMP, a framework that introduces structured dance-rhythm control into a pretrained text-to-music model. Its core component, the Style-Adaptive Rhythm Tokenizer (SART), enables global-to-local rhythm routing. RAMP further adapts a pretrained text-to-music model (ACE-Step 1.5 Base) to multimodal inputs through fine-tuning. In addition, the authors construct the Dance100K dataset and validate the effectiveness of their proposed approach through extensive experiments.

Critical strengths

1、The paper proposes a Style-Adaptive Rhythm Tokenizer (SART), which integrates a kinematically biased expert bank, a style-conditioned routing mechanism, and a motion-conditioned residual routing mechanism. This design captures both clip-level rhythmic preferences and frame-level motion variations, providing structured rhythm features to guide subsequent model adaptation.

2、The paper leverages large language models (LLMs) to construct Dance100K, a large-scale, temporally coherent dance–music dataset that encompasses motion information, music data, dance-style descriptions, and lyric annotations.

3、The paper incorporates rhythm control as complementary conditioning information, preserving the backbone’s native text and lyric generation capabilities while introducing additional general dance tokens and SART rhythm tokens. A multi-stage fine-tuning strategy is employed to obtain the final multimodal model.

Critical weaknesses

1、The overall framework appears relatively simplistic and does not introduce substantial architectural innovations, relying largely on multimodal conditioning and staged fine-tuning. Furthermore, Figure 1 does not sufficiently emphasize the core methodological contributions, and some icons are inconsistent throughout the figure. The figure should be further refined to improve its clarity and visual consistency.

2、The ablation studies should be further strengthened. The choice of (K=4) in SART is primarily justified by heuristic considerations, without a direct evaluation of alternative numbers of experts. Ablation experiments with different values of (K) would help determine whether the expert bank could be simplified or expanded. In addition, sensitivity analyses are needed for key routing hyperparameters, such as αroute and λstyle, to better justify their chosen values.

3、The proposed framework adapts a pretrained text-to-music model, but the individual contributions of different conditioning branches are not sufficiently evaluated. In particular, comparisons are needed to assess the effects of using only general dance features, only SART-based rhythm features, and their combination, in order to clarify the respective contributions and potential complementarity of these inputs.

Questions for authors

Please refer to the weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3iaA — W1: Method Novelty and Revised Figure 1

We thank the reviewer for this comment. We agree that the methodological contributions should be made more explicit. RAMP intentionally retains a pretrained text-to-music backbone to reuse its musical priors. Its methodological contributions are structured dance-rhythm conditioning and a preservation objective for introducing this modality. We detail both contributions below.

(1) SART: structured global-to-local rhythm modeling. SART performs global-to-local rhythm routing over shared kinematically biased experts: a style-conditioned prototype captures clip-level rhythmic preferences, while motion-conditioned residual routing follows frame-level dynamics. The architectural contribution is this coupled style/motion routing design, which explicitly incorporates global rhythmic preferences and local movement dynamics into rhythm-token construction. The conditioning-branch ablation in our response to Critical Weakness 3 supports the utility of this representation: under the same training budget, SART-only exceeds General-only on all three rhythm metrics (ABHS, MDAphys, and Beat F1), and their combination further improves all three.

(2) Teacher-constrained modality expansion: an explicit preservation objective. Adding dance/rhythm control raises a separate challenge: retaining the backbone's established text/lyric-conditioned behavior during adaptation. To address this, Stage 2 jointly optimizes a full-condition branch and a text/lyric-only branch aligned with a frozen Stage-1 teacher under matched text inputs, noisy latents, and flow timesteps. The teacher never receives dance or rhythm conditions, so the preservation target remains independent of the newly added modality. This provides a modality-isolated behavioral reference while the full-condition branch learns dance-conditioned generation. The methodological distinction lies in explicitly constraining pre-existing conditioning behavior during modality expansion.

(3) Figure 1: clearer contributions and consistent visual presentation. Following the reviewer's suggestion, we have redrawn Figure 1 to highlight SART's structured rhythm routing and the teacher-constrained modality-expansion objective, while unifying its icons, arrows, and visual elements for greater clarity and consistency. The revised figure and caption are marked in blue in the updated PDF.

AuthorsOpenReview ↗

Response to Reviewer 3iaA — W2: SART Design and Hyperparameter Ablations

We thank the reviewer for requesting these controlled ablations. Accordingly, we have added expert-count, routing-strength, and style-prior sensitivity experiments. For each configuration, we separately retrain SART and the corresponding downstream generator adaptation parameters; these are not inference-only parameter adjustments. Within each sweep, we vary only the parameter under study, keeping the training data, training epochs, and all other hyperparameters fixed. All variants use the same evaluation protocol.

Expert-count sensitivity. We compare K∈{1,2,4,8}K\in\{1,2,4,8\}, with K=4K=4 being the default model reported in the paper.

KKABHS ↑MDAphys ↑Beat F1 ↑PQ ↑CLAP ↑
10.10580.30040.61946.85970.2413
20.10440.28700.62527.75580.2503
4 (default)0.11170.29780.62247.66860.2923
80.09410.25260.60506.39920.2365

A smaller bank (K=2K=2) remains competitive, leading on Beat F1 and PQ, whereas expanding to K=8K=8 reduces all five metrics relative to K=4K=4. We retain K=4K=4 as a balanced choice: it leads on ABHS and CLAP and remains close to the best on the other three metrics.

Routing-strength sensitivity. We compare αroute∈{0,0.3,0.5}\alpha_{\mathrm{route}}\in\{0,0.3,0.5\}. Setting αroute=0\alpha_{\mathrm{route}}=0 disables the motion-adaptive routing residual, while retaining the global style-prior and bias terms; it does not disable expert routing altogether. Among the tested values, αroute=0.3\alpha_{\mathrm{route}}=0.3 performs best on all five metrics, supporting the benefit of the motion-adaptive residual and our default setting.

αroute\alpha_{\mathrm{route}}ABHS ↑MDAphys ↑Beat F1 ↑PQ ↑CLAP ↑
00.09540.26180.61026.46510.2252
0.3 (default)0.11170.29780.62247.66860.2923
0.50.10160.28660.61286.84970.2658

Style-prior sensitivity. We compare λstyle∈{0,0.5,1.0,1.5}\lambda_{\mathrm{style}}\in\{0,0.5,1.0,1.5\}, keeping αroute=0.3\alpha_{\mathrm{route}}=0.3 fixed. Setting λstyle=0\lambda_{\mathrm{style}}=0 removes the explicit style-prior term from the routing logits, not all style-prior influence: the motion-residual branch remains modulated by the style prior. Among the tested values, λstyle=1.0\lambda_{\mathrm{style}}=1.0 performs best on all five metrics, supporting the contribution of the explicit prior term and our default setting.

λstyle\lambda_{\mathrm{style}}ABHS ↑MDAphys ↑Beat F1 ↑PQ ↑CLAP ↑
00.08040.20360.60266.39400.2057
0.50.10330.27730.60396.22270.2201
10.11170.29780.62247.66860.2923
1.50.09590.24970.60906.52460.2488
AuthorsOpenReview ↗

Response to Reviewer 3iaA — W3: Contributions of Conditioning Branches

We thank the reviewer for this suggestion. Accordingly, we have added General-only and SART-only ablation experiments, trained under the same budget and compared with the Full (General Dance + SART Rhythm) model.

ConditioningABHS ↑MDAphys ↑Beat F1 ↑PQ ↑CLAP ↑
General Dance only0.10840.27580.58937.26700.3071
SART Rhythm only0.11030.28390.59287.20180.3006
Full0.11170.29780.62247.66860.2923

SART-only outperforms General-only on all three rhythm metrics, supporting its role in fine-grained rhythmic conditioning. General-only achieves the highest CLAP score. Full achieves the best rhythm alignment and PQ, supporting the complementarity of the two conditioning branches, while yielding a lower CLAP score than either single branch.

Reviewer_3iaAOpenReview ↗

Response Letter

Dear Authors,

Thank you very much for your efforts to address my concerns.

However, considering some issues raised by other reviewers, I have decided to maintain my rating.

Best

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.