acceptodds
Withdrawn from ICLR 2027

DanceMAR: Local Generation over Retrieved Motion Priors for Full-Body Dance Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Music-driven full-body dance synthesis requires realistic body and hand motion that follows the music and remains temporally coherent. Current methods primarily generate complete sequences or reuse retrieved examples, with less emphasis on local adaptation that preserves retrieved motion context. In this paper, we introduce DanceMAR, a framework that adapts retrieved motion priors through music-conditioned local residual generation. Specifically, we represent retrieved motion with a continuous hierarchy, retaining the lower-resolution context while restricting updates to local regions of the finest body-motion stream. To improve temporal coherence, we train a bidirectional masked model with local perturbations and motion-space supervision to learn residual corrections from the music and surrounding motion. In addition, we combine the adapted body with hand articulation from the same retrieved example to obtain full-body motion. Extensive experiments on FineDance demonstrate that DanceMAR achieves state-of-the-art body and hand motion fidelity while maintaining competitive beat alignment and foot stability. Further analyses show that motion changes remain localized, with smooth transitions between the adapted regions and the surrounding motion.

Primary area:
generative models
Keywords:
Music-driven dance generation, Human motion synthesis, Retrieval-conditioned generation, Masked motion modeling, Full-body motion generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JP8POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DanceMAR, a music-driven full-body dance synthesis framework that locally adapts retrieved motion priors. A music–motion retrieval module selects paired body and hand exemplars from a training-only bank. The model preserves lower-resolution body streams and predicts music-conditioned residual updates within a local interval of the finest stream. Boundary weighting and motion-space supervision promote continuity, while hand articulation is reused from the paired exemplar. Experiments on FineDance and additional AIST++ evaluations report improved motion fidelity with competitive beat alignment and foot stability.

Critical strengths
  1. A clear and practically relevant formulation. Preserving realistic retrieved motion while allowing localized music-conditioned adaptation offers useful control over choreography and motion detail.

  2. Well-controlled evaluation of local adaptation. The retrieval-only baseline uses identical output processing, and the boundary-weighting comparison matches latent update magnitude, providing meaningful evidence for the proposed refinement.

  3. A modular and extensible framework. The motion bank can incorporate additional exemplars without retraining the learned modules, as demonstrated by the AIST++ bank-expansion experiment.

Critical weaknesses
  1. Insufficient comparison with recent and closely related methods. The comparison omits recent music-to-dance methods such as MEGADance, FlowerDance, and TokenDance, as well as closer retrieval-guided approaches including CustomDance, MotionRAG-Diff, and InfiniteDance. The retrieval-only control demonstrates useful local updates, but does not establish superiority over existing retrieval-and-generation strategies. Representative comparisons under compatible settings are needed to support the claimed advantage.
  2. Limited methodological novelty. Retrieval followed by generative refinement already appears in CustomDance and MotionRAG-Diff; hierarchical masked motion modeling is established in MoMask, and music–motion retrieval has been explored in BeatDance. DanceMAR’s main distinction is preserving retrieved lower-resolution streams while restricting residual updates to a local finest-stream interval. This is a reasonable design, but its conceptual advance over these precedents remains insufficiently established.
  3. Self-retrieval during training raises concerns about the learned adaptation ability. Appendix C.2 allows training queries to retrieve their own source windows. Consequently, training may primarily involve restoring perturbed ground-truth motion, whereas inference requires adapting a different exemplar to the target music. The authors should report the self-retrieval rate and provide leave-one-source-out training controls to demonstrate genuine cross-exemplar adaptation.
Questions for authors

See Weaknesses. I will raise my score if my concerns are adequately addressed.

Reviewer_mxugOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The goal of the paper is to synthesize dance movements given a music track. First, it uses a Music–Motion Retrieval (MMR) representation, along with additional acoustic, beat, style, and position features, to retrieve a paired body–hand motion example from a body and hands motion bank. A bidirectional Masked Residual model (MAR) then predicts a music-conditioned residual that modifies only a local region of the finest body-motion stream, while the lower-resolution streams and surrounding motion are retained. Cosine boundary weighting is used to ensure a smooth transition. The authors evaluate the method on the FineDance dataset and compare it against FACT, MNET, Bailando, EDGE, FineNet, LODGE, FODGE, and LODGE++. They conduct both quantitative evaluations and a user study. However, the paper provides only two video visualization results, with no visual comparisons between the proposed method and the baselines. The method improves body and hand distributional fidelity over prior methods, and ablation studies show that local residual adaptation, fine-level adaptation, music conditioning, and boundary weighting all contribute to the results.

Critical strengths
  1. The combination of motion retrieval and residual adaptation is interesting. Retrieving motions with similar rhythmic and stylistic characteristics and then adapting them to new music through a learned residual, rather than generating the entire motion from scratch, is an interesting approach with strong potential for producing high-quality dance motions. The residual and hierarchical formulation is also interesting because it can preserve the structure and quality of an existing motion while adapting it to the target music.

  2. The quantitative evaluation is relatively comprehensive and shows promising results. The paper also includes a user study and a broad ablation study, providing evidence for the contributions of the proposed components.

Critical weaknesses
  1. The qualitative evaluation is insufficient, and the limitations are not adequately discussed. Although the paper provides quantitative comparisons and a user study, it includes very limited qualitative visualization comparing the proposed method with the baselines. In addition, the paper does not clearly discuss the limitations or failure cases of the approach. Since the method involves generating visually complex dance motions, the lack of direct qualitative comparisons makes it difficult to assess whether the reported quantitative improvements correspond to meaningful improvements in the generated motion. The lack of a clear limitations discussion also makes it harder to understand the scope and robustness of the proposed method.

  2. The paper is sometimes difficult to follow due to unclear and sometimes delayed explanations of the methodology. Several variables and parameters (e.g., {Z^E_s, Z^C_s, Z^M_s, Z^F_s}, L, S) are introduced without sufficient explanation or are defined only later, making it difficult to understand the method and follow the relationships between its different components.

Questions for authors
  • How much of the improvement comes specifically from local residual generation, compared with a simpler music-conditioned refinement network operating on the same retrieved motion? Could the authors clarify the motivation?

  • How does the method perform when the retrieved motion is poorly matched to the input music, for example, when the music has a style that is not represented in the motion database?

  • Can you mention the main limitations of the proposed method?

  • The paper provides very limited qualitative visual comparisons with the baselines. The supplementary material appears to include retrieval results and comparisons with other methods, but not direct qualitative comparisons with the baselines. Why are these comparisons not provided?

Reviewer_jDjAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a retrieval-based method for music-conditioned dance generation. For each music window, a paired body and hand clip is retrieved from a bank of training windows. The body clip is encoded into a hierarchy network with four different time resolution. The coarse levels are kept fixed, and a masked transformer predicts a music-conditioned residual on a local interval of the finest level. Hand motion is directly used from the retrieved clip. The goal is to keep the high-level movement of the retrieved dance while adapting its fine details to the target music.

Critical strengths
  • Clear motivation. Locally correcting a retrieved motion according to the condition is a natural design for retrieval-based generation.
  • Extensive experiments, including AIST++ transfer and well-designed ablations such as the music conditioning and adaptation scope.
Critical weaknesses
  • Mismatch between the training objective and the claimed music-conditioned adaptation. The training query of a window equals its own bank key (Eq. 2-4), so retrieval returns the window itself unless it is excluded. In this case Ur=UgtU^{r}=U^{gt}, and the residual target reduces to −ξ/w-\xi/w. This is a denoising task and can be learned without music. At inference,
  • Limited novelty. Concept of 'retrieving motion and refining it' is well studied in audio-driven motion synthesis [1, 2].
  • Most of the performance gain comes from retrieval, not form the music-conditioned adaptation. Retrieval only already outperforms all baselines in body FID. Hands are not adapted at all (Eq. 12), yet they achieve state-of-the-art hand metrics.
  • Writing quality.
    • Figure 2 is confusing. Labels are misaligned, they are typos ("Traning, Retrival"), and the arrow for cmc_m into the MAR block in (3) is missing. The relation between the MMR and the Music Encoder is also unclear.
    • Notation is inconsistent and partly undefined. The meaning of E,C,M,FE, C, M, F is not given in the main text, definition of ξ\xi is missing and so on.
    • "Retrieval-Anchored Span Realization" appears in Figure 2 but is not explained in the text.

[1] Habibie, I., at., el., "Motion Matching-based Framework for Controllable Gesture Synthesis from Speech", SIGGRAPH, 2022 [2] Yang, S., at., el., "QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation", CVPR, 2023

Questions for authors
  1. How often does training retrieval return the source window or another window from the same song? Is self-retrieval prevented during training?
  2. The perturbation ξ\xi in Eq. (8) is not specified. What are its distribution, scale, and sampling scheme?
  3. The supplementary video shows over-smoothed motion, foot skating, and sudden rotation flips. Which components cause these? Could the over-smoothing come from the training mechanism of GθG_{\theta} that I explained at Weakness section, which tends to regress toward an average motion?
  4. Is the deterministic refinement applied to the baselines? It substantially improves BAS and FSR, so the comparison is not fair otherwise. Please also describe the phase adjustment and the local dynamics and energy control.
  5. How is Edit defined?
  6. What does the adaptation actually change? It is unclear whether adding the predicted residual to the retrieved fine tokens provides a real benefit. Could you provide a random residual control, where ΔU\Delta U is replaced by a random residual with the same norm?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.