acceptodds
Under review as a conference paper at ICLR 2027

Which Move, Which Moment? CueDance: Efficient Retrieval-Augmented Music-to-Dance Generation via Choreographic World Modeling

Abstract

People naturally move to music, while choreographic experience helps dancers determine which move fits which moment. This observation motivates us to view music-to-dance generation as two complementary capabilities: continuous responses to music and the selective use of motion experience at key moments. Existing retrieval-augmented methods introduce real motion references, but the temporal allocation of sparse references and sequential refinement dependent on an initial draft still limit their use of memory. We present CueDance, which connects what motion to retrieve with when to introduce it through a choreographic world model. Anticipate–Attribute–Anchor (3A) proposes motion hypotheses from musical transitions, selects reference moments based on explanatory gain (reduction in prediction error), and retrieves real motions as anchors. Music-driven retrieval runs in parallel with flow sampling, while Mid-Flow Anchor Inpainting integrates the references into the same generation pass. A part-aware generator and local-to-global flow supervision further support natural transitions around the references and efficient synthesis. CueDance achieves state-of-the-art dance generation performance on FineDance and AIST++, with a user study and systematic ablations further validating the effectiveness of selective choreographic memory.

Primary area:
generative models
Keywords:
Music-to-Dance Generation, Retrieval-Augmented Generation, Flow Matching, World Models, Character Animation, Avatar

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_5gEUOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CueDance combines music-conditioned dance generation with selective motion retrieval. Its choreographic world model predicts motion hypotheses and uses explanatory gain to select anchor timing. Retrieved motions are incorporated midway through flow sampling, allowing retrieval and generation to proceed in parallel. Experiments on FineDance and AIST++, component ablations, and a user study support the approach.

Critical strengths
  1. A meaningful formulation of selective choreographic memory. The paper treats retrieval as a joint decision about motion content and temporal placement, moving beyond content similarity alone. This provides a useful perspective on when external motion references can complement music-conditioned generation.

  2. A coherent integration of retrieval and continuous generation. CueDance combines sparse motion references with ongoing flow sampling in a single generation process. This connects memory-based guidance with flexible synthesis while reducing dependence on sequential draft generation and refinement.

  3. A promising balance of quality, efficiency, and long-form consistency. The approach improves kinetic motion fidelity while maintaining high throughput and mitigating degradation during extended generation. These complementary benefits make the framework relevant to practical choreography synthesis.

Critical weaknesses
  1. Explanatory gain is not directly validated as anchor utility. It measures improvement in predicting the next music representation rather than the actual benefit of inserting a retrieved motion. Although timing ablations support its overall effectiveness, how well does the score correlate with generation improvements across candidate positions?

  2. The anchor-count comparison is affected by training mismatch. SAC trains with one to three anchors, whereas evaluation compares one, four, and eight. Testing two and three anchors, or training with matched larger budgets, would better establish whether single-anchor selection is preferable.

  3. The NNR comparison changes several factors simultaneously. Query source, retrieval space and distance, and sampling/refinement budget differ between NNR and 3A. Matched comparisons would help distinguish the benefits of music-driven queries from those of the retrieval representation and generation pipeline.

  4. Related-work positioning could be more systematic. Building on the cited literature, the paper could better connect retrieval-guided choreography (ChoreoMaster, MotionRAG-Diff, CustomDance), alternative music-to-dance backbones (Bailando, Transflower, TokenDance), and reference-conditioned motion modeling (ReMoDiffuse, TANGO, and the original, subsequently withdrawn PersonaGesture preprint). A focused comparison of reference selection, temporal placement, and integration into generation would clarify the contribution, while acknowledging differences in task settings.

Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5gEU

Thank you for the constructive review. We have addressed all four points in the revised PDF.

W1. Does explanatory gain track anchor utility?

Appendix C.9 of the revised PDF evaluates anchor utility across gain ranks in Table C.11 and reports the selected-position distribution in Table C.12.

We place one anchor at different ranks in descending explanatory gain within each window, keeping the content rule, checkpoint, motion memory, MAI, and paired initial noise fixed.

Anchor at gain rankHighest (3A)25%50%75%LowestRandom positionNo anchor
FID-k ↓12.3913.5716.0722.9225.5318.8620.78
FID-g ↓16.1716.4817.4519.9122.5717.6918.29

Both FIDs increase across the tested descending gain ranks, while BAS decreases from 0.235 to 0.207. Anchors at the 75% rank and lowest-gain position perform worse than generation without anchors. These results support explanatory gain as a criterion for allocating sparse references. Each FID is evaluated over the complete test set, so this analysis measures the relationship between gain rank and downstream generation quality at the set level.

Table C.12 additionally summarizes 336 selections per rank across 84 test sequences. Each third of the valid target region accounts for 30.95%–36.61% of selections, showing that the selected positions cover the window. Figure C.2 in Appendix C.9 provides a visual example, accompanied by the 8-s MP4 Explanatory Gain for Motion Reference Selection in the supplementary materials.

W2. Two and three anchors

Appendix C.2 and Table C.3 in the revised PDF include two- and three-anchor configurations, both within the SAC training range.

Number of anchors12348
FID-k ↓12.3914.3813.9816.7217.24
FID-g ↓16.1717.8317.7318.7622.31

One anchor achieves the lowest FID-k and FID-g among all tested counts, including the configurations within the training range. Three anchors achieve the highest DIV-k of 9.20, while one anchor achieves the highest DIV-g and BAS.

SAC samples one, two, and three anchors with probabilities of 70%, 20%, and 10%, respectively, as documented in Appendices A.4 and C.2. The results therefore describe the preference under this training distribution.

W3. Matched NNR comparison

Appendix C.12 of the revised PDF examines query source, retrieval representation, and sampling budget, with the comparison in Table C.15.

Every configuration places one anchor at the same 3A-selected position. NNR uses a generated draft and L1 distance between FK joint coordinates. NNR-latent uses the same draft with cosine similarity in the motion latent space.

SettingQuery sourceRetrieval space / distanceModel evaluationsFID-k ↓FID-g ↓
NNRGenerated draftFK joints / L120+1017.9918.25
NNR-latentGenerated draftLatent / cosine20+1019.1017.21
NNR-latent, budget-matchedGenerated draftLatent / cosine10+1020.2416.86
3AMusic-driven hypothesisLatent / cosine2012.3916.17

With retrieval space, distance, anchor position, and total evaluation budget matched, 3A achieves lower FID-k and FID-g than budget-matched NNR-latent. The generation schedules differ. NNR-latent allocates 10 evaluations to draft generation and 10 to refinement, while 3A derives its query directly from music during a single 20-evaluation sampling pass.

W4. Related-work positioning

Sec. 2 of the revised PDF incorporates the suggested works into the discussion of generative backbones, retrieval, planning, and reference conditioning.

Bailando and Transflower are discussed as autoregressive backbones, and TokenDance as a non-autoregressive backbone. ChoreoMaster and MotionRAG-Diff are connected to external retrieval, CustomDance to motion planning, ReMoDiffuse to text-driven reference conditioning, and TANGO and the original, subsequently withdrawn PersonaGesture preprint to co-speech animation.

The revised discussion relates these approaches through reference matching, temporal placement, and integration into generation, while identifying their different task settings. It then explains how CueDance selects sparse reference moments through explanatory gain, retrieves from music alone in parallel with sampling, and incorporates the reference through MAI within the same generation pass.

Reviewer_6np9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CueDance, a music-driven dance generation framework that combines continuous motion generation with the retrieval of real motions at key moments. At its core are a choreographic world model and the Anticipate–Attribute–Anchor (3A) mechanism, which predicts motion hypotheses from adjacent music states, measures explanatory gain as the reduction in prediction error to select reference moments, and retrieves matching motions as anchors. Retrieval runs in parallel with flow sampling, while Mid-Flow Anchor Inpainting incorporates the references into the same generation pass. A part-aware generator and per-sampling-state adversarial supervision further improve motion coordination and generation quality.

Critical strengths
  1. The paper addresses both “what motion to retrieve” and “when to introduce it.” By combining motion hypotheses, explanatory gain, and real-motion retrieval through 3A, it offers a valuable approach that considers the temporal allocation of a limited reference budget beyond content matching.

  2. Music-driven retrieval does not require an initial motion draft and can run in parallel with flow sampling, with MAI incorporating references into the same generation pass. The controlled comparison shows an increase in throughput from 874 to 994 FPS with nearly unchanged generation quality, supporting its efficiency advantage.

  3. The evaluation includes objective metrics, user ratings, and component ablations. The method achieves the best FID-k on both datasets and the highest mean user ratings for quality, synchronization, and creativity. Ablations on reference timing, content, and training objectives further support the effectiveness of the main design choices.

Critical weaknesses
  1. CueDance achieves the best FID-k on both datasets, but its FID-g remains worse than that of some baselines. The temporal modeling ablation also shows inconsistent rankings across the two metrics. Although the paper briefly acknowledges this discrepancy, it does not sufficiently analyze the corresponding kinetic and geometric characteristics. Representative generated examples would help explain these differences and clarify the method’s strengths and limitations in terms of actual motion quality.

  2. The ablations support the effectiveness of retrieving real motions, but do not include a direct music–motion matching baseline; NNR also uses a generated motion draft as its retrieval query. It therefore remains unclear whether a simpler music-driven retrieval approach could achieve similar gains and how much additional benefit comes from CWM’s specific modeling design. A comparison using the same generator, motion memory, anchor budget, and injection mechanism would help establish CWM’s contribution more clearly.

  3. Section 3.1 and Appendix Table A.1 describe a lower-body pose decoding branch, whereas the output side of Figure 2 explicitly labels only the Upper-pose Decoder and Root-Weaver Decoder. This discrepancy makes it difficult to determine the exact decoding path for lower-body poses and could lead to a misunderstanding of the model architecture. Clarification would be helpful.

Questions for authors

Please refer to the weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6np9

Thank you for the detailed review. We have addressed all three points in the revised PDF.

W1. FID-k vs. FID-g

Appendix C.13 of the revised PDF includes the static pose/motion comparison in Figure C.3, the feature statistics in Table C.16, and an explanation of the different metric rankings.

Figure C.3 illustrates how geometric and kinetic similarity can differ. In the independently selected excerpts, MEGADance has a central pose resembling the GT example, but shows substantially larger changes between neighboring poses. Mamba exhibits a different arm configuration and torso tilt, while its torso and hip kinetic statistics are closer to GT than MEGADance's.

Table C.16 complements these examples with statistics over all 84 FineDance test sequences.

MethodFID-k ↓Torso bounceHip bounceFID-g ↓Foot offsetArm away from torso
GT–0.3160.355–58.5%84.1%
ForeGate (CueDance)12.390.3470.36916.1765.8%85.8%
Mamba mixer11.670.3540.37828.4469.2%89.6%
MEGADance50.000.7600.80613.0265.3%84.3%

MEGADance achieves a lower FID-g than ForeGate (13.02 vs. 16.17), alongside a higher FID-k (50.00 vs. 12.39). Conversely, Mamba achieves a lower FID-k (11.67 vs. 12.39), alongside a higher FID-g (28.44 vs. 16.17). Thus, closer geometric matching can coexist with poorer kinetic matching, and closer kinetic matching can coexist with poorer geometric matching. Each FID measures the distribution of its complete feature set across test sequences.

The revised Appendix C.13 discusses these complementary characteristics and identifies improving geometric distribution matching while maintaining kinetic fidelity as future work.

W2. Direct music–motion matching baselines

Appendix C.11 of the revised PDF describes the two direct music-driven retrieval baselines, with the controlled comparison in Table C.14.

We compare nearest-neighbour retrieval using music features with InfiniteDance's music–motion retrieval network, retrained on the same FineDance training split. All configurations share the generator checkpoint, motion memory, one-anchor budget, MAI, and paired initial noise. We cross each content rule with two timing rules. Cells report FID-k / FID-g.

Content ruleBeat/Onset timingExplanatory-gain timing
Music→music NN19.95 / 21.5117.94 / 19.83
Adapted MD-Retrieval18.75 / 20.3916.62 / 17.79
CWM hypothesis15.74 / 16.9412.39 / 16.17 (3A)

Generation without anchors gives 20.78 / 18.29. Under Beat/Onset timing, the direct retrieval baselines reduce FID-k but increase FID-g, whereas the CWM hypothesis reduces both. Explanatory-gain timing further lowers both FIDs for every content rule. With this timing fixed, full 3A reduces FID-k by 4.23 relative to Adapted MD-Retrieval. These controls separately evaluate the contributions of reference content and timing.

W3. Lower-body decoding path

The updated Figure 2 in the revised PDF clarifies the lower-body decoding paths. The architecture is specified in Sec. 3.1, Appendix A.3, and Table A.1.

The model follows the architecture described in Sec. 3.1 and Table A.1. The same lower-body latent feeds two frozen branches. The lower-pose decoder reconstructs lower-body poses, while Root-Weaver predicts local root-displacement increments, as detailed in Appendix A.3.

In the original Figure 2, the label "Root-Weaver decoder" grouped the lower-pose decoder and its root-displacement branch into one lower-body decoding module. This grouping obscured the two output paths. The updated figure shows the Upper-pose decoder, Lower-pose decoder, and Root-Weaver as separate blocks, making explicit the architecture used in the reported experiments.

Reviewer_6np9OpenReview ↗

Response Letter

Dear Authors,

Thank you very much for your efforts to address my concerns. I have changed my rating.

Best

Thank you very much for your kind response and for taking the time to carefully review our manuscript and revisions. We sincerely appreciate your thoughtful and constructive feedback, particularly your suggestions regarding the kinetic and geometric analysis, the direct music–motion retrieval comparisons, and the clarification of our architecture. Your comments have been extremely valuable in helping us strengthen both the experimental analysis and the presentation of our work.

We are truly grateful for your recognition of our efforts and your willingness to reconsider your rating. Thank you again for your time, careful consideration, and support throughout the review process.

Reviewer_3tgDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces CueDance, a retrieval-augmented framework for music-to-dance generation. It presents a Choreographic World Model with an Anticipate–Attribute–Anchor mechanism that determines which motions to retrieve and when to incorporate them based on explanatory gain. Mid-flow anchor inpainting allows retrieved motions to be integrated within a single flow sampling process, while a part-aware generator and local-to-global flow supervision enhance motion quality and efficiency. Experiments on FineDance and AIST++ show competitive generation quality, along with efficient inference and supporting ablation studies.

Critical strengths
  • The proposed explanatory gain offers an interesting mechanism for identifying when motion references are most valuable, rather than relying on fixed positions or simple musical cues. Ablation studies show that it outperforms alternative selection strategies, making it a meaningful contribution to retrieval-augmented dance generation.

  • The proposed mid-flow anchor inpainting allows motion retrieval and generation to run in parallel without requiring additional sampling passes. CueDance achieves strong results on FineDance and AIST++, particularly in FID-k, while maintaining high generation efficiency. Comprehensive ablation studies and user evaluations further support the effectiveness of the framework.

Critical weaknesses
  • Although the paper shows that explanatory gain is effective for anchor selection, it remains unclear what high-gain moments correspond to in actual music. The authors are encouraged to provide concrete examples—for instance, visualizations of explanatory gain alongside musical features, beats, or structural transitions, together with the corresponding retrieved motions. This would help clarify what the proposed metric captures and why these moments are particularly well suited for motion anchoring.
  • The paper describes CueDance as supporting streaming generation, yet the method assumes the music track is available in advance and uses future music states for motion prediction and anchor selection. Does the proposed framework require lookahead during inference? If so, how much future music context is needed? The authors should clarify whether the method supports truly online music-to-dance generation or only window-by-window generation with pre-accessible music, and should report the corresponding algorithmic latency.
Questions for authors

Please refer to the weakness above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3tgD

Thank you for the careful reading. We have addressed both points in the revised PDF.

W1. What high-gain moments correspond to in music

Appendix C.9 of the revised PDF includes the static reference-selection example in Figure C.2 and the anchor-utility analysis in Table C.11. Please also see the corresponding 8-s MP4, Explanatory Gain for Motion Reference Selection, in the supplementary materials.

Explanatory gain measures how much a motion hypothesis reduces next-state prediction error relative to the learned state-persistence predictor. High gain therefore identifies transitions whose prediction benefits strongly from motion conditioning.

In Figure C.2, we visualize explanatory gain, onset strength, Raw Error, and detected beats for an 8-s music window, together with the corresponding retrieved motions. Gain selects 7.07 s, onset strength 5.07 s, and Raw Error 2.00 s, each within 0.07 s of a detected beat. Thus, the largest prediction improvement from motion conditioning occurs at a different moment from either the strongest onset or the largest Raw Error. The supplementary MP4 presents this Figure C.2 example with synchronized music and motion.

Table C.11 tests whether these moments provide useful anchors. With the checkpoint, memory, MAI, and paired initial noise fixed, moving the anchor from the highest-gain position to the median- and lowest-gain positions increases FID-k / FID-g from 12.39 / 16.17 to 16.07 / 17.45 and 25.53 / 22.57. Generation without anchors gives 20.78 / 18.29. These results support selecting the highest-gain transition and show that anchoring at the lowest-gain position can reduce generation quality.

W2. Lookahead and latency

The revised PDF specifies "window-based streaming" in Secs. 2 and 4.2. Appendix C.10 and Table C.13 detail the required music context and measured latency.

CueDance generates window by window from pre-accessible music. Within each window, 3A evaluates valid transitions and ForeGate uses two-sided context, so the complete music-feature context of the current window must be available before generation.

  • Music context. On FineDance, the first window spans 8 s. Each subsequent window retains 1.6 s of history and uses 6.4 s of new music. On AIST++ at 60 FPS, the corresponding durations are 4 s, 0.8 s, and 3.2 s.
  • Latency. On an RTX 3090, parallel 3A takes 194.7 ± 7.9 ms for 192 new frames, compared with 190.8 ± 8.4 ms without 3A and 225.6 ± 9.3 ms for serial 3A. These are means ± sample SD over 84 measurements from 21 repeated runs of one test clip, each contributing four complete subsequent windows. Following Appendix B.3, timing starts when conditioning features are available and ends after pose and root-displacement decoding.

The revised PDF reports the music-context requirements alongside this measured generation latency.

Thank you for the authors' response. My previous concerns and questions have been fully addressed. I believe a score of 3 is appropriate for this paper, and I will therefore maintain my rating.

Thank you very much for your thoughtful feedback and for taking the time to carefully review our manuscript, response, and revisions. We sincerely appreciate your constructive suggestions, particularly regarding the interpretation of explanatory gain and the clarification of streaming latency. Your comments have been invaluable in helping us strengthen the paper and improve the clarity of our presentation.

We are grateful that our revisions have addressed your concerns, and we truly appreciate your time, consideration, and support throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.