acceptodds
Withdrawn from ICLR 2027

DRUM: Real-Time Dancing Robots Using Music and Execution Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive robot dancing requires a humanoid to respond to musical input throughout a performance while maintaining coherent and stable motion. Two representative deployment pipelines fall short of this goal: choreography-specific policies require additional training for new routines, whereas offline music-to-motion generators can produce references poorly matched to a general tracker. When either pipeline executes a fixed sequence, upcoming movements cannot be regenerated to accommodate changing music or deviations in the robot's actual state. We present DRUM, a framework for real-time humanoid dancing that combines streaming music-conditioned generation with execution-feedback training. Our streaming model generates successive dance segments, allowing updated music and motion history to shape upcoming movements. To make these movements compatible with an existing tracker, we post-train the generator while keeping the tracker frozen. This training alternates reward-based adaptation of the generated motion distribution with learning conditioned on the tracker's executed motion histories. The two feedback signals address complementary needs: learning what the tracker can execute and how to continue from the states it actually reaches. Together, they connect musical responsiveness with execution-aware motion continuation, reducing the mismatch between planned and realized dance. Experiments demonstrate state-of-the-art performance in streaming dance generation and execution robustness. Real-world humanoid demonstrations further showcase continuous dancing and responsive transitions under user-driven music changes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
humanoid robot, generative model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_C8XpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DRUM, a real-time humanoid dance generation framework that replaces fixed-sequence generation with streaming synthesis. Short dance segments are generated conditioned on incoming music and recently executed robot motion. The generator is further adapted with execution-feedback training (EFT), which alternates tracker-reward optimization with rehearsal on generated and executed motion histories. Experiments on FineDance and AIST++ and deployment on a real X2 humanoid demonstrate competitive dance-generation quality, real-time generation efficiency, and improved execution stability after EFT.

Critical strengths
  1. The paper is well motivated. The mismatch between offline generated references and the states actually reached by a physical humanoid is an important problem for continuous robot dancing.

  2. Conditioning future generation on executed motion and incorporating executed histories during post-training are well suited to the streaming deployment setting.

  3. The evaluation is relatively comprehensive, including generation benchmarks, three frozen trackers, real-robot experiments, ablations, and a user study.

Critical weaknesses

Weakness

  1. The real contribution over existing work is not sufficiently clear. DRUM combines streaming music-conditioned generation, tracker-based execution feedback, and executed-history rehearsal, but several of these ingredients are closely related to prior work such as DiscoForcing, RLPF, and closed-loop motion generation methods. Although the paper discusses these differences, it remains unclear how much methodological novelty lies in the specific combination and alternating optimization strategy, and which interactions between components are essential to the reported gains.

  2. EFT introduces a substantial trade-off between executability and standard generation quality. In Table 1, the model before EFT consistently achieves better FID, diversity, and music-alignment metrics than the final model, particularly on AIST++. The paper acknowledges this trade-off, but it remains unclear how much degradation is necessary to achieve the reported execution gains. An analysis of intermediate training checkpoints or different EFT strengths would help clarify whether a better balance between generation quality and execution robustness is possible.

Minor Weaknesses

  1. The supplementary demonstrations do not provide robot-execution videos for the main competing dance-generation baselines. Since an important claim is that conventional generators produce references that are less suitable for real humanoid execution, side-by-side robot videos using the same tracker would substantially improve the visual evidence for this claim. Besides, the resolution is relatively low, which makes it difficult to inspect motion smoothness, foot contacts, and small tracking artifacts. Providing higher-resolution videos would improve the qualitative evaluation.
Questions for authors
  1. What is the key contribution of DRUM beyond existing streaming generation, tracker-feedback optimization, and closed-loop motion generation methods? In particular, which component or interaction between components is essential to DRUM?

  2. Table 1 shows that EFT consistently worsens several conventional generation metrics, especially FID and BAS on AIST++. How do the authors interpret this trade-off? Is this degradation intrinsic to optimizing tracker executability, or could the strength of EFT be adjusted to obtain a better Pareto trade-off between generation fidelity/music alignment and execution robustness?

  3. In Table 3, reward-only training obtains lower joint error than the full model, whereas adding rehearsal improves motion quality and other execution characteristics. The paper attributes the reward-only behavior to reduced motion amplitude. How sensitive is the balance between execution accuracy and dance expressiveness to the relative weighting of reward optimization and rehearsal?

  4. Since the paper reports quantitative real-robot results for EDGE and FlowerDance, could the authors provide side-by-side real-robot videos of DRUM and the main baselines under the same frozen tracker and matched music?

  5. The reported 680 FPS measures generator throughput while excluding the tracker. Could the authors report the end-to-end latency of one online generation-execution-feedback cycle, including motion generation, communication, tracker inference, and the feedback update, and clarify the hardware used for the real-time deployment?

Reviewer_3sdROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DRUM, a framework for real-time music-conditioned humanoid dancing. Its main contributions are a) the design of a music-conditioned dance generator, that can stream reference motions in real-time, b) the fine-tuning of said generator to improve physical feasibility, and c) the evaluation of said protocol on the real robot using frozen humanoid motion trackers (including human assessment).

Critical strengths
  • The paper is well-written, and generally easy to follow.
  • From a high-level perspective, the method is relatively simple and addresses a clear problem: most humanoid motion generators do not take the capabilities of a given motion tracker into account.
  • Experimental coverage is quite broad, covers the generator and tracking quality in isolation, uses multiple generators and trackers, and even has a human user study. Moreover, the evaluation protocol handles the different baselines very fairly (all trained on the same data, and same checkpoint selection).
Critical weaknesses
  • The experiments evaluate the generator and tracker mostly in isolation. First, the generator with and without EFT are judged by generated motion quality. Second, with the tracker in the loop, only the success and joint errors are measured (for motions from different generators, with and without EFT). A more thorough evaluation (e.g., including dance quality metrics) of the closed-loop system as a whole would add value to the paper, as it would show whether the tracker introduces any phase lag that may hurt the beat alignment, and if DRUM with EFT can outperform DRUM without EFT, once the tracker is in the loop. This could be done entirely in simulation. Extended real-world evaluation (e.g., including FID, DIV, and BAS metrics in section 4.3) would be even stronger, though I do not consider it necessary.
  • The main contribution (EFT) is only evaluated on a single generator (the proposed DRUM). There is no information on whether the proposed fine-tuning step (or its individual components) could generalize to other generators.
  • Some details are omitted (e.g., inference details, confidence intervals in Table 1, real-robot protocol details, and more). Adding them would strengthen the paper.
Questions for authors
  • From what I can tell, the chosen trackers were originally trained for the Unitree G1 humanoid, and there are no public checkpoints for the X2 robot. Did you retrain each tracker from scratch with the datasets listed in 4.3 for the X2 robot? How long did that training take? Also, in 4.3 EDGE is listed as a dataset that you train on, but from my understanding EDGE is a generator which was trained on AIST. Could you clarify?
  • The user study suggests that EFT mostly improves dance stability, as the other metrics seem to be within noise. Have you tried, instead of fine-tuning the generator, fine-tuning the tracker using the frozen generator? If not, could you provide some intuition whether you see this as a potential different solution?
  • Did you try fine-tuning other motion generators (e.g. EDGE)?
  • How long are the real-world trials per tracker, and how many trials each, and how exactly is success defined?
  • Is inference run on-board? Or tracker on-board and generator off-board? What is the round-trip latency on the physical system?
Reviewer_K4zMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DRUM, a system that enables a humanoid to dance continuously to changing music. It trains a motion generator using feedback from a fixed controller and uses the robot’s actual movements to guide subsequent generation. Experiments on FineDance, AIST++, and a real X2 humanoid report improved execution success across three controllers.

Critical strengths

Real-robot evaluation across multiple trackers. The paper reports consistent execution improvements across three frozen trackers and two datasets on an X2 humanoid, providing useful evidence that generator adaptation can improve compatibility with different controllers.

Critical weaknesses
  1. The technical advance over DART, DiscoForcing, and CLoSD is unclear. The framework largely combines established generation and execution-feedback mechanisms. The paper cites DiscoForcing, the closest streaming music-to-dance method, but omits it from the experiments, leaving the claimed advance over existing streaming systems unclear.
  2. The experiments do not clearly support the claimed benefit of rehearsal for preserving expressiveness. In Table 3, reward-only training matches the full method’s success rate of 0.99 and achieves lower joint error (0.10 versus 0.16). Although rehearsal improves foot sliding and key-body error, FID decreases only slightly from 40.19 to 39.76.
  3. Although Table 4 provides comparative human ratings, the paper and supplementary video lack clearly labeled visual comparisons with baselines and ablations, particularly reward-only training. Such comparisons are important for assessing expressiveness and transition smoothness, which the reported automatic metrics do not fully capture.
  4. The retargeting procedure is unclear. The generation benchmarks use X2-retargeted FineDance and AIST++, yet the paper does not specify the retargeting algorithm or its configuration. Since this preprocessing affects motion quality and physical feasibility, the omission makes the benchmarks difficult to reproduce and the results harder to interpret.
Questions for authors
  1. Table 2 reports real-robot success rates, but the evaluation protocol is unclear. What were the trial count, duration, and success criterion for each method and tracker?
  2. The paper defines streaming using 95th-percentile update time but reports only throughput. What is the measured update latency, and how much audio lookahead and buffering are used?
  3. Table 4 reports standard deviations without specifying the aggregation. Were these computed across individual ratings, participant averages, or clip averages?
Reviewer_jajqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DRUM, a framework for real-time music-conditioned humanoid dancing. It combines a streaming diffusion-based motion generator with a frozen motion tracker. The main contribution is execution-feedback training (EFT), which alternates reward-based optimization using tracker feedback with rehearsal on generated and executed motion histories. Experiments on FineDance, AIST++, and a real X2 humanoid show improvements in tracking accuracy and execution success, although the feedback training also degrades motion-generation FID.

Critical strengths
  1. Meaningful problem with practical relevance. Incorporating execution feedback into continuous generation is a sensible and practically useful direction.

  2. The authors evaluate their method with three different frozen trackers on a real humanoid. The consistent improvements in tracking accuracy and execution success provide useful evidence of the framework's practical effectiveness.

Critical weaknesses
  1. The novelty over existing work is not sufficiently established. The main ingredients of DRUM have been explored separately. DiscoForcing already supports real-time music-conditioned motion generation and interactive music changes, while RLPF studies physical-feedback-based optimization of motion generators. Related methods such as PhyGile and CLoSD also incorporate executed motion histories into subsequent generation. DRUM brings these ideas together, but it remains unclear what new technical insight this combination provides. More importantly, the experiments compare primarily against offline music-to-dance generators, without evaluating against DiscoForcing or another relevant streaming baseline. This makes it difficult to assess the actual improvement over existing real-time approaches.

  2. The necessity of the proposed execution-history rehearsal is not convincingly demonstrated. In Tab3, reward-only training already achieves the same execution success rate as the full method (0.99), with an even lower joint tracking error (0.10 vs. 0.16). The ablation without executed histories does suggest that executed-history rehearsal helps when combined with supervised rehearsal. However, the full method still provides limited improvement over reward-only optimization on the reported execution metrics. The authors argue that reward-only training leads to overly conservative motions and reduced expressiveness, but this claim is mainly supported by qualitative observations. The paper lacks quantitative comparisons of motion diversity, amplitude, or human preferences between reward-only training and the full method. Since preserving motion quality is a central motivation for rehearsal, stronger evidence is needed to establish its contribution.

Questions for authors
  1. How does DRUM compare with DiscoForcing under a comparable streaming setting? Can the authors clarify the advantages over existing real-time music-conditioned generation methods?

  2. Can the authors quantify the claimed degradation of dance expressiveness under reward-only training? A comparison of motion diversity, amplitude, beat alignment, and user preferences would help establish whether executed-history rehearsal is necessary.

  3. How many independent real-robot trials were conducted, and how is execution success defined? In addition, can the authors report the end-to-end latency and quantitatively evaluate responsiveness to live music changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.