acceptodds
Under review as a conference paper at ICLR 2027

HEDGE: Hierarchical Expressive Dance Generation with Long-term Music Structure Guidance

Abstract

Generating dance from music not only requires precise beat alignment but also synchronization with the global musical structure, especially in long sequences containing multiple sections such as intros, verses, and choruses. Existing methods primarily focus on beat-level alignment, but often fail to capture and reflect broader musical dynamics, which limits expressive variation and results in choreography that does not adapt well to section-level transitions. To address these limitations, we introduce hierarchical segmentation for both music and motion, organizing them into section and interval levels to more effectively capture structural and rhythmic details. Building on this representation, we propose HEDGE, a two-level diffusion-based framework that generates dance sequences conditioned on both local rhythmic cues and global musical structure. Our method produces choreography that is not only realistic and musically synchronized, but also dynamically adapts to the expressive changes across musical sections. Extensive experiments demonstrate that HEDGE achieves superior responsiveness to musical structure, offering a significant advancement toward automated choreography that faithfully reflects the full narrative and expressive intent of music.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Music-to-Dance Generation, Music Structure Analysis, Structure-Aware Long-Form Choreography

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fQrhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles long-form music-to-dance generation, where the dance should change character with the musical structure rather than only follow the beat. Music is split into sections (using the pretrained LinkSeg) and inter-beat intervals, and motion is encoded with ACTOR-PAE into phase latents (section level) and periodic latents (interval level). HEDGE then runs two diffusion stages: a section-level one that denoises phase latents given section music features and neighbouring sections, and an interval-level one that denoises periodic latents with three ControlNets (music, transition, phase). A test-time foot-skating optimization is added on top. The authors also propose SJHK, a per-section correlation between motion speed and music tempo. On FineDance, HEDGE reports much lower FID than FACT, Bailando, EDGE, LODGE and SoulNet, the best SJHK, and the lowest foot-skating ratio when the foot-contact optimization is used. Ablations, a user study and a second dataset (JoruriPuppet) are deferred to the supplement.

Critical strengths
  • The problem is well motivated. Most dance generation work stops at beat alignment, and conditioning on musical structure with a section/interval hierarchy is a natural way to go further. The full pipeline hangs together nicely.
  • The FID improvements on FineDance are large, e.g., 60.3/18.4 vs.\ 87.5/47.8 for LODGE, and with foot-contact optimization FSR is the lowest in the table. I also appreciated that the authors point out that BAS above the real-data level (FACT, EDGE) isn't actually good.
Critical weaknesses
  • The central claim isn't isolated in the main paper. Most building blocks (ACTOR-PAE phase latents, 8-interval segments with 4-interval overlap, transition modules, phase mixing, test-time optimization) come from the cited Au et al.\ (2025, 2026). What is new is mostly the LinkSeg section labels/features as conditioning, plus the SJHK metric. The paper argues that section-level modeling is what makes the dance follow the structure, but the main text has no ablation for it. So I can't tell whether the FID gains come from the proposed idea or from the stronger backbone.
  • SJHK doesn't yet convince me as a measure of structural response. It is proposed by the authors, adapted from a metric designed for puppet performance, and the method is explicitly conditioned on sections, so it is favoured by construction. More importantly, real data scores -0.008 in Table 1: ground-truth dances show essentially no within-section speed--tempo correlation, yet HEDGE gets 0.115. Being far above human choreography on a metric isn't obviously a good thing, and there is no evidence in the main text that SJHK matches human judgement. Combined with a single dataset (20 test tracks) and no variance, the evidence for the main claim is thin.
Questions for authors
  • Real data has SJHK =-0.008. How should I interpret this, and does SJHK agree with human ratings of structural alignment in your user study?
  • Foot-contact optimization raises FID$_k$ from 60.3 to 75.3. Could you discuss this trade-off? Also, in Eq.9 the weight σ((py−ϵp)/τ)\sigma((p_y-\epsilon_p)/\tau) is large when the foot is high; is that the intended sign, or should it be ϵp−py\epsilon_p-p_y?
  • How robust is the system to LinkSeg mistakes? let's say wrong boundaries or labels?
1 comment

Concern regarding FID gains and section-level modelling:

The ablation study of the building blocks in the HEDGE framework is reported in Table 3 of the supplementary material. These results indicate that the observed FID gains arise primarily from two components: (1) phase latent diffusion based on ACTOR-PAE, and (2) the use of 8-interval segments in place of fixed-length segments. Removing these components also reduces SJHK, indicating a weaker ability of the generated dance to follow musical structure. Separately, the ablations of the Section-level Diffusion Modules and of the variant without PhaseControl (supplementary Table 3) produce negative SJHK scores. This shows that omitting section-level modelling substantially impairs the model’s ability to generate dance that follows musical structure. The effect of section-level modelling on FID, however, remains marginal.

Concern regarding SJHK=-0.008 on real FineDance data:

On FineDance, the recorded performances are choreographed with changes in motion speed. The accompanying music, however, rarely provides a tempo rise or fall to follow, since most pieces remain at a nearly constant BPM. As a result, SJHK on the real FineDance recordings is poor (SJHK = −0.008). On the JururiPuppet dataset, by contrast, the recorded music exhibits substantial tempo variation. The real dances show a strong correlation with the music, with SJHK higher than that of all generated dances, which is consistent with the expected behaviour of the metric.

Concern regarding foot contact optimization:

Foot-contact optimization slows the motion of a planted foot. This lowers FSR, but it also suppresses the small, energetic steps that remain in contact during real dance. The generated motion therefore moves away from FineDance, and the kinetic FID rises from 60.3 to 75.3. Real FineDance itself has an FSR of 6.62. Under the same criterion, lively footwork produces slight sliding while the foot is down, so a lower FSR is not automatically closer to the data. Because this optimization is a post-processing step, the number of loops sets the operating point between the two metrics. Table 5 of the supplementary material reports this trade-off. Increasing the optimization from 1 to 10 to 100 loops reduces FSR from 6.67 to 4.16 to 1.79, while the kinetic FID rises from 63.85 to 75.27 to 91.79. Equation 9 was also incorrect. The correct form is σ((ϵp−py)/τ)\sigma((\epsilon_p - p_y)/\tau). We thank the reviewer for the careful reading of the formula.

Sensitivity to the section label:

The system is designed around music-section labels and is therefore sensitive to errors in those labels. In Table 4 of the supplementary material, the input section label is fixed to a single category for the whole piece. FID and SJHK vary substantially with the category that is used.

The model is trained on LinkSeg labels and is fitted to the LinkSeg prediction distribution, so a segmentation produced differently from LinkSeg degrades the result. We test this with an alternative segmentation from a multimodal LLM: the music is given to qwen3.8-omni-flash, which is prompted to output section labels. In the table below, HEDGE(LinkSeg) is trained on LinkSeg labels and HEDGE(Qwen) is trained on Qwen labels. When the segmentation used for training matches that used for evaluation, SJHK is strong (0.115 under LinkSeg and 0.076 under Qwen). When the two differ, SJHK falls to around 0, even though the source music is the same piece. This indicates that the model’s SJHK is sensitive to the segmentation on which it is trained.

MetricHEDGE(LinkSeg)HEDGE(Qwen)bailandoedgefactlodgeSoulNet
SJHK(LinkSeg)0.115-0.0050.025-0.015-0.0570.0660.014
SJHK(Qwen)-0.0360.076-0.0090.015-0.044-0.017-0.012
Reviewer_P29GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents HEDGE, a hierarchical diffusion framework for long music-conditioned dance generation. It organizes music and motion into musical sections and overlapping beat-based intervals. Section-level diffusion generates compact motion phase parameters, while interval-level diffusion produces detailed motion using phase guidance, neighboring motion context, and music-conditioned ControlNets. An optional test-time optimization reduces foot skating. The method is evaluated on FineDance and JoruriPuppet, with component ablations and a user study. The paper also introduces Section-aware Jo-Ha-Kyu (SJHK), which averages within-section correlations between joint speed and instantaneous music tempo.

Critical strengths
  1. Modeling musical structure in long dance generation is a meaningful goal, and the section–interval hierarchy provides a reasonable inductive bias.

  2. Compact phase parameters for section-level planning and dense periodic latents for local synthesis provide a sensible connection between temporal scales.

  3. The experiments include two datasets, component ablations, and a user study, with substantial FID improvements over the retrained LODGE baseline on FineDance.

Critical weaknesses
  1. SJHK averages within-section Pearson correlations between joint speed and music tempo. Independently rescaling or shifting the speed sequence within each section leaves these correlations unchanged, while potentially changing the relative dynamics between sections. It therefore measures within-section tempo coupling rather than directly assessing cross-section choreographic changes. Its relationship to perceived musical structure alignment needs validation before the reported improvement can support the paper’s central claim.

  2. The FID improvements are substantial, but distributional similarity does not directly demonstrate better musical structure expression. Without post-processing, HEDGE has a higher FSR than LODGE (7.53 versus 4.31); optimization reduces it to 4.16 while worsening FIDk from 60.310 to 75.268. The user study provides positive evidence, but the number and selection of evaluated clips and the uncertainty of the ratings are insufficiently reported, leaving the consistency and magnitude of the perceptual advantage unclear.

Questions for authors
  1. What evidence shows that SJHK agrees with human judgments of musical structure alignment? Given its invariance to independent affine transformations of joint speed within each section, how does it assess choreographic changes between sections?

  2. How many music tracks and clips were evaluated in the user study, what were their durations, and how were they selected? Which post-processing configurations were used for HEDGE and LODGE? Could the authors report paired rating differences with uncertainty estimates to clarify whether the perceptual advantage is consistent?

2 comments

We thank the reviewer for this careful reading of the metric.

What SJHK measures:

A music–dance pattern can be separated into two components. The first is the average speed of a section. This is the quantity raised in the review, and it is the quantity evaluated by the original JHK metric. The second is the rhythmic pattern of that section, given by how instant tempo and joint speed change from beat to beat. This is the quantity SJHK is designed to measure.

The importance of SJHK follows from the observation that evaluation based on average speed can miss a change in choreography. Music in a typical pop setting, including pieces in FineDance, is usually written around one tempo for the whole track, so intro, verse, and chorus stay close in average BPM and differ in local timing. The corresponding statistical analysis is given in the JHK analysis below. A repetitive motion can therefore score well under JHK while carrying no rhythmic structure within a section. Because JHK computes a single correlation over whole piece, it responds to coarse changes in average speed and can assign a high score to such motion. Reducing each section’s BPM and speed pattern to its average-speed line removes the within-section fluctuations, so the choreographic pattern is no longer visible.

SJHK evaluates the second component. It scores how closely beat-to-beat changes in joint speed follow the instant tempo inside each section. Two sections can share the same average tempo, while the speed inside each section still rises and falls with that section’s instant tempo and forms a choreographic pattern. Invariance to a shift or a rescaling keeps the score on this contour, so the rhythmic pattern of the section is identified on its own. That is the quantity we intend to measure, and it matches what listeners expect on the choreography changes: the music stays around one tempo for whole track, while instant BPM and dance pattern are more uniform in the intro and more varied in the chorus.

BPM variation and JHK scores on FineDance:

To address reviewer’s concern about average speed changes between sections, we first analyze BPM variation of FineDance test music. For each piece, we define BPM range as the difference between the 90th and 10th percentiles. The median of this range is only 5.9 BPM, while the median of the per-piece mean BPM is 117.0. A range of 5.9 BPM around a mean of 117.0 indicates that the tempo within a piece changes little. Because the test music maintains a nearly constant BPM throughout each piece, the original JHK evaluation favors a repetitive dance performed at a constant tempo, and thus does not reflect perceived changes in choreographic pattern.

We further report JHK scores on FineDance, computed with original JHK implementation. The reference dances obtain the lowest score. Professional choreography varies across sections, so its speed contour departs from the nearly constant BPM of FineDance test music and the correlation falls. Methods commonly regarded as repetitive, such as FACT and EDGE, receive substantially higher scores. SoulNet tends to generate constant-speed, repetitive dance because the trained HRVQ module stores speed in a coarse body token, so reusing that token holds one velocity across its frames and produces a flat speed contour. From such evaluation, JHK metric on FineDance therefore favors repetitive dance.

ReferenceHEDGEHEDGE(fc_opt)BailandoEDGEFACTLODGESoulNet
−0.178−0.134−0.131−0.100−0.0201−0.0908−0.1560.106

Foot skating handling between LODGE and HEDGE:

The FSR gap between LODGE and HEDGE comes from where foot skating is handled. LODGE performs forward kinematics inside the Foot Refine Block, so the correction is applied once at each of the 50 DDIM steps. This yields a better foot-skating ratio (FSR) even without post-processing, at the cost of a slower diffusion process. HEDGE moves forward kinematics out of diffusion and into post-processing. Foot skating can then be handled more flexibly, with lower memory use and only 10 optimization iterations. The effect of the number of post-processing iterations is reported in Table 5 of the supplementary material.

User Study:

To further substantiate the perceptual advantage, we recruited 25 additional participants, increasing the sample from 20 to 45. Each entry below is the previous mean ± std (n = 20), followed by the updated mean ± std (n = 45). The user study is being done on 10 tracks randomly sampled from the 20 test tracks. Their durations have a minimum of 42.55 s, a mean of 105.12 s, and a maximum of 192.40 s.

FACTEDGELODGEHEDGE
Naturalness1.95±0.91=>2.44±1.243.16±0.90=>2.96±1.243.16±1.07=>3.44±1.103.68±1.25=>3.98±1.08
Coherence1.74±0.87=>2.31±1.292.74±0.99=>2.87±1.183.26±0.99=>3.58±0.993.74±1.37=>3.93±1.10
Structure Alignment1.95±0.71=>2.42±1.232.68±1.06=>3.07±1.193.16±0.96=>3.62±1.073.68±1.42=>4.02±1.16
Area_Chair_pqRdOpenReview ↗

Reviewer P29G, please respond to authors today

Reviewer P29G,

Today is the last day of the discussion period. The authors have prepared a detailed response to your review. Please reply to them as soon as possible to ensure there is a chance for back and forth discussion before the discussion period ends.

AC

Reviewer_56cFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HEDGE generates long music-conditioned dance with a two-level diffusion framework. Predicted musical sections and beat-based, variable-duration windows organize the sequence; compact phase latents connect section-level guidance to local motion generation. The paper evaluates motion quality and music alignment on FineDance and JoruriPuppet, with ablations and a user study.

Critical strengths
  1. The presented dance has a tangible quality advantage over the displayed baselines: movements are more expansive, with natural pauses, stable poses, and smoother transitions. The folk dance in 00:02:30 is especially brilliant. FineDance FID scores and the user study support the practical importance of this observation.
  2. Unlike a fixed-duration hierarchy, HEDGE aligns its windows with predicted musical sections and beats, using phase latents to pass guidance to local generation. This design enhanced its ability to generate more distinct movements across the sections in one song, comparing to LODGE. Its representation and module ablations offer evidence that this design is useful, even if they do not isolate every contribution.
Critical weaknesses
  1. The structure-alignment claim relies on automatically predicted section labels. In the spotlight examples, labels appear to switch across short phrases that may belong to one continuous chorus. The authors acknowledge a similar LinkSeg failure mode. Without independent boundary validation, responding to these labels may not mean responding to actual musical sections. SJHK, computed within predicted sections, does not settle this.
  2. The improvement over LODGE is difficult to attribute specifically to section-level guidance. The methods also differ in motion representation, model components, and segmentation. The existing ablations are helpful, but removing PhaseControl still leaves section labels in the local model. A capacity-matched local-only comparison would better isolate the central design claim.
  3. Despite clear improvements over the displayed baselines, some generated poses still appear constrained, with noticeably bent legs rather than full extension. Some movement accents also appear misaligned with the musical beats (e.g., around 00:05:00 in the supplementary video). These examples limit the perceived expressiveness and rhythmic precision.
Questions for authors
  1. Are the spotlight labels raw LinkSeg predictions? Can the authors compare their boundaries with independently checked music sections and examine the effect of correcting erroneous labels on representative results?
  2. What are the exact BAS formula, motion-beat extraction, and reference-score capping procedure? Why is real-data SJHK -0.008 in the main paper but -0.087 in the supplement?
1 comment

Independently checked music sections:

The music-motion-segmentation labels used in this work, including those used for generation, are taken directly from the raw LinkSeg predictions. These boundaries are difficult to verify because ground-truth musical sections are not available in most recent music-to-motion datasets, including FineDance and JoruriPuppet. We also considered another music-section method, All-In-One, but it could not be installed. We therefore obtained an alternative segmentation with a multimodal LLM. The music segmentation is given to qwen3.8-omni-flash, which is prompted to return music-section labels. After running both Qwen and LinkSeg on the test music, we adapt the pipeline so that either segmentation can condition dance generation. Because SJHK depends on the section boundaries, each result is evaluated under each segmentation. The scores are reported below.

MetricHEDGE(LinkSeg)HEDGE(Qwen)bailandoedgefactlodgeSoulNet
SJHK(LinkSeg)0.115-0.0050.025-0.015-0.0570.0660.014
SJHK(Qwen)-0.0360.076-0.0090.015-0.044-0.017-0.012

HEDGE trained on LinkSeg attains a strong SJHK when evaluated with LinkSeg. The Qwen-trained model is evaluated in the matched setting, using the Qwen segmentation for both training and evaluation. When the segmentation used for training differs from that used for evaluation, SJHK falls to around 0, even though the source music is the same piece. This indicates that our model’s SJHK is sensitive to the segmentation on which it is trained. The comparison models generate motion autoregressively over short windows and do not use music-section segmentation, so their SJHK remains near 0 under both segmentations.

Comparison without section labels as input:

To isolate the effect of the music section labels, we adjust HEDGE in two ways: the section label is removed from the input, and PhaseControl is removed. Segments are still processed in 8-beat intervals, so the model stays at a local level and is disconnected from the global section structure. The results are shown below. Relative to Original HEDGE, removing the section label gives a higher BAS (0.232 compared with 0.215) and a lower SJHK (−0.012 compared with 0.115). The FID and diversity values stay similar.

HEDGE variantsFIDkFIDgDivkDivgBASFSRSJHK
Original60.31018.3506.7496.9730.2157.530.115
w.o.PhaseControl53.03425.1755.4885.3440.2089.45-0.023
no_sect_label54.16125.6205.7185.7020.2328.20-0.012

Beat extraction and metric calculation:

We extract beats with librosa.beat.beat_track. Specifically, the audio is loaded at a sampling rate of 30 FPS times a hop length of 512. Then, the onset envelope is computed with librosa.onset.onset_strength, and its initial tempo is estimated with librosa.beat.tempo. librosa.beat.beat_track is then called on that envelope with hop_length 512, tightness 100, and this initial tempo as start_bpm. Note that the same beat sequence is used throughout the experiment, including data segmentation, the music input, and the final BAS and SJHK calculations.

The BAS score follows the BAS metrics in FACT. For music beats BmB^{m} and dance beats BdB^{d}:

BAS=1∣Bm∣∑tm∈Bmexp⁡(−min⁡td∈Bd(td−tm)22σ2),σ=3.\mathrm{BAS}=\frac{1}{|B^{m}|}\sum_{t^{m}\in B^{m}}\exp\left(-\frac{\min_{t^{d}\in B^{d}}(t^{d}-t^{m})^{2}}{2\sigma^{2}}\right),\qquad\sigma=3.

The reported score is the mean of this value over the test sequences. We do not use a reference-score capping procedure. On real data, the SJHK score is −0.0087, in agreement with the value reported in the main paper. The value in the supplementary material contains a typo in which a zero was dropped before formatting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.