- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tackles long-form music-to-dance generation, where the dance should change character with the musical structure rather than only follow the beat. Music is split into sections (using the pretrained LinkSeg) and inter-beat intervals, and motion is encoded with ACTOR-PAE into phase latents (section level) and periodic latents (interval level). HEDGE then runs two diffusion stages: a section-level one that denoises phase latents given section music features and neighbouring sections, and an interval-level one that denoises periodic latents with three ControlNets (music, transition, phase). A test-time foot-skating optimization is added on top. The authors also propose SJHK, a per-section correlation between motion speed and music tempo. On FineDance, HEDGE reports much lower FID than FACT, Bailando, EDGE, LODGE and SoulNet, the best SJHK, and the lowest foot-skating ratio when the foot-contact optimization is used. Ablations, a user study and a second dataset (JoruriPuppet) are deferred to the supplement.
Critical strengths
- The problem is well motivated. Most dance generation work stops at beat alignment, and conditioning on musical structure with a section/interval hierarchy is a natural way to go further. The full pipeline hangs together nicely.
- The FID improvements on FineDance are large, e.g., 60.3/18.4 vs.\ 87.5/47.8 for LODGE, and with foot-contact optimization FSR is the lowest in the table. I also appreciated that the authors point out that BAS above the real-data level (FACT, EDGE) isn't actually good.
Critical weaknesses
- The central claim isn't isolated in the main paper. Most building blocks (ACTOR-PAE phase latents, 8-interval segments with 4-interval overlap, transition modules, phase mixing, test-time optimization) come from the cited Au et al.\ (2025, 2026). What is new is mostly the LinkSeg section labels/features as conditioning, plus the SJHK metric. The paper argues that section-level modeling is what makes the dance follow the structure, but the main text has no ablation for it. So I can't tell whether the FID gains come from the proposed idea or from the stronger backbone.
- SJHK doesn't yet convince me as a measure of structural response. It is proposed by the authors, adapted from a metric designed for puppet performance, and the method is explicitly conditioned on sections, so it is favoured by construction. More importantly, real data scores -0.008 in Table 1: ground-truth dances show essentially no within-section speed--tempo correlation, yet HEDGE gets 0.115. Being far above human choreography on a metric isn't obviously a good thing, and there is no evidence in the main text that SJHK matches human judgement. Combined with a single dataset (20 test tracks) and no variance, the evidence for the main claim is thin.
Questions for authors
- Real data has SJHK =-0.008. How should I interpret this, and does SJHK agree with human ratings of structural alignment in your user study?
- Foot-contact optimization raises FID$_k$ from 60.3 to 75.3. Could you discuss this trade-off? Also, in Eq.9 the weight is large when the foot is high; is that the intended sign, or should it be ?
- How robust is the system to LinkSeg mistakes? let's say wrong boundaries or labels?
Concern regarding FID gains and section-level modelling:
The ablation study of the building blocks in the HEDGE framework is reported in Table 3 of the supplementary material. These results indicate that the observed FID gains arise primarily from two components: (1) phase latent diffusion based on ACTOR-PAE, and (2) the use of 8-interval segments in place of fixed-length segments. Removing these components also reduces SJHK, indicating a weaker ability of the generated dance to follow musical structure. Separately, the ablations of the Section-level Diffusion Modules and of the variant without PhaseControl (supplementary Table 3) produce negative SJHK scores. This shows that omitting section-level modelling substantially impairs the model’s ability to generate dance that follows musical structure. The effect of section-level modelling on FID, however, remains marginal.
Concern regarding SJHK=-0.008 on real FineDance data:
On FineDance, the recorded performances are choreographed with changes in motion speed. The accompanying music, however, rarely provides a tempo rise or fall to follow, since most pieces remain at a nearly constant BPM. As a result, SJHK on the real FineDance recordings is poor (SJHK = −0.008). On the JururiPuppet dataset, by contrast, the recorded music exhibits substantial tempo variation. The real dances show a strong correlation with the music, with SJHK higher than that of all generated dances, which is consistent with the expected behaviour of the metric.
Concern regarding foot contact optimization:
Foot-contact optimization slows the motion of a planted foot. This lowers FSR, but it also suppresses the small, energetic steps that remain in contact during real dance. The generated motion therefore moves away from FineDance, and the kinetic FID rises from 60.3 to 75.3. Real FineDance itself has an FSR of 6.62. Under the same criterion, lively footwork produces slight sliding while the foot is down, so a lower FSR is not automatically closer to the data. Because this optimization is a post-processing step, the number of loops sets the operating point between the two metrics. Table 5 of the supplementary material reports this trade-off. Increasing the optimization from 1 to 10 to 100 loops reduces FSR from 6.67 to 4.16 to 1.79, while the kinetic FID rises from 63.85 to 75.27 to 91.79. Equation 9 was also incorrect. The correct form is σ((ϵp−py)/τ). We thank the reviewer for the careful reading of the formula.
Sensitivity to the section label:
The system is designed around music-section labels and is therefore sensitive to errors in those labels. In Table 4 of the supplementary material, the input section label is fixed to a single category for the whole piece. FID and SJHK vary substantially with the category that is used.
The model is trained on LinkSeg labels and is fitted to the LinkSeg prediction distribution, so a segmentation produced differently from LinkSeg degrades the result. We test this with an alternative segmentation from a multimodal LLM: the music is given to qwen3.8-omni-flash, which is prompted to output section labels. In the table below, HEDGE(LinkSeg) is trained on LinkSeg labels and HEDGE(Qwen) is trained on Qwen labels. When the segmentation used for training matches that used for evaluation, SJHK is strong (0.115 under LinkSeg and 0.076 under Qwen). When the two differ, SJHK falls to around 0, even though the source music is the same piece. This indicates that the model’s SJHK is sensitive to the segmentation on which it is trained.