- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SingDance, a unified framework for singing-and-dancing video generation conditioned on a reference image, text prompt, and audio track. The proposed method incorporates explicit vocal-role control to distinguish between on-screen vocal production and off-screen vocals, enabling both singing-and-dancing and dancing-only generation. The framework adopts a two-stage training strategy, first learning speaking/listening behaviors and then introducing music-conditioned body motion. Notably, the model is trained without paired singing-and-dancing videos and achieves the target task through compositional zero-shot generalization. Extensive experiments evaluate dance generation, lip synchronization, vocal-role switching, and visual quality.
Critical strengths
S1:Interesting task formulation. The paper explores singing-and-dancing video generation, extending existing music-driven dance generation to incorporate controllable vocal articulation. The task has practical value for personalized human video generation.
S2: Comprehensive experimental evaluation. The experimental section is well organized and covers multiple aspects of the proposed framework, including zero-shot singing-and-dancing, dancing-only generation, paired vocal-role switching, and speech-driven lip synchronization. The authors also provide ablation studies and human evaluations, making the experimental evaluation relatively thorough.
S3: Clear and practical training strategy. The staged training approach is straightforward and well motivated by the limited availability of paired singing-and-dancing data. Learning vocal articulation and music-conditioned motion separately, then composing them at inference time, provides a practical approach to the target task without requiring paired training samples.
Critical weaknesses
W1. Insufficient discussion of the unique challenges of the proposed task. Although singing-and-dancing video generation is an interesting application, the paper does not sufficiently clarify what makes it a distinct technical problem beyond combining speech-driven lip synchronization and music-driven dance generation. The proposed framework largely treats these two capabilities as separately learnable components that can be composed at inference time. However, singing-and-dancing is not necessarily a straightforward combination of vocal articulation and body motion, as the two may interact in ways that are not captured by independently learning them. I would expect a paper introducing a new task to provide a more detailed discussion of its unique challenges and explain why existing approaches to the two individual tasks are insufficient. Without such an analysis, the motivation for treating singing-and-dancing as a separate research problem remains somewhat limited.
W2. Limited methodological novelty. Most of the technical components employed in SingDance are relatively standard. The framework uses pretrained Wav2Vec 2.0 and MuQ encoders for speech and music representations, learned embeddings for vocal-role control, task-dependent token routing, and cross-attention for audio conditioning. The staged training strategy and replay mechanism are also established techniques. While combining these components into a unified framework is reasonable, the paper does not introduce a particularly novel modeling mechanism. The main contribution therefore appears to lie in the task formulation and the integration of existing techniques, rather than a substantial methodological advance.
W3. Limited analysis of the interaction between singing and dancing. Although the paper provides a comprehensive evaluation of both lip synchronization and motion–beat alignment, it lacks a more detailed analysis of the interaction between these two capabilities. In particular, it remains unclear whether large-amplitude body movements negatively affect lip synchronization performance. Since simultaneous singing and dancing is the central task of this paper, an analysis of lip synchronization under different levels of body motion intensity would further strengthen the experimental evaluation.
Questions for authors
Q1. The decision to exclude paired singing-and-dancing videos from training is understandable given the potential scarcity of such data. Have the authors considered how the proposed model would perform if a small amount of paired singing-and-dancing data were introduced during fine-tuning? Would this substantially improve generation quality, and what would it imply about the benefits and limitations of the proposed compositional training strategy?
Q2. Could the authors clarify the construction of the Action Prompts used for SingDance-Test and Dance-Only-Test, ideally providing representative examples of the actual prompts used for each baseline? In particular, do these prompts contain fine-grained motion descriptions extracted from the corresponding ground-truth videos?
Q3. Why is instrumental dancing-only generation also assigned the listener role, even though no vocal source is present? Does the meaning of the listener role remain consistent across speech interaction, song-based dancing, and instrumental dancing? What advantage does this semantic role formulation offer over a simpler explicit lip-sync ON/OFF condition?
Q4. Could the authors report the number or percentage of valid samples retained for each method when computing the quantitative metrics, particularly those relying on face or pose tracking? Since videos with invalid detections are excluded, these statistics would help assess the reliability and fairness of the reported comparisons.
5 comments
Response to Reviewer SssY (Part 2/5)
(1) Distinct challenges and the factors affecting articulation (continued)
Supplementary controlled analysis. We agree that a new task calls for a closer analysis of what makes it difficult. Following the suggestion in Weakness 3, we change one factor at a time, including the amount of body motion, and measure lip synchronization.
Lip synchronization from talking to singing-and-dancing, changing one factor at a time.
| Row | Setting | LSE-C ↑ | LSE-D ↓ | Upper-body / head speed |
|---|---|---|---|---|
| A | EMTD references, speech | 8.11 | 7.48 | 0.21 / 0.08 |
| B | A, speech replaced by songs | 6.08 | 8.71 | 0.19 / 0.08 |
| C | B, singing-and-dancing configuration | 5.74 | 8.90 | 0.31 / 0.13 |
| D | SingDance-Test references, standing still | 5.13 | 9.09 | 0.18 / 0.11 |
| E | D, gentle swaying | 4.87 | 8.92 | 0.39 / 0.26 |
| F | SingDance-Source (D, original dance prompt)† | 4.94 | 8.94 | 0.76 / 0.24 |
Notes. Source role. This comparison uses its own common valid subset: all values are averaged over the 48 common valid cases, i.e., the cases with valid SyncNet tracks under every setting. † Same setting as SingDance-Source in Table 3; its values differ slightly from Table 3 because of the different common valid subset. A–C use a low-motion subset of 50 EMTD references, so that the talking rows have motion comparable to standing still (A therefore differs from SingDance in Table 4); A and B follow the settings of Table 4, and in B each reference is paired with one SingDance-Test song. C adds the routed music token and the three-pass guidance used for singing-and-dancing. D–F use the SingDance-Test references and songs with the settings of Table 2 and differ only in the action field. Speeds are mean DWPose speeds in shoulder widths per second. Face height is 224–230 px in A–C and 190–198 px in D–F.
- Singing is harder to articulate than speech. From A to B, motion is essentially unchanged, yet LSE-C drops from 8.11 to 6.08, the largest single change in the table. Wav2Vec 2.0 features of the speech and song audio in A and B also form clearly separated clusters (Appendix G, Figure 6), consistent with this drop.
- Full-body framing lowers synchronization further. From C to D, LSE-C drops to 5.13 although motion decreases.
- Body motion also lowers synchronization. Both moving conditions (E, F) have lower LSE-C than standing still (D), while LSE-D changes little.
In summary, singing-and-dancing is a practical task that existing methods cover only in parts, and its articulation is affected by the singing audio, the full-body framing, and the accompanying body motion. SingDance provides a feasible solution, and the analyses above characterize where the difficulty lies.
Response to Reviewer SssY (Part 3/5)
(2) Positioning of the contribution (Response to Weakness 2)
"Weakness 2: Most of the technical components employed in SingDance are relatively standard. … The main contribution therefore appears to lie in the task formulation and the integration of existing techniques, rather than a substantial methodological advance."
SingDance deliberately builds on a pretrained video model and established components, and its contribution lies at the level of the task, its formulation, and the resulting system:
- A practical task that existing methods cover only in parts. The same song should support either dancing-only or singing-and-dancing, which requires song-conditioned dance and controllable articulation in one video. To our knowledge, existing music-conditioned dance methods do not control whether the dancer articulates the vocals. Other reviewers describe this as "a useful control problem" (Reviewer jbib) and the unified handling of four tasks as "novel and practically appealing" (Reviewer xsUx).
- A formulation that makes the task learnable without paired data. Articulation is defined as a vocal role, so the source and listener states learned from speech transfer to songs: the speech-stage checkpoint, trained before any song data is introduced, already reaches LSE-C 5.19 as Source and 1.14 as Listener on SingDance-Test. The conditions are organized so that the held-out Song/Source configuration differs from a trained configuration only in this role, which, as Reviewer jbib notes, "makes the proposed composition concrete".
- A system built with appropriate design and training choices. We adapt Wan2.2-TI2V-5B with separate speech and music pathways and role conditions, and train it in two stages on curated speech and dance data (Appendix C); as the reviewer notes, the staged training is "straightforward and well motivated". The same checkpoint reaches competitive lip synchronization on EMTD with roughly one-third of InfiniteTalk's generation-time parameters (Table 4).
- A measurable benefit of end-to-end composition. With the same inputs, end-to-end generation achieves higher LSE-C and visual quality than applying either of two recent lip-sync models afterward (see (1)), and a single model covers speaking, listening, dancing-only, and singing-and-dancing.
(3) Fine-tuning with a small amount of paired data (Response to Question 1)
"Question 1: … Have the authors considered how the proposed model would perform if a small amount of paired singing-and-dancing data were introduced during fine-tuning? … what would it imply about the benefits and limitations of the proposed compositional training strategy?"
We have not fine-tuned with paired data, because this work tests whether the target behavior can be obtained compositionally without such supervision; adding it would remove the held-out setting that our evaluation relies on. We expect a small amount of in-domain paired data to further improve singing articulation, and the compositional model is a natural initialization for such fine-tuning. The compositional strategy is most useful when paired data are scarce, and paired supervision is complementary to it.
Response to Reviewer SssY (Part 1/5)
Dear Reviewer SssY,
Thank you for the thoughtful review, and for recognizing the practical value of the task, the comprehensive evaluation, and the staged training strategy. Your concerns center on what distinguishes singing-and-dancing from combining two existing tasks, how singing and dancing interact, what the method contributes beyond standard components, and how the evaluation is set up. We address them below with clarifications and supplementary experiments. All revisions are marked in blue in the updated PDF.
(1) Distinct challenges and the factors affecting articulation (Response to Weakness 1 & Weakness 3)
Please refer to Sec. 4.4 (Lines 463–465, Page 9), Appendix G (Lines 1106–1180, Pages 21–22), and Appendix J (Lines 1269–1295, Page 24) of the revised manuscript.We thank the reviewer for these suggestions, which help us present the task more completely. We add a discussion of its distinct challenges and a controlled analysis of articulation, including the effect of body motion, as Appendix G, and cite them in Sec. 4.4. The cascade comparison below is in Appendix J.
A practical task. Short-form dance videos include both dancing-only and singing-and-dancing, and for the same song a creator may want either. A model for this setting therefore has to generate song-conditioned dance and, in the same video, control whether the dancer sings.
Existing work covers the two parts separately. Interactive avatar systems control speaking and listening in conversation (e.g., LPM 1.0, Zeng et al., 2026; StreamAvatar, Sun et al., 2026), and music-conditioned dance methods generate body motion without vocal articulation (e.g., X-Dancer and MusicInfuser). Neither line addresses singing-and-dancing, where one song drives the dance and, depending on the role, the articulation.
Combining them after generation is not sufficient. A natural alternative generates the dance first and then edits the lips with a lip-sync model. We apply two recent lip-sync models with representative designs, OmniSync (Peng et al., 2025b), which edits the whole frame, and LatentSync (Li et al., 2024), which inpaints a face crop, to our own Listener videos, so that body motion before editing is identical. End-to-end generation compares favorably in three respects: