acceptodds
Under review as a conference paper at ICLR 2027

SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning

Abstract

Generating personalized dance videos from a reference image, text prompt, and audio track requires music-conditioned body motion. Singing-and-dancing adds a second requirement: the visible subject must also articulate the vocals. Existing music-conditioned methods focus primarily on choreography, while speech-driven models generally assume that the visible subject produces the input voice, leaving this combined setting largely underexplored. We introduce SingDance, a unified video diffusion framework that formulates controllable vocal articulation as a semantic role: the visible subject is either the source, who produces the vocal signal, or the listener, who receives it from an off-screen performer. Task-defined hard-compact routing selects task-relevant speech, music, and role conditions, which are composed through frame-wise joint audio injection; source and listener retain the same speech pathway. Training uses asymmetric supervision: on-screen speaking and curated off-screen conversational-response videos establish role control, while instrumental and song-based dancing-only videos establish music-conditioned body motion. The target Song/Source configuration is held out from SingDance task training. At inference, assigning the source role to a song composes separately learned articulation and song-conditioned dance capabilities, enabling compositional zero-shot singing-and-dancing. Experiments demonstrate strong motion–beat alignment and visual fidelity, reliable paired switching of vocal articulation while preserving music-aligned body motion, and highly competitive lip synchronization with substantially fewer generation-time parameters than the strongest speech-driven baseline evaluated. Video samples are available at https://anonymous4submit-pages.github.io/submission-demo/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video generation, audio-driven human animation, music-conditioned dance generation, lip synchronization, video diffusion models

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_SssYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SingDance, a unified framework for singing-and-dancing video generation conditioned on a reference image, text prompt, and audio track. The proposed method incorporates explicit vocal-role control to distinguish between on-screen vocal production and off-screen vocals, enabling both singing-and-dancing and dancing-only generation. The framework adopts a two-stage training strategy, first learning speaking/listening behaviors and then introducing music-conditioned body motion. Notably, the model is trained without paired singing-and-dancing videos and achieves the target task through compositional zero-shot generalization. Extensive experiments evaluate dance generation, lip synchronization, vocal-role switching, and visual quality.

Critical strengths

S1:Interesting task formulation. The paper explores singing-and-dancing video generation, extending existing music-driven dance generation to incorporate controllable vocal articulation. The task has practical value for personalized human video generation.

S2: Comprehensive experimental evaluation. The experimental section is well organized and covers multiple aspects of the proposed framework, including zero-shot singing-and-dancing, dancing-only generation, paired vocal-role switching, and speech-driven lip synchronization. The authors also provide ablation studies and human evaluations, making the experimental evaluation relatively thorough.

S3: Clear and practical training strategy. The staged training approach is straightforward and well motivated by the limited availability of paired singing-and-dancing data. Learning vocal articulation and music-conditioned motion separately, then composing them at inference time, provides a practical approach to the target task without requiring paired training samples.

Critical weaknesses

W1. Insufficient discussion of the unique challenges of the proposed task. Although singing-and-dancing video generation is an interesting application, the paper does not sufficiently clarify what makes it a distinct technical problem beyond combining speech-driven lip synchronization and music-driven dance generation. The proposed framework largely treats these two capabilities as separately learnable components that can be composed at inference time. However, singing-and-dancing is not necessarily a straightforward combination of vocal articulation and body motion, as the two may interact in ways that are not captured by independently learning them. I would expect a paper introducing a new task to provide a more detailed discussion of its unique challenges and explain why existing approaches to the two individual tasks are insufficient. Without such an analysis, the motivation for treating singing-and-dancing as a separate research problem remains somewhat limited.

W2. Limited methodological novelty. Most of the technical components employed in SingDance are relatively standard. The framework uses pretrained Wav2Vec 2.0 and MuQ encoders for speech and music representations, learned embeddings for vocal-role control, task-dependent token routing, and cross-attention for audio conditioning. The staged training strategy and replay mechanism are also established techniques. While combining these components into a unified framework is reasonable, the paper does not introduce a particularly novel modeling mechanism. The main contribution therefore appears to lie in the task formulation and the integration of existing techniques, rather than a substantial methodological advance.

W3. Limited analysis of the interaction between singing and dancing. Although the paper provides a comprehensive evaluation of both lip synchronization and motion–beat alignment, it lacks a more detailed analysis of the interaction between these two capabilities. In particular, it remains unclear whether large-amplitude body movements negatively affect lip synchronization performance. Since simultaneous singing and dancing is the central task of this paper, an analysis of lip synchronization under different levels of body motion intensity would further strengthen the experimental evaluation.

Questions for authors

Q1. The decision to exclude paired singing-and-dancing videos from training is understandable given the potential scarcity of such data. Have the authors considered how the proposed model would perform if a small amount of paired singing-and-dancing data were introduced during fine-tuning? Would this substantially improve generation quality, and what would it imply about the benefits and limitations of the proposed compositional training strategy?

Q2. Could the authors clarify the construction of the Action Prompts used for SingDance-Test and Dance-Only-Test, ideally providing representative examples of the actual prompts used for each baseline? In particular, do these prompts contain fine-grained motion descriptions extracted from the corresponding ground-truth videos?

Q3. Why is instrumental dancing-only generation also assigned the listener role, even though no vocal source is present? Does the meaning of the listener role remain consistent across speech interaction, song-based dancing, and instrumental dancing? What advantage does this semantic role formulation offer over a simpler explicit lip-sync ON/OFF condition?

Q4. Could the authors report the number or percentage of valid samples retained for each method when computing the quantitative metrics, particularly those relying on face or pose tracking? Since videos with invalid detections are excluded, these statistics would help assess the reliability and fairness of the reported comparisons.

5 comments
AuthorsOpenReview ↗

Response to Reviewer SssY (Part 1/5)

Dear Reviewer SssY,

Thank you for the thoughtful review, and for recognizing the practical value of the task, the comprehensive evaluation, and the staged training strategy. Your concerns center on what distinguishes singing-and-dancing from combining two existing tasks, how singing and dancing interact, what the method contributes beyond standard components, and how the evaluation is set up. We address them below with clarifications and supplementary experiments. All revisions are marked in blue in the updated PDF.

(1) Distinct challenges and the factors affecting articulation (Response to Weakness 1 & Weakness 3)

"Weakness 1: … the paper does not sufficiently clarify what makes it a distinct technical problem beyond combining speech-driven lip synchronization and music-driven dance generation. … I would expect a paper introducing a new task to provide a more detailed discussion of its unique challenges and explain why existing approaches to the two individual tasks are insufficient."

"Weakness 3: … it remains unclear whether large-amplitude body movements negatively affect lip synchronization performance. … an analysis of lip synchronization under different levels of body motion intensity would further strengthen the experimental evaluation."

Please refer to Sec. 4.4 (Lines 463–465, Page 9), Appendix G (Lines 1106–1180, Pages 21–22), and Appendix J (Lines 1269–1295, Page 24) of the revised manuscript.

We thank the reviewer for these suggestions, which help us present the task more completely. We add a discussion of its distinct challenges and a controlled analysis of articulation, including the effect of body motion, as Appendix G, and cite them in Sec. 4.4. The cascade comparison below is in Appendix J.

A practical task. Short-form dance videos include both dancing-only and singing-and-dancing, and for the same song a creator may want either. A model for this setting therefore has to generate song-conditioned dance and, in the same video, control whether the dancer sings.

Existing work covers the two parts separately. Interactive avatar systems control speaking and listening in conversation (e.g., LPM 1.0, Zeng et al., 2026; StreamAvatar, Sun et al., 2026), and music-conditioned dance methods generate body motion without vocal articulation (e.g., X-Dancer and MusicInfuser). Neither line addresses singing-and-dancing, where one song drives the dance and, depending on the role, the articulation.

Combining them after generation is not sufficient. A natural alternative generates the dance first and then edits the lips with a lip-sync model. We apply two recent lip-sync models with representative designs, OmniSync (Peng et al., 2025b), which edits the whole frame, and LatentSync (Li et al., 2024), which inpaints a face crop, to our own Listener videos, so that body motion before editing is identical. End-to-end generation compares favorably in three respects:

  • Lip synchronization. On their common valid cases, it obtains the highest LSE-C (5.01, vs. 4.79 for LatentSync and 3.46 for OmniSync) and an LSE-D comparable to LatentSync (8.93 vs. 8.95).
  • Visual quality. It obtains the highest reference-fidelity and visual-quality scores, whereas OmniSync's whole-frame editing lowers them the most (e.g., Subject 0.9646 → 0.9382, Background 0.9757 → 0.9502).
  • Robustness. LatentSync's face-crop inpainting depends on face detection and could not process 2 videos because the dancer's arm briefly covers the face, which can be common in dance; end-to-end generation needs no such step.
AuthorsOpenReview ↗

Response to Reviewer SssY (Part 2/5)

(1) Distinct challenges and the factors affecting articulation (continued)

Supplementary controlled analysis. We agree that a new task calls for a closer analysis of what makes it difficult. Following the suggestion in Weakness 3, we change one factor at a time, including the amount of body motion, and measure lip synchronization.

Lip synchronization from talking to singing-and-dancing, changing one factor at a time.

RowSettingLSE-C ↑LSE-D ↓Upper-body / head speed
AEMTD references, speech8.117.480.21 / 0.08
BA, speech replaced by songs6.088.710.19 / 0.08
CB, singing-and-dancing configuration5.748.900.31 / 0.13
DSingDance-Test references, standing still5.139.090.18 / 0.11
ED, gentle swaying4.878.920.39 / 0.26
FSingDance-Source (D, original dance prompt)†4.948.940.76 / 0.24

Notes. Source role. This comparison uses its own common valid subset: all values are averaged over the 48 common valid cases, i.e., the cases with valid SyncNet tracks under every setting. † Same setting as SingDance-Source in Table 3; its values differ slightly from Table 3 because of the different common valid subset. A–C use a low-motion subset of 50 EMTD references, so that the talking rows have motion comparable to standing still (A therefore differs from SingDance in Table 4); A and B follow the settings of Table 4, and in B each reference is paired with one SingDance-Test song. C adds the routed music token and the three-pass guidance used for singing-and-dancing. D–F use the SingDance-Test references and songs with the settings of Table 2 and differ only in the action field. Speeds are mean DWPose speeds in shoulder widths per second. Face height is 224–230 px in A–C and 190–198 px in D–F.

  • Singing is harder to articulate than speech. From A to B, motion is essentially unchanged, yet LSE-C drops from 8.11 to 6.08, the largest single change in the table. Wav2Vec 2.0 features of the speech and song audio in A and B also form clearly separated clusters (Appendix G, Figure 6), consistent with this drop.
  • Full-body framing lowers synchronization further. From C to D, LSE-C drops to 5.13 although motion decreases.
  • Body motion also lowers synchronization. Both moving conditions (E, F) have lower LSE-C than standing still (D), while LSE-D changes little.

In summary, singing-and-dancing is a practical task that existing methods cover only in parts, and its articulation is affected by the singing audio, the full-body framing, and the accompanying body motion. SingDance provides a feasible solution, and the analyses above characterize where the difficulty lies.

AuthorsOpenReview ↗

Response to Reviewer SssY (Part 3/5)

(2) Positioning of the contribution (Response to Weakness 2)

"Weakness 2: Most of the technical components employed in SingDance are relatively standard. … The main contribution therefore appears to lie in the task formulation and the integration of existing techniques, rather than a substantial methodological advance."

SingDance deliberately builds on a pretrained video model and established components, and its contribution lies at the level of the task, its formulation, and the resulting system:

  • A practical task that existing methods cover only in parts. The same song should support either dancing-only or singing-and-dancing, which requires song-conditioned dance and controllable articulation in one video. To our knowledge, existing music-conditioned dance methods do not control whether the dancer articulates the vocals. Other reviewers describe this as "a useful control problem" (Reviewer jbib) and the unified handling of four tasks as "novel and practically appealing" (Reviewer xsUx).
  • A formulation that makes the task learnable without paired data. Articulation is defined as a vocal role, so the source and listener states learned from speech transfer to songs: the speech-stage checkpoint, trained before any song data is introduced, already reaches LSE-C 5.19 as Source and 1.14 as Listener on SingDance-Test. The conditions are organized so that the held-out Song/Source configuration differs from a trained configuration only in this role, which, as Reviewer jbib notes, "makes the proposed composition concrete".
  • A system built with appropriate design and training choices. We adapt Wan2.2-TI2V-5B with separate speech and music pathways and role conditions, and train it in two stages on curated speech and dance data (Appendix C); as the reviewer notes, the staged training is "straightforward and well motivated". The same checkpoint reaches competitive lip synchronization on EMTD with roughly one-third of InfiniteTalk's generation-time parameters (Table 4).
  • A measurable benefit of end-to-end composition. With the same inputs, end-to-end generation achieves higher LSE-C and visual quality than applying either of two recent lip-sync models afterward (see (1)), and a single model covers speaking, listening, dancing-only, and singing-and-dancing.

(3) Fine-tuning with a small amount of paired data (Response to Question 1)

"Question 1: … Have the authors considered how the proposed model would perform if a small amount of paired singing-and-dancing data were introduced during fine-tuning? … what would it imply about the benefits and limitations of the proposed compositional training strategy?"

We have not fine-tuned with paired data, because this work tests whether the target behavior can be obtained compositionally without such supervision; adding it would remove the held-out setting that our evaluation relies on. We expect a small amount of in-domain paired data to further improve singing articulation, and the compositional model is a natural initialization for such fine-tuning. The compositional strategy is most useful when paired data are scarce, and paired supervision is complementary to it.

Reviewer_xsUxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Task Description

  • This paper addresses singing-and-dancing video generation, where the goal is to generate a video from a reference image, a text prompt, and a song, such that the subject dances in sync with the music while also lip-syncing the vocals. The same model should also control whether the visible subject articulates the vocals, covering speaking, listening, dancing-only, and singing-and-dancing.

Research Motivation

  • Existing music-driven dance video methods model only body motion and ignore vocal articulation, while speech-driven video avatar models assume that the input voice is always produced by the visible subject. In addition, paired singing-and-dancing training data is scarce. This paper is motivated by the idea that treating vocal articulation as a semantic role, i.e., the subject either produces the vocals (source) or listens to them from off-screen (listener), allows articulation learned from speech and dance learned from songs to be composed without any singing-and-dancing data.

Main Contributions

  • The paper introduces SingDance, a unified video diffusion framework built on Wan2.2-TI2V-5B. It specifies the vocal role through both a text prompt field and learned role embeddings. It uses (i) task-defined hard-compact routing that selects speech, music, and role tokens for each task, and (ii) frame-wise joint audio injection into the DiT. Through staged training with asymmetric supervision (speaking/listening videos, then dancing-only videos), the held-out singing-and-dancing configuration is achieved at inference in a compositional zero-shot manner by switching only the vocal role.
Critical strengths
  • S1. (Presentation) The paper is clearly written. The problem scope is well defined relative to prior work, and the proposed method is clearly distinguished from existing approaches, making its technical contributions easy to identify.
  • S2. (Informativeness) Handling four tasks (speaking, listening, dancing-only, and singing-and-dancing) within a single unified model is novel and practically appealing.
  • S3. (Informativeness) By changing only the role conditioning (learned global role embedding and local role token) through the routing module, the method enables singing-and-dancing generation without any singing-and-dancing training data. The design is technically simple yet effective.
Critical weaknesses
  • W1. (Soundness) The evidence for the claimed compositional zero-shot generalization is slightly weak. The method relies on combining lip articulation learned from speech with song-conditioned dance, yet there is no analysis of why this transfers to singing articulation. For example, the paper does not examine how Wav2Vec representations of singing vocals differ from those of ordinary speech.
  • W2. (Soundness) The comparative baselines are insufficient. For singing-and-dancing, the only comparison is Wan-S2V. A natural cascade baseline, which generates a dance video first and then applies a separate lip-sync model, is missing. For instance, a dance video generation model such as MusicInfuser or X-Dancer could be combined with a recent lip-sync model such as OmniSync [R1].
  • W3. (Soundness) The small scale of the experiments limits the assessment of generalization and reproducibility. The SingDance-Test and Dance-Only-Test sets contain only 50 and 100 cases, respectively, and the user study involves only 15 participants. Given that a substantial portion of the training data is proprietary, it is difficult to judge the practical value, general applicability, and reproducibility of the proposed method.

References

  • [R1] Peng, Ziqiao, et al. "Omnisync: Towards universal lip synchronization via diffusion transformers." NeurIPS, 2025.
Questions for authors

Q1. (Related to W1) What evidence supports the claim that articulation learned only from speech transfers to singing?

  • Source-role LSE-C on SingDance-Test (4.97) is much lower than on EMTD talking (7.995). Is this gap mainly caused by the speech–singing domain shift in the Wav2Vec features?
  • An analysis of how Wav2Vec representations of singing vocals differ from those of speech would also strengthen the claim.

Q2. (Related to W2) Can the authors provide results against stronger baselines on SingDance-Test?

  • A cascade baseline can be SingDance in Listener mode (and X-Dancer / MusicInfuser), followed by several existing lip-sync models such as OmniSync [R1]. Using SingDance-Listener as the first stage could keep body motion identical, so this comparison would directly isolate the benefit of end-to-end composition.
  • Please report both LSE-C/D and BeatAlign/MBCR, so that any motion degradation caused by post-hoc lip editing is visible.
  • Section 4.4 argues that articulation and motion "must appear in the same video rather than be combined after generation", but this claim is not tested empirically.
6 comments
AuthorsOpenReview ↗

Response to Reviewer xsUx (Part 1/4)

Dear Reviewer xsUx,

Thank you for the careful review, and for recognizing the clarity of the paper, the unified handling of four tasks in a single model, and the simplicity of the design. Your concerns center on why articulation learned from speech transfers to singing, how SingDance compares with cascade baselines, and the scale and reproducibility of the evaluation. We address them below with clarifications and supplementary experiments. All revisions are marked in blue in the updated PDF.

(1) Transfer of articulation from speech to singing (Response to Weakness 1 & Question 1)

"Weakness 1: The evidence for the claimed compositional zero-shot generalization is slightly weak. … there is no analysis of why this transfers to singing articulation. For example, the paper does not examine how Wav2Vec representations of singing vocals differ from those of ordinary speech."

"Question 1: What evidence supports the claim that articulation learned only from speech transfers to singing? Source-role LSE-C on SingDance-Test (4.97) is much lower than on EMTD talking (7.995). Is this gap mainly caused by the speech–singing domain shift in the Wav2Vec features? An analysis of how Wav2Vec representations of singing vocals differ from those of speech would also strengthen the claim."

Please refer to Lines 463–465 on Page 9 (Sec. 4.4), Appendix G (Lines 1106–1180, Pages 21–22), Appendix I (Lines 1223–1239, Page 23), and Appendix C.1 (Lines 790–791, Page 15) of the revised manuscript.

Evidence of transfer.

  • Before any song training. The speech-stage checkpoint is trained only on speech data, curated as talking and conversational videos without singing clips (Appendix C.1). With the same reference images, songs, prompts, seeds, and sampling settings, it already reaches LSE-C 5.19 and LSE-D 8.86 on SingDance-Test, better than Wan-S2V (4.28 and 9.45) on the same inputs.
  • Paired role switching. With all other inputs fixed, Source mode yields LSE-C 4.97 and Listener mode 1.18 (Table 3), and switching from Source to Listener lowers LSE-C and raises LSE-D in every case. In a supplementary blind five-rater study, 98% of the Source videos were judged to lip-sync to the vocals throughout. The articulation therefore follows the song vocals.

Origin of the gap to EMTD (first point of Question 1). We decompose the gap by changing one factor at a time.

Controlled decomposition of the lip-synchronization gap between EMTD and SingDance-Test (Source role). Each row changes one factor relative to the previous row. This comparison uses its own common valid subset: all values are averaged over the 48 common valid cases, i.e., the cases with valid SyncNet tracks under every setting. The EMTD rows use a low-motion subset of 50 EMTD references, so that the talking rows have motion comparable to standing still (the speech row therefore differs from SingDance in Table 4); in the song rows, each reference is paired with one SingDance-Test song. The first two rows follow the settings of Table 4; the singing-and-dancing configuration adds the routed music token (S + R_src + M) and uses three-pass guidance (s_t = 5, s_a = 4). The SingDance-Test rows use the settings of Table 2, and only the action field of the prompt differs among them. † Same setting as SingDance-Source in Table 3; its values differ slightly from Table 3 because of the different common valid subset. Speeds are DWPose landmark speeds in shoulder widths per second (upper body: shoulders, elbows, wrists; head: nose, eyes, ears). Face height is the median SyncNet face-box height.

SettingChanged factorLSE-C ↑LSE-D ↓Upper-body speedHead speedFace height (px)
EMTD references, speech—8.117.480.210.08230
EMTD references, songsaudio6.088.710.190.08228
+ singing-and-dancing configurationrouted music token and guidance5.748.900.310.13224
SingDance-Test references, standing stillreference images and framing5.139.090.180.11198
SingDance-Test references, gentle swayingbody motion4.878.920.390.26191
SingDance-Source†body motion4.948.940.760.24190
AuthorsOpenReview ↗

Response to Reviewer xsUx (Part 2/4)

(1) Transfer of articulation from speech to singing (continued)

  • Audio. Replacing speech with songs on the same EMTD references lowers LSE-C from 8.11 to 6.08 and raises LSE-D from 7.48 to 8.71, while motion is essentially unchanged (upper-body speed 0.21 vs. 0.19).
  • Inference configuration. Adding the routed music token and three-pass guidance lowers LSE-C to 5.74 and also increases body motion (upper-body speed 0.19 → 0.31).
  • Framing. Moving to the full-body SingDance-Test references with the subject standing still lowers LSE-C to 5.13, although motion decreases in this step.
  • Body motion. Relative to standing still, both moving conditions lower LSE-C (4.87 and 4.94 vs. 5.13).

The audio change accounts for the largest share of the gap: 64% of the LSE-C drop and 84% of the LSE-D increase. This is consistent with the reviewer's hypothesis of a speech–singing domain shift. Full-body framing and body motion lower synchronization further.

Wav2Vec representations of speech and songs (second point of Question 1). Using the speech and song audio of the first two rows above, we extract the Wav2Vec 2.0 features with the layer weights learned by our speech pathway. Clip-level features of the two domains form clearly separated clusters (Appendix G, Figure 6), and a linear classifier separates them in cross-validation over held-out source groups (ROC-AUC 1.00). In the fused features, song clips also change more slowly from frame to frame than speech clips (mean adjacent-frame cosine distance 0.249 vs. 0.274). This representation difference is consistent with the drop in the audio step, where only the audio changes. Despite it, the speech-stage checkpoint, trained without singing clips, already reaches LSE-C 5.19 on the songs, so the learned articulation transfers across a clear shift in the input representation.

To summarize, the gap to EMTD mainly reflects the change of audio domain. Source-role articulation learned from speech already transfers to songs before any song training, as the speech-stage result shows, and in the final model the paired role switching shows that articulation follows the song vocals.

AuthorsOpenReview ↗

Response to Reviewer xsUx (Part 3/4)

(2) Cascade baselines (Response to Weakness 2 & Question 2)

"Weakness 2: … A natural cascade baseline, which generates a dance video first and then applies a separate lip-sync model, is missing. …"

"Question 2: Can the authors provide results against stronger baselines on SingDance-Test? A cascade baseline can be SingDance in Listener mode (and X-Dancer / MusicInfuser), followed by several existing lip-sync models such as OmniSync. … Please report both LSE-C/D and BeatAlign/MBCR, so that any motion degradation caused by post-hoc lip editing is visible. Section 4.4 argues that articulation and motion "must appear in the same video rather than be combined after generation", but this claim is not tested empirically."

Please refer to Lines 460–462 on Page 9 (Sec. 4.4) and Appendix J (Lines 1269–1295, Page 24) of the revised manuscript.

We thank the reviewer for suggesting this cascade baseline, which shows the advantage of end-to-end composition more clearly. We add the comparison as Appendix J and cite it in Sec. 4.4.

Following the reviewer's design, we use our SingDance-Listener outputs as the first stage, so that body motion before lip editing is identical, and apply two recent lip-sync models with representative designs: OmniSync (Peng et al., 2025b), which the reviewer suggested, edits the whole frame generatively while concentrating the changes around the mouth, and LatentSync (Li et al., 2024) inpaints a masked face crop and blends it back into the original frame. OmniSync's code and weights are not publicly released, so we run it through the official online service linked from the OmniSync paper; for LatentSync, we use its official open-source implementation. We re-run pose detection on the edited videos.

Cascade vs. end-to-end generation on SingDance-Test. This comparison uses its own common valid subset: LSE-C/LSE-D and BeatAlign/MBCR are averaged over the 48 common valid cases, i.e., the cases that all four settings can be evaluated on, so the SingDance values differ slightly from Table 3; visual-quality metrics use all generated videos of each setting.

MethodLSE-C ↑LSE-D ↓BeatAlign ↑MBCR ↑Subject ↑Background ↑Aesthetic ↑Imaging ↑Smoothness ↑
SingDance-Listener (first stage)1.1612.460.27130.27350.96460.97570.58110.66680.9891
+ OmniSync3.4610.160.26380.26730.93820.95020.56900.66090.9876
+ LatentSync4.798.950.26810.26920.96300.97400.57520.66130.9895
SingDance-Source (end-to-end)5.018.930.27170.25470.96820.97720.58460.66880.9899

End-to-end generation obtains the highest LSE-C (5.01, vs. 4.79 for LatentSync and 3.46 for OmniSync) and an LSE-D comparable to LatentSync (8.93 vs. 8.95). Relative to their first-stage input, both cascades slightly lower BeatAlign and MBCR, which is the motion degradation from post-hoc editing that the reviewer asked about. Each design also has its own cost:

  • Whole-frame editing (OmniSync) lowers reference fidelity and visual quality the most (e.g., Subject 0.9646 → 0.9382, Background 0.9757 → 0.9502).
  • Face-crop inpainting (LatentSync) keeps visual scores close to its input, but it depends on face detection and could not process 2 videos: in both, the dancer's arm briefly covers the face, so no valid face is detected in those frames. Such occlusions can be common in dance videos. These 2 videos are also why this table uses its own common valid subset rather than that of Table 3.

We use SingDance-Listener rather than X-Dancer or MusicInfuser as the first stage because it keeps body motion identical, which isolates the effect of end-to-end composition as the reviewer suggests.

In summary, both representative lip-sync designs synchronize less well than end-to-end generation, and each adds a cost of post-hoc editing. Generating articulation together with motion and appearance avoids these costs and gives the highest reference-fidelity and visual-quality scores.

We thank the reviewer again for this suggestion, which makes the evaluation of our work more complete.

Reviewer_Yc4FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2(−1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SingDance, a unified video generation framework that combines music-conditioned body motion and vocal articulation through explicit vocal-role conditioning. By learning speaking/listening behaviors and music-conditioned dancing from separate training configurations, the model enables compositional zero-shot singing-and-dancing generation without direct supervision for the target task.

Critical strengths
  1. The paper presents an interesting formulation of singing-and-dancing generation as a compositional zero-shot task, combining vocal articulation and music-conditioned body motion without direct supervision for their joint generation.

  2. The vocal-role intervention experiments provide convincing evidence that lip synchronization can be controlled while largely preserving music-aligned body motion.

  3. The proposed framework achieves competitive generation quality and lip synchronization with relatively low inference-time computational requirements.

Critical weaknesses
  1. The ablation studies do not sufficiently isolate the contributions of key design choices, such as hard-compact routing and staged training, making it difficult to determine which components are essential for zero-shot composition.

  2. The compositional generalization claim is primarily validated on a single held-out task configuration, leaving its applicability to more diverse unseen combinations unclear.

  3. The quantitative comparison for singing-and-dancing generation is limited to Wan-S2V, making it difficult to establish the method's advantages over other relevant approaches.

  4. The motion–beat alignment metrics do not directly assess choreography quality or motion diversity, limiting the evaluation of the generated dance movements.

Questions for authors
  1. The proposed framework uses a shared speech pathway for both source and listener roles. What motivates this choice over separate role-specific pathways, and how does sharing the representation facilitate compositional generalization?

  2. The model retains speech features during song-based dancing-only generation, even when the visible subject does not articulate the vocals. What specific information do these features contribute to body motion beyond the music representation?

  3. Appendix Table 6 suggests that vocal articulation is primarily controlled by the learned role condition rather than the textual role field. What is the intended contribution of the textual role field, and under what circumstances does it meaningfully affect generation?

  4. The model uses a fixed task-defined routing strategy rather than dynamically selecting relevant audio conditions. How would this framework handle ambiguous audio inputs, such as songs containing spoken dialogue or multiple vocal sources?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Yc4F (Part 1/4)

Dear Reviewer Yc4F,

Thank you for the constructive review, and for recognizing the compositional zero-shot formulation, the vocal-role intervention as convincing evidence of controllable lip synchronization, and the competitive quality at low inference cost. Your concerns center on what each design choice contributes, the scope of the composition, the baselines for singing-and-dancing, and the evaluation of choreography, together with four questions on specific design choices. We address them below with clarifications and supplementary experiments. All revisions are marked in blue in the updated PDF.

(1) Contributions of the design choices (Response to Weakness 1)

"Weakness 1: The ablation studies do not sufficiently isolate the contributions of key design choices, such as hard-compact routing and staged training, making it difficult to determine which components are essential for zero-shot composition."

Please refer to Line 252 on Page 5 (Sec. 3.3), Lines 484–488 on Pages 9–10 (Sec. 4.5), and Appendix I (Lines 1217–1266, Pages 23–24) of the revised manuscript.

We thank the reviewer for this question. Our presentation may have given the impression that hard-compact routing and staged training are each required for the zero-shot composition, which we do not intend to claim. The composition comes from how complementary supervision and shared conditions are organized: the held-out Song/Source configuration differs from the trained Song/Listener configuration only in the role state, so it can reuse capabilities learned in other configurations. The following analyses on SingDance-Test, with inputs and sampling settings fixed, show what each part contributes:

PartComparisonFinding
Speech stagespeech-stage checkpoint, Sourcesource-role articulation of songs is already present before dance training (LSE-C 5.19, vs. 4.28 for Wan-S2V)
Speech stagespeech-stage checkpoint, Source vs. Listener (table below)role control is already present before dance training (LSE-C 5.19 vs. 1.14)
Dance stagespeech-stage checkpoint vs. w/o M, same S + R_src conditionsadds beat coverage (MBCR 0.2006 → 0.2386) while LSE-C changes by 1%
Music tokenw/o M vs. full model (Table 5)improves beat timing (BeatAlign 0.2462 → 0.2723)
Speech condition in song danceListener with vs. without S (see (6))mainly increases beat coverage (MBCR 0.2449 → 0.2787) at similar beat timing
Learned role conditionTable 6articulation follows the learned role condition; switching only the textual field changes LSE-C little (4.97 vs. 4.98)

Role switching before and after dance-stage training on SingDance-Test (same format as Table 6; same cases as Table 3).

CheckpointRoleRouted tokensLSE-CLSE-D
Speech stageSourceS + R_src5.198.86
Speech stageListenerS + R_lis1.1413.04
FinalSourceS + R_src + M4.978.94
FinalListenerS + R_lis + M1.1812.41

The learned role condition is therefore the component that controls articulation. Both source-role articulation of songs and role control are already learned in the speech stage, while the dance stage and the music token add music-aligned motion. The staged order reflects the asymmetric supervision available to the two capabilities (Sec. 3.3), and we have retitled Sec. 3.3 as "Staged Training and Held-Out Composition" so that it does not suggest that staging is required for composition. Hard-compact routing specifies which conditions each task uses, so that every training configuration and the held-out inference configuration are explicit; we present it as the organization that makes the composition well defined rather than as its cause.

AuthorsOpenReview ↗

Response to Reviewer Yc4F (Part 2/4)

(2) Scope of the composition and ambiguous audio (Response to Weakness 2 & Question 4)

"Weakness 2: The compositional generalization claim is primarily validated on a single held-out task configuration, leaving its applicability to more diverse unseen combinations unclear."

"Question 4: … How would this framework handle ambiguous audio inputs, such as songs containing spoken dialogue or multiple vocal sources?"

Please refer to the caption of Table 1 (Line 218, Page 5) and the Limitations paragraph (Lines 500–504, Page 10) of the revised manuscript.

A single held-out configuration. Our description of the task space may not have made it clear that Song/Source is the only unseen combination, and we now state this in the caption of Table 1. The vocal role is defined only when vocals are present: speech and songs each admit Source and Listener, while instrumental music admits only the non-source state. Of the resulting five configurations (Table 1), four are used for training, so the held-out configuration follows from the structure of the task space rather than being selected from several candidates.

Ambiguous audio. Because the route is specified by the task rather than inferred from the audio, ambiguous audio does not change which conditions are used: a song with spoken dialogue still follows the song route, and the role assigns the entire vocal signal to the visible subject (Source) or to an off-screen performer (Listener). The current formulation does not handle audio in which only part of the vocals belongs to the visible subject, such as a duet or a song interleaved with another speaker. We state this in a Limitations paragraph, together with broader compositional settings such as additional audio types and multiple vocal sources. Extending the role from a per-clip state to a time-varying assignment, similar to the speaking/listening masks used in interactive avatar systems, is a natural direction.

(3) Baselines for singing-and-dancing (Response to Weakness 3)

"Weakness 3: The quantitative comparison for singing-and-dancing generation is limited to Wan-S2V, making it difficult to establish the method's advantages over other relevant approaches."

Please refer to Lines 460–462 on Page 9 (Sec. 4.4) and Appendix J (Lines 1269–1295, Page 24) of the revised manuscript.

Beyond Wan-S2V, a natural alternative to end-to-end generation is a cascade that generates the dance first and then edits the lips with a lip-sync model. To broaden the comparison, we add cascade baselines (Appendix J, Table 18). We apply two recent lip-sync models with representative designs, OmniSync (Peng et al., 2025b), which edits the whole frame, and LatentSync (Li et al., 2024), which inpaints a face crop, to our own Listener outputs, so that body motion before editing is identical. End-to-end generation compares favorably in three respects:

  • Lip synchronization. On their common valid cases, it obtains the highest LSE-C (5.01, vs. 4.79 for LatentSync and 3.46 for OmniSync) and an LSE-D comparable to LatentSync (8.93 vs. 8.95).
  • Visual quality. It obtains the highest reference-fidelity and visual-quality scores, whereas OmniSync's whole-frame editing lowers them the most (e.g., Subject 0.9646 → 0.9382, Background 0.9757 → 0.9502).
  • Robustness. LatentSync's face-crop inpainting depends on face detection and could not process 2 videos because the dancer's arm briefly covers the face, which can be common in dance; end-to-end generation needs no such step.

MusicInfuser and X-Dancer generate dance without vocal articulation, so we compare with them on dancing-only (Table 2).

AuthorsOpenReview ↗

Response to Reviewer Yc4F (Part 3/4)

(4) Choreography quality (Response to Weakness 4)

"Weakness 4: The motion–beat alignment metrics do not directly assess choreography quality or motion diversity, limiting the evaluation of the generated dance movements."

We agree that beat-alignment metrics measure the timing of motion relative to the music rather than choreography quality or motion diversity. A reliable assessment of choreography would require expert raters such as professional dancers or choreographers, which is beyond the scope of our current study. For the automatic evaluation, we follow common practice in dance video generation: BeatAlign is the beat-alignment score of AIST++ (Li et al., 2021), which recent dance video work such as X-Dancer, MACE-Dance (Yang et al., 2025a), and OmniDance (Yang et al., 2026a) also reports, and MBCR follows the coverage direction of TMD-Bench (Yang et al., 2026b). To reflect perceived rhythm beyond these scores, we also run a blind human preference study on motion–music rhythm, where SingDance is preferred over Wan-S2V in 61.33% (singing-and-dancing) and 68.00% (dancing-only) of 300 votes each (Table 2). The audio-enabled videos on the demo page linked in the abstract also allow choreography and motion variety to be inspected directly.

(5) Shared speech pathway (Response to Question 1)

"Question 1: The proposed framework uses a shared speech pathway for both source and listener roles. What motivates this choice over separate role-specific pathways, and how does sharing the representation facilitate compositional generalization?"

Sharing the speech pathway is a deliberate minimal design rather than a claim of optimality, motivated by three considerations. First, it reflects how vocal roles are distributed in our data. About 12K of roughly 300K speech clips are Listener, and every song clip that uses the speech pathway is Song/Listener, since Song/Source is held out. With role-specific encoders, the Source encoder would be trained only on speech and would first encounter singing vocals at inference, whereas the shared encoder is trained on both speech and song vocals. Second, because Song/Listener is trained with this same pathway, the held-out Song/Source configuration differs from a trained configuration only in the vocal-role state. This also keeps the paired interventions in Table 3 and Table 6 single-variable changes; with role-specific pathways, switching the role would also switch the encoder. Third, role-specific pathways come with their own design decisions. StreamAvatar (Sun et al., 2026) injects Wav2Vec features through separate Talk and Listen audio-attention modules gated by TalkNet-derived speaking/listening masks, and explicitly balances speaking and listening clips in its training set. LPM 1.0 (Zeng et al., 2026) adds about 3B parameters of interleaved speak/listen audio cross-attention and trains on tens of millions of clips with separated speaking and listening audio streams. Both target full-duplex conversation, where speaking and listening alternate within a clip; in our setting, a single vocal signal is assigned to one role per clip, and our role distribution is far more imbalanced. Since the shared design already achieves the held-out composition (Tables 2 and 3), we did not pursue role-specific alternatives here. We thank the reviewer for this constructive suggestion and will explore such pathways in future work.

Reviewer_jbibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SingDance generates human videos from a reference image, text, and audio, with an explicit Source/Listener condition controlling whether the visible subject articulates the vocals. The model combines speech and music representations through task-dependent routing and frame-wise audio injection. Training first establishes speaking and listening behavior, then introduces music-conditioned dancing while retaining speech examples. Song/Source is held out from task training and used at inference to generate singing-and-dancing videos. The paper evaluates dance generation and speech-driven animation, with additional experiments on paired role switching, music conditioning, and the respective effects of textual and learned role conditions.

Critical strengths
  1. The paper addresses a useful control problem: the same song should support either dancing alone or singing-and-dancing, depending on whether the visible person is assigned the vocals. Retaining the vocal features in both cases is a sensible choice, since they may still inform body motion. The training configuration makes the proposed composition concrete by withholding Song/Source while learning articulation and song-conditioned dance separately. The authors also appropriately limit the zero-shot claim to their task training.

  2. The paired role intervention provides meaningful evidence of control. In Table 3, switching roles reduces SingDance’s LSE-C from 4.9672 to 1.1768, while BeatAlign remains close at 0.2723 and 0.2704. The inputs, checkpoint, seed, and other inference settings are held fixed. Appendix B further separates textual and learned role conditions, showing that the learned condition accounts for most of the synchronization change. These experiments support a useful capability beyond the aggregate comparison with Wan-S2V.

Critical weaknesses
  1. The contribution of the training and conditioning design remains insufficiently established. The held-out composition is interesting, but the current comparisons leave its source unclear. Wan-S2V is a useful system baseline, although differences in training data and role supervision remain. Table 6 establishes which conditions control the final model’s output, while Table 5 establishes a benefit from music conditioning. Neither shows how much of the target behavior was already available after speech-stage training. It is also unclear whether w/o M removes music tokens at inference or denotes a separately trained model. A comparison with the existing speech-stage checkpoint would help quantify what dance training adds, with articulation and dance alignment evaluated together. Given that much of the audio-conditioning machinery builds on prior work, this distinction matters to my assessment of the contribution.

  2. The evaluation does not yet establish the reliability of the combined task and role switching. The SyncNet changes are substantial, but lower synchronization in Listener mode does not establish that the output avoids inappropriate mouth movements; unrelated mouth motion or a frozen mouth can also score poorly on synchronization. The human study evaluates rhythm and overall visual quality, without directly assessing articulation or role correctness. Tables 2 and 3 use the same Source outputs, which is appropriate, but separate averages still leave unclear how often convincing articulation and dance occur together. SingDance-Test contains only 50 cases, and the effective sample counts after face-track exclusions are not reported. It is also unclear whether Source and Listener LSE scores use the same valid paired subset. These omissions make it difficult to judge how consistently the reported control works.

Questions for authors
  1. Evaluation clarification and supporting evidence: Are the Source and Listener results in Table 3 computed on the same face-valid cases? Please report the valid counts and failure rates. Could the existing paired outputs be assessed directly for song articulation, inappropriate mouth motion in Listener mode, and successful singing-and-dancing within the same video? Per-case results and paired uncertainty estimates would help address the reliability concern.

  2. Training comparison: How is w/o M implemented in Table 5? If the speech-stage checkpoint is available, how does it perform on SingDance-Test using the same reference images, songs, prompts, and sampling settings? Reporting both lip synchronization and motion–beat alignment would clarify the gain from the dance stage, although this comparison alone would not establish an advantage of the particular conditioning architecture.

  3. Data-audit clarification: Appendix C describes substantial filtering, but the coverage of the synchronization and caption audits over the final approximately 30K Song/Listener clips is unclear. Was the final training set independently checked for residual visible singing or lip synchronization? Please provide the audit sample size and observed errors, and clarify whether the Speech/Source subset received corresponding checks. This question concerns the exclusion of the target configuration from task training, not the backbone’s pretraining data.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jbib (Part 1/3)

Dear Reviewer jbib,

Thank you for the detailed review, and for recognizing the usefulness of the control problem and the paired role intervention as meaningful evidence of control. Your concerns center on how much of the target behavior is already available after speech-stage training, how reliably the role switch and the combined task work, and whether the final training set was checked for residual visible singing. We address them below with clarifications and supplementary experiments. All revisions are marked in blue in the updated PDF.

(1) What the speech stage provides and what dance training adds (Response to Weakness 1 & Question 2)

"Weakness 1: … Neither shows how much of the target behavior was already available after speech-stage training. It is also unclear whether w/o M removes music tokens at inference or denotes a separately trained model. A comparison with the existing speech-stage checkpoint would help quantify what dance training adds, with articulation and dance alignment evaluated together. Given that much of the audio-conditioning machinery builds on prior work, this distinction matters to my assessment of the contribution."

"Question 2: How is w/o M implemented in Table 5? If the speech-stage checkpoint is available, how does it perform on SingDance-Test using the same reference images, songs, prompts, and sampling settings? …"

Please refer to Sec. 4.5 (Lines 484–488, Pages 9–10) and Appendix I (Lines 1217–1266, Pages 23–24) of the revised manuscript.

We thank the reviewer for suggesting this comparison, which shows directly what each training stage contributes. We add it as Appendix I and summarize it in Sec. 4.5.

Implementation of w/o M. Our original description did not make this clear, and Sec. 4.5 now states it: w/o M is an inference-time ablation. With the same final checkpoint, prompts, seeds, and sampling settings, we remove only the routed music token and keep all other conditions unchanged. For song inputs, this leaves S + R_src, the same token structure as the speech routes used in training.

Speech-stage checkpoint. We evaluate the speech-stage checkpoint on SingDance-Test with the same reference images, songs, prompts, seeds, and sampling settings. Because it has no music pathway, it uses the same S + R_src configuration as w/o M, so rows 1 and 2 isolate the effect of dance-stage training, and rows 2 and 3 isolate the effect of the music token.

Speech-stage checkpoint vs. final model on SingDance-Test (Source role). LSE-C/LSE-D use the common valid cases of all settings, and BeatAlign/MBCR use the same cases as Tables 2, 3, and 5, so the SingDance and Wan-S2V rows match Table 3.

SettingCheckpointRouted tokensLSE-C ↑LSE-D ↓BeatAlign ↑MBCR ↑
Speech-stagespeech stageS + R_src5.19288.85620.23720.2006
w/o MfinalS + R_src5.14618.89770.24620.2386
SingDancefinalS + R_src + M4.96728.93630.27230.2558
Wan-S2V (Table 3)——4.28099.44620.26400.1983
  • Already available after the speech stage. Source-role articulation of songs is already present (LSE-C 5.19, above Wan-S2V's 4.28), and so is role control: in Listener mode, the speech-stage checkpoint lowers LSE-C to 1.14 on the same inputs.
  • Added by dance training. Music-aligned dance is largely added by the dance stage. Relative to the speech-stage checkpoint, the final model raises BeatAlign by 15% (0.2372 → 0.2723) and MBCR by 28% (0.2006 → 0.2558). Dance-stage training mainly increases beat coverage (MBCR 0.2006 → 0.2386), and the music token mainly improves beat timing (BeatAlign 0.2462 → 0.2723).
  • Cost to articulation. LSE-C decreases by 4% (5.19 → 4.97) and remains above Wan-S2V. Dance-stage training also keeps speech articulation: on EMTD, LSE-C is 8.047 for the speech-stage checkpoint and 7.995 for the final model (LSE-D 7.446 vs. 7.551).

The singing-and-dancing outputs therefore combine articulation and role control obtained in the speech stage with music-aligned motion added in the dance stage.

Relation to prior audio conditioning. As the reviewer notes, the speech conditioning follows Wan-S2V (layer-weighted Wav2Vec 2.0 features, a temporal encoder, and AdaLN-modulated frame-wise cross-attention), and we do not claim these mechanisms as contributions. What we add is how the conditions are organized: a separate MuQ music pathway, a vocal-role condition supplied both as a global embedding and as a local token within the same frame-wise key–value sequence, and task-defined routing under which the held-out Song/Source configuration differs from a trained configuration only in the role state. We agree that the speech-stage comparison does not by itself establish an advantage of the particular conditioning architecture; as the reviewer suggests, its purpose is to quantify what dance training adds.

AuthorsOpenReview ↗

Response to Reviewer jbib (Part 2/3)

(2) Reliability of role switching and of the combined task (Response to Weakness 2 & Question 1)

"Weakness 2: … lower synchronization in Listener mode does not establish that the output avoids inappropriate mouth movements; unrelated mouth motion or a frozen mouth can also score poorly on synchronization. … separate averages still leave unclear how often convincing articulation and dance occur together. SingDance-Test contains only 50 cases, and the effective sample counts after face-track exclusions are not reported. It is also unclear whether Source and Listener LSE scores use the same valid paired subset."

"Question 1: Are the Source and Listener results in Table 3 computed on the same face-valid cases? Please report the valid counts and failure rates. Could the existing paired outputs be assessed directly for song articulation, inappropriate mouth motion in Listener mode, and successful singing-and-dancing within the same video? Per-case results and paired uncertainty estimates would help address the reliability concern."

Please refer to Sec. 4.1 (Lines 343–344 and 371–372, Page 7), the caption of Table 3 (Lines 422–426, Page 8), Sec. 4.3 (Lines 439–442, Page 9), Appendix D.1 and Table 11 (Lines 983–1020, Page 19), and Appendix H (Lines 1183–1215, Pages 22–23) of the revised manuscript.

We understand the concern about how reliably the averages reflect the behavior of the model, and we address it with per-case evidence below.

Same valid subset and failure rates. Yes. In Table 3, the Source and Listener rows are evaluated on the same cases, as its caption now states. For LSE-C/LSE-D, the failure rate is zero: every output of both methods has a valid SyncNet face track under both roles. For BeatAlign/MBCR, every SingDance output passes the pose check, while Wan-S2V passes 92% of the cases in the Source role; Table 3 therefore uses the common valid cases of Table 2, so SingDance's scores are identical across Tables 2, 3, and 5. VBench-based metrics use all generated videos. Table 11 reports the valid and failure counts of every method for each test set and metric family.

Per-case results for articulation. Switching SingDance from Source to Listener lowers LSE-C and raises LSE-D in every case.

Direct per-video assessment of articulation and dance. Following the reviewer's suggestion, we ran a five-rater human study on the existing SingDance outputs on SingDance-Test under both roles during the rebuttal period. Raters watched each video with its song, blind to the assigned role. For each video they answered whether the person lip-syncs to the vocals throughout and whether the person moves in time with the music. When the answer to the first question was no, they chose the mouth state that fit best: natural non-singing, occasional or ambiguous lip-sync, or a frozen face or mouth motion unrelated to the vocals. Each video is labeled by majority vote (at least three of five raters); question wording and aggregation rules are in Appendix H. Inter-rater agreement (Fleiss' κ) is 0.89 for lip-sync and 0.59 for moving in time with the music.

  • Source. 98% of the videos were judged to lip-sync throughout (95% Wilson interval 89.5–99.6%), and 80% to both lip-sync throughout and move in time with the music (67.0–88.8%), which is the joint success rate the reviewer asks about.
  • Listener. 4% of the videos were judged to lip-sync throughout (1.1–13.5%), and 20% when occasional or ambiguous lip-sync is also counted as leakage. Among the Listener ratings that judged the person as not lip-syncing throughout, 86% chose natural non-singing and 14% occasional or ambiguous lip-sync; no rating chose a frozen face or mouth motion unrelated to the vocals.
  • Per pair. The role switch succeeds (the Source video lip-syncs throughout and the Listener video does not) in 94% of pairs (83.8–97.9%). Listener videos move in time with the music as often as Source videos (86% vs. 82%), and both videos of a pair do so in 74% of pairs, so the role switch changes articulation without removing the dance.

The pair-level rates are computed per Source–Listener pair, and the intervals of all per-video and per-pair rates are in Appendix H (Table 14). The absence of frozen or unrelated mouth motion is consistent with how the listener role is supervised: by genuine listening and reacting clips toward an off-screen interlocutor rather than by a frozen-mouth target (Sec. 1).

AuthorsOpenReview ↗

Response to Reviewer jbib (Part 3/3)

(3) Data audits of the final training set (Response to Question 3)

"Question 3: … the coverage of the synchronization and caption audits over the final approximately 30K Song/Listener clips is unclear. Was the final training set independently checked for residual visible singing or lip synchronization? Please provide the audit sample size and observed errors, and clarify whether the Speech/Source subset received corresponding checks. …"

Exclusion of the target configuration. The zero count for Song/Source in Table 9 refers to the training conditions: no training clip is presented with song audio and the visible subject assigned as the vocal source. Every accepted dance clip with vocals is trained under the Listener role, so the target configuration is excluded from task training by how the routes are defined. Whether a Song/Listener clip still contains residual visible singing is a question of data quality, which the checks below address.

Synchronization and caption audits. A dance candidate is accepted only if it passes both the synchronization and caption audits, so every final Song/Listener clip has passed both. The counts reported for these audits in Appendix C.2 refer to the proprietary collection, whereas Table 7 merges it with MA-Data. Throughout curation, we also reviewed sampled clips at each stage and turned recurring failure patterns into additional exclusion rules.

Independent check of the final training set. We randomly sampled 200 clips from the final Song/Listener training set and manually checked whether the visible subject sings or lip-syncs to the vocals at any point. None of the 200 clips showed visible singing or lip-sync.

Speech/Source. The speech data are collected as talking and conversational videos, and singing clips are not included (Appendix C.1). In addition, the structured descriptions of all Speech/Source clips, from both the proprietary collection and HuMoSet, are screened for singing-related terms (e.g., sing, rap, karaoke, and a cappella), and any clip whose description matches is excluded from training.

Thank you again for the constructive review. We hope these clarifications and supplementary results address your concerns, and we would be grateful if you would take them into account in your final assessment.

AuthorsOpenReview ↗

Response to reviewer jbib

Dear Reviewer jbib,

Thank you again for suggesting the comparison with the speech-stage checkpoint. Measuring both lip synchronization and motion–beat alignment under matched inference conditions helped us quantify what dance-stage training adds and strengthened our evaluation.

Following your other suggestions, we conducted a per-case study to directly assess vocal articulation and role correctness on the paired outputs. We also independently checked a random sample of clips from the final Song/Listener training set for residual visible singing or lip-sync to the vocals.

We’ve incorporated these results into the revised PDF and will retain them in the final version. We hope they will be helpful in your final assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Dear Reviewers and ACs,

We thank all four reviewers for their constructive and detailed reviews. We are glad that the reviewers found singing-and-dancing a useful control problem with practical value (jbib, SssY), the compositional zero-shot formulation interesting (Yc4F, SssY), the single model for four tasks novel and practically appealing (xsUx), the paired role intervention meaningful evidence of control (jbib, Yc4F), and the generation quality and lip synchronization competitive at low inference cost (Yc4F).

We have uploaded a revised PDF with all changes in blue. The main changes are:

  • Speech-stage checkpoint (jbib, Yc4F). Before any dance training, it already lip-syncs songs as Source (LSE-C/D 5.19/8.86, vs. 4.28/9.45 for Wan-S2V) and suppresses lip-sync as Listener (1.14/13.04). Dance training and the music token add music-aligned motion: BeatAlign +15% and MBCR +28%, while LSE-C decreases by 4% (Sec. 4.5, App. I).
  • Speech features in song-based dance (Yc4F). Removing the speech conditions in Listener mode lowers MBCR by 12% at similar BeatAlign (App. I).
  • Cascade baselines (xsUx, Yc4F, SssY). We apply OmniSync and LatentSync to our Listener outputs. End-to-end generation has the highest LSE-C and an LSE-D comparable to LatentSync (5.01/8.93, vs. 4.79/8.95 for LatentSync and 3.46/10.16 for OmniSync), the highest reference-fidelity and visual-quality scores, and needs no face detection; LatentSync fails on 2 videos where an arm covers the face (Sec. 4.4, App. J).
  • Valid cases and per-case results (jbib, SssY, xsUx). Sec. 4.1 and the table captions now state which cases each metric uses, and Table 11 gives the valid and failure counts of every method; the protocol is unchanged. Source and Listener use the same cases, and switching to Listener lowers LSE-C and raises LSE-D in every case (App. D.1).
  • Per-video human evaluation (jbib, xsUx). Five blind raters judged 98% of Source videos to lip-sync throughout and 80% to also move in time with the music. The role switch succeeds in 94% of pairs, 4% of Listener videos lip-sync throughout, and no rating reports a frozen or unrelated mouth (Sec. 4.3, App. H).
  • Controlled analysis of articulation (SssY, xsUx). Going from EMTD talking to SingDance-Test one factor at a time, replacing speech with songs causes the largest drop (LSE-C/D 8.11/7.48 → 6.08/8.71); full-body framing and body motion lower synchronization further. Wav2Vec features of speech and songs form clearly separated clusters (Sec. 4.4, App. G).
  • Test prompts (SssY). App. F describes how the test prompts are built, with examples and the inputs of each baseline.
  • Scope of the composition (Yc4F). Sec. 3.3 and the caption of Table 1 clarify the held-out composition, and the conclusion now includes a Limitations and future work paragraph.
AuthorsOpenReview ↗

Thank You to the ICLR Reviewing Committee

We sincerely thank all four reviewers for the time and care they put into reviewing SingDance. During the discussion period, we worked to address their concerns as fully as possible and incorporated the resulting experiments, analyses, and clarifications into the revised manuscript. Taken together, the reviewers’ constructive suggestions substantially strengthened our evaluation and made the paper more complete. Here is a summary of their replies and score changes:

  • Reviewer xsUx wrote that our rebuttal had fully resolved their questions and concerns, encouraged us to retain the additional analyses and results in the final version, and maintained their positive rating of 3 (Weak acceptance).
  • Reviewer SssY raised their overall rating from 2 (Weak rejection) to 3 (Weak acceptance).

We are encouraged by this feedback and thank all the reviewers again for helping us improve the paper. We also extend our gratitude to the Area Chairs, Senior Area Chairs, and Program Chairs for the time and effort they have devoted to this year’s review process, especially given the substantial increase in submissions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.