- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies sign-language video dubbing: generating speech from a silent signing video and an available English translation, with vocal characteristics compatible with the signer's appearance and delivery matched to visible rhythm and emotion. SignDubber adapts the MMAudio architecture, using frozen SpeechT5 text representations and Synchformer video features to condition flow matching in an audio VAE latent space. The authors construct approximately 200,000 training triplets through separate LTX-2 video and audio generation branches, followed by IndexTTS2 emotion enhancement. Evaluation on real EmoSign videos reports improved temporal alignment and subjective ratings over several baselines, while content accuracy remains below that of the TTS baselines. CLIP/Synchformer ablations and a comparison with raw LTX-2 outputs provide additional evidence.
Critical strengths
- Clear formulation of an underexplored task.
Separating translated linguistic content from visually conditioned delivery provides a clear formulation for investigating how signing rhythm and expression can influence synthesized speech.
- Substantial data construction and encouraging real-video results.
The decoupled synthesis pipeline addresses several competing generation requirements, including emotional expression, speech-like lip-motion shortcuts, and audio duration. Evaluating on real EmoSign videos provides useful evidence of transfer beyond generated videos. The consistent improvements in subjective ratings and the controlled CLIP/Synchformer ablations support the usefulness of the chosen conditioning design within the reported setting.
Critical weaknesses
- The practical need and added value of the task requirements are insufficiently established.
The paper does not establish which intended users need detailed motion-speech synchronization or appearance-conditioned voices. With a supplied translation, conventional TTS already enables access to the content. The TTS baselines appropriately receive video-derived style and emotion prompts, but the comparison does not establish the additional benefit over such systems with simple duration matching. The study measures perceived quality rather than comprehension, communication of the signer's intent, or usefulness to signers and viewers. SignDubber-small has a WER of 5.90%, versus 1.59% for Qwen3-TTS; the practical value of the perceptual gains relative to this loss in content accuracy remains unclear. Since the architecture largely adapts existing components, demonstrating the application benefit is particularly important. Related work also omits co-speech gesture research such as BEAT (Liu et al., ECCV 2022; arXiv:2203.05297), which studies speech-motion relationships involving semantics and emotion. Although the generation direction and linguistic domain differ, explaining the challenges specific to sign-language dubbing would clarify the contribution.
- The data and evaluation do not adequately establish linguistic fidelity or the scope of the claimed alignment.
Training regenerates signing videos with altered facial expressions while retaining the original translations. Yet the paper notes that facial expression, head motion, and posture can carry linguistic information. Pose conditioning alone does not establish preservation of the source utterance or compatibility with all five imposed emotions. I could not find a linguistic assessment by fluent ASL signers. In evaluation, boundary error measures endpoints, while CRC-W1 compares normalized motion activity and audio energy; neither establishes that corresponding concepts are expressed at compatible times. Video-text-audio pairing may support implicit semantic alignment, but the model can also obtain content entirely from text and use video mainly for rhythm and affect. The 14-video, 12-participant MOS study does not resolve these questions, and participant sign-language proficiency is unspecified. The evidence supports improved perceived synchronization, but is insufficient to establish faithful transfer of sign-language expression and intent. The claims should reflect this distinction.
Questions for authors
-
Practical benefit and evaluation population. What is the deployment scenario and evidence of benefit beyond video-informed emotional TTS with duration matching? Did signers or intended users participate, what was their ASL proficiency, and how should the content-accuracy versus perceptual-quality trade-off be interpreted?
-
Linguistic validity of synthetic supervision. What checks establish that regenerated videos remain valid ASL matching their translations after expression changes? Please report any existing fluent-signer assessment or other evidence of linguistic preservation, separately from generation-quality and timing filters.
-
Scope of temporal alignment. Does the synchronization claim refer only to motion activity and acoustic rhythm, or also to correspondence between signed concepts and spoken content? If the latter, what evidence supports that correspondence beyond endpoint agreement, energy-distribution similarity, and the illustrative example in Figure 3?
-
Attribution and positioning of the contribution. What evidence isolates the SpeechT5 substitution and data construction stages beyond the CLIP/Synchformer ablations? How does the contribution relate to MMAudio and prior speech-motion research such as BEAT?
2 comments
Thank you for the detailed responses and additional experiments. The evaluation with ASL users provides useful evidence of expressive fidelity, and the comparison with F5-TTS using duration information addresses an important baseline concern. I also appreciate the clarification that synchronization concerns local pace and rhythm rather than strict word-level alignment. These additions alleviate several of my original concerns.
However, my concern regarding the significance of the contribution remains. The nine-clip study strengthens the evidence for perceptual improvements, but does not yet establish whether these improvements lead to more effective communication, particularly given the reported content-accuracy gap relative to TTS. I acknowledge the value of the additional evaluation while remaining uncertain about the practical benefit of the proposed requirements.
Given the limited methodological novelty, the case for acceptance rests primarily on the new task, data construction, and demonstrated value. A new task can constitute a substantial contribution, but I am not yet convinced that the current evidence establishes sufficient significance and new knowledge to support acceptance at ICLR. I therefore maintain my Weak Reject recommendation, while recognizing that the rebuttal has strengthened the empirical support for the approach.
Practical benefit in comparison to TTS baselines with video-derived style and emotion prompts (with duration matching)
Our method targets Deaf and hard-of-hearing users, translating sign language into personalized speech so hearing listeners can better grasp communicative intent—timbre, emotion, and rhythm—beyond text semantics. Compared with TTS baselines, our practical advantages are threefold: (1) TTS baselines accept only text prompts to specify rhythm, emotion, and timbre, limiting fidelity to the signer’s performance shown in the video. (2) MLLM-derived style and emotion prompts are typically global descriptions that fail to capture temporal variations. In our experiments, MLLM-based emotion extraction from sign videos was inaccurate and vague—far less reliable than the emotion correspondence learned from our training data. (3) Duration only aligns overall length, not local expressive rhythm. Current TTS systems do not reliably follow specified durations, and post-hoc stretching introduces noticeable artifacts.
Reviewers are highly encouraged to compare the audio-visual videos provided in the supplementary materials.
How should content-accuracy versus perceptual quality trade-off be interpreted
Linguistic validity of synthesized sign videos
Scope of temporal alignment
Attribution and positioning of the contribution