acceptodds
Under review as a conference paper at ICLR 2027

SignDubber: Personalized and Synchronized Speech Synthesis from Sign Language Videos

Abstract

Sign language translation has largely focused on converting signing videos into text, while the more practically desired task of synthesizing faithful and expressive speech from sign language remains underexplored. To bridge this gap, we introduce a new task: generating personalized speech that matches the signer’s appearance (timbre), signing rhythm, and visible emotion from the sign video and its translated texts. To tackle this, we present SignDubber, a multimodal conditional speech synthesis framework that jointly models fine‑grained video understanding and cross‑modal alignment for expressive speech generation. Specifically, we adopt a flow-matching-based audio generation model conditioned on text embeddings from SpeechT5 and video features extracted by an audio-synchronization-aware video encoder, achieving effective cross-modal alignment. Since existing datasets mostly provide neutral signing with text annotations and lack paired expressive speech, we design a creative data synthesis pipeline that leverages pre-trained audio‑visual generation models to obtain qualified training pairs. Extensive experiments show that our proposed method generalizes robustly to real‑world videos, accurately capturing identity, rhythm, and emotion. Both evaluation metrics and user studies confirm that our approach significantly outperforms multiple representative baselines, demonstrating its practical value.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal, Speech synthesis, Sign language

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jgEPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies sign-language video dubbing: generating speech from a silent signing video and an available English translation, with vocal characteristics compatible with the signer's appearance and delivery matched to visible rhythm and emotion. SignDubber adapts the MMAudio architecture, using frozen SpeechT5 text representations and Synchformer video features to condition flow matching in an audio VAE latent space. The authors construct approximately 200,000 training triplets through separate LTX-2 video and audio generation branches, followed by IndexTTS2 emotion enhancement. Evaluation on real EmoSign videos reports improved temporal alignment and subjective ratings over several baselines, while content accuracy remains below that of the TTS baselines. CLIP/Synchformer ablations and a comparison with raw LTX-2 outputs provide additional evidence.

Critical strengths
  • Clear formulation of an underexplored task.

Separating translated linguistic content from visually conditioned delivery provides a clear formulation for investigating how signing rhythm and expression can influence synthesized speech.

  • Substantial data construction and encouraging real-video results.

The decoupled synthesis pipeline addresses several competing generation requirements, including emotional expression, speech-like lip-motion shortcuts, and audio duration. Evaluating on real EmoSign videos provides useful evidence of transfer beyond generated videos. The consistent improvements in subjective ratings and the controlled CLIP/Synchformer ablations support the usefulness of the chosen conditioning design within the reported setting.

Critical weaknesses
  • The practical need and added value of the task requirements are insufficiently established.

The paper does not establish which intended users need detailed motion-speech synchronization or appearance-conditioned voices. With a supplied translation, conventional TTS already enables access to the content. The TTS baselines appropriately receive video-derived style and emotion prompts, but the comparison does not establish the additional benefit over such systems with simple duration matching. The study measures perceived quality rather than comprehension, communication of the signer's intent, or usefulness to signers and viewers. SignDubber-small has a WER of 5.90%, versus 1.59% for Qwen3-TTS; the practical value of the perceptual gains relative to this loss in content accuracy remains unclear. Since the architecture largely adapts existing components, demonstrating the application benefit is particularly important. Related work also omits co-speech gesture research such as BEAT (Liu et al., ECCV 2022; arXiv:2203.05297), which studies speech-motion relationships involving semantics and emotion. Although the generation direction and linguistic domain differ, explaining the challenges specific to sign-language dubbing would clarify the contribution.

  • The data and evaluation do not adequately establish linguistic fidelity or the scope of the claimed alignment.

Training regenerates signing videos with altered facial expressions while retaining the original translations. Yet the paper notes that facial expression, head motion, and posture can carry linguistic information. Pose conditioning alone does not establish preservation of the source utterance or compatibility with all five imposed emotions. I could not find a linguistic assessment by fluent ASL signers. In evaluation, boundary error measures endpoints, while CRC-W1 compares normalized motion activity and audio energy; neither establishes that corresponding concepts are expressed at compatible times. Video-text-audio pairing may support implicit semantic alignment, but the model can also obtain content entirely from text and use video mainly for rhythm and affect. The 14-video, 12-participant MOS study does not resolve these questions, and participant sign-language proficiency is unspecified. The evidence supports improved perceived synchronization, but is insufficient to establish faithful transfer of sign-language expression and intent. The claims should reflect this distinction.

Questions for authors
  • Practical benefit and evaluation population. What is the deployment scenario and evidence of benefit beyond video-informed emotional TTS with duration matching? Did signers or intended users participate, what was their ASL proficiency, and how should the content-accuracy versus perceptual-quality trade-off be interpreted?

  • Linguistic validity of synthetic supervision. What checks establish that regenerated videos remain valid ASL matching their translations after expression changes? Please report any existing fluent-signer assessment or other evidence of linguistic preservation, separately from generation-quality and timing filters.

  • Scope of temporal alignment. Does the synchronization claim refer only to motion activity and acoustic rhythm, or also to correspondence between signed concepts and spoken content? If the latter, what evidence supports that correspondence beyond endpoint agreement, energy-distribution similarity, and the illustrative example in Figure 3?

  • Attribution and positioning of the contribution. What evidence isolates the SpeechT5 substitution and data construction stages beyond the CLIP/Synchformer ablations? How does the contribution relate to MMAudio and prior speech-motion research such as BEAT?

2 comments
  • Practical benefit in comparison to TTS baselines with video-derived style and emotion prompts (with duration matching)

    • Our method targets Deaf and hard-of-hearing users, translating sign language into personalized speech so hearing listeners can better grasp communicative intent—timbre, emotion, and rhythm—beyond text semantics. Compared with TTS baselines, our practical advantages are threefold: (1) TTS baselines accept only text prompts to specify rhythm, emotion, and timbre, limiting fidelity to the signer’s performance shown in the video. (2) MLLM-derived style and emotion prompts are typically global descriptions that fail to capture temporal variations. In our experiments, MLLM-based emotion extraction from sign videos was inaccurate and vague—far less reliable than the emotion correspondence learned from our training data. (3) Duration only aligns overall length, not local expressive rhythm. Current TTS systems do not reliably follow specified durations, and post-hoc stretching introduces noticeable artifacts.

      Reviewers are highly encouraged to compare the audio-visual videos provided in the supplementary materials.

  • How should content-accuracy versus perceptual quality trade-off be interpreted

    • First, it should be noted that dubbing tasks are more challenging than TTS in achieving high content accuracy, because they must satisfy the additional constraint imposed by the video modality. This phenomenon also widely exists in the general video dubbing community. Our proposed SignDubber already achieves better accuracy than dubbing baselines.
    • In terms of practical application, we recognize that content accuracy is more important than perceptual quality. Nevertheless, as a personalized speech translation capability that better facilitates communication between Deaf individuals and hearing people, our technical paradigm, SignDubber, still represents a technical direction with unique value. We expect that, as the technology advances, it will gradually close the gap with TTS in content accuracy.
  • Linguistic validity of synthesized sign videos

    • Thanks for this constructive suggestion. In our data synthesis pipeline, the facial expression from the original sign video will be modified under the guidance of emotion tag, but the head motion and posture that dominate the linguistic semantics will be retained via the extracted 2D human skeletons.
    • Anyway, considering the potential errors of skeleton estimation and conditional video synthesis, it may disturb the linguistic validity of the synthesized sign videos to some extent, as evaluated through opensource SLT method (i.e. Uni-Sign model trained on How2Sign), giving comparison on the original and re-generated sign videos respectively. We believe this is acceptable since our method still has translated text as semantic anchor, while the sign video mainly serves to provide perceptual attributes for the target speech, like timbre, rhythm, and emotion. Anyway, we expect the linguistic quality of our data synthesis pipeline will be increasingly improved as the estimation and synthesis technique advances. | | BLEU1↑ | ROUGE↑ | BLEURT↑ | |---|---|---|---| | Original videos | 6.43 | 12.79 | 28.57 | | Re-generated | 5.65 | 10.87 | 24.64 |
  • Scope of temporal alignment

    • Please see "Common Response" above
  • Attribution and positioning of the contribution

    • We acknowledge that the main contribution of our method does not lie in technical novelty per se. Rather, we propose a practically important yet under-explored task and make the first attempt to validate its feasibility by developing a technical framework and a data-construction pipeline.
    • Compared with MMAudio and BEAT, it is difficult to claim that our target task is inherently more challenging, since all these methods address cross-modal correspondences from different aspects and dimensions. We emphasize, however, that the cross-modal mapping our model aims to establish is not covered by any existing work and thus contributes novel insight to the community.

Thank you for the detailed responses and additional experiments. The evaluation with ASL users provides useful evidence of expressive fidelity, and the comparison with F5-TTS using duration information addresses an important baseline concern. I also appreciate the clarification that synchronization concerns local pace and rhythm rather than strict word-level alignment. These additions alleviate several of my original concerns.

However, my concern regarding the significance of the contribution remains. The nine-clip study strengthens the evidence for perceptual improvements, but does not yet establish whether these improvements lead to more effective communication, particularly given the reported content-accuracy gap relative to TTS. I acknowledge the value of the additional evaluation while remaining uncertain about the practical benefit of the proposed requirements.

Given the limited methodological novelty, the case for acceptance rests primarily on the new task, data construction, and demonstrated value. A new task can constitute a substantial contribution, but I am not yet convinced that the current evidence establishes sufficient significance and new knowledge to support acceptance at ICLR. I therefore maintain my Weak Reject recommendation, while recognizing that the rebuttal has strengthened the empirical support for the approach.

Reviewer_rLbqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SignDubber, a framework for generating personalized and synchronized speech from sign language videos and their corresponding text translations. The method combines SpeechT5, Synchformer, and a flow-matching-based audio generation model to capture speech content, signing rhythm, and emotional expression. The authors also construct a synthetic training dataset and evaluate the method on real sign language videos.

Critical strengths
  1. The paper explores an interesting and practical task by extending traditional sign language translation from text output to personalized speech generation with emotion and temporal synchronization.
  2. The experiments cover multiple baselines and ablation studies, and the authors provide numerous generated examples for qualitative comparison.
Critical weaknesses
  1. The proposed framework is not an end-to-end sign-to-speech system, as it still requires the translated text as input. It is essentially a downstream extension of existing SLT systems.
  2. The methodological novelty is somewhat limited. The model mainly combines existing components, including MMAudio, SpeechT5, Synchformer, and flow matching, without introducing substantial architectural innovations or new training objectives.
  3. The necessity of strict synchronization between signing motion and generated speech is not sufficiently justified. Since sign and spoken languages differ in their linguistic and temporal structures, encouraging speech to follow signing motion may introduce unnatural speaking rates or pauses.
Questions for authors

See the weaknesses above.

2 comments
  • Position of our proposed task and the value of our work
    • For personalized speech translation of sign-language videos, an end-to-end sign-to-speech pipeline would indeed be a more direct solution, but no such study currently exists. Our proposed SignDubber serves as the downstream task of existing SLT to realize this unique function, which contributes clear value. This is analogous to video dubbing, which also requires the original video and the spoken script rather than directly predicting speech from the original video. We provide this viewpoint for the reviewer to recognize the value of our work as the first to propose this task and provide a feasibility validation, rather than treat this as a weakness.
  • The necessity of strict synchronization between signing motion and generated speech
    • Please see "Common Response" above

Thank you for the detailed response. I appreciate the value of the proposed task and acknowledge the effectiveness of the approach. However, I do not think the rebuttal fully addresses my main concern regarding the limited methodological novelty. While the work has its merits, I still feel that the overall contribution falls short of the acceptance bar for ICLR.

Reviewer_bCa2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces sign language video dubbing. Given a silent sign-language video and its English translation, the goal is to generate speech whose content matches the text and whose timbre, rhythm and emotion fit the signer's appearance, signing motion and visible expression. No paired data exists, so the authors synthesise about 0.2M training triplets from OpenASL:

  • LTX-2, conditioned on the first frame and the DWPose sequence, generates emotional silent videos and, in a separate pass, speech audio;
  • IndexTTS2 then adds the target emotion to that audio.

The model, SignDubber, is MMAudio with the CLIP streams removed and the text encoder replaced by SpeechT5, with Synchformer video features modulating the audio frame by frame. It is trained from scratch at 0.1B and 0.8B parameters. On 200 real EmoSign videos it reaches much lower boundary error (0.08 s vs about 0.75 s) and better rhythm correlation than the TTS and omni-model baselines. It also gets the highest MOS in a 14-video, 12-listener study, at the cost of higher WER (5.9% vs 1.6% for TTS).

Critical strengths
  1. A new, practically motivated task with a workable way to get training data. Moving sign-language translation from subtitles to speech that reflects the signer's timing and affect is a meaningful accessibility goal that prior SLT and dubbing work has not addressed. The decoupled LTX-2 pipeline is a creative answer to the complete lack of paired data:

    • a silent-lip emotional video branch;
    • a separate pose-conditioned audio branch;
    • IndexTTS2 emotion enhancement.
  2. Clear evidence that frame-level video conditioning is what drives synchronisation. The ablation (Table 2) trains each variant from scratch under identical settings. Removing Synchformer raises boundary error from 0.08 to 0.58 s and lowers all MOS scores by 1.2–1.7 points, while adding CLIP hurts both WER and synchronisation. Real-video tests and the comparison with the raw LTX-2 generator suggest the model is more than a copy of its data generator.

Critical weaknesses
  1. The supervision and the evaluation are largely circular, so it is unclear what "matching the signer" means.
    • Every training target is synthetic. The "right" timbre for a face and the "right" rhythm for a motion are whatever LTX-2 hallucinated, and emotion comes from an IndexTTS2 setting chosen per prompt. No real sign-to-speech data, interpreter voicing, or perceptual study validates these targets.
    • The synthetic data is filtered by WER, Bnd-Err and CRC-W1, the same metrics used for evaluation. The model is therefore trained on data selected to score well on the test metrics.
    • Bnd-Err and CRC-W1 are heuristics the authors propose themselves. They mainly reward speech that fills the signing interval and follows its motion-energy envelope. Because ASL and English differ in word order and timing, whether local "rhythm alignment" between signs and English words is even meaningful is not discussed or validated.
    • The identity-compatibility metric is judged manually by the authors, who are not blind to the method, and only removes "severely inconsistent" voices.

Please report: validation against real voiced interpretations, ideally with signers or Deaf and interpreter participants, and metrics that do not share a definition with the data filter.

  1. The baselines are disadvantaged on exactly the dimension where SignDubber wins, and the human evaluation is too small to make up for it.
    • The main gains (Bnd-Err 0.08 vs about 0.75 s, CRC-W1, Rhythm MOS) come largely from duration. The TTS baselines receive no target duration; the authors note the public IndexTTS2 release lacks duration control. In Fig. 3 they simply finish early.
    • Obvious baselines are missing:
      • TTS output time-stretched or segment-aligned to the detected signing interval;
      • a duration-controllable TTS such as F5-TTS or E2 TTS with a target length;
      • SignDubber trained on the "naive" TTS-generated pairs that section 4 dismisses, which would show whether the expensive LTX-2 pipeline is necessary.
    • The user study covers only 14 hand-picked "emotionally rich" videos and 12 listeners. Their signing or Deaf background is not reported, and "timbre" and "naturalness" are merged into one score. The large model is worse than the small one on most metrics, and no variance is reported for the objective metrics.
Questions for authors

See weaknesses

2 comments
  • Filtering training samples with testing metrics causes no hacking issues

    • We clarify that the metrics like WER and Bnd-Err are used to filter low-quality AI-generated data rather than used as training loss terms, so no evaluation hacking risk exists. Note that, deep learning models are generally trained on paired samples with ground-truth supervision that shall be perfect on their target evaluation metrics. This viewpoint could justify our reasonability of selecting training data with good score on the test metrics.
  • As ASL and English differ in word order and timing, whether local "rhythm alignment" between signs and English words is even meaningful?

    • Although inverted constructions do occur in sign language with respect to natural-language expression, this does not affect their synchronized correspondence at the expression level. For example, after a sentence is inverted, it can still be expressed with the same tone and speaking rate. In our task, synchronization between translated speech and sign motions mainly operates at the level of local utterance unit, i.e. matching signing pace to speech rate, rather than requiring strict word-by-word alignment.
  • Comparison with TTS baselines with length/duration customized.

    • In our earlier experiments, we did implement Qwen3-TTS with a duration-stretching trick (via ffmpeg), but found that the resulting speech audio suffers from noisy perceptual artifacts. In addition, F5-TTS achieves the target duration mainly by increasing pause duration at commas or periods, rather than naturally adjusting speaking rate. We conducted a re-evaluation. The results for the two additional baselines are provided in the table below: | Method | WER(%)↓ | Bnd-Err(s)↓ | CRC-W1↓ | Timbre/Nat.-MOS↑ | Emotion-MOS↑ | Rhythm/Sync-MOS↑ | | -------------------- | ------------- | ------------- | ------------- | ---------------- | ------------- | ---------------- | | Qwen3-TTS | 1.59±0.12 | 0.76±0.05 | 0.17±0.02 | 3.62±0.18 | 3.30±0.17 | 3.15±0.18 | | Qwen3-TTS(stretched) | 2.46±0.18 | 0.12±0.01 | 0.08±0.01 | 2.51±0.17 | 2.77±0.16 | 2.42±0.16 | | IndexTTS2 | 1.66±0.13 | 0.75±0.05 | 0.13±0.02 | 3.55±0.16 | 3.43±0.16 | 3.06±0.18 | | F5-TTS | 1.63±0.13 | 0.51±0.04 | 0.10±0.01 | 3.44±0.16 | 3.27±0.14 | 3.14±0.15 | | Qwen3.5-Omni-Plus | 1.98±0.15 | 0.93±0.06 | 0.18±0.02 | 3.48±0.16 | 3.14±0.15 | 2.76±0.16 | | FunCineForge | 16.14±1.20 | 0.08±0.01 | 0.07±0.01 | 3.50±0.15 | 3.16±0.17 | 3.08±0.16 | | SignDubber-small | 5.90±0.45 | 0.08±0.01 | 0.06±0.01 | 3.97±0.17 | 3.84±0.17 | 3.79±0.16 | | SignDubber-large | 6.71±0.44 | 0.08±0.01 | 0.06±0.01 | 3.99±0.14 | 3.86±0.17 | 3.72±0.17 |
  • Since the TTS baselines fail to produce satisfactory results, there is no reason to expect that training our model on data synthesized by them would yield better results. We therefore omit the comparison with a SignDubber baseline trained on these “naive” TTS-generated pairs.

  • About variance of statistic results in quantitative evaluation.

    • Thanks for this valuable suggestion. We have added std for the statistics results in Tab. 2, which indicates the significant level of our superiority over competitors. See the table above.
  • "Timbre" and "naturalness" are merged into one score.

    • We define "naturalness" as whether the speech sounds like a human voice rather than a machine-synthesized voice. As a further aspect of this dimension, "timbre" determines whether the human voice and its identity appearance are coherent and harmonious.
  • The identity-compatibility metric is judged manually by the authors, who are not blind to the method, and only removes "severely inconsistent" voices.

    • Thank you for this keen observation. For Table 3 in our supplementary materials, we have redone the comparative evaluation: we invited four participants with good hearing and vision to evaluate over 100 samples (presented anonymously), and the new evaluation results are shown in the table below. | Method | WER(%)↓ | Bnd-Err(s)↓ | CRC-W1↓ | Timbre/Nat.-MOS↑ | Rhythm/Sync-MOS↑ | | ---------------- | ------------- | ------------- | ------------- | ---------------- | ---------------- | | LTX-2 | 10.86±0.88 | 0.35±0.04 | 0.10±0.01 | 3.73±0.14 | 3.57±0.15 | | SignDubber-small | 6.33±0.54 | 0.09±0.01 | 0.06±0.01 | 4.05±0.15 | 3.83±0.16 | | SignDubber-large | 7.17±0.49 | 0.09±0.01 | 0.06±0.01 | 4.08±0.16 | 3.81±0.14 |

Thank you for your response. I will keep my score since it is already positive.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Responses

  • 【R1, R3】: User-study participants were hearing individuals with no ASL proficiency.
    • Our proposed SignDubber converts sign-language expression into personalized speech to help deaf and hard-of-hearing individuals be effectively understood by hearing people, particularly with respect to perceptual attributes such as emotion and rhythm. Because the input to our model already includes the translated text, our evaluation focuses primarily on whether the speech’s rate, rhythm, and emotional variation match and synchronize with the signer’s performance in the video. Accordingly, the participants in our user study were hearing individuals without ASL knowledge; they were not required to understand the semantics of the sign motions in order to make these judgments. Semantic accuracy is evaluated separately through text-level translation metrics.
    • If ASL interpreters are required, we additionally provide the subjective evaluation below, which more reliably assesses whether the synthesized speech faithfully conveys the expressive attributes of the signed video.
  • 【R1, R3】: Subjective evaluation by ASL individuals with normal hearing
    • We recruited a group of ASL teachers through Italki, an online language education platform. With their help and the help of their community, we conducted a subjective evaluation with 12 normal-hearing ASL users. Using nine short emotional ASL clips from the TV series Switched at Birth, we asked each participant to rate the fidelity of the synthesized speech to the sign video. This evaluation directly assessed whether the synthesized speech faithfully conveyed the expressive attributes of the signed video. The results are as follows (the Input column indicates the input required in addition to the translated text). Our model achieves the best performance while requiring only the most minimal and accessible input. | Method | Input | Timbre/Nat.-MOS↑ | Emotion-MOS↑ | Rhythm/Sync-MOS↑ | | ----------------- | ---------------------- | ---------------- | ------------- | ---------------- | | F5-TTS | Ref auido+Duration | 3.64±0.19 | 3.65±0.19 | 3.99±0.16 | | FunCineForge | Ref audio+Video+Prompt | 3.48±0.19 | 3.47±0.21 | 3.65±0.20 | | IndexTTS2.5 | Ref auido+Prompt | 3.48±0.18 | 3.63±0.19 | 3.64±0.19 | | Qwen3.5-Omni-Plus | Video+Prompt | 3.31±0.22 | 3.52±0.20 | 3.46±0.20 | | Qwen3-TTS | Prompt | 3.61±0.19 | 3.62±0.18 | 3.38±0.21 | | SignDubber-small | Video | 3.94±0.20 | 4.13±0.18 | 4.29±0.16 |
  • 【R1, R2, R3】: The necessity of strict synchronization between signing motion and generated speech
    • We clarify that our task aims to establish the synchronization between translated speech and sign motions at the level of local utterance unit, i.e. matching signing pace to speech rate and motion rhythm to tone of voice, rather than requiring strict word-by-word alignment. For example, urgent expression generally involves faster speech and may also involve faster signing; this is a way in which the signer conveys emotion and intent in communication. This mapping principle is learned from such training pairs, as our dataset synthesis pipeline does.
    • To verify the cross-modal alignment, reviewers are strongly encouraged to inspect the audio-visual results in our supplementary material.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.