acceptodds
Under review as a conference paper at ICLR 2027

Direct Translation Between Sign Languages

Abstract

Sign language translation has made substantial progress between sign and spoken languages, while translation across sign languages remains less explored. Translating directly between sign languages could support communication across signing communities without requiring a shared written language. A cascade of sign-to-text, spoken-language translation, and text-to-sign models offers one route, but can propagate intermediate errors and requires three sequential translation stages. We develop direct sign-to-sign translation, whose training is limited by the scarcity of parallel signing across languages. To address this obstacle, we adapt back-translation to construct cross-lingual pairs from existing text–sign corpora: the source signing is synthesized through a text bridge, while the target remains the gold sign from the original corpus. Using these pairs, we jointly train a single Qwen3-based model for text-to-sign and sign-to-sign translation. The latter generates target signing directly from source signing without an intermediate transcript. Experiments cover six directions among American, Chinese, and German Sign Language, using back-translation pairs and existing cross-lingual sign pairs. On synthetic sources, direct translation improves BLEU-4 in five directions and lowers overall motion error relative to the cascade. On existing test pairs, it improves BLEU-4 over the cascade by 1.91–3.87 points across all six directions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sign-to-sign translation, Multilingual sign language translation, Back-translation, Sign language generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_do4aOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies direct translation between American, Chinese, and German Sign Language without an intermediate transcript at inference. To address the scarcity of parallel sign-language data, the authors construct synthetic-source, gold-target pairs by translating target transcripts and generating source signing with a fixed text-to-sign model. A shared Qwen3-based model is then jointly trained for text-to-sign and sign-to-sign translation using VQ-VAE motion tokens. Experiments compare direct translation with a sign-to-text, machine translation, and text-to-sign cascade on synthetic inputs and existing cross-lingual sign pairs, reporting motion error, back-translated BLEU, and inference latency.

Critical strengths
  1. The paper addresses a meaningful and relatively underexplored task: direct translation between sign languages without an intermediate transcript at inference(Sign2Sign) following the previous work: How to align multiple signed language corpora for better sign-to-sign translations?

  2. The synthetic-source, gold-target pairing strategy provides a practical way to expand training data without collecting new parallel signing.

  3. The paper transparently discusses experimental limitations and failure cases, including training-clip overlap, constant cascade outputs, and the scope of latency measurements.

Critical weaknesses

(1) It would benefit from broader coverage of sign language translation and production, particularly multilingual modeling and synthetic-data approaches, and a clearer explanation of how the proposed method differs from existing work.

(2) Section 2 introduces the construction of synthetic training pairs, while Section 4 returns to dataset construction and filtering for evaluation. Although these sections serve different purposes, their distinction and relationship should be made more explicit.

(3) The VQ-VAE tokenization and autoregressive language-model generation closely resemble SOKE, while direct cross-lingual sign translation has already been explored by Inan et al. The clearest contribution appears to be the synthetic training corpus and task extension, but the experiments do not isolate their effects from model architecture and joint training.

(4) Tables 4 and 7 contain a numerical inconsistency. Equation (5) defines the overall error as (12Cbody+21Cleft+21Cright)/54(12C_{\mathrm{body}}+21C_{\mathrm{left}}+21C_{\mathrm{right}})/54. For Direct DGS→ASL, the reported component errors of 6.78, 2.12, and 2.06 yield approximately 3.13, whereas both tables report 2.14. Furthermore, weighting Table 7’s reported overall errors by anchor counts gives approximately 2.22, rather than the reported average of 2.25. These discrepancies require clarification.

(5) The semantic reliability of the synthetic training pairs remains insufficiently validated. The three corpora cover different domains, so translated transcripts may fall outside the source-language T2S model’s training distribution. Consequently, synthetic source signing may omit or misrepresent content present in the gold target. Although the paper acknowledges this limitation, it does not quantify the resulting semantic mismatch or assess synthetic-source fidelity through human evaluation. This raises concerns about the quality of the synthetic sign-to-sign corpus, which constitutes a central contribution of the paper.

Questions for authors

see above

4 comments

We thank you for the detailed feedback and for recognizing the importance of direct S2S translation.

Contribution beyond Inan: data that enables effective S2S learning.

Inan et al. pioneered direct S2S with 317–2,274 matched training pairs per language pair. Our construction removes the requirement for matching recordings across corpora: we translate a recorded target’s transcript, synthesize source signing in another language, and retain the original target. This yields 112,324 directed training pairs across six directions among ASL, CSL, and DGS.

This corpus supplies the supervision needed to train our six-direction direct S2S model at substantially larger scale. Our model reports BLEU-4 scores of 7.66–13.55, compared with 0.00–6.81 reported by Inan et al. Under our paired, shared-evaluator comparison, our model also improves BLEU-4 over the cascade by 1.91–3.87 points in all six directions on recorded Inan-Test inputs (Table 3), despite training with synthetic sources.

Distinction from prior BT and contribution of joint training.

Zhou et al. and Moryossef et al. construct sign–text or gloss–text supervision; we construct cross-language sign–sign supervision for motion-to-motion translation. Our method makes existing text–sign resources usable for direct S2S training without a pre-existing reverse S2S model. Established tokenization and multilingual modeling provide the foundation for this contribution.

Table 6 additionally compares the best-performing checkpoints obtained with T2S-only, S2S-only, and joint training. The joint model achieves lower synthetic-source S2S motion error for all three target languages than either single-task model.

Synthetic-source reliability.

We appreciate the concern about cross-domain synthesis. We deliberately synthesize only the source, retaining recorded target signing as supervision. Following the rationale of back-translation, imperfect synthetic inputs can expand useful training data while avoiding generated targets. The recorded-input results support this supervision’s utility. Cross-domain MT/T2S noise is discussed in Section 2 and the limitations.

Dataset clarity and numerical correction.

We agree that the dataset roles can be clearer. Section 2 describes training-pair construction; Section 4 describes held-out BT-input evaluation and evaluation on existing recordings. Inan-Strict is a separate diagnostic with training-clip overlap. We will strengthen the cross-reference.

Thank you for identifying the typo in Tables 4 and 7. Direct DGS→ASL should be 3.14, not 2.14. We have corrected both entries and clarified the weighting. This widens the deficit relative to the cascade in that direction while leaving the overall average unchanged.

Dear Reviewer do4a,

As the discussion period draws to a close, we wanted to follow up on our response. We clarified how our approach to constructing training pairs differs from prior work and corrected the inconsistency you identified in Tables 4 and 7. We would welcome your thoughts on these revisions and would be happy to address any remaining concerns.

Thank you again for your time and careful reading of our paper.

Thanks for the rebuttal

Dear Reviewer do4a,

Thank you for taking the time to read our rebuttal and reconsider your assessment. We appreciate your careful reading of the paper and your feedback throughout the review process.

Reviewer_jZCxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies direct translation among ASL, CSL and DGS (six directions). To address the scarcity of parallel signing across languages, the authors adapt back-translation: the transcript of a gold text–sign sample is machine-translated into another spoken language, a fixed T2S model synthesizes the source signing, and the result is paired with the original gold signing, giving 112,324 training pairs. A Qwen3-0.6B model with a shared VQ-VAE motion tokenizer is trained jointly on T2S and S2S. Compared with an S2T→MT→T2S cascade, direct translation obtains higher BLEU-4 in all six directions on Inan-Test and in five directions on each of the synthetic-input and Inan-Strict evaluations. With pose extraction and rendering excluded, the reported direct inference time is approximately 1/2.6 of the sum of the separately timed cascade stages.

Critical strengths
  1. The problem has practical significance. Back-translation produces about 112K training pairs (Table 1), which alleviates the scarcity of parallel signing across languages, and it does not require matching equivalent clips across corpora.

  2. S2S training uses only synthetic source signing, yet on the existing clips of Inan-Test the model outperforms the cascade in BLEU-4 in all six directions, by 1.91–3.87 points (Table 3).

Critical weaknesses
  1. There is no source-sensitivity control, and there are cases where different inputs produce identical outputs (Sec. 5.4). Language and corpus domain are confounded, and there is no evaluation by native signers. It is therefore unclear how much the model relies on the source semantics, and whether the gains in automatic metrics correspond to actual translation quality.

  2. The robustness of the comparison with the cascade still needs to be verified. The CSL front end produces constant outputs, and once CSL inputs are excluded from Inan-Strict, the motion-error advantage of direct translation is reversed (Sec. 5.2). In Table 6, the T2S motion error of the joint model is higher than that of the T2S-only model, but cascade results with the generator replaced are not reported. In addition, the distribution of generated lengths and a length-normalized comparison are missing, and some aggregate values are inconsistent with Table 7.

Questions for authors
  1. Within each direction, how do scores change after reassigning intact, length-matched source clips while keeping references fixed? How often do different inputs produce identical outputs?

  2. How much does Inan-Test overlap with component training sets? Could you provide non-overlapping results and a small-scale native-signer evaluation?

  3. Do conclusions hold on the main evaluation cohorts after fixing the CSL front end and, in separate controls, adapting the front end (Table 8) or using the T2S-only generator?

  4. Could you report generated lengths, DTW-path-normalized errors and paired bootstrap intervals for BLEU differences?

2 comments

We thank you for the constructive questions. Our additional analyses show:

  • Direct responds to source changes in 81% of evaluated requests.
  • Inan-Test has no clip-ID overlap with native training/validation splits.
  • Direct leads both cascade configurations on BLEU in all six directions.
  • Direct’s aggregate Strict motion advantage persists after path normalization.

1. Source dependence and repeated outputs

Replacing source clips within each direction changed generated motion in 81% of evaluated requests, with references fixed and token length matched, alongside approximate frame-length matching. This demonstrates clear sensitivity to source input.

After deduplicating source-clip/target-language requests, 165 input units (17%) shared an identical output with another distinct input, mainly in CSL-related directions; this pattern warrants further analysis.

2. Training overlap and signer evaluation

All 817 distinct clips in the 3,542 paired Inan-Test requests belong to native test splits, with no clip-ID overlap with native training/validation lists or released Inan training pairs. Appendix A.2 documents this audit and distinguishes Inan-Test from Strict, a mixed-split diagnostic. Evaluation by qualified signers remains future work.

3. T2S-only cascade comparison

New Table 9, Appendix A.3.2, replaces the cascade’s joint generator with the T2S-only checkpoint. Requests, S2T front ends, MT outputs, tokenizer, decoding settings, and target evaluators remain fixed.

Direct achieves higher BLEU than both cascades in all six directions, exceeding the better cascade score by 1.02–3.41 points. It also has lower cumulative DTW-PA-MPJPE than both cascades in four directions; T2S-only performs best on this motion metric in the two directions targeting ASL.

4. Length sensitivity and statistical uncertainty

Inan-Test outputs average 95.8 frames for Direct and 100.2 for cascade, versus 110.7 frames for processed references.

For Strict, we normalize each body/hand DTW cost by its own alignment-path length, retaining the original selected references and joint weights:

Motion metric ↓DirectCascade
Cumulative2.25112.4386
Path-normalized0.02010.0226

Direct’s aggregate advantage persists under path normalization. Inan-Test path normalization and paired confidence intervals for the reported BLEU differences remain unverified.

Tables 4 and 7 also correct Direct DGS→ASL from 2.14 to 3.14; the aggregate calculated from unrounded scores remains 2.25.

Finally, language and domain co-vary across these corpora; broader cross-domain evaluation remains future work.

Dear Reviewer jZCx,

As the discussion period draws to a close, we wanted to follow up on our responses above, particularly the comparison you suggested between Direct and a cascade with a T2S-only generator. We would appreciate your thoughts on the additional results and would be happy to clarify any points that remain open.

Thank you again for your time and thoughtful review.

Reviewer_6yFROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of translation between 2 different sign languges directly(instead of cascaded). It mainly leverages back-translation and achives state-of-the-art performances on several datasets. They additionally form a large dataset of 100K pairs, and also shows their system is 2.6x faster than cascading.

Critical strengths
  1. Direct sign language translation is severly under-studied in the sign lanugage processing field, and is indeed an important problem.
  2. The paper is well written and easy to follow.
  3. State-of-the art in many benchmarks.
  4. Speed : the paper shows it's faster than cascanded and this is important towards real-time conversations/translation.
Critical weaknesses
  1. Usage of mostly intepreted signing videos. No natural signing
  2. Lack of robust evalutions like BLEURT. Also, a difference of 2 BLEU at this low scores isnt usually even very significant.
  3. Lack of novelty on its own. Using back-translation for sign laguage translation is quite old and many works did that.
Questions for authors
  1. Can the authors do a small analyssi even on small part of naural signing from YouTube-SL-25 ?
  2. Can the BLEURT also be provided ?
5 comments

We thank you for recognizing the importance of direct sign-to-sign translation, our empirical results, and the efficiency benefits.

1. Natural signing: an ongoing research direction

We are already working on this direction through a separate multilingual dataset and benchmark project. YouTube-SL-25 provides sign–text data, while reliable cross-sign-language evaluation requires constructing and validating corresponding references across sign languages. This is a substantial additional research effort that we are pursuing as a separate study. Our future work will include human evaluation by qualified signers to assess meaning preservation and signing quality, alongside evaluation on natural-signing sources.

2. BLEURT and the evaluation pipeline

Our text-based evaluation follows this chain:

Generated target signing → sign-to-text model → predicted text → comparison with reference text.

Both BLEU and BLEURT operate at the final step. Their scores therefore depend on the sign-to-text model’s recognition accuracy, and cannot directly assess signing features absent from its textual output. A low score can reflect errors in either the generated signing or the evaluator. Replacing BLEU with BLEURT changes the text comparison but does not resolve this upstream limitation.

We are adding BLEURT-20 as complementary evidence, together with paired bootstrap confidence intervals that account for repeated source clips. We will provide the results in a follow-up update.

3. Contribution: scalable cross-language S2S supervision

Our contribution is to construct cross-language sign–sign supervision at scale. Earlier BT studies augment sign–text or gloss–text data; İnan et al. construct S2S pairs by matching existing recordings across corpora. We synthesize source signing in another language while retaining the recorded target, removing the requirement to find matching recordings. This yields 112,324 training pairs across six directions and supports a single multilingual model with direct sign→sign inference. The contribution combines this supervision strategy with demonstrated translation and efficiency gains, building on established generation components.

Following up on the BLEURT evaluation promised in our response, we have completed BLEURT-20 scoring for all 3,542 paired Inan-Test requests. We compare Direct with the joint-generator cascade from Table 3, using text recovered from their generated target signing by the fixed target-language S2T evaluators, the same references, and uniform text cleaning.

Higher scores are better. The 95% confidence intervals use 10,000 paired bootstrap resamples grouped by source clip; directional intervals are not adjusted for multiple comparisons.

DirectionDirectCascade (joint)Direct−cascade, 95% CI
ASL→CSL0.510.39+0.12 [0.06, 0.18]
ASL→DGS0.540.49+0.05 [0.01, 0.10]
CSL→ASL0.680.58+0.10 [0.06, 0.15]
CSL→DGS0.590.46+0.13 [0.05, 0.20]
DGS→ASL0.670.55+0.12 [0.07, 0.17]
DGS→CSL0.510.42+0.09 [0.04, 0.15]
Request-weighted overall0.580.48+0.10 [0.08, 0.13]

Direct achieves higher BLEURT in all six directions and overall. The overall improvement is +0.10, with a 95% confidence interval entirely above zero. Each directional interval also excludes zero at the reported, unadjusted level.

These results provide complementary evidence that the text recovered from Direct’s generated signing agrees more closely with the reference meaning. They support the reported BLEU improvements using a learned metric that captures semantic similarity beyond lexical overlap.

The evaluation still depends on an intermediate S2T model, so its scores remain affected by recognition errors. Direct assessment by qualified signers remains necessary to establish the comprehensibility and linguistic fidelity of the generated signing.

Dear Reviewer 6yFR,

As the discussion period draws to a close, we wanted to follow up on the BLEURT evaluation you suggested. We reported results for all six translation directions, along with paired confidence intervals. We would welcome your feedback on whether these results address your question and would be happy to discuss any remaining concerns.

Thank you again for your time and helpful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.