- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies direct translation between American, Chinese, and German Sign Language without an intermediate transcript at inference. To address the scarcity of parallel sign-language data, the authors construct synthetic-source, gold-target pairs by translating target transcripts and generating source signing with a fixed text-to-sign model. A shared Qwen3-based model is then jointly trained for text-to-sign and sign-to-sign translation using VQ-VAE motion tokens. Experiments compare direct translation with a sign-to-text, machine translation, and text-to-sign cascade on synthetic inputs and existing cross-lingual sign pairs, reporting motion error, back-translated BLEU, and inference latency.
Critical strengths
-
The paper addresses a meaningful and relatively underexplored task: direct translation between sign languages without an intermediate transcript at inference(Sign2Sign) following the previous work: How to align multiple signed language corpora for better sign-to-sign translations?
-
The synthetic-source, gold-target pairing strategy provides a practical way to expand training data without collecting new parallel signing.
-
The paper transparently discusses experimental limitations and failure cases, including training-clip overlap, constant cascade outputs, and the scope of latency measurements.
Critical weaknesses
(1) It would benefit from broader coverage of sign language translation and production, particularly multilingual modeling and synthetic-data approaches, and a clearer explanation of how the proposed method differs from existing work.
(2) Section 2 introduces the construction of synthetic training pairs, while Section 4 returns to dataset construction and filtering for evaluation. Although these sections serve different purposes, their distinction and relationship should be made more explicit.
(3) The VQ-VAE tokenization and autoregressive language-model generation closely resemble SOKE, while direct cross-lingual sign translation has already been explored by Inan et al. The clearest contribution appears to be the synthetic training corpus and task extension, but the experiments do not isolate their effects from model architecture and joint training.
(4) Tables 4 and 7 contain a numerical inconsistency. Equation (5) defines the overall error as . For Direct DGS→ASL, the reported component errors of 6.78, 2.12, and 2.06 yield approximately 3.13, whereas both tables report 2.14. Furthermore, weighting Table 7’s reported overall errors by anchor counts gives approximately 2.22, rather than the reported average of 2.25. These discrepancies require clarification.
(5) The semantic reliability of the synthetic training pairs remains insufficiently validated. The three corpora cover different domains, so translated transcripts may fall outside the source-language T2S model’s training distribution. Consequently, synthetic source signing may omit or misrepresent content present in the gold target. Although the paper acknowledges this limitation, it does not quantify the resulting semantic mismatch or assess synthetic-source fidelity through human evaluation. This raises concerns about the quality of the synthetic sign-to-sign corpus, which constitutes a central contribution of the paper.
Questions for authors
see above
4 comments
Dear Reviewer do4a,
As the discussion period draws to a close, we wanted to follow up on our response. We clarified how our approach to constructing training pairs differs from prior work and corrected the inconsistency you identified in Tables 4 and 7. We would welcome your thoughts on these revisions and would be happy to address any remaining concerns.
Thank you again for your time and careful reading of our paper.
Thanks for the rebuttal
Dear Reviewer do4a,
Thank you for taking the time to read our rebuttal and reconsider your assessment. We appreciate your careful reading of the paper and your feedback throughout the review process.
We thank you for the detailed feedback and for recognizing the importance of direct S2S translation.
Contribution beyond Inan: data that enables effective S2S learning.
Inan et al. pioneered direct S2S with 317–2,274 matched training pairs per language pair. Our construction removes the requirement for matching recordings across corpora: we translate a recorded target’s transcript, synthesize source signing in another language, and retain the original target. This yields 112,324 directed training pairs across six directions among ASL, CSL, and DGS.
This corpus supplies the supervision needed to train our six-direction direct S2S model at substantially larger scale. Our model reports BLEU-4 scores of 7.66–13.55, compared with 0.00–6.81 reported by Inan et al. Under our paired, shared-evaluator comparison, our model also improves BLEU-4 over the cascade by 1.91–3.87 points in all six directions on recorded Inan-Test inputs (Table 3), despite training with synthetic sources.
Distinction from prior BT and contribution of joint training.
Zhou et al. and Moryossef et al. construct sign–text or gloss–text supervision; we construct cross-language sign–sign supervision for motion-to-motion translation. Our method makes existing text–sign resources usable for direct S2S training without a pre-existing reverse S2S model. Established tokenization and multilingual modeling provide the foundation for this contribution.
Table 6 additionally compares the best-performing checkpoints obtained with T2S-only, S2S-only, and joint training. The joint model achieves lower synthetic-source S2S motion error for all three target languages than either single-task model.
Synthetic-source reliability.
We appreciate the concern about cross-domain synthesis. We deliberately synthesize only the source, retaining recorded target signing as supervision. Following the rationale of back-translation, imperfect synthetic inputs can expand useful training data while avoiding generated targets. The recorded-input results support this supervision’s utility. Cross-domain MT/T2S noise is discussed in Section 2 and the limitations.
Dataset clarity and numerical correction.
We agree that the dataset roles can be clearer. Section 2 describes training-pair construction; Section 4 describes held-out BT-input evaluation and evaluation on existing recordings. Inan-Strict is a separate diagnostic with training-clip overlap. We will strengthen the cross-reference.
Thank you for identifying the typo in Tables 4 and 7. Direct DGS→ASL should be 3.14, not 2.14. We have corrected both entries and clarified the weighting. This widens the deficit relative to the cascade in that direction while leaving the overall average unchanged.