- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a pose-based streaming framework for real-time sign language translation. It combines a confidence-aware masked pose autoencoder, a part-specific vector-quantized tokenizer, and prototype-based sign retrieval. At inference, sliding-window predictions are stabilized through past-only majority voting and consecutive deduplication. A causal boundary head groups the resulting gloss stream into segments, which a Transformer translates into text without revising previously emitted output. Prototype banks are constructed from training clips using external segmentation procedures. Experiments on German, Chinese, and English benchmarks report improvements in recognition and translation quality, while evaluations on PHOENIX-2014T examine latency and the quality and coverage of committed text.
Critical strengths
- The paper addresses the important challenge of producing stable translations from short, noisy video windows without revising emitted text. Its integration of confidence-aware pose encoding, prototype retrieval, and past-only voting provides a coherent approach to this setting.
- The evaluation covers multiple sign languages and includes controls for segmentation quality, dictionary coverage, pose detectors, and individual model components. Reporting results across five seeds and runtime costs including pose extraction also strengthens the experimental assessment.
Critical weaknesses
- The writing can be improved. The manuscript is difficult to follow. Technical details, architectural components, training objectives, and evaluation claims are densely presented with insufficient hierarchy and weak transitions. The relationships between the three stages and their respective training and inference procedures are particularly unclear. Substantial restructuring is needed to make the methodology and experimental reasoning accessible.
- The reported performance requires stronger verification, and several quantitative results need clarification. Table 2 reports 63.15 BLEU-1 on PHOENIX-2014T, substantially exceeding the listed baselines despite using pose-only input. These gains, together with improvements on other benchmarks, warrant more transparent checkpoints, evaluation scripts, and supervision details. High scores alone do not establish an error, but there are additional reporting concerns: Table 9’s correlations cannot be reproduced from its displayed values, and Table 4 reports 8.2 WER under oracle-gloss input, where the recognition WER should be zero or inapplicable under the stated definition.
- The paper omits recent competitive approaches [1-6]. Particularly relevant is RTG-Net [6], which explicitly studies real-time sign language recognition and translation on edge devices. Its efficiency results should be discussed and, where feasible, compared under a common runtime protocol, while distinguishing throughput-based real-time processing from causal streaming.
- The baseline’s reported recognition algorithmic latency requires careful distinction from translation TTFT. Matching the recognition window size alone does not establish matched translation latency; Table 7 actually compares 0.37 s against 0.32 s. Furthermore, the mapping from forced-aligned gloss timestamps to translated text tokens is unspecified, making token latency and prefix-reference construction difficult to reproduce. The claimed five-window startup delay is also not explicitly enforced in Algorithm 1. Cross-language matched-latency conclusions exceed the PHOENIX-only evidence.
- Sentence-level clips are assigned a single gloss label despite generally containing multiple glosses, without explaining how this supervision is constructed. The translator is trained using complete retrieved sequences paired with complete transcripts, but inference translates shorter boundary-delimited segments; the corresponding segment-level targets are unspecified. The construction of boundary labels and handling of remaining output at stream termination also require clarification. Finally, the SOKE citation does not adequately specify the adapted dictionary-based DTW segmentation procedure used for the English datasets.
- In Table 2, the authors report a BLEU-1 score of 63.15 on PHOENIX-2014T, which is highly questionable. Their results on other datasets also substantially exceed those of previous methods despite using pose-only input.
Questions for authors
See the weaknesses.