- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SignXray, a plug-and-play sign video encoder that integrates five heterogeneous human-centric priors into a unified Face-Hand-Body representation. The method first learns a sign-aware latent space through multi-prior fusion and semantic supervision, then distills this representation into an RGB-based ViT encoder for downstream inference without external prior extractors. Experiments across sign language recognition and translation benchmarks demonstrate consistent improvements over multiple baseline architectures, together with reported inference acceleration.
Critical strengths
- The paper integrates heterogeneous visual information into a unified 2048-dimensional representation and employs two-stage distillation to enable RGB-only inference, offering practical value for downstream applications.
- The paper provides extensive experimental evaluation, covering continuous sign language recognition, isolated sign language recognition, and sign language translation across multiple model architectures and datasets.
Critical weaknesses
- Compared with existing multi-stream sign language representation learning methods such as Uni-Sign, this paper mainly relies on existing visual prior extractors, feature fusion, and knowledge distillation techniques. What are the substantial technical innovations of the proposed unified Face-Hand-Body representation?
- Does the proposed multi-prior fusion method outperform simpler approaches, such as direct feature concatenation?
- The paper lacks sufficient independent ablation studies on semantic supervision, contrastive learning, feature distillation, and dimension masking, making it difficult to assess the individual contribution of each component to the overall performance.
- Although the paper reports significant inference speed improvements, it lacks a detailed runtime breakdown of the encoder, temporal modeling module, and decoder. The effects of changes in feature sequence length and the contributions of different components to the overall acceleration remain unclear.
Questions for authors
- Is the pretrained SignXray encoder frozen or fine-tuned when transferred to different downstream tasks and datasets? How does this choice affect its generalization across sign languages?
- In Appendix C.2, the GlossAlign operation uses ground-truth gloss annotations for temporal alignment. How is this alignment handled for datasets without gloss annotations, particularly in gloss-free sign language translation?
Official Comment by Reviewer 7Z7M
Thank you for the detailed responses and for providing the additional experiments and clarifications. However, I still have the following concerns.
I do not fully agree with the claim that “SignXray introduces a new paradigm that compiles heterogeneous expert pipelines into a single RGB encoder.” DUNE [1] has already explored distilling complementary knowledge from multiple heterogeneous experts into a single visual encoder. Similarly, SignRep [2] uses training-time priors to enhance RGB representations while eliminating the need for prior extractors during inference. Therefore, the general idea has already been explored in previous work. I acknowledge that SignXray differs in the diversity and heterogeneity of the prior information it incorporates. However, What is the substantive technical advance of the proposed method in terms of its technical mechanism? Specifically, does fusing the five priors through the proposed approach actually learn new complementary relationships, rather than simply providing additional supervision signals? Why are these five priors fused specifically through the proposed approach? Table 5 only presents five leave-one-out experiments. At least reasonable subset comparisons are needed to sufficiently support the claim of complementarity. I would also like to clarify the comparison with Uni-Sign. To my understanding, Uni-Sign does not use only 2D keypoints. In its RGB-pose setting, it also extracts RGB hand-region features and employs Prior-Guided Fusion (PGF) to integrate RGB and pose features. Therefore, Uni-Sign already incorporates a multimodal representation combining Face-Hand-Body pose information and RGB features. [1] Sarıyıldız M B, Weinzaepfel P, Lucas T, et al. Dune: Distilling a universal encoder from heterogeneous 2d and 3d teachers[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 30084-30094. [2] Wong R, Camgoz N C, Bowden R. Signrep: Enhancing self-supervised sign representations[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 22804-22814.
Regarding Weakness 4: Although the authors provide an additional runtime breakdown, an important issue remains unclear. The SignXray encoder still outputs a sequence of length (T), and the reduction from (T) to (T/4) occurs inside the downstream temporal module. Therefore, it has not been demonstrated that the SignXray encoder itself reduces the number of tokens passed to the downstream modules. Please report, in as much detail as possible, the actual number of tokens at the output of the visual encoder, the sequence length entering the Temporal Module, and the sequence length entering the Decoder for both models. In addition, what operations are included in the reported 12 ms/frame preprocessing time?
I therefore maintain my original score.