acceptodds
Withdrawn from ICLR 2027

SignXray: Plug-and-Play Sign Encoding in a Face-Hand-Body Multi-Prior Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sign languages convey information through coordinated facial expressions, hand articulation, body motion, and their temporal interactions. Existing human-representation models expose complementary aspects of this signal through heterogeneous outputs that are difficult to reuse through a common video interface. We present **SignXray**, a plug-and-play sign video encoder learned in a unified Face-Hand-Body multi-prior space. During training, SignXray consolidates supervision from SMPLer-X, DWPose, MediaPipe, PrimeDepth, and Sapiens Segmentation into a compact, sign-aware representation and distills this heterogeneous prior knowledge into an RGB ViT. At inference time, the resulting encoder maps RGB video directly into the unified representation, giving downstream models access to the complementary information of all five prior formats through a single reusable feature interface. Experimental results demonstrate that SignXray achieves state-of-the-art performance on continuous SLR and translation tasks, with up to a 50× acceleration over the evaluated pixel-space baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sign Language Translation, Sign Language Recognition, Multi-Prior Representation Learning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7Z7MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SignXray, a plug-and-play sign video encoder that integrates five heterogeneous human-centric priors into a unified Face-Hand-Body representation. The method first learns a sign-aware latent space through multi-prior fusion and semantic supervision, then distills this representation into an RGB-based ViT encoder for downstream inference without external prior extractors. Experiments across sign language recognition and translation benchmarks demonstrate consistent improvements over multiple baseline architectures, together with reported inference acceleration.

Critical strengths
  1. The paper integrates heterogeneous visual information into a unified 2048-dimensional representation and employs two-stage distillation to enable RGB-only inference, offering practical value for downstream applications.
  2. The paper provides extensive experimental evaluation, covering continuous sign language recognition, isolated sign language recognition, and sign language translation across multiple model architectures and datasets.
Critical weaknesses
  1. Compared with existing multi-stream sign language representation learning methods such as Uni-Sign, this paper mainly relies on existing visual prior extractors, feature fusion, and knowledge distillation techniques. What are the substantial technical innovations of the proposed unified Face-Hand-Body representation?
  2. Does the proposed multi-prior fusion method outperform simpler approaches, such as direct feature concatenation?
  3. The paper lacks sufficient independent ablation studies on semantic supervision, contrastive learning, feature distillation, and dimension masking, making it difficult to assess the individual contribution of each component to the overall performance.
  4. Although the paper reports significant inference speed improvements, it lacks a detailed runtime breakdown of the encoder, temporal modeling module, and decoder. The effects of changes in feature sequence length and the contributions of different components to the overall acceleration remain unclear.
Questions for authors
  1. Is the pretrained SignXray encoder frozen or fine-tuned when transferred to different downstream tasks and datasets? How does this choice affect its generalization across sign languages?
  2. In Appendix C.2, the GlossAlign operation uses ground-truth gloss annotations for temporal alignment. How is this alignment handled for datasets without gloss annotations, particularly in gloss-free sign language translation?
Reviewer_mks3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SignXray, a sign language video encoder trained through multi-prior knowledge distillation. The framework integrates five pretrained human-representation models to capture complementary body, hand, face, depth, and segmentation cues. This method consists of two training stages. In Stage 1, heterogeneous visual priors are integrated into a unified 2,048-dimensional representation space with auxiliary gloss-related semantic supervision. In Stage 2, an RGB-based ViT is trained to reproduce the frozen teacher representations through feature regression, removing the need to execute the external prior extractors during inference. SignXray is evaluated on continuous sign language recognition (CSLR), isolated sign language recognition (ISLR), and sign language translation (SLT). The paper reports improvements across multiple benchmarks and downstream architectures.

Critical strengths

<1>The paper addresses a relevant practical problem, which consolidates information from multiple pretrained human-representation models into a single RGB video encoder. The trained model is plug-and-play and avoids the execution of external prior extractors during inference. <2> The evaluation covers CSLR, ISLR, and SLT across multiple datasets and downstream architectures. The reported performance improvements suggest that the learned representations can be useful beyond a single task-specific model.

Critical weaknesses

<1>The paper identifies heterogeneous human-representation outputs as a central challenge but does not sufficiently explain why existing techniques are inadequate for addressing this problem.

<2>Insufficient methodological clarity and controlled validation of the core design choices. Figure 2 does not clearly represent several essential components of the training pipeline, such as the explicit relationship between teacher and student features. What’s more, although the paper includes hyperparameter analyses, component-level comparisons in Figure 5, and leave-one-prior-out ablations in Table 5, these experiments do not adequately isolate the necessity of the key design choices.

Questions for authors

<1>Does replacing an existing visual encoder with SignXray require retraining or fine-tuning the downstream task head? Which components are frozen or updated?

<2>Could the authors provide additional comparisons under matched visual backbones, gloss supervision, and training budgets?

<3>Appendix E.1 reports a WLASL-2000 train/development/test split of 8,618/479/478 samples, which is substantially smaller than the commonly reported size of the original WLASL-2000 dataset. Could the authors clarify the filtering procedure and confirm whether the comparisons use identical evaluation protocols?

Reviewer_KAr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a plug and play video encoder composing prior heterogeneous human-representation priors that connects a unified face-body-hand latent representation. The framework is divided into two stages. In stage 1, a 2048 dimensional representation is achieved via source-specific encoders and attention-based fusion of SMPLer-X, DWPose, MediaPipe, PrimeDepth, and Sapiens Segmentation. This frozen representation is used as supervision target in stage 2 where an RGB-based spatiotemporal ViT is trained. This allowed authors to avoid running the five computationally expensive prior extractor during inference. They have evaluated the method in ASLLRP, PHOENIX2014-T, CSL-Daily, WLASL-2000, OpenASL, How2Sign and multilingual SignSuisse configurations.

In SLTUNET, BLEU-4 jumped from 28.47 to 29.91 in PHOENIX2014-T, 25.01 to 28.58 in CSL-Daily, and in Uni-Sign, 23.14 to 24.92 in OpenASL. The recognition accuracy increased to 68.29% from 64.82% in WLASL-2000. The reported inference acceleration is 48.4 times in SLTUNET configuration. The main contribution of the paper is a reusable multi-prior representation learning framework which can be used in many downstream architectures.

Critical strengths
  1. Tables 2-4, 8-11 shows the most important strength of the paper by the improvements on nearly every reported metric in different sign language datasets, tasks, and downstream architectures, which implies generalizability.
  2. The second important strength is to convert heterogeneous human representation models into a single RGB encoder, replacing expensive prior extractors at inference and makes the encoder deployable.
Critical weaknesses
  1. The methodological contribution of the unified distillation of the heterogeneous priors is not clear yet. Fig. 5(b) does not separate the stage-1 codebook semantic supervision from the priors themselves. Table 5 shows the marginal cost of each prior, but neither experiment compares against a single-prior teacher or plain concatenation.
  2. The architecture and metric of the reported bootstrap mean gain and confidence interval under PHOENIX2014-T are not clearly identified. The efficiency assessment of SLTUNET also lacks a detailed computational breakdown of the baseline and the proposed configuration.
Questions for authors
  1. Please provide a controlled comparison with the strongest single-prior teacher, simple concatenation and multi-prior fusion keeping the same RGB ViT capacity and training budget. Specifically, show a variant without semantic supervision of stage-1 codebook.
  2. Eq. 2 and Appendix F shows Z as Tx2048 on a shared temporal grid implying one feature per frame. Sec. 3.3 says each output feature covers approximately 5-10 input frames and Fig. 8 shows downsampling. Which is correct and in which stage does that reduction occur?
  3. Please clarify whether the five test evaluations are on independent training checkpoints or repeated inference of the same checkpoint. Furthermore, which architecture and metric does the +3.27 bootstrap gain refer to?
  4. Please provide the latency breakdown of each component for the SLTUNET baseline mentioned in Table 6 and report the number of GPUs used in each configuration of the mentioned eight-GPU system.

Comments on the paper

1 comment
Reviewer_7Z7MOpenReview ↗

Official Comment by Reviewer 7Z7M

Thank you for the detailed responses and for providing the additional experiments and clarifications. However, I still have the following concerns.

I do not fully agree with the claim that “SignXray introduces a new paradigm that compiles heterogeneous expert pipelines into a single RGB encoder.” DUNE [1] has already explored distilling complementary knowledge from multiple heterogeneous experts into a single visual encoder. Similarly, SignRep [2] uses training-time priors to enhance RGB representations while eliminating the need for prior extractors during inference. Therefore, the general idea has already been explored in previous work. I acknowledge that SignXray differs in the diversity and heterogeneity of the prior information it incorporates. However, What is the substantive technical advance of the proposed method in terms of its technical mechanism? Specifically, does fusing the five priors through the proposed approach actually learn new complementary relationships, rather than simply providing additional supervision signals? Why are these five priors fused specifically through the proposed approach? Table 5 only presents five leave-one-out experiments. At least reasonable subset comparisons are needed to sufficiently support the claim of complementarity. I would also like to clarify the comparison with Uni-Sign. To my understanding, Uni-Sign does not use only 2D keypoints. In its RGB-pose setting, it also extracts RGB hand-region features and employs Prior-Guided Fusion (PGF) to integrate RGB and pose features. Therefore, Uni-Sign already incorporates a multimodal representation combining Face-Hand-Body pose information and RGB features. [1] Sarıyıldız M B, Weinzaepfel P, Lucas T, et al. Dune: Distilling a universal encoder from heterogeneous 2d and 3d teachers[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 30084-30094. [2] Wong R, Camgoz N C, Bowden R. Signrep: Enhancing self-supervised sign representations[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 22804-22814.

Regarding Weakness 4: Although the authors provide an additional runtime breakdown, an important issue remains unclear. The SignXray encoder still outputs a sequence of length (T), and the reduction from (T) to (T/4) occurs inside the downstream temporal module. Therefore, it has not been demonstrated that the SignXray encoder itself reduces the number of tokens passed to the downstream modules. Please report, in as much detail as possible, the actual number of tokens at the output of the visual encoder, the sequence length entering the Temporal Module, and the sequence length entering the Decoder for both models. In addition, what operations are included in the reported 12 ms/frame preprocessing time?

I therefore maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.