acceptodds
Withdrawn from ICLR 2027

Real-Time Sign Language Translation by Prototype-Assisted Causal Streaming

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sign language translation (SLT) for live captioning must produce text while the signer is still signing: each decision may use only frames already observed, and text shown to the user cannot be revised. Decisions are therefore made from short windows in which a sign may be only partly visible and pose detectors lose joints, so window-level recognition must be stabilised before it is committed. We propose a real-time, strictly causal, pose-based SLT pipeline with three components. A confidence-aware masked pose autoencoder uses detector confidence to gate unreliable joints and unreliable reconstruction targets. A part-specific vector-quantised tokenizer, with separate body, hand, and face codebooks trained by video-level contrast, block-to-video alignment, and angular-margin and centre objectives, maps short windows to discriminative sign units. A causal streaming decoder retrieves the nearest sign prototype for each window, aggregates retrievals by past-only voting, and passes the gloss stream through a causal boundary head to a Transformer translator; it reads no future frames and commits each output once. We further formalise streaming SLT through window end-times, zero lookahead, and commit-on-emit, and report time-to-first-token, token and end-to-end latency, and the quality and coverage of committed text. On Phoenix-2014, Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL, the system improves BLEU-4 over the strongest online baseline by 5.55 on Phoenix-2014T and 6.29 on CSL-Daily, keeps a 2.59 BLEU-4 lead at matched time-to-first-token, and has committed 84% of its final output after 80% of the input. Controlled ablations isolate each component, the train-side word segmentation used to build the prototype bank, and the pose detector.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
sign language translation, real-time, causal streaming pipeline, prototype retrieval, vector quantization, discriminative training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZECnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a pose-based streaming framework for real-time sign language translation. It combines a confidence-aware masked pose autoencoder, a part-specific vector-quantized tokenizer, and prototype-based sign retrieval. At inference, sliding-window predictions are stabilized through past-only majority voting and consecutive deduplication. A causal boundary head groups the resulting gloss stream into segments, which a Transformer translates into text without revising previously emitted output. Prototype banks are constructed from training clips using external segmentation procedures. Experiments on German, Chinese, and English benchmarks report improvements in recognition and translation quality, while evaluations on PHOENIX-2014T examine latency and the quality and coverage of committed text.

Critical strengths
  1. The paper addresses the important challenge of producing stable translations from short, noisy video windows without revising emitted text. Its integration of confidence-aware pose encoding, prototype retrieval, and past-only voting provides a coherent approach to this setting.
  2. The evaluation covers multiple sign languages and includes controls for segmentation quality, dictionary coverage, pose detectors, and individual model components. Reporting results across five seeds and runtime costs including pose extraction also strengthens the experimental assessment.
Critical weaknesses
  1. The writing can be improved. The manuscript is difficult to follow. Technical details, architectural components, training objectives, and evaluation claims are densely presented with insufficient hierarchy and weak transitions. The relationships between the three stages and their respective training and inference procedures are particularly unclear. Substantial restructuring is needed to make the methodology and experimental reasoning accessible.
  2. The reported performance requires stronger verification, and several quantitative results need clarification. Table 2 reports 63.15 BLEU-1 on PHOENIX-2014T, substantially exceeding the listed baselines despite using pose-only input. These gains, together with improvements on other benchmarks, warrant more transparent checkpoints, evaluation scripts, and supervision details. High scores alone do not establish an error, but there are additional reporting concerns: Table 9’s correlations cannot be reproduced from its displayed values, and Table 4 reports 8.2 WER under oracle-gloss input, where the recognition WER should be zero or inapplicable under the stated definition.
  3. The paper omits recent competitive approaches [1-6]. Particularly relevant is RTG-Net [6], which explicitly studies real-time sign language recognition and translation on edge devices. Its efficiency results should be discussed and, where feasible, compared under a common runtime protocol, while distinguishing throughput-based real-time processing from causal streaming.
  4. The baseline’s reported recognition algorithmic latency requires careful distinction from translation TTFT. Matching the recognition window size alone does not establish matched translation latency; Table 7 actually compares 0.37 s against 0.32 s. Furthermore, the mapping from forced-aligned gloss timestamps to translated text tokens is unspecified, making token latency and prefix-reference construction difficult to reproduce. The claimed five-window startup delay is also not explicitly enforced in Algorithm 1. Cross-language matched-latency conclusions exceed the PHOENIX-only evidence.
  5. Sentence-level clips are assigned a single gloss label despite generally containing multiple glosses, without explaining how this supervision is constructed. The translator is trained using complete retrieved sequences paired with complete transcripts, but inference translates shorter boundary-delimited segments; the corresponding segment-level targets are unspecified. The construction of boundary labels and handling of remaining output at stream termination also require clarification. Finally, the SOKE citation does not adequately specify the adapted dictionary-based DTW segmentation procedure used for the English datasets.
  6. In Table 2, the authors report a BLEU-1 score of 63.15 on PHOENIX-2014T, which is highly questionable. Their results on other datasets also substantially exceed those of previous methods despite using pose-only input.
Questions for authors

See the weaknesses.

Reviewer_FyatOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a three-stage framework for real-time sign language translation based on pose representations. The method combines confidence-aware masked pose encoding, part-specific vector quantization with discriminative supervision, and a prototype-assisted causal streaming decoder. During inference, gloss predictions are obtained through sliding-window prototype retrieval, stabilized using past-only majority voting and consecutive deduplication, and subsequently translated into natural language through a causal boundary predictor and a gloss-to-text Transformer.

The authors evaluate the framework on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL, reporting competitive translation performance under a zero-lookahead, commit-on-emit setting. The results are promising, but several details of the training objectives and evaluation procedures remain unclear.

Critical strengths

S1. A practical approach to streaming sign language translation.

Real-time sign language translation requires balancing translation quality against latency, which makes this a worthwhile problem to address. The zero-lookahead and commit-on-emit constraints are practically meaningful, and the proposed framework explicitly considers both recognition stability and incremental text generation.

S2. Combining pose representation learning with prototype retrieval.

The combination of confidence-aware pose encoding, part-specific VQ codebooks, discriminative representation learning, and prototype retrieval offers a structured approach to pose-based sign recognition. The use of past-only majority voting further offers an intuitive mechanism for stabilizing predictions in a streaming environment.

S3. Evaluation across multiple datasets and settings.

The experiments cover several sign language datasets and report recognition accuracy, translation quality, and latency-related metrics. The paper also includes component ablations, analyses of training-side segmentation quality, and robustness experiments. These experiments provide useful evidence of the framework's potential effectiveness.

Critical weaknesses

W1. Unclear assignment of gloss labels to sentence-level clips.

Sec. 3.3 states that word-level and sentence-level clips carrying the same gloss label are treated as positives and share a class, an ArcFace centre, and a prototype. However, a sentence-level video generally contains multiple glosses, making the assignment of a single gloss label to an entire sentence ambiguous.

This matters because the proposed supervised contrastive and ArcFace objectives operate on a single class label for each training instance. Without a clear definition of how sentence-level instances are constructed and labelled, it is difficult to determine whether the resulting supervision consistently encourages gloss-discriminative representations.

Sec. 3.4 also defines each gloss prototype as the mean embedding over both word-level and sentence-level clips carrying that label. If sentence-level representations contain information from multiple glosses, their contribution to a single-gloss prototype requires additional justification.

As multi-granularity supervision is central to the proposed representation learning framework, these missing details make it difficult to assess the validity and reproducibility of the method.

W2. An apparent inconsistency in the Stage-2 reconstruction loss.

According to Sec. 3.2, the reconstruction decoder is discarded after Stage-1 pretraining and only the encoder is retained. However, the Stage-2 objective in Eq. (7) explicitly includes a reconstruction loss.

It is therefore unclear how this loss is computed during Stage-2 training. In particular, the paper does not explain whether the reconstruction decoder is retained, reintroduced, or replaced by another prediction head.

This concerns the actual optimization procedure rather than a minor implementation detail, as the reconstruction objective forms part of the reported Stage-2 loss. A precise explanation is necessary to reproduce the training process and understand the role of reconstruction supervision in Stage 2.

W3. Unclear WER results in the oracle-gloss and deduplication experiments.

Table 4 reports 8.2% WER for the "oracle gloss in (translator only)" condition. However, Appendix D.4 states that this condition replaces the prototype-bank output with the official gloss sequence and evaluates only the gloss-to-text Transformer. Under standard gloss-level WER evaluation, directly substituting the reference gloss sequence should result in zero recognition errors, unless additional processing or a different evaluation protocol is involved.

Table 19 also reports that removing consecutive deduplication increases WER only from 18.3% to 21.4%. Given that the streaming recognizer produces highly overlapping predictions using a four-frame stride, removing deduplication would normally be expected to introduce substantial repeated-gloss insertions. The relatively small increase in WER is therefore difficult to interpret without further details about the evaluated prediction sequence.

These observations do not necessarily indicate incorrect experimental results. However, the current descriptions are insufficient to reconcile the reported WER values with the stated inference and oracle-evaluation procedures. Since these experiments are used to support the effectiveness of key components, the evaluation protocol should be made explicit.

Overall Assessment

The paper addresses an important problem, and the proposed streaming translation framework shows encouraging results. However, the unclear sentence-level supervision makes it difficult to evaluate a central part of the method. The apparent inconsistency in the reconstruction objective and the questions surrounding recognition evaluation also leave some uncertainty about the training procedure and the interpretation of the experimental results.

These issues need to be clarified before the proposed training procedure and empirical conclusions can be properly assessed.

Questions for authors

Q1. How are sentence-level clips labelled and used to construct prototypes? (W1)

Could the authors clarify how a sentence-level training clip is assigned a single gloss label, particularly when the sentence contains multiple glosses? Are sentence-level samples constructed as full videos, target-gloss-specific contextual clips, or through another procedure? How are these labels used in the supervised contrastive and ArcFace objectives, and how are sentence-level embeddings incorporated into the corresponding gloss prototypes?

Q2. How is the reconstruction loss computed during Stage 2? (W2)

Sec. 3.2 states that the reconstruction decoder is discarded after Stage-1 pretraining, while Eq. (7) includes a reconstruction loss in the Stage-2 objective. Could the authors clarify whether a reconstruction decoder remains active during Stage-2 training? If so, how is it initialized and what reconstruction targets and masking strategy are used?

Q3. How are WER values calculated in the oracle-gloss and deduplication experiments? (W3)

Why does the oracle-gloss condition in Table 4 report 8.2% WER when Appendix D.4 states that the model receives the official gloss sequence? At which point is the oracle sequence substituted, and is WER measured before or after any subsequent processing? Additionally, when consecutive deduplication is disabled in Table 19, how are the overlapping window predictions converted into the final gloss sequence used for WER evaluation?

Q4. How is TTFT measured, particularly during the initial voting period?

Could the authors clarify the exact starting point used to measure time-to-first-token (TTFT)? Is the timer started when the first video frame arrives or when the first complete input window becomes available? In particular, with a 16-frame window, a four-frame stride, and a five-window voting horizon, how is the reported TTFT of 0.66 seconds obtained? Eq. (9) permits voting over fewer than five windows during initialization, whereas Sec. 4.5 states that the first token is committed once the five-window vote has formed. It would be helpful to clarify how the initial voting period, boundary-triggered emission, and the TTFT values reported in Table 7 are handled.

Reviewer_j3JwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a three-stage, pose-based pipeline for strictly causal sign language translation. Stage 1 employs confidence-gated masked pose reconstruction; Stage 2 combines part-specific vector quantization with multiple discriminative objectives to learn short-window embeddings; Stage 3 retrieves gloss prototypes, applies past-only majority voting and deduplication, and feeds the resulting stream to a causal boundary detector and a gloss-to-text Transformer. The authors formalize window end-times and irreversible emission, and report recognition, translation, and latency results on Phoenix-2014/2014T, CSL-Daily, How2Sign, and OpenASL. On Phoenix-2014T, they report 29.24 BLEU-4 and 18.3 WER at a 0.66 s time-to-first-token (TTFT), with 26.28 BLEU-4 at a lower-latency operating point.

Critical strengths
  1. The work does not merely relabel offline SLT as “online”, it explicitly specifies past-only window access and immutable emissions, and reports TTFT, processing latency, and prefix-level quality/coverage. This is a relevant evaluation direction for live captioning, and the accuracy–latency trade-off in Table 7 is potentially valuable if the timing convention is validated.
Critical weaknesses
  1. My major concern is that the streaming timing and comparison protocol do not yet substantiate the central claim. With K=16 and stride s=4, five complete voting windows become available only at frame K+4s=32. At a 30-fps input rate, this is approximately 1.07 s of observation before processing. Yet Section 4.5 states that the first token is emitted once a five-window vote has formed while reporting TTFT=0.66 s; the same table gives TTFT=0.32 s for a baseline using a 16-frame window, shorter than the approximately 0.53 s required to observe 16 frames at 30 fps. Algorithm 1 permits voting over fewer than five windows at startup, so the discrepancy may stem from an undocumented convention, but it must be resolved with a frame-by-frame event-time trace. Further, Algorithm 1 has no end-of-stream flush for pending glosses, and its boundary head is evaluated only when a new deduplicated gloss arrives. The paper therefore does not establish what happens to a final uncommitted segment. The token-latency reference (Section 4.1 and Appendix C.2) also aligns official glosses but does not explain how those gloss timestamps map onto potentially different target-language output tokens. These are central, not peripheral, problems for an irreversible real-time system.
  2. The core representation supervision and attribution are not sufficiently defined. Section 3.3 (lines 197–201) states that both word-level and “sentence-level” clips carry a single gloss class g, share an ArcFace centre, and contribute to the prototype of that gloss (Section 3.4, Eq. 8). A complete sentence ordinarily contains multiple glosses, so the rule for assigning one class, defining positive pairs, and averaging sentence embeddings into sign-specific prototypes is unclear. Without this, the principal training objective cannot be reliably reproduced or assessed. Moreover, the loss- and MAE-specific ablations (Appendix D, Tables 17–18) are run on WLASL2000 isolated-sign recognition rather than on the paper’s primary continuous SLT tasks. Consequently, the degree to which the elaborate discriminative/VQ stack, rather than training resources or simple prototype retrieval, drives the headline SLT gains remains uncertain. The combination is also mostly built from known components; a stronger task-matched novelty/attribution test is needed.
Questions for authors
  1. Precisely how can a full sentence clip carry one gloss label? Please specify the sampling rule, gloss assignment, positive/negative sets, and prototype construction for multi-gloss sentences, with a worked training example. Please report a continuous-SLT comparison using word-only prototypes and a non-VQ embedding baseline under the same supervision and compute.
  2. What are the raw input video frame rates, time origin, buffering policy, warm-up condition, and precise definition of TTFT for every row of Tables 6–7? Please show one reproducible timestamp trace from input frame arrival through five-window voting, boundary activation, translation, and first emission. Does the first token use fewer than L=5 votes? How can the K=16 baseline emit at 0.32 s under the stated zero-lookahead regime?
  3. For comparison fairness, please provide an exact supervision/resource table for every online baseline, including whether the baseline BLEU scores come from its original paper or from attaching/retraining a gloss-to-text translator. Which systems use the same pose front-end, external sign dictionaries, pretrained recognizers, training segmentation, gloss annotations, and visual fine-tuning? In particular, justify “Vis.Ft. = No” for the proposed system in Table 2 given discriminative training on labeled sign clips, and include controlled comparisons with matched auxiliary data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.