acceptodds
Under review as a conference paper at ICLR 2027

RefineSLT: Learning What to Recover and When to Refine in Sign Language Translation

Abstract

Existing sign language translation (SLT) models can produce fluent sentences while omitting or mistranslating content essential to the signed message, such as entities, actions, or temporal information. Addressing these errors requires identifying potentially missing or mistranslated content and directing refinement toward its visual evidence. We introduce RefineSLT, a framework that couples semantic cue prediction with learned refinement decisions for pretrained SLT models. CueMiner derives semantic cue supervision from training translations, and CueGrounder learns to predict these cues and associate them with temporal visual evidence. CueRefiner combines cue confidence with estimated coverage in the initial translation to control bounded, temporally weighted feature updates. The pretrained decoder then generates the final translation from the refined features. The backbone encoder and decoder remain frozen, and inference requires neither reference translations nor ground-truth glosses. With enhancement modules trained for each backbone, the framework is designed to accommodate different architectures, translation capabilities, and sign languages, offering a common basis for enhancing existing translators and adapting to future models. An analysis of baseline outputs on PHOENIX-2014T illustrates how errors in a few content-bearing words or phrases can alter the intended message, motivating semantic cue learning and selective refinement. Experiments on PHOENIX-2014T, How2Sign, and Auslan-Daily demonstrate consistent BLEU-4 and ROUGE-L improvements across all evaluated backbones, including relative BLEU-4 gains of 12.1% on PHOENIX-2014T and 54.2% on Auslan-Daily Communication with CV-SLT.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sign language translation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9V4SOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RefineSLT, a framework for improving semantic fidelity in pretrained sign language translation models while keeping their encoders and decoders frozen. It consists of three components: CueMiner extracts semantic cue pseudo-labels from training translations using masking-based embedding changes and an IDF prior; CueGrounder predicts cue representations and their temporal visual support from video features and initial decoding states; and CueRefiner uses cue confidence and estimated translation coverage to control residual updates to visual features. The original decoder then generates a refined translation in a second pass. Experiments on PHOENIX-2014T, How2Sign, and Auslan-Daily report consistent BLEU and ROUGE improvements across multiple backbones, evaluated on test subsets excluding reference translations of three words or fewer.

Critical strengths

(1) RefineSLT enhances pretrained translators while keeping their encoders and decoders frozen and reusing the original decoder. This makes the framework potentially useful for improving existing systems without retraining the entire backbone. (2) The paper evaluates multiple backbones across three sign language datasets and reports consistent translation improvements. Ablations with a parameter-matched adapter, random cues, and IDF-based cues provide useful controls for assessing the proposed components.

Critical weaknesses

(1) Missing comparisons with recent competitive methods. The experiments omit recent methods[1-6] and the related-work discussion overlooks [1-6], which address contextual representation learning, efficient multi-stream modeling, and feature enhancement for SLT. Discussing these approaches would better position the proposed refinement framework.
(2) The paper does not directly establish that CueMiner identifies meaning-critical content, that hidden-state similarity reliably measures translation coverage, or that cue-conditioned attention localizes the corresponding visual evidence. The ablations demonstrate component utility but provide limited support for these interpretations. Direct cue-quality, coverage-detection, and localization evaluations are needed to substantiate the central claims. (3) BLEU, ROUGE, and reference-cue exact matching do not adequately measure successful semantic correction or harmful changes to initially correct content. The reported 92% success rate also lacks a sufficiently precise definition of attempted corrections and their coverage. Furthermore, excluding references of three words or fewer limits comparability with standard benchmarks.
(4) Important details concerning cue extraction, loss formulations, training stages, and backbone configurations are missing. \textbf{ The “without gloss supervision” description and shared I3D implementation also require clarification given the original backbones’ supervision and architectures. } Finally, the lightweight claim lacks parameter and runtime measurements despite requiring two decoding passes, while limited ablation scope and missing uncertainty estimates leave experimental reliability insufficiently assessed. (5) The writing could be improved, and Figure 2 is suspected to have been generated by AI. [1]Learning effective sign features without text for gloss-free sign language translation,
[2] Mixsigngraph: A sign sequence is worth mixed graphs of nodes [3] ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation [4] Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues [5] Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction [6] SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale [7] Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation

Questions for authors

See the weaknesses.

Reviewer_3LNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RefineSLT, a framework for improving pretrained sign language translation models through semantic cue-guided feature refinement. CueMiner extracts keyword- and phrase-level semantic cues from training translations as pseudo-labels, introducing finer-grained supervision beyond sentence-level translation targets. CueGrounder learns to predict these cues and associate them with temporal visual evidence using video features and initial decoder states. CueRefiner then combines cue confidence and estimated coverage in the initial translation to guide localized residual updates to visual features, followed by a second decoding pass. The original backbone remains frozen. Experiments on PHOENIX-2014T, How2Sign, and Auslan-Daily report consistent translation improvements across multiple backbones, supported by ablation studies of the proposed components.

Critical strengths

1.RefineSLT extracts keyword- and phrase-level semantic cues from reference translations and uses them to supervise video-conditioned cue prediction and temporal grounding. This introduces finer-grained supervision without requiring additional manual cue or temporal annotations. By incorporating visual evidence, the framework is designed to recover semantic content that may be absent from the initial translation. 2.Rather than applying uniform feature updates, CueRefiner combines predicted cue confidence, estimated semantic coverage, and temporal attention to control localized residual refinement. The original backbone remains frozen, and the framework improves translation performance across multiple architectures and sign languages. The ablation results further support the contributions of cue selection, temporal grounding, and coverage-aware refinement.

Critical weaknesses

1.The relationship between semantic cue supervision and actual translation error correction remains insufficiently established. CueMiner identifies important words or phrases in reference translations, but these cues do not necessarily correspond to content omitted or mistranslated by the initial decoder. Similarly, CueRefiner estimates semantic coverage from hidden-state similarity rather than directly supervised error identification. Although the ablations support the usefulness of these components, the reported correction success rate relies on normalized exact matching and does not directly establish cue prediction accuracy, temporal grounding quality, or the reliability of missing-content detection. Consequently, it remains unclear to what extent the improvements result from accurate semantic correction rather than additional fine-grained supervision and feature adaptation. 2.The reported results exclude test samples whose reference translations contain three or fewer words, without providing the retained sample counts or results on the complete official test sets. Moreover, the paper provides limited details about how the framework accommodates substantially different backbone architectures and does not quantify the computational overhead of the additional modules and second decoding pass. The description of Table 1 as being without gloss supervision is also potentially misleading, since some original backbones were trained with gloss supervision. These issues limit the comparability and practical interpretation of the reported improvements.

Questions for authors

<1>How accurately do the predicted semantic cues identify content that is actually missing or mistranslated in the initial translation? Can the authors provide a direct evaluation of cue prediction, temporal grounding, and correction reliability, including cases where refinement changes previously correct content? <2>Can the authors further distinguish the contribution of keyword-level semantic supervision from that of coverage-aware feature refinement? In particular, a comparison with a parameter-matched, cue-supervised feature adaptation baseline would help clarify whether the proposed selective refinement mechanism provides benefits beyond the additional supervision. <3>Can the authors report results on the complete official test sets, clarify the supervision histories and architectural interfaces of the evaluated backbones, and quantify the additional parameters and inference cost introduced by RefineSLT?

Reviewer_DWN3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents RefineSLT, a post-hoc enhancement framework for pretrained sign language translation (SLT) systems. A training-only component, CueMiner, selects potentially meaning-critical words or spans from reference translations using the change in a frozen sentence embedding after masking, combined with an IDF prior. CueGrounder then learns to predict these cues from visual features and initial decoder states, and assigns them soft temporal evidence. CueRefiner estimates whether a cue is already covered by the initial translation and applies confidence- and coverage-gated residual updates to the corresponding visual features. The original frozen decoder is run a second time to produce the final translation; neither references nor gloss labels are needed at inference. The authors evaluate the approach on PHOENIX-2014T, How2Sign, and two Auslan-Daily subsets, shows clear improvements across all evaluated backbones.

Critical strengths
  1. The improvements across all evaluated backbones are impressive. The same refinement recipe is assessed on three sign languages and several different translation backbones. For example, PHOENIX-2014T test BLEU-4 increases from 28.15 to 31.56 with CV-SLT, and Auslan-Daily Communication BLEU-4 increases from 7.16 to 11.04 with CV-SLT. The breadth makes the framework a potentially useful model-enhancement strategy, rather than an isolated gain for one architecture.
Critical weaknesses
  1. The main semantic-fidelity claim is not established by the evaluation. The paper motivates the method using meaning-changing errors involving entities, numbers, negation, and other critical content, yet its main outcomes are BLEU-1–4 and ROUGE-L (Sec. 4.1). The “Success” metric in Table 3 is normalized exact inclusion of a reference cue only among attempted corrections, not an independent measure of whether the signed message was faithfully recovered. It does not reveal the fraction of missed cues attempted, false interventions, newly introduced semantic errors, or changes that preserve meaning through paraphrase. Likewise, selecting cues via sentence-embedding change and IDF (Eq. 1) is not validated against human judgments of semantic importance; embedding similarity may be insensitive to precisely the numerical, polarity, and relational distinctions motivating the work. The qualitative cases in Fig. 3 are promising but cannot resolve this issue.
  2. The effectiveness of the key components is not sufficiently demonstrated. CueGrounder learns soft attention with sentence-level contrastive supervision (Eq. 2); no cue-prediction precision/recall, annotated temporal-localization evaluation, attention-quality analysis, or calibrated coverage/gate measurements are reported. The global-pooling ablation establishes that a particular learned variant is useful, not that its attended frames constitute correct evidence. Moreover, the generic-adapter baseline in Table 3 scores below the frozen backbone (27.10 versus 27.62 BLEU-4), so it is not a convincing competitive control for the two-pass trainable adaptation afforded to RefineSLT. Comparisons to stronger equally trained refinement or deliberation controls, tests with shuffled cue/frame associations, and the costs of a second decoder pass are missing. Finally, all reported test sets are filtered using reference length (>3 words; Sec. 4), without the retained sample counts, unfiltered scores, or variability across runs. These omissions make the magnitude, cause, and deployment relevance of the gains difficult to assess.
Questions for authors
  1. On a manually annotated, held-out sample, what are CueMiner’s precision/recall for meaning-critical cues, especially numbers, named entities, negation, and semantic-role changes? How much of the initial semantic error mass is actually corrected, and how often are initially correct facts degraded? Please report an independent semantic-faithfulness assessment (ideally involving qualified sign-language evaluators) rather than only n-gram overlap and exact cue inclusion.
  2. Can the authors quantitatively evaluate predicted cue accuracy, temporal localization against a modest annotated subset, and the calibration of confidence p_i and coverage ρ_i? In Eq. 3, why should maximal similarity to any decoder state reliably distinguish a correct mention of a cue from a contradictory or incomplete one? What happens when visual grounding weights or cue assignments are randomly shuffled?
  3. For each dataset, how many test examples were excluded by the reference-length filter, and what are the results on the full official test sets? Can the authors report results across multiple random seeds or paired bootstrap intervals, particularly on the relatively small/low-BLEU Auslan splits? Were all original backbones independently rerun under identical feature extraction, tokenization, and decoding, and are all their weights frozen during refinement training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.