acceptodds
Withdrawn from ICLR 2027

SignRAG: Unified Retrieval-Augmented Gloss-Free Sign Language Translation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contemporary decoder-only large language models (LLMs) have demonstrated strong capabilities across a wide range of domains. However, existing pretraining paradigms for gloss-free sign language translation (SLT) are largely designed around conventional encoder-decoder pretrained language models, which limits their direct applicability to decoder-only LLMs. To address this limitation, we propose SignRAG, a unified framework combining hierarchical pretraining, target-domain retrieval augmentation, and retrieval-aware reinforcement fine-tuning. Hierarchical pretraining first learns linguistically grounded sign representations and then jointly aligns the sign encoder with an LLM, mitigating cross-modal optimization imbalance. For downstream adaptation, SignRAG complements parameter-based fine-tuning with a target-domain retrieval gallery that provides instance-specific translation cues. To ensure that retrieved contexts are used appropriately, we further introduce Retrieval Utility-Guided Reinforcement Fine-Tuning (RUG-RFT), which combines translation-quality and retrieval-utility rewards to encourage beneficial retrieval use while suppressing harmful reliance. Experiments on multiple SLT benchmarks establish new state-of-the-art performance. In particular, to the best of our knowledge, SignRAG is the first gloss-free approach to outperform gloss-supervised methods across all reported metrics on CSL-Daily. Our code and models of different sizes will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sign Language Translation;Multimodal Pre-training;Retrieval-Augmented Generation;MLLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZCtGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SignRAG proposes a unified framework for gloss-free sign language translation using decoder-only LLMs. It combines hierarchical pretraining for visual-language alignment, target-domain retrieval augmentation for additional translation cues, and retrieval utility-guided reinforcement fine-tuning to improve the use of retrieved information. Experiments on four SLT benchmarks demonstrate good performance.

Critical strengths

1.Strong empirical results across multiple SLT benchmarks. 2.Effectively integrating retrieval augmentation and reinforcement learning into gloss-free SLT.

Critical weaknesses
  1. Although the combination of the three core components achieves strong empirical results, the paper does not clearly establish its novelty over prior work. The proposed framework appears more like an integration of existing techniques than a substantially new methodological contribution.
  2. The experiments do not sufficiently isolate the proposed contributions, and some comparisons are not fully controlled.
  3. The RUG-RFT reward design does not clearly demonstrate that it effectively encourages the use of beneficial retrieval information, rather than simply rewarding outputs that the model already favors.
  4. It remains unclear whether the substantial performance gains from hierarchical pretraining stem from its specific design or primarily from additional supervision and training stages.
Questions for authors

1.The problem addressed by Hierarchical Pretraining is very similar to that of FLa-LLM. Why is the staged training strategy in FLa-LLM insufficient, and what additional challenges are addressed by the architectural differences introduced in SignRAG? 2.What are the novel technical contributions of Target-domain Retrieval compared with existing retrieval-augmented methods? Furthermore, the paper does not sufficiently demonstrate that the proposed Attention-weighted Similarity outperforms simpler retrieval strategies, such as global feature cosine similarity. 3.RUG-RFT measures retrieval utility through likelihood differences, but an increase in likelihood does not necessarily indicate that the retrieved information contributes to correct translation. Furthermore, the reward only retains positive likelihood differences without explicitly penalizing harmful reliance on incorrect retrievals. Can the authors demonstrate that RUG-RFT effectively distinguishes between beneficial and harmful retrieved information, rather than merely increasing the model's confidence in its generated outputs?

Reviewer_saJ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to use a decoder-only LLM (Qwen3-8B) for gloss-free sign language translation. The authors argue that when a sign encoder is directly connected to a strong LLM, the language prior dominates the optimization, which they call cross-modal optimization imbalance. To address this, they propose SignRAG. First, a hierarchical pretraining is performed: the skeleton-based sign encoder is aligned with text, and then jointly pretrained with the LLM. Second, during downstream fine-tuning, translations of similar sign samples are retrieved from the target-domain training set and added to the prompt. Third, the model is further fine-tuned with GRPO (RUG-RFT), where the reward combines a translation quality term and a retrieval utility term. The authors report gloss-free state-of-the-art results on four SLT benchmarks, and on CSL-Daily the method also outperforms gloss-based methods. The ablation studies are conducted only on CSL-Daily.

Critical strengths
  1. Strong empirical results under a careful retrieval protocol. On CSL-Daily, SignRAG-RFT reaches 32.91 BLEU-4, compared with 27.42 of Geo-Sign; on OpenASL it reaches 26.62, compared with 23.14 of Uni-Sign (Table 1). The retrieval gallery is built only from the training split, the query itself is excluded, and samples with identical translations are filtered during training (Sec. 4.1, App. C.7). Without this filtering, BLEU-4 drops to 11.21 (Table 5). So the authors have clearly considered the shortcut of copying the answer from the gallery, and I think the reported gains are credible. This is the main reason I see value in this work. I also note that the gains on How2Sign and PHOENIX-2014T are much smaller, and the comparison with gloss-based methods on CSL-Daily relies on external pretraining data (CSL-News).

  2. The paper adapts the pretrain–SFT–RL pipeline of recent LLMs to SLT in a systematic way, with stage-wise evidence. With the same Qwen3-8B backbone, the SFT baseline only gets 23.46 BLEU-4 on CSL-Daily, which is lower than the previous SOTA. Adding pretraining, retrieval and RUG-RFT improves it to 29.72, 31.22 and 32.91, respectively. This shows that the improvement does not simply come from a larger LLM.

Critical weaknesses
  1. The design of the retrieval-utility reward is problematic. It compares "the current policy with retrieval" against "the frozen SFT model without retrieval", so both the model and the input are changed at the same time. When the reward goes up, it may be because retrieval helps, or simply because the policy becomes more confident after training. The two cannot be separated. The experiments show that adding this reward improves the scores, but they do not show whether the model really learns to tell useful retrieved contexts from misleading ones, which is exactly the problem RUG-RFT is supposed to solve.
  2. The pretraining comparison in Table 6 is hard to understand. Encoder-only pretraining gets 3.20 BLEU-4 and end-to-end pretraining gets 9.53, while the SFT baseline without any pretraining already gets 23.46 (Table 2a). It is not normal that pretraining makes the model so much worse. The authors should clarify how these settings differ in initialization, trainable modules and training budget. Since this experiment is the main evidence for the claim of cross-modal optimization imbalance, I do not find it convincing in its current form.
Questions for authors
  1. For the same model, how large is the performance gap with and without retrieval? When misleading retrieved contexts are given, is there any difference between the model trained with the full reward and the one trained with the quality reward only?

  2. Why are the two pretraining baselines in Table 6 lower than the SFT baseline in Table 2? Are their training settings and compute budgets comparable?

  3. Besides exact duplicates, how much near-duplication is there between test and training sentences? If the test set is grouped by similarity to the training set, how does the gap with and without retrieval change? Does retrieval still help on test sentences with low similarity?

Reviewer_nst7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the gloss-free sign language translation (SLT) task and the authors identified the challenges when applying decoder-only LLMs to this task. In light of this, the authors proposed SignRAG, a unified framework combining hierarchical pretraining, target-domain retrieval augmentation, and retrieval-aware reinforcement fine-tuning. Experiments on multiple SLT benchmarks validate the effectiveness of the proposed framework.

Critical strengths
  1. The authors argued the optimization challenge between the visual and text modalities, which is interesting.
  2. The authors designed a unified framework combining hierarchical pretraining, target-domain retrieval augmentation, and retrieval-aware reinforcement fine-tuning.
  3. Extensive experiments across multiple STL benchmarks validate the effectiveness of the proposed method.
Critical weaknesses
  1. The contribution is limited, and this work is more like an engineering project. The proposed method consists of pre-training, retrieval-augmented strategy and fine-tuning, which are general strategies in the AI domain. The technical novelty is somewhat limited.
  2. The authors identified the challenge of cross-modal optimization imbalance. However, it is unclear whether the performance improvement comes from the mitigation of the optimization imbalance. Although the authors provided an intuitive explanation in Appendix D.3, they should provide more comprehensive theoretical and experimental analyses to verify the effectiveness of the proposed method.
  3. The proposed attention-weight sign-sign similarity should be compared with alternative retrieval strategies, such as global feature pooling, cosine similarity or temporal alignment-based methods.
  4. The authors highlight the domain adaption as a key challenge. The experiments mainly focus on individual target datasets. I am wondering how well the proposed method generalizes to unseen domains. The authors are suggested to provide cross-domain evaluations to verify the generalization ability.
  5. Table 1 shows that the proposed method has limited performance gains over the baselines. Meanwhile, SignRAG-RFT is not the best method on OPENASL with R-L (43.87 VS 43.92), while the authors highlighted their method in bold as the best.
Questions for authors

See Weaknesses above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.