acceptodds
Withdrawn from ICLR 2027

SignRet: Sign Language Retrieval via Fine-Grained Gesture Descriptions and Ranking Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sign language retrieval aims to establish bidirectional correspondence between sign language videos and natural language sentences. Existing methods typically match global video–text representations or aggregate local clip–word similarities. Global matching may overlook subtle gesture differences, while local matching relies on how well these details are encoded in the visual features. Moreover, both paradigms typically determine the final ranking from independently computed query–candidate similarities, without explicitly modeling the score and rank structure of the retrieved candidate list. Consequently, they may struggle to identify the correct match when multiple candidates exhibit similar visual patterns. To address these limitations, we propose SignRet, a sign language retrieval framework consisting of Sign Language Description Enhancement and Candidate Ranking Refinement. The former incorporates frame-level gesture descriptions into temporally corresponding visual features, enriching video representations with explicit fine-grained cues for more discriminative cross-modal matching. The latter revisits the top candidates obtained from the initial matching and models their contextual relationships to recalibrate candidate relevance, thereby improving the reliability of the final retrieval results. Experiments on PHOENIX-2014T, How2Sign, and CSL-Daily demonstrate that SignRet consistently improves both retrieval directions and achieves strong performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
sign language retrieval, video-text retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KpSxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SignRet for bidirectional sign-language–text retrieval, combining (i) Sign Language Description Enhancement (SDE), which injects MLLM-generated frame-level gesture descriptions into RGB clip features via temporally aligned local cross-attention, and (ii) Candidate Ranking Refinement (CRR), which applies listwise contextual re-scoring to top-K candidates using relative scores, reverse-matching statistics, and global similarities. Experiments on PHOENIX-2014T, How2Sign, and CSL-Daily show consistent gains over SEDS (e.g., +4.5/+3.8 R@1 on PHOENIX-2014T) with modest latency overhead.

Critical strengths

The core strength is the complementary two-level design: SDE addresses the representation bottleneck (fine-grained gesture cues missing from RGB) while CRR addresses the ranking bottleneck (listwise context ignored by pairwise scoring). This cleanly disentangles two distinct failure modes, and the component ablations (Tables 2, 6, 7) convincingly show they are complementary (SDE alone +2.1/+3.1, CRR alone +3.8/+0.7, both +6.5/+4.7 R@1 on CSL-Daily RGB). Second, the method is practically efficient — descriptions are generated offline and CRR adds only ~2.6–2.8 ms/query over the baseline (Table 8), making the gains cheap to obtain. Together these make the approach consequential: it is a realistic, deployable refinement of a strong existing backbone rather than a costly redesign.

Critical weaknesses
  1. The English language needs to be revised, as there are obvious traces of AI translation. The abstract contains garbled citation strings and machine-translation artifacts (e.g., "(Jang et al.,2025;Guan et al.,2025;Guo et al., 2025;\ i;\ \ ;\ \ {i t\ \ l a l,\ 025…"), equations are broken in multiple places (e.g., "D__{Ω_{l}},,", "V22" vs. V2T, "δ>0\ s the required margin"), and spacing/punctuation errors recur throughout (e.g., "videos.Natural"). Given ICLR's explicit scrutiny of AI-polished text, this level of drafting care undermines the paper's credibility and requires full revision.
  2. Table 1 should include the references for the comparison methods, and it would be even better if the publication years of these comparison methods could also be clearly indicated. Currently the main comparison table lists SA-SR, SA-CM, SA-COMB, GFSLT-VLP, CiCo, UPRet, and SEDS without citations or years, forcing readers to guess their provenance and recency. As the central evidence for the claimed state-of-the-art status, Table 1 must be self-contained and traceable.
  3. The comparative experiment was only conducted in comparison with one method involving pose (SEDS), which is clearly insufficient. The pose-based ablation (Table 5) appears only on PHOENIX-2014T, and the main tables contain no other pose-based competitor. Consequently, the gains attributed to SignRet could be confounded with the contribution of the pose stream itself; without multiple pose-based baselines (e.g., pose-enhanced variants of CiCo or C2RL), the source of improvement is not cleanly isolated.
  4. Due to the lack of recent work, the advancement of this paper has been questioned. Most baselines date from 2023–2024 (CiCo, SEDS, UPRet); recent sign-language retrieval advances (e.g., SignCLIP's multilingual contrastive pretraining, hard-negative mining approaches cited in the related work) and strong recent generic text–video retrieval methods with demonstrated transferability are cited but never evaluated. The claimed SOTA position is therefore not convincingly established against the current frontier.
  5. There is a lack of quality and robustness analysis for the reliance on the descriptions generated by MLLM. The frame-level descriptions are taken wholesale from MMSLT's LLaVA-OneVision setup, yet there is no analysis of description accuracy, failure modes, or sensitivity to the prompt — even though a single mis-described frame (e.g., wrong handshape) directly contaminates the enhanced clip features. Nor is there an experiment replacing LLaVA-OneVision with another MLLM to show the framework is not tied to one captioner. Combined with the transductive use of full-split ranking statistics in CRR (potential train/test information leakage without discussion), these omissions weaken the robustness claims.
Questions for authors
  1. The English language needs to be revised, as there are obvious traces of AI translation. The abstract contains garbled citation strings and machine-translation artifacts (e.g., "(Jang et al.,2025;Guan et al.,2025;Guo et al., 2025;\ i;\ \ ;\ \ {i t\ \ l a l,\ 025…"), equations are broken in multiple places (e.g., "D__{Ω_{l}},,", "V22" vs. V2T, "δ>0\ s the required margin"), and spacing/punctuation errors recur throughout (e.g., "videos.Natural"). Given ICLR's explicit scrutiny of AI-polished text, this level of drafting care undermines the paper's credibility and requires full revision.
  2. Table 1 should include the references for the comparison methods, and it would be even better if the publication years of these comparison methods could also be clearly indicated. Currently the main comparison table lists SA-SR, SA-CM, SA-COMB, GFSLT-VLP, CiCo, UPRet, and SEDS without citations or years, forcing readers to guess their provenance and recency. As the central evidence for the claimed state-of-the-art status, Table 1 must be self-contained and traceable.
  3. The comparative experiment was only conducted in comparison with one method involving pose (SEDS), which is clearly insufficient. The pose-based ablation (Table 5) appears only on PHOENIX-2014T, and the main tables contain no other pose-based competitor. Consequently, the gains attributed to SignRet could be confounded with the contribution of the pose stream itself; without multiple pose-based baselines (e.g., pose-enhanced variants of CiCo or C2RL), the source of improvement is not cleanly isolated.
  4. Due to the lack of recent work, the advancement of this paper has been questioned. Most baselines date from 2023–2024 (CiCo, SEDS, UPRet); recent sign-language retrieval advances (e.g., SignCLIP's multilingual contrastive pretraining, hard-negative mining approaches cited in the related work) and strong recent generic text–video retrieval methods with demonstrated transferability are cited but never evaluated. The claimed SOTA position is therefore not convincingly established against the current frontier.
  5. There is a lack of quality and robustness analysis for the reliance on the descriptions generated by MLLM. The frame-level descriptions are taken wholesale from MMSLT's LLaVA-OneVision setup, yet there is no analysis of description accuracy, failure modes, or sensitivity to the prompt — even though a single mis-described frame (e.g., wrong handshape) directly contaminates the enhanced clip features. Nor is there an experiment replacing LLaVA-OneVision with another MLLM to show the framework is not tied to one captioner. Combined with the transductive use of full-split ranking statistics in CRR (potential train/test information leakage without discussion), these omissions weaken the robustness claims.
Reviewer_snbSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces two independent methods for sentence-level sign language retrieval, where sign language videos are matched with their natural-language sentence translations.

The first contribution, Sign Language Description Enhancement (SDE), uses an MLLM to generate frame-level sign language descriptions and incorporates these descriptions into visual representations through temporally aligned attention.

The second contribution, Candidate Ranking Refinement (CRR), is a reranking method that uses relative and reverse-normalized similarity scores, together with other matching features, to refine the top retrieved candidates.

Although presented as a unified framework, the two contributions appear relatively independent, with distinct motivations and training stages. I find the connection between them rather loose, which makes it difficult to identify a coherent methodological contribution beyond combining two improvements to the retrieval pipeline.

Critical strengths

The paper reports consistent improvements on three sign language retrieval benchmarks, and the component ablations provide evidence that both proposed modifications contribute to the final performance:

I particularly found SDE reasonably well motivated. While the novelty over Kim et al. (2025) is limited, adapting these descriptions to sentence-level retrieval through local temporal alignment is a meaningful direction.

Critical weaknesses
  1. Limited novelty and unclear connection between contributions.

The SDE component builds directly on Kim et al. (2025), which already introduced MLLM-generated descriptions of sign language components and their integration with visual representations. The main distinction here appears to be the local temporal fusion mechanism for retrieval. While useful, I find this to be a relatively narrow contribution over the existing work.

The CRR component addresses a substantially different problem, and the two methods are trained in separate stages. The paper does not clearly establish why these two contributions should be considered together beyond their complementary empirical improvements. I would have appreciated a stronger common motivation or a more explicit methodological connection between them.

  1. Dataset-specific choice of the SDE fusion weight.

Equation 2 introduces a fusion weight, γ, which is set to 0.05 for PHOENIX-2014T and 0.02 for How2Sign and CSL-Daily. No sensitivity analysis is provided to establish how strongly the results depend on this dataset-specific hyperparameter.

  1. Insufficient discussion of prior context-based rescoring and normalization methods.

The motivation behind CRR, particularly the use of reverse-normalized similarities and other candidates' scores to refine retrieval predictions, is related to earlier context-based rescoring methods for sign language retrieval. [1] previously explored using predictions for other keywords to rescore word-level retrieval in sign-language videos, through both statistical context modeling and a learned MLP. From what I understand, the main differences are that (i) [1] addresses word-level rather than sentence-level retrieval, and (ii) [1] learns its context model from training data and applies it over a fixed keyword vocabulary, whereas SignRet's CRR uses similarity statistics over the complete evaluation collection, introducing an additional closed-set dependency.

This dependency is particularly questionable given that prior work in multimodal retrieval [2, 3] has already introduced normalization strategies that do not require access to the complete test query set. It is unclear why CRR instead relies on statistics from the complete evaluation collection.

I find the absence of this prior work from the discussion surprising, given the paper's emphasis on contextual rescoring. The authors should clarify how CRR relates to these earlier approaches and why requiring such a closed-set dependency is preferable to alternatives that avoid it.

  1. CRR relies on a closed evaluation set, which does not align well with the retrieval task motivation.

This is my main concern.

As I understand Equation 4, CRR computes reverse-normalized similarities using statistics over the complete retrieval set of the evaluation split. In addition, Equation 7 combines corrections from both text-to-video and video-to-text candidate lists. Consequently, the final retrieval scores depend on access to the full collection of evaluation sentences and videos.

This assumption seems to conflict with the intended motivation of sign language retrieval as discussed in the paper's introduction. In a realistic retrieval system, a user may submit an arbitrary natural-language query to search an existing collection of sign language videos. The complete set of future queries, and therefore their similarity distributions against the video collection, would not be available in advance.

While the proposed method may be appropriate for closed-set retrieval benchmarks, this assumption is not sufficiently discussed or reconciled with the motivation in the introduction.

I think an additional ablation is necessary to understand whether the improvements from CRR reflect a generally useful ranking mechanism or primarily depend on transductive access to the complete evaluation collection.

One possible experiment would be to construct a fixed reference set of natural-language sentences independently of the test set, for example by prompting an external language model to generate diverse sentences similar in domain and style, but without using the exact test translations. The authors could then compute reverse-normalization statistics using this reference set and evaluate independently arriving test queries without access to the complete test query collection. This would also require adapting the reverse-direction score corrections that depend on the unavailable test sentences and ensuring that the modified CRR is trained under the same reference-set protocol.

Such an experiment would help establish whether CRR remains effective in a more realistic open-query setting. I consider this distinction important for assessing the broader applicability of the proposed method.

[1] Tamer and Saraçlar (2020), Cross-Lingual Keyword Search for Sign Language.

[2] Bogolin et al. (2022), Cross Modal Retrieval with Querybank Normalisation.

[3] Wang et al. (2023), Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks.

Questions for authors
  1. Closed-set dependency of CRR. Does CRR require access to the complete evaluation query collection at inference time, as Equations 4 and 7 appear to imply? If so, can the authors evaluate whether its improvements persist when using a fixed, test-independent reference set instead?
Reviewer_YdueOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SignRet, a framework for bidirectional sign language retrieval that combines fine-grained gesture description enhancement with candidate ranking refinement. Specifically, Sign Language Description Enhancement (SDE) uses a frozen multimodal language model to generate frame-level gesture descriptions and integrates them into visual representations through temporally aligned cross-attention. Candidate Ranking Refinement (CRR) then uses relative matching scores, ranking statistics, and a listwise Transformer to refine the initial retrieval results. Experiments on PHOENIX-2014T, How2Sign, and CSL-Daily demonstrate consistent improvements over the evaluated baselines, supported by component ablations and additional analyses.

Critical strengths
  1. Clear motivation. The method is clearly presented, and the additional analyses of feature fusion, description granularity, and retrieval efficiency provide empirical evidence.
  2. Consistent empirical improvements. The proposed framework achieves improvements across three sign language retrieval datasets in both text-to-video and video-to-text retrieval.
Critical weaknesses
  1. Limited novelty. Both SDE and CRR have direct precedents (Cap4Video, NarVid, CoreCaption for description-enhanced retrieval; Wang et al. 2019 and Yuan et al. 2022 for reranking), but the paper only cites them without comparison. The contribution is closer to "combining existing ideas for sign language retrieval" than a methodological advance.
  2. Insufficient validation of the proposed ranking refinement mechanism. The main experimental comparisons are largely centered on established retrieval baselines such as SEDS. Although the paper discusses more recent approaches, including sign-aware hard-negative mining (2026) and description-enhanced retrieval methods, these are not adequately evaluated under comparable settings.
  3. Incomplete comparisons. The paper cites SignCLIP (2024b), C²RL (2025), and Lee et al. (2026) in Related Work, but none appears in Table 1. The most recent method actually compared is SEDS (2024a), so the empirical comparison does not cover the 2025–2026 developments.
  4. Description reliability is unverified, and SDE's causal attribution is ambiguous. The paper provides no quantitative measure of how accurately the MLLM describes handshapes and gestures, and no shuffled/generic-description control, so it cannot distinguish gains from accurate gesture cues versus an auxiliary-text regularization effect.
  5. Offline cost is not quantified. Appendix E reports only online latency and explicitly excludes LLaVA-OneVision-7B inference. Running a 7B model on every sampled frame may dominate total pipeline cost, yet no GPU-hours, storage, or scaling with gallery size is reported.
Questions for authors
  1. Novelty: How do SDE and CRR fundamentally differ from existing description-enhanced retrieval and listwise reranking methods?
  2. Recent baselines: Can the authors provide comparisons with more recent sign language retrieval methods, particularly the 2026 sign-aware hard-negative mining approach?
  3. CRR effectiveness: Does CRR outperform simpler reranking methods, such as MLP-based score refinement, under identical settings?
  4. Description reliability: How reliable are the MLLM-generated descriptions in capturing subtle hand configurations and temporal gestures?
  5. Efficiency: What are the computational costs of offline description generation and retrieval on larger galleries?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.