acceptodds
Withdrawn from ICLR 2027

Multi-Granular Semantic Matching with Uncertainty-Aware Calibration for Sign Language Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sign language retrieval requires precise cross‑modal alignment between sign and textual semantics. However, existing methods typically rely on either global or fine‑grained alignment between text and sign video, making it difficult to jointly capture overall event semantics and subtle motion details. Moreover, given a text (respectively a sign video), there are multiple semantically consistent sign videos (respectively texts), and the same sign articulation may convey distinct semantics across different contexts. This correspondence multiplicity and contextual ambiguity introduce uncertainty into sign‑text matching, leading to sub‑optimal semantic alignment. To address these challenges, we propose MGUC, a multi‑granular semantic matching framework with uncertainty‑aware calibration for sign language retrieval. Specifically, coarse‑grained global matching captures overall event semantics and ensures holistic sign‑text consistency. Fine‑grained local matching captures subtle motion details by jointly modeling cross‑modal relevance and local feature interactions through bidirectional similarity aggregation, with learnable [Null] tokens mitigating spurious matches arising from transitional clips and words without explicit sign gestures. We further introduce an uncertainty‑aware modeling and calibration module to capture uncertainty in sign‑text correspondences and refine cross‑modal alignment. It represents sign videos and texts as multivariate Gaussian distributions and uses Bhattacharyya distances to quantify their distributional discrepancy. The relative distance between unpaired and paired samples is used to adaptively calibrate the negative contributions during contrastive learning. This reduces excessive repulsion from potentially semantically related samples, thereby refining text‑to‑sign alignment in the shared embedding space. Extensive experiments on three widely used benchmarks demonstrate that MGUC achieves competitive retrieval performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sign Language Retrieval, Cross-Modal Semantic Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_BQGbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MGUC, a multi-granular semantic matching framework with uncertainty-aware calibration for sign language retrieval. It combines global semantic matching with fine-grained local matching, using bidirectional interaction-aware similarity aggregation and learnable [Null] tokens to mitigate spurious local correspondences. Sign videos and text queries are also represented as Gaussian distributions, whose relative Bhattacharyya distances calibrate the negative contributions of unpaired samples during contrastive learning. Experiments provide empirical support for effectiveness.

Critical strengths
  1. The global and fine-grained branches capture complementary aspects of sign–text correspondence, combining overall event semantics with local motion details.
  2. PDM and UACC form a well-motivated training strategy: Gaussian representations guide the adaptive weighting of unpaired samples, helping reduce excessive repulsion from semantically related candidates.
  3. Results on three benchmarks, together with ablations and the supplementary human evaluation, support the effectiveness of the method.
Critical weaknesses
  1. The paper should discuss PCME++ (ICLR 2024), which also studies probabilistic representations, closed-form distances, and false negatives. The differences and additional benefits of MGUC’s distributional modeling and calibration need to be clearer.
  2. Table 2 and Appendix A.7 support UACC, but the contribution of learned covariances remains unclear. Comparing against mean-only distance-based calibration, while retaining PDM and the remaining training setup, would help isolate this contribution.
  3. The method is motivated by multiple semantically valid matches, while the main metrics evaluate annotated pairs. The relevance analysis is helpful, but comparison with baselines using multi-positive retrieval metrics would more directly test this motivation.
Questions for authors

See weakness. Two minor points:

  1. In Figures 6 and 7, are the weight and uncertainty bins formed using equal sample counts or equal-width value intervals? Please clarify the boundaries.
  2. Line 723 contains “Table ??”.
Reviewer_vc2POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MGUC for bidirectional sign language video–text retrieval. It combines global semantic matching with fine-grained clip–word matching through bidirectional interaction-aware aggregation. Learnable [Null] tokens accommodate clips and words without clear cross-modal counterparts. Experiments cover How2Sign, PHOENIX-2014T, and CSL-Daily under RGB-only and RGB+Pose settings, with component ablations, computational analysis, and a human evaluation of the relationship between negative weights and semantic relevance.

Critical strengths
  1. Effective local matching with supporting ablations. On How2Sign, adding dual [Null] tokens improves Bi-SA providing direct evidence for the usefulness of handling unmatched local content.

  2. Substantial empirical validation. The evaluation covers three datasets and two visual-input settings, with detailed component ablations. The blinded human study of 410 video–text pairs further supports the calibration mechanism

Critical weaknesses
  1. Unclear benefit specifically from uncertainty modeling. Tables 2–5 lack a fixed-covariance or deterministic-distance calibration control, leaving the specific value of learned uncertainty unclear.

  2. Unclear provenance of the strongest baseline. Table 1 lists SEDS’s PHOENIX-2014T V2T R@1 as 76.3, whereas its public manuscript reports 78.7; MGUC obtains 77.4. This discrepancy needs reconciliation under a comparable evaluation protocol before the reported improvement can be assessed reliably.

Questions for authors
  1. How are the alternative distances in Table 5 estimated, normalized, and tuned to ensure a fair comparison?
Reviewer_xDXDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper tries to approach sign language retrieval by matching at both granularities at once, compared to existing methods which focus more on either global embeddings or clip-word similarities. The paper starts from the fact that previous work treats every unpaired sample in the batch as a negative even when it is semantically close to the query, and from there the authors propose MGUC. It fuses a global cosine similarity with a fine-grained clip-word similarity aggregated by a learned bidirectional weighting, adds a learnable null token so transitional clips and filler words can be ignored, and models each video and text as a Gaussian so that the Bhattacharyya distance of an unpaired sample relative to the paired one can down-weight it in the contrastive loss. The method reports the best R@1 and Rsum on How2Sign, PHOENIX-2014T, and CSL-Daily in both RGB-only and RGB plus pose settings.

Critical strengths
  1. The paper is well written and nicely motivated.
  2. The authors put real work into ablations and into justifying design choices. The analysis is thorough and clearly presented across many tables and figures in the main text and the appendix.
  3. The null token is an interesting idea. It is well motivated for sign language, where transitions and filler words carry no sign, it has a clean ablation, and it should suit other sequence matching problems as well.
Critical weaknesses

I have significant concerns about the positioning and the baselines. The authors describe their citations as recent work, but the related work section has recent papers only in general cross-modal alignment. The cited sign language retrieval methods and the cited probabilistic methods stop at 2023 and 2024. Looking at more recent work such as CMCM [1], which also combines coarse and fine-grained alignment and a Gaussian component for the same task on the same three datasets, the novelty looks thin. The null token and the negative reweighting are the only parts that appear new. The fine-grained module and its bidirectional version are almost identical to the Interactive Similarity Aggregation and Bi-ISA of UCoFiA [2], and Figure 3 looks like literally a copied version of UCoFiA's [2] Figure 3? However, UCoFiA [2] is not cited, not positioned against, and not used as a baseline. The baselines are also old, from 2022 to 2024. More recent work such as C2RL [3] and CMCM [1] should be compared against, both in method and in results, or the omission should be justified.

The method's design choices looks generally good and the ablations are reasonable, but the positioning is quite unclear, and it is hard to see where the novelty is and whether what is proposed adds clear value over existing recent work.

References:

[1] X.-H. Yang, D. Wei, W. Li, H. Hu, "Causality-inspired multi-grained cross-modal sign language retrieval," Computer Vision and Image Understanding, vol. 264, 104631, 2026, doi: 10.1016/j.cviu.2025.104631.

[2] Z. Wang, Y.-L. Sung, F. Cheng, G. Bertasius, M. Bansal, "Unified Coarse-to-Fine Alignment for Video-Text Retrieval," ICCV 2023, pp. 2804-2815, doi: 10.1109/ICCV51070.2023.00264.

[3] Z. Chen et al., "C2RL: Content and Context Representation Learning for Gloss-Free Sign Language Translation and Retrieval," IEEE TCSVT, vol. 35, no. 9, pp. 8533-8544, 2025, doi: 10.1109/TCSVT.2025.3553052.

Questions for authors
  1. Please compare the proposed framework against UCoFiA, C2RL, and CMCM in terms of method and novelty. What exactly is new relative to each?
  2. UCoFiA looks very close to the fine-grained module. Please add it as a baseline, or state which row of Table 3 corresponds to it and justify why the full model was omitted. C2RL and CMCM should also be added to Table 1 or their omission explained.

I am willing to increase my score depending on how the novelty looks after positioning against these works and how the gains look there.

Reviewer_ukYMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-granularity semantic matching and uncertainty-aware calibration framework (MGUC) for sign language retrieval. This method combines coarse-grained global semantic matching (GSM) and fine-grained local semantic matching (FSM), enhancing the semantic alignment capability between text and sign language videos. Additionally, the authors model the cross-modal semantic uncertainty using Gaussian distribution, estimate the distribution differences between samples through Bhattacharyya Distance, and propose an uncertainty-aware contrast calibration (UACC) mechanism to dynamically adjust the contribution of potential false negative samples in contrast learning. Experiments on three benchmark datasets, How2Sign, PHOENIX-2014T, and CSL-Daily, demonstrate that this method achieves competitive bidirectional retrieval performance.

Critical strengths
  1. The issues of global semantic absence, local matching deviation, and uncertainty in cross-modal semantic correspondence were fully considered.
  2. GSM and FSM complement each other, enabling the simultaneous modeling of overall event semantics and fine-grained action information.
  3. The relative Bhattacharyya Distance is used to adaptively adjust the weight of negative samples, providing a new solution to alleviate the problem of false negative samples.
Critical weaknesses
  1. A smaller distribution distance does not necessarily imply semantic relevance. UACC may simultaneously reduce the training weight of real difficult negative samples.
  2. The relationship between Gaussian variance and true semantic ambiguity is not clear, and there is a lack of direct probability calibration verification.
  3. The paper emphasizes one-to-many semantic correspondence, but mainly adopts the single positive example Recall metric; existing manual evaluations have not directly verified the quality of multi-positive example retrieval.
  4. Its fixed-sized learnable interaction matrix may depend on the position of local features and the sequence length, lacking robustness analysis for temporal changes and variable-length inputs.
  5. Fine-grained matching requires calculating the local interaction between the query and the candidate videos, but the paper does not fully evaluate the actual retrieval cost under a large-scale candidate library.
Questions for authors
  1. UACC assigns the weights of negative samples based on the relative Bhattacharyya Distance between unpaired samples and positive samples. However, true difficult negative samples with similar semantics may also receive lower weights. How does the author distinguish between false negative samples and true difficult negative samples? Can the weight distribution and retrieval error analysis under different semantic relevance levels be provided?
  2. Why can the variance of the diagonal Gaussian distribution effectively represent the semantic correspondence uncertainty between sign language and text, rather than just sample difficulty or feature noise? Besides the correlation between variance and Recall, can the reliability of this uncertainty estimation be verified through probability calibration experiments or data with ambiguous annotations?
  3. One of the main motivations of the paper is to solve the one-to-many semantic correspondence problem. However, the existing Recall@K only evaluates the unique paired samples. Can the author use the manual relevance annotations in Appendix A.7 to further report indicators such as nDCG@K and mAP that support multiple positive examples or hierarchical correlations? Additionally, is using only 20 queries sufficient to support this conclusion?
  4. Bi-SA employs learnable matrices WvW_v and WtW_t related to the maximum clip/token quantity. Will this design learn position-related aggregation biases rather than true semantic interaction? Can the author further test the performance under different video lengths, sampling densities, and time perturbations, and compare it with attention aggregation methods with similar parameter quantities?
  5. Appendix Table 6 only reports FLOPs and inference time at the single-query level. However, Bi-SA involves local interactions between queries and candidate samples. Can the author further report end-to-end retrieval latency, GPU memory consumption, and throughput under different retrieval library scales (such as 1K and 10K candidate videos), and clarify which computations can be preprocessed offline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.