acceptodds
Under review as a conference paper at ICLR 2027

SignAlign: Structure-Aware Alignment for Cross-Modal Representation Learning in Sign Language Retrieval

Abstract

Sign language retrieval enables bidirectional search between natural-language text and signing videos based on their semantic correspondence. Existing methods compute video–text similarity by comparing global representations or aggregating local similarities between video tokens and text tokens. Despite their effectiveness, two limitations remain. First, contrastive training emphasizes annotated video–text associations without explicitly preserving semantic relations among texts. As text representations adapt to videos, their pretrained relative distances can change. If texts with different meanings become too close, a video associated with one may be incorrectly retrieved for the other. retrieved for the other. Second, local alignment computes weights independently for each text token. A video token with high similarity to several text tokens can increase the retrieval score through all of them, even when the video does not express the remaining content of the text. To address these limitations, we propose SignAlign, a structure-aware alignment framework for cross-modal representation learning. It preserves semantic relations among sentences while jointly aligning video tokens and text tokens. To preserve sentence relations during adaptation, Text Semantic Structure Alignment (TSSA) uses a frozen pretrained text encoder as a reference. It aligns normalized pairwise distances and triplet-wise angles among sentence embeddings with those derived from this reference. Complementing this sentence-level alignment, Cross-Modal Token Structure Alignment (CTSA) jointly determines token alignment weights through entropy-regularized optimal transport. Text tokens aligned with the same video token share its available weight, limiting repeated emphasis in retrieval scoring. The resulting joint alignment score is fused with independently weighted local similarity scores during training and inference. Experiments on three public sign language retrieval benchmarks demonstrate improved retrieval in both directions over strong baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sign Language Retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Xg5qOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SignAlign achieves improved sign language retrieval by addressing the problem of preserving the dissimilarity among different text embeddings through the TSSA module and the problem of the model paying higher attention to few common video embeddings through the CTSA module. The motivation of this work is clear and the experimental results verify the ability of the proposed approach to achieve improved results. Furthermore, an extensive ablation study justifies the selection of the values of the hyperparameters used in the work.

Critical strengths

One of the strengths of this work is their clear motivation and contributions. They propose two modules, each serving a specific purpose to address specific limitations of previous methodologies. Thus, the reader can clearly follow the reasoning behind the proposed work. A second strength of this work is the experimental results and ablation study. By testing the proposed work in 3 well-known large sign language datasets, the authors show the improvements achieved by their method. In addition, an extensive ablation study justifies the value assigned to each hyperparameter employed by the proposed method.

Critical weaknesses

None

Questions for authors

The following should be addressed by the authors:

  • The RGB encoder and the vision transformer should be clarified. Which models are used?
  • The acronym EOT (line 205) should be defined.
  • Tables 5,6 & 7 should clearly present the V2T and T2V columns for better understanding.
  • Some minor English errors “retrieved for the other” appears twice in line 20 and “assigns” in line 116.
3 comments
AuthorsOpenReview ↗

Response to Reviewer Xg5q

We sincerely thank the reviewer for the support and encouraging assessment of our work!

Q1: Which RGB encoder and vision Transformer are used?

A1: The RGB encoder is I3D; we use the publicly released, fixed 1024-dimensional I3D segment features from CiCo (Cheng et al., 2023). The vision Transformer is a trainable temporal Transformer with 12 layers, hidden width 768, and 12 attention heads. Its compatible Transformer blocks, class token, layer normalizations, and output projection are initialized from the visual tower of CLIP ViT-B/32 (Radford et al., 2021). We have clarified these details in Appendix A.1 (Lines 767–775).

Q2: What does EOT mean?

A2: EOT stands for “end-of-text.” TSSA uses the EOT-token hidden state followed by CLIP’s pretrained text projection as the sentence representation. We have defined EOT at its first occurrence in line 205.

Q3: Unclear labeling of the T2V and V2T columns in Tables 5–7.

A3: We have explicitly labeled the two R@1 columns in Tables 5–7 as “T2V R@1 ↑” and “V2T R@1 ↑,” respectively, from left to right.

Q4: Minor English errors.

A4: We have removed the duplicated “retrieved for the other,” corrected “assigning weights” to “assigns alignment weights,” and proofread the manuscript.

Thank you very much! My comments have been addressed. I'll keep my score

We sincerely thank the reviewer for the positive score and encouraging assessment, as well as the valuable suggestions for improving our paper.

Reviewer_HqVVOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SignAlign addresses sign language retrieval through two complementary components. Text Semantic Structure Alignment (TSSA) preserves normalized pairwise distances and triplet-wise angles between sentence embeddings, using a frozen pretrained text encoder as the reference. Cross-Modal Token Structure Alignment (CTSA) computes an entropy-regularized optimal transport score between text and video tokens and combines it with existing local matching scores. The framework achieved impressive results on popular datasets.

Critical strengths
  1. Clear presentation. The paper is well organized, and the overall framework and training objectives are easy to follow.
  2. Impressive results. The reported retrieval results are impressive across three benchmarks, showing improvements over strong baselines. The ablation studies also demonstrate the benefits of combining TSSA and CTSA.
  3. Interesting geometric constraints. Preserving the relative geometry of pretrained text representations is an interesting idea and provides a relatively novel perspective for sign language retrieval. Combining distance and angular constraints allows the representations to adapt while retaining relational information from the pretrained encoder.
Critical weaknesses
  1. Incomplete initialization details. The initialization and freezing of the reference text encoder are clearly described. However, the initialization of the video Transformer and video-side projection layers, as well as the source of the pre-extracted RGB features, remains unclear. These details are needed to understand the starting point of joint optimization and the method’s dependence on pretrained representations.
  2. Insufficient analysis of optimization stability. The angular loss divides by the norms of sentence-difference vectors, while the distance loss normalizes distances by their batch mean. These operations may become sensitive when representations are close to each other. CTSA also backpropagates through a limited number of Sinkhorn iterations at a low temperature. However, the paper does not provide loss or gradient diagnostics, to assess training stability.
  3. Limited justification for the necessity and novelty of OT. CTSA adopts a standard entropy-regularized OT formulation, and transport-based alignment has already been explored in prior sign language retrieval work. The specific contribution of OT therefore needs further clarification. In Table 7, transport-only scoring performs comparably to native matching, while combining the two improves retrieval. This suggests that the scoring functions are complementary, but does not directly establish whether the improvement comes from more accurate structural alignment. Table 8 provides useful normalization controls, although the single-seed results leave uncertainty about their robustness. The inference improvement using the same checkpoint in Table 9 is also relatively modest, increasing mR@1 from 73.68 to 74.14.
  4. Limited direct evidence of semantic geometry preservation. The TSSA experiments mainly evaluate downstream retrieval performance, without directly measuring how well the text geometry is preserved. Metrics such as distance correlations, angular discrepancies, and neighborhood preservation would help compare the frozen encoder, the adapted baseline, and TSSA. Figure 1(b) provides a t-SNE-style visualization to motivate the method, but how this figure was generated is not sufficiently explained, and the corresponding representations obtained with TSSA are not included. The current results therefore support the effectiveness of TSSA more directly than its proposed mechanism.
Questions for authors

Q1. Which video-side parameters use pretrained weights, and which are randomly initialized? Please identify the RGB feature-extraction checkpoint and clarify whether training starts from pretrained encoders or an already trained retrieval model. Q2. How are near-zero distances handled in the geometric losses? Could the authors provide weighted loss curves, gradient-norm curves, multi-seed results, and Sinkhorn marginal-error statistics under the main configuration? Q3. Could simpler coverage or marginal regularizers achieve comparable improvements with matched tuning and multiple seeds? Does the benefit persist when gradients through the transport plan are stopped? This would help distinguish the contribution of transport scoring from that of differentiating through the transport computation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer HqVV (1/3)

We sincerely appreciate your valuable comments and recognition of our work. Below, we provide detailed responses to your questions.

Q1: Incomplete initialization details.

A1: We clarify the following three aspects:

  • Video Transformer and projection layers. Compatible weights of the video Transformer and its output projection are initialized from the visual encoder of CLIP ViT-B/32 (Radford et al., 2021). The input projection (1024 to 768 dimensions) and temporal positional embeddings are randomly initialized.
  • RGB features. We use the publicly released I3D RGB features provided by CiCo (Cheng et al., 2023) as fixed inputs during retrieval training.
  • Training starting point. The retrieval text encoder is initialized from pretrained CLIP and jointly trained with the video Transformer described above. Training does not start from an already trained retrieval model.

We have added these details to Appendix A.1, “Additional Implementation Details” (Lines 767–776).

Q2: Insufficient analysis of optimization stability.

A2: We provide the following analysis under the main PHOENIX-2014T configuration, with a transport temperature of 0.01 and ten Sinkhorn iterations for CTSA.

(1) Handling near-zero distances in the geometric losses. Geometric losses use FP32. Squared distances are clamped to 1e-12 before taking square roots, and distances are normalized by the batch mean of nonzero pairwise distances. Difference-vector normalization uses ε = 1e-12. Global gradients are clipped at norm 1. These safeguards reduce sensitivity near zero and limit large gradients.

(2) Weighted-loss and gradient-norm diagnostics. Appendix Figure 7 presents weighted-loss curves and pre-clipping gradient-norm curves over 200 epochs, including epoch-wise median and 95th-percentile gradient norms. Losses and gradient norms decrease overall, and losses and gradients remain finite throughout all 11,200 steps in each recorded run. These observations support stable optimization in the recorded runs.

(3) Sinkhorn marginal errors. CTSA uses log-domain updates for numerical stability. Across three seeds, mean L1 marginal errors over 642 ground-truth pairs are 0.2358–0.2542 (visual) and 2.10e-7–2.18e-7 (text). These results quantify the degree of convergence toward the prescribed marginals within the fixed Sinkhorn iteration budget.

(4) Multi-seed results. To assess stability across random seeds, we repeat training with three paired seeds and report mean ± standard deviation below. The improvements persist across seeds, supporting stable retrieval performance.

ModelT2V R@1 ↑V2T R@1 ↑
Baseline70.56 ± 0.4171.03 ± 0.20
Baseline + TSSA73.31 ± 0.4673.45 ± 0.42
Baseline + TSSA + CTSA74.92 ± 0.9175.59 ± 0.56
AuthorsOpenReview ↗

Response to Reviewer HqVV (2/3)

Q3: Limited justification for the necessity and novelty of OT.

A3: We clarify CTSA’s contribution and address each experimental concern below. All newly added tables report mR@1 on PHOENIX-2014T as mean ± standard deviation.

(1) Specific contribution of CTSA.

CTSA uses a standard OT solver; its novelty lies in how the transport plan is used. UPRet uses transport as a training objective. CTSA instead uses the transport plan to compute a retrieval score, fused with native matching in both training and inference, limiting repeated emphasis on a few video segments across text tokens. Table 7 shows that transport-only scoring performs comparably to native matching, while their fusion improves retrieval, supporting the complementary use of these scores.

(2) Alignment rather than mere scoring complementarity (Table 7).

We permute the transport plan’s text columns on trained CTSA checkpoints at inference. The permuted plan remains an additional score with the same marginals; only word–segment alignment is disrupted. This reduces mR@1 by 1.47 points on average, showing that the gain comes from more accurate structural alignment rather than merely from complementary scoring.

Scoring methodmR@1 ↑
Native + CTSA74.10 ± 0.38
Native + permuted CTSA72.63 ± 0.35

(3) Multi-seed normalization controls (Table 8).

To address the single-seed concern, we compare global, row-only, and column-only normalization with CTSA, which jointly constrains row and column sums, across three seeds (42, 0, and 1). Within each seed, the encoded features and blending coefficient are held fixed. Results are reported as mean ± standard deviation. CTSA achieves the highest mean mR@1, exceeding the stronger one-sided alternative, column-only normalization, by 2.18 percentage points, showing that the advantage of joint constraints is stable across multiple random seeds.

Added scoring componentmR@1 ↑
Global normalization70.40 ± 0.28
Row-only normalization70.90 ± 0.34
Column-only normalization71.92 ± 0.42
CTSA transport plan74.10 ± 0.38

(4) Performance in Table 9.

With TSSA disabled, most of CTSA’s gain comes from improving the encoders during training, not from the transport score at inference. A CTSA-trained model using native scoring alone reaches 73.68 mR@1, versus 70.87 for the baseline. Table 9 isolates the extra gain from adding the transport score at inference (73.68 → 74.14), which is small but improves both retrieval directions.

(5) Comparison with simpler coverage and marginal regularizers.

We add a coverage regularizer (coverage-KL) and a marginal regularizer (marginal-L2) to the baseline, tune each coefficient over {0.001, 0.01, 0.1} using Dev mR@1, and report the mean ± standard deviation across three seeds for the selected setting.

Under matched tuning and multiple seeds, the coverage regularizer performs on par with the baseline (70.92 vs. 70.91), and the marginal regularizer slightly underperforms it (70.60). CTSA exceeds both by at least 2.76 points. Simpler coverage or marginal regularizers therefore cannot achieve comparable improvements.

MethodmR@1 ↑
Baseline70.91 ± 0.06
Coverage-KL70.92 ± 0.28
Marginal-L270.60 ± 0.50
CTSA73.68 ± 0.28

(6) Effect of stopping gradients through the transport plan.

With TSSA disabled, we compare full differentiation with a variant that stops gradients through the transport plan across three paired seeds. In this variant, the plan is recomputed at every forward pass but detached during backpropagation, while gradients still flow through token similarities. Both variants combine native matching and transport scoring at inference. Stopping gradients through the plan (73.31 ± 0.28 mR@1) performs close to full differentiation (73.68 ± 0.28) and clearly outperforms the baseline (70.91 ± 0.06) in Q3(5). Thus, the retrieval benefit persists when gradients are stopped, since the contribution comes mainly from transport scoring, while differentiating through the transport computation adds only a small further improvement.

Training variantmR@1 ↑
Full gradients73.68 ± 0.28
Stopped gradients73.31 ± 0.28
AuthorsOpenReview ↗

Response to Reviewer HqVV (3/3)

Q4: Limited direct evidence of semantic geometry preservation.

A4: We provide direct quantitative evidence below, complemented by empirical visualizations in the appendix.

(1) Direct measurement of geometry preservation. We compare the frozen encoder, the adapted baseline, and TSSA in the original 512-dimensional space, using 630 unique PHOENIX-2014T test sentences. Distance Spearman, Angle MAE, and Neighbor retention@10 measure distance correlation, angular discrepancy, and neighborhood preservation, respectively.

RepresentationDistance Spearman ↑Angle MAE (°) ↓Neighbor retention@10 ↑
Frozen reference encoder1.0000.00100.00%
Adapted Baseline0.169 ± 0.0159.43 ± 0.1817.68 ± 0.55%
TSSA0.859 ± 0.0063.59 ± 0.0356.99 ± 0.21%

The adapted baseline substantially departs from the reference geometry, while the model with TSSA improves agreement across all three metrics, including distance correlation from 0.169 to 0.859. These measurements directly support TSSA’s geometry-preservation objective.

(2) Generation of Figure 1(b). Figure 1(b) is a schematic illustration of semantic relation drift, rather than a projection of learned embeddings. We have clarified this in its caption.

(3) Empirical visualization with TSSA. Appendix Figure 8 presents t-SNE visualizations of the same test sentences from the frozen reference encoder, the adapted baseline, and the model with TSSA. We apply k-means to the frozen reference’s 512-dimensional embeddings to define eight color groups (n_init = 20, random_state = 1729). These group assignments remain fixed across panels, while t-SNE is applied separately to each representation set. Colors indicate reference-space clusters, not ground-truth semantic classes.

Figure 8 illustrates increased mixing of reference-defined groups after baseline adaptation and better retention with TSSA. The accompanying distance correlation and neighborhood-retention metrics are computed in the original 512-dimensional space, independently of the two-dimensional projections.

Together, these measurements and visualizations provide direct evidence of text-geometry preservation during adaptation, complementing the downstream retrieval results.

Reviewer_BpknOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for sign language retrieval where the goal is to have this bidirectional way of retrieving a sign language video from text, and vice versa. Their system called SignAlign introduces TSSA (to maintain the geometry of the embeddings) + CTSA (to match the tokens to the segments) , which collectively reaches state-of-the-art performances on 3 interpreted signing datasets on this specific task compared to previous works.

Critical strengths
  • The paper is well written and easy to follow.
  • The TSSA and CSTA proposed are well motivated and look interesting.
Critical weaknesses
  • On a personal take, this is a very niche topic. Nothing wrong with that, but atleast if we want to have such topic at a main conference conference it should really be nailed down well in terms of maybe practical application or theoritical study and etc. To me, this sounds more like just slightly extending what could be done prevously on this task. Nothing wrong again with this, but wouldnt see it worth a main conference audience. It's just adding one more row to the table and increasing some numbers slightly more.

  • The whole study is based on old and unnatural signign dataset dataset like Phoenix, How2Sign and CLS-Daily. There has been several increasing concerns from the community that these datasets are not ideal. How does all this work with actual natural signing. With signing in the wild, and etc. Which makes it actually useful in real-life. Givem the recent advances, I think we need to elevate the field collectively and these main conferences should highlight more of this as a signal.

Questions for authors

Could you explicity say how usable is this whole system in real life. On real signing.

I will personally highly encourgage the authors to move this a specialized workshop with the right audience .

7 comments
AuthorsOpenReview ↗

Response to Reviewer Bpkn (1/3)

We thank the reviewer for noting that the paper is well written and that TSSA and CTSA are well motivated. Below, we address the practical value of sign language retrieval (Q1), our methodological contribution (Q2), and performance on natural, in-the-wild signing (Q3), with additional experiments on OpenASL (Shi et al., 2022) and CSL-OpenWorld (Gan et al., 2026).

Q1: Practical value and relevance to ICLR.

A1:

(1) A large community. According to the World Federation of the Deaf, more than 70 million deaf people worldwide use over 300 sign languages, making sign language research worth broad attention (United Nations, n.d.).

(2) Sign language research at ICLR. Sign language research has previously been published at the ICLR main conference, including Sign2GPT (ICLR 2024; Wong et al., 2024), Uni-Sign (ICLR 2025; Li et al., 2025), and Grounding or Guessing? (ICLR 2026; Hamidullah et al., 2026). SignAlign addresses representation learning and multimodal alignment within this research area.

(3) Retrieval under limited annotation. Much signed content, such as news and educational videos, remains difficult to find because its meaning is conveyed in the signing itself rather than in titles or metadata. Providing comprehensive textual descriptions requires expert annotation, which is costly to scale (Yin et al., 2021), or automatic translation, which still faces reliability challenges (Hamidullah et al., 2026). Retrieval offers a practical path: it makes video collections searchable without requiring textual annotations for every video in the collection, and it only needs to rank relevant videos highly rather than translate them accurately sentence by sentence. Two representative applications are:

  • Information access: Deaf users can find signed news, announcements, and community updates through text queries, even when titles or metadata omit the relevant content.
  • Resource management: Broadcasters, archives, and schools can search large signing-video collections to locate materials for teaching and public services.

(4) Beyond generic video retrieval. Sign languages convey meaning through coordinated hand, facial, and body cues, so retrieval must match linguistic meaning rather than shared scenes or actions, which motivates dedicated methods (Yin et al., 2021; Cheng et al., 2023).

AuthorsOpenReview ↗

Response to Reviewer Bpkn (2/3)

Q2: Methodological contribution.

A2: SignAlign identifies and addresses two previously unexamined failure modes of cross-modal adaptation in sign language retrieval, rather than adding modules to an existing pipeline.

(1) Identifying failure modes.

Existing methods improve video encoders, contrastive objectives, or local matching, but overlook two problems in cross-modal alignment:

  • Semantic drift across text queries (Fig. 1b). Contrastive training pulls texts toward their paired videos but ignores text–text relations. As training reshapes text embeddings to match videos, their pretrained semantic relations drift, so texts with different meanings may become too similar and a video matching one text can be wrongly retrieved for another.
  • Concentrated evidence within pairs (Fig. 1c). In existing local matching, each text token selects its matching video segments independently. As a result, several words may all match the same few segments, and a video can score highly even if it conveys only part of the sentence.

(2) Proposing distinct designs.

  • TSSA: Unlike relational distillation, which transfers structure between models, TSSA anchors a model to its own pretrained geometry, allowing adaptation while retaining semantic distinctions.
  • CTSA: While UPRet uses transport as a training objective, CTSA uses the transport plan to compute a retrieval score and fuses it with native matching in both training and inference.

(3) Demonstrating complementary mechanisms.

  • TSSA: preserving inter-query structure. TSSA aligns sentence distances and angles with the frozen pretrained encoder, limiting semantic drift during adaptation. It raises distance Spearman correlation with the pretrained encoder from 0.169 (baseline) to 0.859 (see response to Reviewer HqVV, Q4), confirming the preservation of pretrained semantic relations.
  • CTSA: coordinating intra-pair alignment. coordinating intra-pair alignment. CTSA jointly allocates transport weights across text tokens and video segments, limiting repeated emphasis on a few segments. Permuting the plan’s text columns while preserving visual coverage reduces mR@1 by 1.47 points (see response to Reviewer HqVV, Q3(2)), confirming the benefit of coordinated alignment within each pair.

These components address inter-query semantic structure and intra-pair token alignment, respectively. Their complementary benefits are demonstrated in Table 4: on CSL-Daily, TSSA and CTSA individually improve mR@1 by 1.0 and 0.4 points, while their combination improves it by 4.4 points. SignAlign also outperforms UPRet in mR@1 across all three datasets (Tables 1–3).

These questions of preserving pretrained structure and allocating alignment evidence concern representation learning and multimodal alignment, which fall within ICLR’s scope.

AuthorsOpenReview ↗

Response to Reviewer Bpkn (3/3)

Q3: Dataset selection and performance on natural, in-the-wild signing.

A3:

(1) Dataset selection.

We acknowledge that some of these datasets rely on controlled laboratory recordings and do not fully represent signing as it naturally occurs in everyday communication. Nevertheless, PHOENIX-2014T, How2Sign, and CSL-Daily remain among the most established and widely used datasets across sign-language recognition, translation, generation, and retrieval. We therefore use them to enable direct comparison with established methods.

(2) Additional evaluation on natural, in-the-wild signing.

To address the concern about “old and unnatural” datasets, we additionally evaluate SignAlign on two in-the-wild datasets, OpenASL (Shi et al., 2022) and CSLOpenWorld (Gan et al., 2026). Unlike laboratory recordings, their videos are collected from real world online sources such as vlogs and everyday content shared by signers. They feature hundreds of signers with natural variation in signing style, as well as uncontrolled backgrounds, camera viewpoints, and lighting.

  • OpenASL (Shi et al., 2022) contains 98,417 ASL–English pairs (288 hours) from over 200 signers, including personal vlogs alongside online news, with varied signing styles and recording conditions.
  • CSL-OpenWorld (Gan et al., 2026) contains over 100,000 Chinese Sign Language–Chinese text pairs (216 hours) from online platforms, covering everyday life, education, culture, and news, with diverse camera viewpoints and backgrounds.

Experimental setup and results. Given the limited rebuttal timeframe, we randomly sampled 15,000 OpenASL pairs (38.08 hours) and 18,901 CSL-OpenWorld pairs (35.43 hours) from downloadable video–text pairs. RGB features are extracted using a frozen I3D encoder pretrained on BSL-1K. All methods share the same subsets, splits, and features, and are independently trained with three random seeds. For each run, the checkpoint is selected by Dev mR@1; the table reports mean Test performance across the three seeds.

DatasetMethodT2V R@1 ↑T2V R@10 ↑V2T R@1 ↑V2T R@10 ↑mR@1 ↑
OpenASLCiCo†14.8640.9113.2641.0314.06
OpenASLUPRet†16.2337.4914.7438.9715.49
OpenASLSignAlign40.1465.6338.0363.3439.09
CSL-OpenWorldCiCo†17.1943.4115.2642.0516.22
CSL-OpenWorldUPRet†15.6943.6216.5544.2716.12
CSL-OpenWorldSignAlign38.2264.7336.5863.6537.40

†CiCo denotes the native-matching baseline underlying SignAlign. UPRet is adapted from its official implementation.

Natural signing is more challenging: all methods perform substantially lower scores on OpenASL and CSL-OpenWorld due to diverse signers, uncontrolled recording, and open-domain content. Yet SignAlign’s advantage widens, for two reasons. (1) Semantic geometry preservation. Unlike narrow-domain benchmarks, open-domain content covers diverse topics with freely translated subtitles, so test sentences rarely resemble training ones and must be matched by meaning. This relies on pretrained semantic geometry, where similar sentences lie close together, which contrastive training on limited data distorts and TSSA preserves. (2) Coordinated evidence allocation. Frequent generic signs, such as pointing and pronouns, resemble many words and are repeatedly emphasized under independently normalized alignment weights; CTSA constrains the total matching weight of each video segment.

Q4: Real-life usability.

A4: On natural signing, SignAlign can serve as a candidate retrieval tool, returning a short list of relevant videos for users to confirm, and new videos can be indexed without annotation. We acknowledge that top-1 accuracy still leaves room for improvement and will discuss this as a limitation in the revised paper.

We share the reviewer’s view that the field should move toward natural signing, and we have included these results in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.