acceptodds
Under review as a conference paper at ICLR 2027

Towards Multi-View Sign Language Understanding: A Benchmark Dataset and Baseline

Abstract

Most existing Sign Language Understanding (SLU) methods are developed under fixed-view settings and remain vulnerable to viewpoint changes, while the scarcity of multi-view data hinders systematic research on viewpoint robustness. To address this gap, we introduce MVSign, a benchmark spanning three sign languages and seven viewpoints. MVSign combines original frontal recordings from existing datasets with additional views synthesized via D whole-body motion reconstruction and target-view rendering, while preserving the source splits and annotations. We further propose CanonSLU, a framework for multi-view SLU. This framework employs a canonical-view-guided strategy, using the frontal view as a semantic anchor. Specifically, CanonSLU incorporates Canonical-View Knowledge Transfer (CKT) to transfer canonical knowledge from the frontal view to non-frontal views, and Motion Relation Aggregation (MRA) to aggregate motion-related features across frames and strengthen temporal representations under viewpoint changes. Extensive experiments on MVSign demonstrate competitive performance with gains on non-frontal views, establishing CanonSLU as an effective baseline for multi-view SLU.

Primary area:
datasets and benchmarks
Keywords:
Continuous Sign Language Recognition, Canonical-View Prior, Multi-view Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_PumAOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the critical and underexplored challenge of viewpoint robustness in Sign Language Understanding (SLU). To bridge the lack of multi-view continuous sign language resources, the authors introduce MVSign, a benchmark spanning three sign languages and seven viewpoints. MVSign combines original frontal recordings with non-frontal views synthesized via a robust three-stage pipeline (whole-body 3D parameter extraction using SMPLer-X, root-driven viewpoint transformation, and skeleton-guided adaptive rendering using DWPose and StableAnimator) while preserving source splits and annotations. Furthermore, the authors propose CanonSLU, a canonical-view-guided framework featuring Motion Relation Aggregation (MRA) for robust temporal representation and Canonical-View Knowledge Transfer (CKT) for frontal-to-non-frontal semantic distillation. Extensive experiments across Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT) tasks demonstrate that CanonSLU achieves state-of-the-art performance on non-frontal and aggregate splits.

Critical strengths
  1. High Practical Significance & Novel Benchmark: Viewpoint variation is a major bottleneck for real-world SLU deployment. Constructing MVSign—a systematic multi-view benchmark covering multiple sign languages, diverse camera angles, and both continuous recognition and translation tasks—is a timely and valuable contribution to the community.

  2. Rigorous Validation of Synthetic Data: Recognizing the potential risks of synthetic data, the authors provide thorough quantitative pose-fidelity evaluations (NME, PCK) and a comprehensive human perceptual study (Table 11), confirming that the generated videos maintain high visual realism and temporal-motion consistency.

Critical weaknesses
  1. Concerns Regarding Error Propagation in the Data Pipeline: The data generation pipeline is multi-staged (SMPLer-X →\rightarrow Root Transformation →\rightarrow DWPose →\rightarrow StableAnimator). Each module is prone to errors, particularly SMPLer-X and DWPose under fast finger articulations, self-occlusions, and close hand-body interactions inherent in sign language. The paper lacks a detailed discussion or limitation analysis on how artifacts or pose-estimation failures in the synthetic data affect downstream SLU training.

  2. Justification of Viewpoint Sampling Strategy: The chosen multi-view set (R90∘,R45∘,L30∘,L60∘,U30∘,D30∘R90^\circ, R45^\circ, L30^\circ, L60^\circ, U30^\circ, D30^\circ) is asymmetric (e.g., R90∘R90^\circ and R45∘R45^\circ vs. L30∘L30^\circ and L60∘L60^\circ). While we acknowledge that native datasets like How2Sign provide R90∘R90^\circ recordings, could the authors provide a deeper justification for this specific asymmetric sampling strategy? Specifically, given that sign language involves hand dominance (handedness) and body orientation bias, how do these chosen viewpoints adequately cover potential left-right disparities, and why are symmetric counterparts (e.g., L90∘L90^\circ or R30∘R30^\circ) not uniformly sampled across all subsets? The rationale behind this specific asymmetric sampling strategy requires deeper clarification.manuscript.

Questions for authors

None

Reviewer_BCkjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The topic is valuable, and the motivation is solidly argued. The authors conducted controlled experiments using natively captured paired frontal and side-view videos, quantitatively demonstrating the substantial performance degradation caused by viewpoint changes.

Critical strengths

The comparisons are fair, and the experimental workload is sufficient. All nine CSLR comparison methods were retrained on MVSign according to the protocols in their original papers. Systematic ablations were conducted on the individual and combined effects of the two modules, the insertion layer positions, and the sensitivity to distillation weights.

The results are reported honestly. Unfavorable results—such as WER as high as 58% at side viewpoints and BLEU-4 of only 6.6 on natively captured data—are presented truthfully. Its quantitative data on viewpoint sensitivity (e.g., WER at R90° is about 20 percentage points higher than at the frontal viewpoint) are valuable to the community.

Critical weaknesses
  1. There is a lack of direct evidence for the validity of the synthetic viewpoint data. The experiment in the introduction demonstrating that “multi-view data are useful” is based on natively captured R90° videos from How2Sign, which demonstrates the value of natively captured data. However, all models on PT14/CSL are trained on data mixed with synthetic viewpoints, and there is no data ablation with a frontal-only training baseline.

  2. The robustness conclusions are built on synthetic test data, and the gap between synthetic and natively captured viewpoints has not been quantified. All non-frontal test samples in PT14/CSL are synthetic, so the paper’s core claim is currently supported only by synthetic data.

  3. There are key omissions in the related work. MVDT, proposed by Guan et al. (IET Computer Vision, 2025), has constructed a five-view continuous sign language dataset (not publicly released) and achieved viewpoint-invariant sign language translation based on teacher-student distillation, which is highly similar to the core idea of CKT in this paper; the synthetic sign language data work of How2Sign-Synth3D (CVPR 2026) is also directly relevant to the data construction pipeline in this paper. None of the above works is cited or discussed in the paper, causing the statement in the introduction that “viewpoint variation is an unexplored challenge” to be inconsistent with the existing literature, and making the incremental contributions of CKT and MVSign unclear. The authors are advised to add comparisons and analyses of differences with the above works.

  4. The practical usability of the main contribution (the dataset) is questionable, and there is a release dilemma. The core contribution of this paper is the dataset, but the way it is delivered is constrained at both ends: if the synthetic videos are released directly, as derivative data of the source dataset (under an application-based license), they lack authorization from the original dataset; if only the generation scripts are released, the randomness of the generation process and model version drift will cause different users to obtain inconsistent data instances, and the reproducibility of the benchmark as a fixed evaluation standard cannot be guaranteed. Either way, the community will find it difficult to actually use it, directly undermining the core contribution.

  5. All experiments use a single seed, raising concerns about significance. In Table 2, the advantage over the strongest comparison method is only 1.45–1.78 WER, which is of the same order of magnitude as typical seed variance in CSLR. It is recommended that, at a minimum, the mean ± standard deviation across multiple seeds be reported for the key comparison with CanonSLU.

Questions for authors
  1. Could the authors add a frontal-only training baseline and test it on non-frontal viewpoints to quantify the contribution of the synthetic viewpoint data itself?

  2. Since H2S contains both natively captured and synthetic R90° samples, could the authors report the performance gap of the same model on the two within the same dataset? This would be the most direct evidence for whether evaluation conclusions based on synthetic viewpoints transfer to natively captured viewpoints, without confounding from domain shift.

  3. What is the essential difference between the viewpoint distillation scheme of CKT in this paper and that of MVDT (Guan et al., IET CV 2025)? What is the relationship between MVSign and its five-view continuous sign language dataset?

  4. What is the final release format of the dataset? If the video data themselves are released, how will the redistribution authorization for derivative data and signer likeness consent be addressed? If only scripts are released, how will the consistency of data generated by different users and the comparability of results be ensured?

Comments on the paper

2 comments

Thank you for the detailed response. I have also considered the comments and concerns raised by the other reviewers.

Figure 2 reports 112,665 videos in total, with 88% labeled as original and 12% as generated. This appears to correspond to 99,165 inherited/native videos and 13,500 newly synthesized videos. Could the authors confirm that 13,500 is the complete number of newly generated samples, and clarify whether the remaining videos include the native How2Sign R90° recordings? Please also clarify throughout the paper whether reported dataset sizes refer to the total benchmark size or only the newly synthesized augmentation, so that these two quantities are not conflated.

Thank you for the detailed clarification. I have also considered the comments from the other reviewers. From the benchmark-design perspective, I find MVSign valuable: it provides a systematic framework for studying viewpoint variation and combines native and synthesized multi-view data in a useful way. I therefore view the work as close to Weak Accept on its scientific motivation and benchmark design.

However, I remain concerned about its suitability for the dataset track. The clarification shows that only 13,500 of the 112,665 videos are newly synthesized, while the majority are inherited from existing datasets. More importantly, the redistribution permissions for PHOENIX-2014T- and CSL-Daily-derived videos have not yet been confirmed. Thus, the current version presents a unified benchmark whose practical release and reproducibility are not yet guaranteed.

I therefore maintain my original rating. I encourage the authors to obtain the necessary source-specific permissions, clearly separate inherited data from newly synthesized data, and resubmit the work in a future venue once the release and licensing conditions are fully resolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.