- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the critical and underexplored challenge of viewpoint robustness in Sign Language Understanding (SLU). To bridge the lack of multi-view continuous sign language resources, the authors introduce MVSign, a benchmark spanning three sign languages and seven viewpoints. MVSign combines original frontal recordings with non-frontal views synthesized via a robust three-stage pipeline (whole-body 3D parameter extraction using SMPLer-X, root-driven viewpoint transformation, and skeleton-guided adaptive rendering using DWPose and StableAnimator) while preserving source splits and annotations. Furthermore, the authors propose CanonSLU, a canonical-view-guided framework featuring Motion Relation Aggregation (MRA) for robust temporal representation and Canonical-View Knowledge Transfer (CKT) for frontal-to-non-frontal semantic distillation. Extensive experiments across Continuous Sign Language Recognition (CSLR) and Sign Language Translation (SLT) tasks demonstrate that CanonSLU achieves state-of-the-art performance on non-frontal and aggregate splits.
Critical strengths
-
High Practical Significance & Novel Benchmark: Viewpoint variation is a major bottleneck for real-world SLU deployment. Constructing MVSign—a systematic multi-view benchmark covering multiple sign languages, diverse camera angles, and both continuous recognition and translation tasks—is a timely and valuable contribution to the community.
-
Rigorous Validation of Synthetic Data: Recognizing the potential risks of synthetic data, the authors provide thorough quantitative pose-fidelity evaluations (NME, PCK) and a comprehensive human perceptual study (Table 11), confirming that the generated videos maintain high visual realism and temporal-motion consistency.
Critical weaknesses
-
Concerns Regarding Error Propagation in the Data Pipeline: The data generation pipeline is multi-staged (SMPLer-X Root Transformation DWPose StableAnimator). Each module is prone to errors, particularly SMPLer-X and DWPose under fast finger articulations, self-occlusions, and close hand-body interactions inherent in sign language. The paper lacks a detailed discussion or limitation analysis on how artifacts or pose-estimation failures in the synthetic data affect downstream SLU training.
-
Justification of Viewpoint Sampling Strategy: The chosen multi-view set () is asymmetric (e.g., and vs. and ). While we acknowledge that native datasets like How2Sign provide recordings, could the authors provide a deeper justification for this specific asymmetric sampling strategy? Specifically, given that sign language involves hand dominance (handedness) and body orientation bias, how do these chosen viewpoints adequately cover potential left-right disparities, and why are symmetric counterparts (e.g., or ) not uniformly sampled across all subsets? The rationale behind this specific asymmetric sampling strategy requires deeper clarification.manuscript.
Questions for authors
None
Thank you for the detailed response. I have also considered the comments and concerns raised by the other reviewers.
Figure 2 reports 112,665 videos in total, with 88% labeled as original and 12% as generated. This appears to correspond to 99,165 inherited/native videos and 13,500 newly synthesized videos. Could the authors confirm that 13,500 is the complete number of newly generated samples, and clarify whether the remaining videos include the native How2Sign R90° recordings? Please also clarify throughout the paper whether reported dataset sizes refer to the total benchmark size or only the newly synthesized augmentation, so that these two quantities are not conflated.