acceptodds
Withdrawn from ICLR 2027

TriViS: A Large-Scale Multi-View Benchmark for Vietnamese Sign Language Recognition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This paper presents TriViS, the first large-scale multi-view Vietnamese dataset for Continuous Sign Language Recognition and Translation, designed as a controlled benchmark for studying dataset design in underrepresented sign languages. TriViS contains 12,000 sentences and 84,453 synchronized videos captured from three viewpoints (frontal, left, and right) across both controlled studio recordings and outdoor environments. The synchronized multi view setup enables systematic analysis of viewpoint sensitivity, view complementarity, and other data collection factors that are difficult to isolate in uncontrolled web-scale corpora. Along with the dataset, we introduce a simple multi-view fusion framework that integrates synchronized visual streams into a unified representation for sentence-level recognition and translation. Using state-of-the-art CSLR and CSLT baselines, we conduct extensive experiments to evaluate the contribution of individual viewpoints, multi-view fusion strategies, sentence-independent generalization, and the effect of LLM scale. Our results show that multi-view observations consistently outperform single-view baselines and provide new empirical insights into how viewpoint configuration and controlled data collection affect continuous sign language understanding. Our code and dataset are publicly available on https://github.com/Anonymous-Submission-abcxyz2501/TriVIS_Submission_ICLR2027.gitGitHub and https://huggingface.co/datasets/Abcxyz2501/Trivis_Representative_SamplesHugging Face.

Primary area:
datasets and benchmarks
Keywords:
Vietnamese Sign Language, Multi-view Learning, Continuous Recognition, Benchmark Dataset, Visual Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YHY4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TriViS, a dataset for continuous Vietnamese sign language recognition and translation. It contains 12,000 sentences and 84,453 video clips recorded from three synchronized camera views in lab and real-world settings. The authors describe how they collected, annotated, and checked the data. They also extend SpaMo to combine the three views in a model called SpaMo-MV. The experiments compare camera views, test performance on unseen sentences, and study the effect of language model size. The appendix also tests SpaMo on unseen signers. The results show clear gains from multi-view input on lab data, while real-world settings and unseen signers remain difficult.

Critical strengths
  1. TriViS fills an important gap in data for continuous Vietnamese sign language recognition and translation. Most existing resources focus on a small set of languages. TriViS provides videos, sentences, and gloss annotations across several topics. I think this dataset will be useful for training and testing future models for Vietnamese sign language. This is the main reason I lean toward acceptance.

  2. The authors put care into data collection and annotation. Deaf-education specialists and Deaf signers helped prepare and check the gloss annotations. The data cover lab and real-world settings, with changes in recording sessions and clothing. The authors also describe how they split the videos into sentences, crop the signer, and check video quality. Recording the same signing performance from three views makes it possible to study what each view adds. In Table 3, three-view SpaMo-MV reaches 25.28 BLEU-4 on the lab subset, compared with 11.09 for single-view SpaMo. This supports the value of the recording setup.

Critical weaknesses
  1. The dataset includes only four signers, and the original split has the same signers in training and testing. This limits what the results tell us about new users. I appreciate the leave-one-signer-out evaluation in the appendix. However, Table 6 shows that SpaMo's BLEU-4 drops from 11.09 on the original split to 5.00 ± 0.91 on unseen signers. The test also covers only SpaMo, so it does not show how much multi-view fusion helps with this problem. The paper should make this limit clear when discussing its main results.

  2. The claims about multi-view benefits should be more specific. The largest gains come from lab data with three views available in both training and testing. In Table 3, three-view SpaMo-MV scores 2.60 BLEU-4 on real-world data. This is better than single-view SpaMo at 1.00, but below single-view GloFE at 6.03. When training uses three views but testing uses only the front view, lab BLEU-4 rises from 11.09 to 12.26, and not all other metrics improve. These results support a more limited claim than consistent gains across settings. The dataset is still useful, but the paper should be clear about when multiple views help and what this means for practical use.

Questions for authors
  1. How are the overlap and unseen-sentence splits built? Please report the training, validation, and test sizes, and explain what is kept separate between them. Do these splits share signers? Are the three synchronized videos of each recording always kept in the same split? These details would help distinguish performance on new sentences from performance on new signers or camera views.

  2. For the lab and real-world results in Table 3, are separate models trained for each setting? Or is one model trained on shared data and then tested on the two subsets? Please clarify this setup. It matters for understanding whether the performance gap comes from differences in training data, recording conditions, or other factors.

Reviewer_j6zxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

TriViS is a Vietnamese Sign Language (VSL) corpus of 12,000 sentences from four Deaf signers, filmed from three synchronized viewpoints (84,453 clips, studio and outdoor). The paper adds SpaMo-MV, which extends SpaMo with view-specific adapters and attention pooling, and reports that multi-view input beats single-view baselines (+14.19 BLEU-4 on the lab subset) and generalizes better to unseen sentences. The experiments report BLEU, ROUGE-L and WER on lab and real-world test files. They also include an unseen-sentence test and an LLM-size sweep (Fig. 4), GPT-5 and Gemini-2.5 API baselines, and a leave-one-signer-out evaluation (App. A.2).

Critical strengths
  1. I found no earlier sentence-level multi-view VSL video corpus. The public file list matches the stated counts (28,151 recordings × 3 views; 72,453 lab and 12,000 real-world clips), and glosses come from six deaf-education specialists and four Deaf signers, cross-checked between the groups (p.4 l.181-189). This corpus is what could make a revised version a useful benchmark. As submitted, it cannot offset the problems below, because the corpus is not openly usable and the evaluation is unsound.
Critical weaknesses
  1. The released evaluation does not test the paper's claims. The build script copies 600 of 1,200 test and 600 of 1,200 validation recordings into train.csv, checkpoints are selected on that validation set, and test_unseen.csv holds 574 training recordings. The paper gives no split sizes and never defines its overlap and unseen-sentence protocols (p.5 l.265-269). Its one split description ("the original random split with mixed signers", p.13 l.675-676) does not match the files, where val and test are single-signer blocks. The released scorer runs sacreBLEU with tokenize='none' on "|"-separated gloss strings, although the paper describes direct sign-to-text translation (p.3 l.133-135). Separators make up 33.4% (lab) and 34.7% (real-world) of reference tokens, so a constant output "zzz | zzz | zzz | zzz | zzz." scores BLEU-1 28.67 and WER 84.31 on the real-world test file, better than every Table 3 method. Since the paper documents neither its split nor its scoring settings, Table 3 and Fig. 4 rest either on these files or on an undocumented protocol. [Partially verified] This is the main reason for my rating. The central multi-view claim remains untested, and testing it means retraining and rescoring every method on a clean split.

  2. The benchmark is not openly available. The paper links only a 51-clip sample. The full set, linked only from the README, sits behind a Hugging Face login with no license, and the repository's LICENSE reads "For now, all rights are reserved by the authors." For a benchmark paper this blocks independent use and verification, so it is a second, independent reason for rejection.

Questions for authors
  1. Clarification: Which split files and which scorer (sacreBLEU signature; gloss strings or sentences) produced Tables 3-4 and Fig. 4?
  2. Validation experiment: What do Table 3's SpaMo and SpaMo-MV rows give when retrained without the duplicated recordings and scored on the sign-to-text targets the paper describes?
  3. Clarification: Will the full corpus be released under an open license without a login gate, and do the signers' consent terms cover public release of their faces?
Reviewer_vjHnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents TriViS as a benchmark for continuous Vietnamese sign language recognition and translation, reporting 12,000 sentences and 84,453 camera-view clips from four Deaf signers recorded with synchronized frontal, left, and right cameras in laboratory and outdoor settings. It also introduces SpaMo-MV, a SpaMo extension with view-specific adapters and attention pooling. The reported experiments examine viewpoint effects, unseen-sentence performance, and language-model scale, with an additional single-view signer-independent evaluation in the appendix. The claimed contribution combines a Vietnamese sign language resource with a controlled platform for understanding viewpoint complementarity and informing future data collection.

Critical strengths
  1. Continuous Vietnamese signing has a meaningful data requirement. Translating the text annotations of another sign language dataset would not provide the corresponding Vietnamese signing, and isolated-sign recordings do not directly supply the transitions and sentence-level supervision needed for continuous understanding. Collecting these performances therefore has potential value independently of whether the accompanying model introduces a new architecture.

  2. Synchronized recordings provide a useful basis for controlled viewpoint analysis. Observing the same signing performance from several cameras allows comparisons that hold content and timing fixed. This is a useful property for investigating what additional views reveal, rather than comparing unrelated performances captured from different angles. The involvement of Deaf signers and deaf-education specialists in preparing and checking the corpus is also a positive aspect of the collection process.

Critical weaknesses
  1. The linguistic scope and gloss standardization are insufficiently specified. The manuscript describes the source materials and expert involvement, but does not clearly identify which sign languages or regional varieties the four participants use, or how these sources and individual practices are converted into a consistent gloss inventory. In particular, it is unclear whether differences across signers are retained as variants, mapped to shared labels, or standardized during sentence preparation and recording. Shared gloss labels can be appropriate, but their interpretation depends on explicit conventions. These choices affect both what variation the models are expected to learn and how broadly the dataset’s vocabulary-coverage claims should be interpreted. This is a question about the linguistic meaning and scope of the annotations, rather than simply the number of participants.

  2. The resource contribution and the broader multi-view claims need clearer differentiation. How2Sign (Duarte, et al., CVPR 2021) already provides multi-view continuous signing, while Multi-VSL (Dinh, et al., WACV 2025) studies three-view Vietnamese isolated-sign recognition. TriViS adds a potentially useful combination of language and task coverage. However, the need for Vietnamese sign language data does not itself establish a VSL-specific multi-view mechanism, and the current experiments do not establish one. Such a mechanism is not a prerequisite for a valuable dataset. The concern is that language coverage and evidence about how multiple views improve continuous understanding are different contributions. The reported aggregate comparisons do not identify which continuous-signing phenomena or error types explain the gains, or what new understanding follows beyond the general observation that additional viewpoints can help. A targeted analysis of the relevant visual ambiguities would better connect the resource contribution to the broader scientific claims.

  3. The marginal value of the third camera is not established. The introduction emphasizes data collection under realistic constraints, but the main comparison between one and three views does not show whether two cameras already provide most of the benefit. Without frontal-plus-left and frontal-plus-right comparisons, the reader cannot determine how much the third camera adds, which failures it resolves, or whether that improvement warrants the additional acquisition and processing effort. Since the cameras record simultaneously, recording time should also be distinguished from equipment, synchronization, storage, preprocessing, and model-computation costs. A focused comparison using the existing recordings and matched model configurations would make the proposed collection guidance more actionable. Otherwise, the claim of providing a blueprint for future collection should be narrowed.

  4. The effect of sentence simplification on continuous-signing coverage is unclear. Section 3.1 states that sentences considered “overly complex” or incompatible with VSL grammar were removed or revised. Adapting unsuitable written-language constructions into natural signing is reasonable, but this should be distinguished from removing linguistically meaningful structures because they are difficult to annotate or perform. The paper does not explain what “overly complex” means, how many sentences were affected, or which contextual and compositional phenomena remain after this process. The relatively short average sentence length is not, by itself, evidence of low dataset value. What is needed is a transparent account of the deletion and revision rates, representative before/after examples, and an expert-supported description of the retained linguistic phenomena. This would help establish the contribution beyond isolated-sign resources and the scope of the practical applications emphasized in the paper.

  5. Ethical oversight, dataset governance, and reviewer anonymity require clarification. TriViS includes identifiable recordings and detailed metadata from four adult Deaf signers, but the manuscript does not document ethical review or informed consent. These omissions limit assessment of participant protections without establishing that approval or consent was absent. The authors should clarify the basis for approval, exemption, or not requiring review, and whether consent covers identifiable imagery, personal attributes, dataset distribution, downstream reuse, and any external API processing. Describing Table 5 as “anonymized” also requires justification given its detailed individual attributes and the visible facial imagery. Source-material permissions, dataset reuse terms, and withdrawal or removal procedures should be specified. Finally, the linked Hugging Face dataset currently requires visitors to share their username and email with the dataset maintainers to obtain access. This creates a reviewer-anonymity concern, and an alternative access arrangement should be provided that preserves both reviewer anonymity and appropriate participant privacy safeguards.

Questions for authors
  1. Which sign languages or regional varieties do the four signers represent? What linguistic references and annotation guidelines define the gloss inventory, and are differences across signers preserved, mapped to common labels, or standardized during corpus preparation and recording?

  2. Beyond expanding Vietnamese sign language coverage, what specific empirical insight does TriViS establish about continuous multi-view understanding? Can the authors identify the relevant sentence-level phenomena or visual ambiguities and show, through targeted examples and error analysis, when an additional view resolves them?

  3. Under the same training and evaluation conditions, how do frontal-plus-left and frontal-plus-right inputs compare with all three views? What is the third camera’s marginal benefit, on which examples does it help, and what additional acquisition or processing costs accompany it?

  4. How were “overly complex” sentences identified, and what proportions were removed and revised for this reason? Please distinguish grammatical adaptation from complexity reduction, provide representative before/after examples, and explain which phenomena requiring context or compositional understanding remain represented.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.