- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces vision-only sign language sentence segmentation (VSLSS), which splits long multi-sentence signing videos into sentence-level clips without text at inference, as a preprocessing step for sentence-level SLT. The proposed FVS combines a pose branch (regional GCN + dual-path dilated TCN) with a Skeletal Motion Cepstral Coefficient (SMCC) branch, an MFCC-style pipeline applied to confidence-weighted joint motion energy. The two are fused by a learnable gate followed by a Transformer, and trained with prior-guided losses (Dice, temporal continuity, and a differentiable sentence-count loss). FVS reports large gains in F1@50 and SER on PHOENIX-2014T, How2Sign, and OpenASL, and recovers most of the BLEU-4 gap between unsegmented and ground-truth-segmented input for Uni-Sign on How2Sign.
Critical strengths
- The problem is practical and well motivated. Sentence-level SLT models do fail on long input (BLEU-4 2.98 → 13.35 with FVS segmentation), so a lightweight vision-only segmenter is useful to the community.
- Gains at strict thresholds are substantial on the naturally continuous datasets (F1@50 +16.7 on How2Sign, +15.8 on OpenASL) and SER is much lower than all baselines.
Critical weaknesses
- PHOENIX-2014T construction may introduce artificial boundary cues. Long videos are built by concatenating independently trimmed sentence clips, so concatenation points likely contain pose jumps or discontinuities that a motion-energy feature like SMCC can trivially detect. This setting does not reflect real continuous signing. A simple non-learned baseline, such as thresholding frame-to-frame keypoint displacement, would show how much of the PHOENIX performance comes from these artifacts.
- Ground-truth quality and task definition. “Sentence” boundaries in How2Sign and OpenASL come from subtitle or translation-unit timestamps, which are known to be imperfectly aligned with signing. The paper should discuss how this label noise affects the evaluation and what a “sentence” means in VSLSS.
Questions for authors
Concern 1 (PHOENIX construction): The long videos for PHOENIX-2014T are built by concatenating independently trimmed sentence clips. Concatenation points are therefore likely to contain artificial cues, such as pose discontinuities or frame jumps, which do not occur in natural continuous signing. A motion-energy feature like SMCC is especially sensitive to such cues, so the model may partly be detecting editing cuts rather than real sentence boundaries. This matters because all ablations, including the key evidence for SMCC in Table 2, are conducted only on PHOENIX dev. I would ask the authors to report a simple non-learned baseline, such as thresholding frame-to-frame keypoint displacement, on PHOENIX, and to repeat the main ablations on How2Sign or OpenASL.
Concern 2 (ground truth and task definition): On How2Sign and OpenASL, the “sentence” boundaries come from subtitle timestamps rather than annotated boundaries in the signing itself. Subtitle timing is often misaligned with the signing, and a subtitle unit is not necessarily a complete signed sentence. This makes the F1@50 and SER numbers hard to interpret, since the model is evaluated against noisy targets. It also means the task in practice is learning to reproduce subtitle segmentation. The authors should state this explicitly, discuss how label noise affects the results, and clarify whether the 1–2 minute chunks are cut at sentence boundaries, since that would provide free boundaries at the chunk endpoints.
3 comments
Response to Q1.
We sincerely thank you for raising this important concern. To investigate whether our method remains effective on naturally continuous videos, we repeated the main ablation experiments on OpenASL and have included the results in Appendix A.9. These results further demonstrate the effectiveness of FVS on OpenASL, which consists of naturally continuous videos.
To directly examine the potential impact of artificial cues in PHOENIX-2014T, we have clarified the concatenation procedure and the corresponding quality checks in Appendix A.3 to make our data construction process more transparent. More importantly, as suggested by the reviewer, we have reported a simple non-learned baseline—thresholding frame-to-frame keypoint displacement in Appendix A.8 of the revised manuscript. The experimental results show that this simple displacement-thresholding baseline performs extremely poorly on both the artificially concatenated PHOENIX-2014T and the naturally continuous OpenASL.
The experimental results are provided in the following table:
Table: Performance comparison between the non-learned motion threshold baseline and FVS on the PHOENIX-2014T dataset. (Corresponds to Table 18 in the paper.)
| Method | dev F1@10 | dev F1@25 | dev F1@50 | dev SER | test F1@10 | test F1@25 | test F1@50 | test SER |
|---|---|---|---|---|---|---|---|---|
| Motion Threshold | 63.94 | 59.92 | 29.96 | 0.532 | 61.69 | 56.66 | 26.91 | 0.774 |
| FVS (ours) | 91.90 | 91.56 | 90.05 | 0.088 | 92.37 | 92.26 | 90.98 | 0.062 |
Table: Performance comparison between the non-learned motion threshold baseline and FVS on the OpenASL dataset. (Corresponds to Table 19 in the paper.)
| Method | dev F1@10 | dev F1@25 | dev F1@50 | dev SER | test F1@10 | test F1@25 | test F1@50 | test SER |
|---|---|---|---|---|---|---|---|---|
| Motion Threshold | 53.10 | 48.31 | 27.83 | 0.634 | 52.30 | 47.28 | 24.60 | 0.668 |
| FVS (ours) | 89.39 | 88.26 | 86.17 | 0.055 | 89.32 | 88.07 | 85.94 | 0.078 |
Table: Impact of frequency representation. (Corresponds to Table 20 in the paper.)
| Representation | F1@10 | F1@25 | F1@50 | SER |
|---|---|---|---|---|
| Pose-only | 84.82 | 83.53 | 78.14 | 0.206 |
| Motion Energy | 83.01 | 82.68 | 76.82 | 0.223 |
| SFFT-based | 87.24 | 86.73 | 82.16 | 0.151 |
| Mel-based | 88.19 | 87.07 | 84.70 | 0.144 |
| SMCC | 88.27 | 87.46 | 85.10 | 0.113 |
| Pose+SMCC | 89.39 | 88.26 | 86.17 | 0.055 |
Table: Impact of the proposed Prior-Guided Losses (PGL). indicates that the corresponding loss component is removed. (Corresponds to Table 21 in the paper.)
| Cnt. | Dice | Temp | AUX | F1@10 | F1@25 | F1@50 | SER |
|---|---|---|---|---|---|---|---|
| × | × | × | × | 80.12 | 78.98 | 73.61 | 0.262 |
| × | 83.71 | 84.12 | 79.85 | 0.182 | |||
| × | 82.41 | 83.12 | 79.09 | 0.194 | |||
| × | 81.26 | 80.42 | 77.47 | 0.223 | |||
| × | 81.87 | 82.79 | 78.02 | 0.201 | |||
| 89.39 | 88.26 | 86.17 | 0.055 |
Table: Impact of multi-dilation temporal convolution module. (Corresponds to Table 22 in the paper.)
| Variant | F1@10 | F1@25 | F1@50 | SER |
|---|---|---|---|---|
| w/o TCN | 72.17 | 70.52 | 64.07 | 0.319 |
| Single-path TCN | 86.16 | 85.50 | 80.39 | 0.175 |
| Reverse-path TCN | 84.11 | 83.84 | 76.90 | 0.227 |
| DM-TCN | 89.39 | 88.26 | 86.17 | 0.055 |
Table: Impact of fusion strategy. (Corresponds to Table 23 in the paper.)
| Fusion | F1@10 | F1@25 | F1@50 | SER |
|---|---|---|---|---|
| Fixed | 86.37 | 85.01 | 79.79 | 0.184 |
| Concatenation + FC | 86.85 | 85.74 | 80.91 | 0.161 |
| Learnable Gate | 89.39 | 88.26 | 86.17 | 0.055 |
Response to W2.
We thank you for raising this concern. We would like to clarify an important aspect of the annotations in How2Sign and OpenASL. The sentence boundaries used in our experiments are manually annotated by sign-language experts based on the video content, rather than automatically inferred from subtitle timestamps. Therefore, the boundary annotations do not inherit potential temporal misalignment errors from subtitle-to-video alignment.
We further clarify that a “sentence” in VSLSS refers to a sentence-level unit defined by the ground-truth annotations. The objective of VSLSS is to detect the boundaries of these annotated sentence units directly from continuous signing videos. For How2Sign and OpenASL, these ground-truth boundaries are manually annotated by sign-language experts through inspection of the video content.
Manual sentence boundary annotation may still involve some degree of subjectivity, particularly when the transitions between consecutive signing sentences are visually ambiguous. Nevertheless, these annotations are based on direct inspection of the signing videos and are independent of subtitle timestamp alignment. Thus, the potential annotation uncertainty discussed by you does not arise from treating subtitle timestamps as ground-truth sentence boundaries.
Response to Q2.
Thanks for pointing out that our original description of the annotation process and chunk construction was insufficiently clear, which may have led to a misunderstanding. How2Sign and OpenASL consist of naturally continuous long videos. The sentence boundaries in these datasets are manually annotated by sign language users with relevant expertise, based on the video content, rather than being inferred from subtitle timestamps.
Regarding chunk construction, the 1–2-minute chunks are obtained by randomly partitioning the videos along the temporal dimension, independently of sentence boundaries. In other words, the chunk start and end points are not deliberately aligned with sentence boundaries and therefore do not provide additional boundary cues to the model. This design is intended to prevent the model from exploiting manually imposed chunk endpoints as shortcuts. It also allows us to evaluate whether FVS can reliably identify both intra-sentence and inter-sentence temporal boundaries in continuous video segments, which may contain only partial sentences. In Appendix A.3 of the revised manuscript, we have revised the manuscript to explicitly clarify the source of the ground-truth annotations and the chunk construction strategy. We appreciate the reviewer’s feedback.
Thank you for your valuable and constructive feedback.
To cU6p
We are grateful for the insightful comments and clarify the concerns as follows.
Response to W1.
Thanks for raising this important concern. Regarding video construction, we follow the original recording dates and continuous sequence indices to concatenate the sentence-level clips, rather than arbitrarily or randomly stitching them. This preserves the original temporal ordering of the recordings and helps avoid artificial discontinuities caused by incorrect clip ordering. In addition, we conducted random manual inspections of the concatenated videos with the assistance of sign language users, focusing particularly on transitions between adjacent clips. The inspected transitions were generally natural, and no obvious pose discontinuities, abrupt inter-frame transitions, or other artificial artifacts were observed.
To better demonstrate the generalization ability of our method across different datasets, we conducted additional ablation experiments on OpenASL, which consists of naturally continuous long-form signing videos. In contrast to PHOENIX-2014T, these videos do not require concatenating independently trimmed sentence clips to construct continuous sequences. The additional results in Appendix A.8 of the revised manuscript show that motion-frequency representations achieve similar performance improvements on naturally continuous long-form sign language videos.