acceptodds
Under review as a conference paper at ICLR 2027

When Frequency Meets Sign: Exploiting Complementary Signals for Vision-Only Sign Language Sentence Segmentation

Abstract

Most existing sign language translation (SLT) systems are developed for sentence-level video clips, limiting their ability to handle long videos containing multiple sentences in real-world scenarios. To bridge this gap, we introduce the Vision-only Sign Language Sentence Segmentation (VSLSS) setting, which partitions long multi-sentence videos into sentence-level clips solely from visual signals. As a preprocessing step, VSLSS allows existing sentence-level SLT models to process multi-sentence videos without retraining on multi-sentence data. However, sign language often lacks explicit motion pauses between sentences, making accurate sentence boundary localization from visual information alone challenging. To address this challenge, we propose FVS, a frequency-aware vision-only sign language sentence segmentation framework. To better exploit visual cues for sentence boundary localization, FVS introduces a Skeletal Motion Cepstral Coefficient (SMCC) extractor to capture additional high-frequency motion information beyond pose features. The pose and motion-frequency features are processed by two parallel branches and adaptively fused through a learnable Gated Fusion module. To alleviate under-segmentation, over-segmentation, and fragmented predictions, we further design Prior-Guided Losses (PGL). Extensive experiments demonstrate the effectiveness of FVS, and downstream experiments further validate its utility for sentence-level SLT.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Sign Language Sentence Segmentation, Temporal Segmentation, Frequency-Domain Representation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cU6pOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces vision-only sign language sentence segmentation (VSLSS), which splits long multi-sentence signing videos into sentence-level clips without text at inference, as a preprocessing step for sentence-level SLT. The proposed FVS combines a pose branch (regional GCN + dual-path dilated TCN) with a Skeletal Motion Cepstral Coefficient (SMCC) branch, an MFCC-style pipeline applied to confidence-weighted joint motion energy. The two are fused by a learnable gate followed by a Transformer, and trained with prior-guided losses (Dice, temporal continuity, and a differentiable sentence-count loss). FVS reports large gains in F1@50 and SER on PHOENIX-2014T, How2Sign, and OpenASL, and recovers most of the BLEU-4 gap between unsegmented and ground-truth-segmented input for Uni-Sign on How2Sign.

Critical strengths
  1. The problem is practical and well motivated. Sentence-level SLT models do fail on long input (BLEU-4 2.98 → 13.35 with FVS segmentation), so a lightweight vision-only segmenter is useful to the community.
  2. Gains at strict thresholds are substantial on the naturally continuous datasets (F1@50 +16.7 on How2Sign, +15.8 on OpenASL) and SER is much lower than all baselines.
Critical weaknesses
  1. PHOENIX-2014T construction may introduce artificial boundary cues. Long videos are built by concatenating independently trimmed sentence clips, so concatenation points likely contain pose jumps or discontinuities that a motion-energy feature like SMCC can trivially detect. This setting does not reflect real continuous signing. A simple non-learned baseline, such as thresholding frame-to-frame keypoint displacement, would show how much of the PHOENIX performance comes from these artifacts.
  2. Ground-truth quality and task definition. “Sentence” boundaries in How2Sign and OpenASL come from subtitle or translation-unit timestamps, which are known to be imperfectly aligned with signing. The paper should discuss how this label noise affects the evaluation and what a “sentence” means in VSLSS.
Questions for authors

Concern 1 (PHOENIX construction): The long videos for PHOENIX-2014T are built by concatenating independently trimmed sentence clips. Concatenation points are therefore likely to contain artificial cues, such as pose discontinuities or frame jumps, which do not occur in natural continuous signing. A motion-energy feature like SMCC is especially sensitive to such cues, so the model may partly be detecting editing cuts rather than real sentence boundaries. This matters because all ablations, including the key evidence for SMCC in Table 2, are conducted only on PHOENIX dev. I would ask the authors to report a simple non-learned baseline, such as thresholding frame-to-frame keypoint displacement, on PHOENIX, and to repeat the main ablations on How2Sign or OpenASL.

Concern 2 (ground truth and task definition): On How2Sign and OpenASL, the “sentence” boundaries come from subtitle timestamps rather than annotated boundaries in the signing itself. Subtitle timing is often misaligned with the signing, and a subtitle unit is not necessarily a complete signed sentence. This makes the F1@50 and SER numbers hard to interpret, since the model is evaluated against noisy targets. It also means the task in practice is learning to reproduce subtitle segmentation. The authors should state this explicitly, discuss how label noise affects the results, and clarify whether the 1–2 minute chunks are cut at sentence boundaries, since that would provide free boundaries at the chunk endpoints.

3 comments
AuthorsOpenReview ↗

Thank you for your valuable and constructive feedback.

To cU6p

We are grateful for the insightful comments and clarify the concerns as follows.

Response to W1.

Thanks for raising this important concern. Regarding video construction, we follow the original recording dates and continuous sequence indices to concatenate the sentence-level clips, rather than arbitrarily or randomly stitching them. This preserves the original temporal ordering of the recordings and helps avoid artificial discontinuities caused by incorrect clip ordering. In addition, we conducted random manual inspections of the concatenated videos with the assistance of sign language users, focusing particularly on transitions between adjacent clips. The inspected transitions were generally natural, and no obvious pose discontinuities, abrupt inter-frame transitions, or other artificial artifacts were observed.

To better demonstrate the generalization ability of our method across different datasets, we conducted additional ablation experiments on OpenASL, which consists of naturally continuous long-form signing videos. In contrast to PHOENIX-2014T, these videos do not require concatenating independently trimmed sentence clips to construct continuous sequences. The additional results in Appendix A.8 of the revised manuscript show that motion-frequency representations achieve similar performance improvements on naturally continuous long-form sign language videos.

Response to Q1.

We sincerely thank you for raising this important concern. To investigate whether our method remains effective on naturally continuous videos, we repeated the main ablation experiments on OpenASL and have included the results in Appendix A.9. These results further demonstrate the effectiveness of FVS on OpenASL, which consists of naturally continuous videos.

To directly examine the potential impact of artificial cues in PHOENIX-2014T, we have clarified the concatenation procedure and the corresponding quality checks in Appendix A.3 to make our data construction process more transparent. More importantly, as suggested by the reviewer, we have reported a simple non-learned baseline—thresholding frame-to-frame keypoint displacement in Appendix A.8 of the revised manuscript. The experimental results show that this simple displacement-thresholding baseline performs extremely poorly on both the artificially concatenated PHOENIX-2014T and the naturally continuous OpenASL.

The experimental results are provided in the following table:

Table: Performance comparison between the non-learned motion threshold baseline and FVS on the PHOENIX-2014T dataset. (Corresponds to Table 18 in the paper.)

Methoddev F1@10dev F1@25dev F1@50dev SERtest F1@10test F1@25test F1@50test SER
Motion Threshold63.9459.9229.960.53261.6956.6626.910.774
FVS (ours)91.9091.5690.050.08892.3792.2690.980.062

Table: Performance comparison between the non-learned motion threshold baseline and FVS on the OpenASL dataset. (Corresponds to Table 19 in the paper.)

Methoddev F1@10dev F1@25dev F1@50dev SERtest F1@10test F1@25test F1@50test SER
Motion Threshold53.1048.3127.830.63452.3047.2824.600.668
FVS (ours)89.3988.2686.170.05589.3288.0785.940.078

Table: Impact of frequency representation. (Corresponds to Table 20 in the paper.)

RepresentationF1@10F1@25F1@50SER
Pose-only84.8283.5378.140.206
Motion Energy83.0182.6876.820.223
SFFT-based87.2486.7382.160.151
Mel-based88.1987.0784.700.144
SMCC88.2787.4685.100.113
Pose+SMCC89.3988.2686.170.055

Table: Impact of the proposed Prior-Guided Losses (PGL). ×\times indicates that the corresponding loss component is removed. (Corresponds to Table 21 in the paper.)

Cnt.DiceTempAUXF1@10F1@25F1@50SER
××××80.1278.9873.610.262
×83.7184.1279.850.182
×82.4183.1279.090.194
×81.2680.4277.470.223
×81.8782.7978.020.201
89.3988.2686.170.055

Table: Impact of multi-dilation temporal convolution module. (Corresponds to Table 22 in the paper.)

VariantF1@10F1@25F1@50SER
w/o TCN72.1770.5264.070.319
Single-path TCN86.1685.5080.390.175
Reverse-path TCN84.1183.8476.900.227
DM-TCN89.3988.2686.170.055

Table: Impact of fusion strategy. (Corresponds to Table 23 in the paper.)

FusionF1@10F1@25F1@50SER
Fixed g=0.5g=0.586.3785.0179.790.184
Concatenation + FC86.8585.7480.910.161
Learnable Gate89.3988.2686.170.055

Response to W2.

We thank you for raising this concern. We would like to clarify an important aspect of the annotations in How2Sign and OpenASL. The sentence boundaries used in our experiments are manually annotated by sign-language experts based on the video content, rather than automatically inferred from subtitle timestamps. Therefore, the boundary annotations do not inherit potential temporal misalignment errors from subtitle-to-video alignment.

We further clarify that a “sentence” in VSLSS refers to a sentence-level unit defined by the ground-truth annotations. The objective of VSLSS is to detect the boundaries of these annotated sentence units directly from continuous signing videos. For How2Sign and OpenASL, these ground-truth boundaries are manually annotated by sign-language experts through inspection of the video content.

Manual sentence boundary annotation may still involve some degree of subjectivity, particularly when the transitions between consecutive signing sentences are visually ambiguous. Nevertheless, these annotations are based on direct inspection of the signing videos and are independent of subtitle timestamp alignment. Thus, the potential annotation uncertainty discussed by you does not arise from treating subtitle timestamps as ground-truth sentence boundaries.

Response to Q2.

Thanks for pointing out that our original description of the annotation process and chunk construction was insufficiently clear, which may have led to a misunderstanding. How2Sign and OpenASL consist of naturally continuous long videos. The sentence boundaries in these datasets are manually annotated by sign language users with relevant expertise, based on the video content, rather than being inferred from subtitle timestamps.

Regarding chunk construction, the 1–2-minute chunks are obtained by randomly partitioning the videos along the temporal dimension, independently of sentence boundaries. In other words, the chunk start and end points are not deliberately aligned with sentence boundaries and therefore do not provide additional boundary cues to the model. This design is intended to prevent the model from exploiting manually imposed chunk endpoints as shortcuts. It also allows us to evaluate whether FVS can reliably identify both intra-sentence and inter-sentence temporal boundaries in continuous video segments, which may contain only partial sentences. In Appendix A.3 of the revised manuscript, we have revised the manuscript to explicitly clarify the source of the ground-truth annotations and the chunk construction strategy. We appreciate the reviewer’s feedback.

Reviewer_XgsxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce FVS for vision-only Sign Language Sentence Segmentation, which segments long sign-language videos into sentences without captions at inference by combining pose and motion-frequency features.

Critical strengths
  1. A well-defined task with direct downstream evidence. VSLSS is a clean preprocessing formulation, and the paper shows that FVS clips bring a frozen sentence-level translator close to its ground-truth-segmentation performance. This is stronger than a segmentation benchmark alone: it shows that the predicted boundaries are useful to the downstream system, not merely close to the labels.

  2. A simple, interpretable, and reusable motion-frequency representation. SMCC adapts the MFCC pipeline to skeletal motion in a lightweight way, and the representation ablation is monotonic, e.g., motion energy, SFFT, Mel, and SMCC each add. The combined pose+SMCC model is clearly stronger than either branch alone. The idea is easy to reuse in other pose-based temporal tasks.

Critical weaknesses

The headline comparison is not controlled, so the source of the margin is unidentified. The paper reports large F1@50 gains over action-segmentation and sign-language baselines, but it does not document how those baselines were adapted to VSLSS: whether they received the same pose input, the same boundary labels, the same training completeness, and the same post-processing. The baselines are also not given the SMCC representation, even though SMCC is the paper's main contribution and accounts for a large part of FVS's gain. The reported margins therefore cannot distinguish a better representation, a better temporal model, and a better-tuned pipeline. This matters because the paper's central empirical claim is the size of the advantage over prior methods, not merely that FVS works.

Questions for authors
  1. How were MS-TCN, ASFormer, FACT, DiffAct, and SignBD adapted to VSLSS? Via same pose input, same boundary labels, same training completeness, same post-processing? And what do they achieve when given the same pose+SMCC representation as FVS?

  2. Segmental F1 at overlap thresholds is a segment-level metric. What is the boundary-level precision/recall within a temporal tolerance, and does the advantage over baselines hold?

8 comments
AuthorsOpenReview ↗

Thank you for your valuable and constructive feedback.

To Xgsx

We are grateful for the insightful comments and clarify the concerns as follows.

Response to W1.

We sincerely thank you for highlighting the importance of controlled baseline comparisons. As also kindly pointed out by the other reviewers, VSLSS is a newly introduced task that aims to segment continuous sign language videos into sentence-level units using visual information alone. To adapt these baselines to VSLSS, we adopted the same boundary labels, the same training budget, and the same post-processing. We agree that evaluating all methods with identical input representations would provide a more controlled comparison and help disentangle the contributions of input representations and temporal architectures. However, achieving such a comparison requires substantial modifications to the original implementations. The selected methods were originally developed under different task formulations and architectural assumptions, and their input interfaces are not directly interchangeable. Such modifications would also require additional implementation and computational resources, while potentially introducing discrepancies from the original methods.

We would also like to clarify that the primary contribution of FVS is not simply a different temporal segmentation architecture. Rather, we introduce a vision-only formulation for sign language sentence segmentation and explore how complementary visual motion representations can improve boundary localization. Specifically, FVS combines pose features, which capture the overall signing structure, with SMCC features, which encode fine-grained skeletal motion variations. These representations are adaptively integrated through Gated Fusion and jointly optimized with Prior-Guided Losses to improve transition localization and prediction consistency. In this process, the pose features and SMCC features are fused through a specifically designed architecture, rather than through simple feature concatenation.

We acknowledge that the current comparison does not fully isolate the contribution of each input representation from that of the temporal architecture. We have therefore clarified the scope of the baseline comparison in the revised manuscript to avoid over-interpreting the performance differences as purely architectural gains. Following the your valuable suggestion, we believe that future work on a unified benchmarking protocol, where different temporal architectures are evaluated using identical Pose and Pose+SMCC inputs under matched training and post-processing settings, would provide a more comprehensive understanding of the respective contributions of representation learning and temporal modeling. We appreciate this helpful suggestion, which also points to an important direction for further investigation.

Response to Q1.

We sincerely thank you for pointing out that the description of our baseline comparison settings in the original manuscript was insufficiently detailed. To adapt the baseline models to the VSLSS task, we adopted the same training budget, boundary annotations, and post-processing strategy for all models to ensure a consistent evaluation setting. One important distinction, however, is that the baseline models take the original RGB videos as input, whereas our proposed FVS model takes 2D pose sequences extracted from the videos as input.

It is worth emphasizing that 2D pose sequences provide a structured and more compact representation of human motion in the original RGB videos. Compared with RGB videos, they retain more task-relevant information about signing movements while discarding substantial appearance and background information that is less directly relevant to boundary localization. Therefore, FVS does not benefit from a richer visual input. Instead, despite using a more compact input representation and a smaller model, FVS achieves superior performance, further demonstrating the effectiveness of our representation and model design. Furthermore, our proposed SMCC feature, is essentially a frequency-domain representation obtained by filtering the 2D pose sequences. It does not introduce any additional external information. Meanwhile, in FVS, the pose sequences and SMCC features are not simply concatenated. They are integrated through a specifically designed network architecture that enables more effective interaction and fusion between the two complementary representations.

Based on these considerations, and to respect the original designs of the baseline methods while avoiding unnecessary modifications to their architectures, we retain the RGB video inputs used in the original baseline papers rather than replacing them with pose sequences or SMCC features. We believe this setting better reflects the performance of each method under its original input configuration.

Finally, the ablation studies presented in Section 5.4 of the revised manuscript further validate the performance advantages of FVS from multiple perspectives. The results demonstrate that both the SMCC features and the effective fusion of pose sequences and SMCC contribute to the overall performance improvement. This suggests that the advantage of FVS does not stem solely from any individual input representation, but rather from the combination of a more effective feature representation and a carefully designed fusion mechanism. In Section 5.3 of the revised manuscript, we have further clarified these experimental settings and design motivations in the revised manuscript to improve the transparency of our comparisons.

Response to Q2.

We sincerely thank you for this important and constructive suggestion. In addition to overall segmentation performance, evaluating temporal localization accuracy and boundary detection quality can provide a more comprehensive assessment of a model’s performance on the VSLSS task.

In the original manuscript, following Sentence-level Segmentation for Long Sign Language Videos with Captions and evaluation practices in the action segmentation literature, we primarily adopted F1@10/25/50 and SER as our main evaluation metrics to assess segmentation performance.

Meanwhile, to further evaluate the temporal localization accuracy of predicted boundaries, we have supplemented Appendix A.7 with Boundary MAE and Precision/Recall metrics under fixed temporal tolerances, including [email protected], [email protected], [email protected], and [email protected]. These metrics provide a more detailed assessment of model performance in terms of boundary localization errors and boundary detection quality under different tolerance thresholds.

The experimental results are provided in the following table:

Table: Boundary-level temporal localization performance on the Dev and Test sets of PHOENIX-2014T, How2Sign, and OpenASL. (Corresponds to Table 17 in the paper.)

DatasetDev [email protected]Dev [email protected]Dev [email protected]Dev [email protected]Dev MAETest [email protected]Test [email protected]Test [email protected]Test [email protected]Test MAE
PHOENIX-2014T93.0891.7793.2692.950.03194.0093.8494.3694.210.029
How2Sign72.3965.1683.9577.280.08371.2464.3181.7576.400.082
OpenASL72.9568.1093.2187.810.04271.8567.4792.6786.830.046

Regarding the reporting of metrics for baseline models, we referenced the experimental results from the paper by Sentence-level Segmentation for Long Sign Language Videos with Captions. However, their original paper did not report the aforementioned boundary-level metrics. Consequently, it is currently difficult to directly obtain fully comparable evaluation results. We also hope that future papers related to the VSLSS task will publicly report the aforementioned metrics that directly measure temporal offsets.

Reviewer_ZKrvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the task of pure visual sign language sentence segmentation (Vision-only Sign Language Sentence Segmentation, VSLSS), aiming to divide continuous long videos into sentence-level segments without relying on text information. To this end, the authors propose the Frequency-aware Vision-only Segmentation (FVS) framework. It extracts frequency-domain features of skeletal motion using Skeletal Motion Cepstral Coefficients (SMCC), and combines Dual-Path Multi-Expansion Rate Temporal Convolution (DM-TCN), Gated Fusion, and Prior Guidance Loss (PGL) to improve the accuracy of sentence boundary positioning. Experiments show that this method outperforms existing visual baselines on three datasets, PHOENIX-2014T, How2Sign, and OpenASL, and can significantly improve the performance of downstream sign language translation.

Critical strengths
  1. It addresses the problem of continuous sign language sentence segmentation without text assistance, thereby overcoming the application limitations of existing sentence-level sign language translation systems.
  2. By combining skeleton postures and motion frequency domain information, it provides an effective approach for capturing sentence boundary cues at different time scales.
  3. The FVS outperforms the compared pure visual baselines in terms of F1@10, F1@25, F1@50 and SER metrics on three datasets, especially showing a significant improvement in F1@50.
Critical weaknesses
  1. PHOENIX-2014T is manually stitched, and VBA uses a fixed proportion to expand the inter-sentence regions, which may introduce unnatural boundaries and annotation deviations, and lacks manual verification.
  2. F1@10/25/50 and SER mainly reflect the error in fragment overlap and quantity, and cannot directly prove the accuracy of the predicted boundaries in terms of time position.
  3. At 25 FPS, the Mel scale is approximately linear within the effective frequency range, and the 0.8-second window and 0.2-second step size may blur the fine boundaries.
  4. SMCC is based on MFCC, while DM-TCN has two paths that only reverse the order of expansion rates and have the same theoretical receptive field. Their complementary mechanism has not been fully proven.
Questions for authors
  1. Will the manual stitching and fixed proportion VBA annotations of PHOENIX-2014T introduce easily identifiable artificial boundaries? Can the evaluation and analysis of the impact of different VBA proportions on the results be conducted on the natural continuous video for manual verification of sentence boundaries?
  2. Since the main goal of this paper is to precisely locate sentence boundaries, why is there no report of direct indicators for measuring time offsets, such as Boundary MAE or Precision/Recall under fixed time tolerances (±0.2s, ±0.5s)?
  3. Considering that the Nyquist frequency of the 25 FPS skeleton signal is only 12.5 Hz, why did the authors choose the Mel filter bank derived from speech processing? Can the linear frequency filter bank, other frequency domain representations, and different window lengths and strides be compared to prove the necessity of the SMCC design?
  4. The two paths of DM-TCN have the same theoretical receptive field. How did the authors prove that the reverse arrangement expansion rate can learn truly complementary local and global features, rather than merely benefiting from the increase in model capacity? Can a parameter quantity-matched control experiment be provided?
5 comments
AuthorsOpenReview ↗

Thank you for your valuable and constructive feedback.

To ZKRV

We are grateful for the insightful comments and clarify the concerns as follows.

Response to W1.

Thanks for pointing out the potential concerns regarding the construction of PHOENIX-2014T and the use of VBA annotations. Regarding video construction, we follow the original recording dates and continuous sequence indices to concatenate the sentence-level clips, rather than arbitrarily or randomly stitching them. This preserves the original temporal ordering of the recordings and helps avoid artificial discontinuities caused by incorrect clip ordering. In addition, we conducted random manual inspections of the concatenated videos with the assistance of sign language users, focusing particularly on transitions between adjacent clips. The inspected transitions were generally natural, and no obvious pose discontinuities, abrupt inter-frame transitions, or other artificial artifacts were observed.

Regarding VBA, we adopt the same annotation proportion V=0.2V=0.2 as in Sentence-level Segmentation for Long Sign Language Videos with Captions. VBA expands each sentence boundary into a multi-frame transition region to provide frame-level supervision and alleviate the class imbalance between transition and intra-sentence frames. Noted, this proportion controls the extent of the training transition region rather than directly specifying the final predicted boundary. During inference, the final boundary is determined by selecting the frame with the highest predicted transition probability within each candidate transition region. Therefore, for the PHOENIX-2014T dataset, manually concatenating the clips and using VBA to expand the inter-sentence regions do not introduce unnatural boundaries or annotation bias, and this process has been manually verified by sign language users.

Response to Q1.

For the annotation of the PHOENIX-2014T dataset, we follow the Sentence-level Segmentation for Long Sign Language Videos with Captions and adopt Variable-length Boundary Annotation (VBA) rather than fixed-length boundary annotation. Therefore, boundary positions are not divided according to a preset fixed time interval or uniform length. Additionally, artificial boundary cues that are easily identifiable by the model are not directly introduced by fixed-length segmentation. In Appendix A.11 of the revised manuscript, following your suggestion, we compared different VBA annotation proportions on the PHOENIX-2014T and OpenASL datasets. The results show that both datasets achieve the best performance at a proportion of 0.2. However, even with the optimal VBA proportion, the performance on OpenASL remains lower than that obtained with ground-truth annotations. This effectively rules out the possibility that the fixed-proportion VBA annotation introduces easily identifiable artificial boundaries.

To further examine whether the effectiveness of FVS depends on the potential artifacts of concatenated videos, we repeated the main ablation experiments on OpenASL, which consists of naturally continuous sign language videos. The results, reported in Appendix A.9 of the revised manuscript, show that the proposed components, including SMCC, DM-TCN, the fusion strategy, and PGL, remain effective in this natural-video setting. This provides additional evidence that the effectiveness of FVS is not limited to artificially concatenated PHOENIX-2014T videos.

The experimental results are provided in the following table:

Table: Effect of different VBA proportions (β\beta) on the PHOENIX-2014T and OpenASL development sets. (Corresponds to Table 26 in the paper.)

VBA Proportion(β\beta)PHOENIX-2014T F1@10PHOENIX-2014T F1@25PHOENIX-2014T F1@50PHOENIX-2014T SEROpenASL F1@10OpenASL F1@25OpenASL F1@50OpenASL SER
0.0584.1882.7877.290.22977.9876.4471.010.261
0.1089.0787.8384.000.18483.3282.9878.970.208
0.2091.9091.5690.050.08887.6986.5284.140.162
0.3090.0688.9486.240.11585.8384.5380.740.173

Table: Impact of frequency representation. (Corresponds to Table 20 in the paper.)

RepresentationF1@10F1@25F1@50SER
Pose-only84.8283.5378.140.206
Motion Energy83.0182.6876.820.223
SFFT-based87.2486.7382.160.151
Mel-based88.1987.0784.700.144
SMCC88.2787.4685.100.113
Pose+SMCC89.3988.2686.170.055

Table: Impact of the proposed Prior-Guided Losses (PGL). ×\times indicates that the corresponding loss component is removed. (Corresponds to Table 21 in the paper.)

Cnt.DiceTempAUXF1@10F1@25F1@50SER
××××80.1278.9873.610.262
×83.7184.1279.850.182
×82.4183.1279.090.194
×81.2680.4277.470.223
×81.8782.7978.020.201
89.3988.2686.170.055

Table: Impact of multi-dilation temporal convolution module. (Corresponds to Table 22 in the paper.)

VariantF1@10F1@25F1@50SER
w/o TCN72.1770.5264.070.319
Single-path TCN86.1685.5080.390.175
Reverse-path TCN84.1183.8476.900.227
DM-TCN89.3988.2686.170.055

Table: Impact of fusion strategy. (Corresponds to Table 23 in the paper.)

FusionF1@10F1@25F1@50SER
Fixed g=0.5g=0.586.3785.0179.790.184
Concatenation + FC86.8585.7480.910.161
Learnable Gate89.3988.2686.170.055

Response to W2.

We agree that segmental F1@10/25/50 and SER evaluate segmentation quality from the perspectives of segment overlap and sentence-count estimation. But they do not directly quantify the temporal localization accuracy of individual predicted boundaries. To address this limitation, we have supplemented our evaluation with boundary-level metrics in the revised manuscript. Specifically, we report boundary Precision and Recall under two fixed temporal tolerances (±0.2 s and ±0.5 s), denoted as [email protected], [email protected], [email protected], and [email protected] in Appendix A.7 of the revised manuscript. These metrics measure the precision and recall of predicted boundaries when their temporal deviations from the corresponding ground-truth boundaries fall within the specified tolerances. We further introduce Boundary MAE to directly quantify the average absolute temporal deviation between matched predicted and ground-truth boundaries.

Response to Q2.

In the original manuscript, we mainly followed Sentence-level Segmentation for Long Sign Language Videos with Captions and related evaluation practices in the action segmentation field, using F1@10/25/50 and SER as the main evaluation metrics to measure the overall segmentation performance of the model.

Following your suggestion, we have further added boundary-level evaluation metrics in Appendix A.7 of the revised manuscript. It includes Boundary MAE and Precision and Recall computed under fixed temporal tolerances, specifically [email protected], [email protected], [email protected], and [email protected]. These metrics provide complementary evaluations of the model's temporal localization capability from the perspectives of boundary localization deviation and boundary detection performance under different temporal tolerances.

The experimental results are provided in the following table:

Table: Boundary-level temporal localization performance on the Dev and Test sets of PHOENIX-2014T, How2Sign, and OpenASL. (Corresponds to Table 17 in the paper.)

DatasetDev [email protected]Dev [email protected]Dev [email protected]Dev [email protected]Dev MAETest [email protected]Test [email protected]Test [email protected]Test [email protected]Test MAE
PHOENIX-2014T93.0891.7793.2692.950.03194.0093.8494.3694.210.029
How2Sign72.3965.1683.9577.280.08371.2464.3181.7576.400.082
OpenASL72.9568.1093.2187.810.04271.8567.4792.6786.830.046

Regarding the reporting of metrics, we referenced the experimental results from the paper by Sentence-level Segmentation for Long Sign Language Videos with Captions. However, their original paper did not report the aforementioned boundary-level metrics. Consequently, it is currently difficult to directly obtain fully comparable evaluation results. We also hope that future papers related to the VSLSS task will publicly report the aforementioned metrics that directly measure temporal offsets.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Review and Reviewer–Author Discussion Summary

We sincerely thank all reviewers for their careful evaluation of our manuscript and for their constructive and insightful comments. We greatly appreciate the time and effort they have devoted to understanding our work. We acknowledge that the feedback has helped us identify several aspects of our work that require further clarification and empirical validation. In response, we have carefully revisited the reviewers' concerns and conducted additional experiments and analyses to further examine the effectiveness, robustness, and design choices of our proposed FVS framework. The main questions raised during the review process and our corresponding responses are summarized as follows.

  • Robustness to artificial boundary cues and evaluation on naturally continuous videos (Reviewers ZKrv and cU6p). We clarified that PHOENIX-2014T videos are concatenated according to their original recording dates and continuous sequence indices, rather than in a random order. We also conducted manual inspections of randomly sampled concatenation points. The inspected videos generally exhibit natural transitions, without obvious pose discontinuities or abrupt frame-to-frame movements that could serve as artificial boundary cues. More importantly, we repeated the main ablation studies on OpenASL, which consists of naturally continuous videos. The results further support the effectiveness of the proposed components in this setting. In addition, we introduced a non-learned motion-threshold baseline based on frame-to-frame keypoint displacement. Its substantially lower performance on both PHOENIX-2014T and OpenASL suggests that simple motion discontinuities alone are insufficient to explain the performance of FVS. These results have been included in Appendix A.8 and A.9.

  • Annotation protocols, sentence definitions, and VBA sensitivity (Reviewers ZKrv, Xgsx, and cU6p). We clarified the annotation sources and video construction procedures for the three datasets. For How2Sign and OpenASL, sentence boundaries are manually annotated by sign-language experts based on video content, rather than inferred from subtitle timestamps. Their long-form videos are divided into contiguous 1–2-minute segments while preserving the original temporal order. The segments are non-overlapping and gap-free, and their endpoints are not aligned with predefined sentence boundaries. For PHOENIX-2014T, we follow the existing protocol and apply the Variable Boundary Assignment (VBA) strategy with a proportion of β=0.2\beta=0.2. We further clarified that VBA expands the frame-level transition region around an original ground-truth boundary. It does not change the source or identity of the underlying boundary annotation. All methods evaluated on the same dataset use the same dataset-specific ground-truth boundary locations and supervision protocol. Additional sensitivity experiments show that β=0.2\beta=0.2 performs best among the tested settings on both PHOENIX-2014T and OpenASL. Detailed analysis and experiments are provided in Appendix A.3 and A.11.

  • More direct evaluation of boundary localization (Reviewers ZKrv and Xgsx). In addition to segmental F1 and Segment Edit Score (SER), we introduced boundary-level precision and recall under temporal tolerances of ±0.2\pm 0.2 s and ±0.5\pm 0.5 s, together with Boundary Mean Absolute Error (MAE). On the test sets, FVS achieves boundary MAEs of 0.029 s, 0.082 s, and 0.046 s on PHOENIX-2014T, How2Sign, and OpenASL, respectively. These additional metrics provide a more direct assessment of temporal boundary localization. We also acknowledged that further improvements in localization accuracy and stability remain worthwhile directions. These results have been included in Appendix A.7.

  • Sensitivity and design choices of the SMCC representation (Reviewer ZKrv). We expanded the analysis of the frequency representation and temporal window configuration used by SMCC. Specifically, we compared linear-frequency and Mel-frequency representations, as well as different window lengths and hop sizes. The results show that the selected configuration, with a window length of 0.8 s and a hop size of 0.2 s, achieves the best overall performance among the tested settings. Detailed analysis and experiments are provided in Appendix A.10.

  • Complementarity of the two DM-TCN paths (Reviewer ZKrv). We clarified the motivation behind the two dilation orders in DM-TCN. Although the increasing- and decreasing-dilation paths have the same theoretical receptive field, they apply temporal transformations in different orders. The increasing-dilation path progressively aggregates fine-grained patterns into broader contextual representations, whereas the decreasing-dilation path first incorporates broader context and then refines the representations through finer-grained transformations. Since these paths consist of successive learnable nonlinear transformations, reversing the dilation order can lead to different feature representations. We further conducted ablation experiments comparing the full DM-TCN with single-path and reverse-path variants. The results support the effectiveness of the proposed dual-path design. We have included detailed derivations and experimental results in section 4.3 and 5.4.

  • Baseline adaptation and fairness of comparisons (Reviewer Xgsx). We provided additional details on the implementation and adaptation of the temporal segmentation baselines. We clarified that the compared methods are evaluated under the same dataset-specific boundary annotations and evaluation metrics, while their input representations may differ. In particular, conventional baselines use visual features extracted from RGB videos, whereas FVS operates on 2D pose sequences together with the corresponding SMCC features. We acknowledged that the main comparison should be interpreted as an overall-method comparison, rather than a strictly controlled architecture-only comparison.

  • Boundary localization versus downstream translation utility (Reviewer ZKrv). We further discussed the distinction between accurately localizing a sentence boundary and producing useful sentence-level video segments for downstream translation. In continuous signing, sentence transitions may span a temporal region rather than correspond to a single unambiguous frame. In addition, certain boundary errors may occur around coordinated or subordinate clauses, where the resulting segmentation can still preserve much of the semantic content. We emphasized that downstream translation utility provides complementary evidence of practical relevance, but should not be regarded as a substitute for direct boundary-localization evaluation. We also acknowledged the need for further improvements in precise and stable boundary detection.

We sincerely appreciate the reviewers' constructive comments, which have helped us improve the experimental validation, clarify the annotation protocols, and better articulate the scope and limitations of our approach. We believe the comprehensive rebuttal and added experiments have fully resolved all concerns and significantly enhanced the completeness and robustness of this work.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.