acceptodds
Under review as a conference paper at ICLR 2027

ZipSign: Training-Free Spatio-Temporal Token Compression for Sign Language Translation

Abstract

Sign language translation (SLT) models have made steady progress by improving visual representation learning and translation modeling. However, existing SLT models usually process redundant visual tokens across spatial and temporal dimensions, leading to substantial computational costs and limiting their deployment in resource-constrained and real-time scenarios. In this work, we propose ZipSign, to the best of our knowledge, the first training-free spatio-temporal token compression method for sign language translation. ZipSign adopts a two-stage token compression strategy. The first stage performs similarity-based spatial token merging, which preserves sign-critical hand and face regions while merging redundant background tokens within each frame. The second stage performs confidence-guided temporal token pruning, which removes less informative frame-level visual tokens before they are fed into the translation module. ZipSign introduces no additional learnable parameters and can be directly plugged into existing SLT models during inference. Extensive experiments on standard SLT benchmarks demonstrate that ZipSign effectively reduces computational cost and inference latency while maintaining translation performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sign Language Translation, Token Compression, Efficient Inference, Spatio-Temporal Modeling, Training-Free Acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xjYBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ZipSign, a training-free compression framework for sign language translation. ZipSign combines hand- and face-preserving spatial token merging with recognition-confidence-guided temporal pruning. The experiments on TwoStream and Sign2GPT across PHOENIX14T and CSL-Daily show inference speedups with limited translation-quality degradation.

Critical strengths

A key strength is the training-free design, which accelerates existing SLT models without retraining. Across two backbones and two datasets, the reported 1.7–2.0× speedups at moderate compression, with limited translation-quality degradation.

Critical weaknesses
  1. Spatial merging combines region protection, norm-directed fusion, weighted pooling, and resolution-adaptive budgets. A weighted-pooling-only baseline and separate ablations of these components are needed to identify which mechanisms produce the gains.
  2. On CSL-Daily, ZipSign reduces Sign2GPT latency by 41.4% while estimated GFLOPs remain nearly unchanged. Although FLOPs and latency need not scale proportionally, please provide component-wise profiling and clarify whether FLOP estimates cover the complete autoregressive decoding process, with matched decoding settings and reported output lengths.
  3. The manuscript would benefit from careful proofreading. Examples include “we splits … and performs” → “we split … and perform” (L222), “temporal redundant” → “temporally redundant” (L254), and “a segments” → “segments” (L271).
Questions for authors

The related work would benefit from discussing “Sign Language Translation with Iterative Prototype" (IP-SLT, ICCV 2023).

2 comments
AuthorsOpenReview ↗

Additional Component Ablations and Inference Efficiency Analysis of ZipSign.

Response to Weaknesses 1 Insufficient Component Ablations of Spatial Merging: We thank the reviewer for this valuable comment.

To isolate the contributions of individual spatial components, we conducted incremental ablations on TwoStream with the PHOENIX14T test set. Starting from weighted-pooling-only baseline, we progressively introduce region protection, resolution-adaptive budgets, and norm-directed fusion. All experiments use the same frozen checkpoint, with temporal pruning disabled.

As shown in Table 8, introducing region-protected merging improves BLEU-4 by 0.50. Resolution-adaptive budgets further improve BLEU-4 by 0.28. Adding norm-directed fusion yields an additional 1.14 BLEU-4 improvement. Overall, the complete spatial module improves BLEU-4 by 1.92 and ROUGE-L by 0.89 over weighted pooling alone, demonstrating the complementary contributions of these components.

Table 8: Ablation of spatial merging components on TwoStream with the PHOENIX14T test set (temporal pruning disabled).

MethodSpatial RetentionB4 ↑R-L ↑
Weighted Pooling Only100%26.8152.55
Pooling + Region Protection50%27.3152.85
Pooling + Region Protection + Resolution-Adaptive Budgets50%27.5952.93
Ours(Pooling + Region Protected + Resolution-Adaptive Budgets + Norm-Directed Fusion)50%28.7353.44

Response to Weaknesses 2: Thanks.

(1) Component-wise Profiling.

We conducted component-wise profiling on Sign2GPT with CSL-Daily using an RTX A6000 GPU. As shown in Table 9, ZipSign reduces total latency by 41.4% (378.58 → 221.78 ms/sample), primarily through a 47.7% reduction in autoregressive decoding time (339.59 → 177.74 ms/sample). Spatial pooling incurs an additional 5.06 ms, while feature fusion saves 1.98 ms and temporal selection introduces 2.15 ms overhead. The visual backbone remains nearly unchanged. Overall, the 161.85 ms reduction in decoder latency substantially outweighs the additional processing overhead.

This acceleration is achieved by temporal pruning, which shortens the visual sequence supplied to the pretrained language model (XGLM), reducing the visual context processed during autoregressive decoding. Thus, decoder acceleration dominates the overall inference speedup despite relatively small changes in estimated GFLOPs.

Table 9: Component-level inference latency breakdown on Sign2GPT with CSL-Daily test set (ms/sample, mean ± std).

ComponentBaselineZipSign (50%)
Visual Backbone22.64 ± 3.5022.46 ± 5.94
Spatial Pooling5.25 ± 2.4410.31 ± 5.05
Feature Fusion11.10 ± 2.619.12 ± 2.78
Temporal Token Selection—2.15 ± 5.66
Translation Decoder339.59 ± 4.10177.74 ± 6.03
Total Latency378.58 ± 9.68221.78 ± 7.52

(2) GFLOPs Accounting, Decoding Settings, and Output Lengths.

  1. GFLOPs Accounting. Our FLOP estimates cover the complete autoregressive decoding process. Total GFLOPs change little because the computationally expensive visual backbone remains unchanged after token compression. In contrast, XGLM decoding is sequential and memory-bound, so it accounts for little of the FLOPs but much of the latency. Fewer visual tokens shorten each decoding step, which reduces latency without notably changing FLOPs.

  2. Decoding settings. Both methods use the same frozen checkpoint and identical XGLM generation settings (decoding strategy, beam size, maximum length, stopping criteria). Latency covers full generation, excluding keypoint extraction.

  3. Output lengths. Output lengths are nearly identical on PHOENIX14T (19.16 vs. 19.35) and CSL-Daily (12.24 vs. 12.22) (Table 10), so the speedup comes from lower per-step cost rather than shorter outputs.

Table 10: Average generated-output lengths using the Sign2GPT backbone on PHOENIX14T and CSL-Daily test sets.

DatasetBaselineZipSign (50%)
PHOENIX14T19.1619.35
CSL-Daily12.2412.22

Response to Weaknesses 3 Grammatical Errors and Proofreading Issues: We thank the reviewer for pointing out these grammatical errors. We have corrected all the identified issues and carefully proofread the manuscript to improve its grammatical accuracy and overall clarity. All corrections are highlighted in green in the revised manuscript.

Response to Question 1: We thank the reviewer for this suggestion. We have added a discussion and citation of IP-SLT (ICCV 2023) in the Related Work section to provide a more comprehensive overview of existing SLT methods.

Reviewer_iNUjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ZipSign is a training free method for compressing visual tokens in sign language translation models, aimed at cutting inference cost without retraining or changing the architecture. It works in two stages. The spatial stage uses keypoints to locate hand and face regions, leaves those tokens untouched, and merges only the remaining background tokens through bipartite cosine matching in the style of ToMe, with merge direction set by feature norm so that the larger norm token becomes the destination. The frame representation is then formed by a weighted pooling that favours sign critical tokens. The temporal stage prunes frame level tokens before they reach the translation module, using gloss confidence from a CTC branch, or pseudo gloss confidence where no gloss branch exists. Consecutive frames sharing a prediction form a segment, the highest confidence token in each segment is kept as a key token, neighbouring tokens are added to preserve local motion, and any remaining budget is filled by confidence. Experiments cover two backbones, the gloss based TwoStream and the gloss free Sign2GPT, on PHOENIX14T and CSL-Daily, compared against Uniform sampling, MaxPool and STTM at temporal keep ratios of 50 and 30 percent. The central claim is that ZipSign gives a better trade off between translation quality and inference latency than those alternatives.

Critical strengths
  1. The method works clearly in its best setting. On the gloss based TwoStream backbone at the aggressive 30 percent keep ratio the gains are large and cannot be explained by noise. PHOENIX14T test gives 25.27 BLEU-4 against 20.97 for STTM, and CSL-Daily test gives 22.16 against 16.70 for MaxPool, so 4.30 and 5.46 points respectively. Margins of that size on test sets of 642 and 1,176 samples are real. The finding that confidence based frame selection beats uniform sampling and pooling once the budget is tight is credible and worth knowing.

  2. The problem matters and the solution is easy to adopt. Inference efficiency in SLT really is underexplored. The method needs no training and no architecture change, and it drops into frozen checkpoints. With a stronger empirical case this would be an attractive contribution.

Critical weaknesses
  1. No variance reporting, and most gains are smaller than likely noise.

Every number in every table is a single run. No seeds, no confidence intervals, no significance tests. That matters because many of the headline gains are tiny. On the gloss free Sign2GPT backbone the BLEU-4 advantage over plain Uniform sampling is 0.27 on PHOENIX14T at 50 percent, 0.29 on CSL-Daily at 50 percent, and 0.42 on CSL-Daily at 30 percent. Only PHOENIX14T at 30 percent reaches 1.78.

Run to run variation in BLEU-4 on a 642 sample test set is around a point. Three of the four Sign2GPT results are therefore indistinguishable from the simplest baseline available, and Uniform is faster in all three. At PHOENIX14T 50 percent, Uniform runs at 280.45 ms against 288.95 ms, giving 2.03 times speedup against 1.97. So I do not accept the claim that ZipSign generalises well to the gloss free backbone. What the evidence supports is that the method gives large gains on gloss based models under aggressive compression and no measurable advantage over uniform sampling on the gloss free model tested.

  1. The main results rely on a tuned spatial scheme that the main paper does not disclose.

Table 1 and Table 2 both state in their captions that the spatial token keep ratio is fixed at 50 percent, and Section 4.1 repeats it. Appendix Section 7.1 says otherwise. The TwoStream implementation uses a resolution adaptive budget with hand chosen offsets of minus 0.05 at the 28 by 28 level, plus 0.10 at 14 by 14 and plus 0.30 at 7 by 7, and Table 6 gives the resulting background merge ratios. Table 7 shows what this is worth. At a nominal 50 percent keep ratio the fixed scheme reaches 27.01 BLEU-4 on PHOENIX14T test while the adaptive scheme reaches 28.73, a difference of 1.72 points from the allocation choice alone. That is larger than the margin ZipSign holds over STTM in most cells of Table 1, and much larger than every Sign2GPT margin.

A tuned component worth more than the reported advantage should not live only in the appendix. Two things need fixing. The main text should state that the spatial budget is tuned per feature level and say which split the three offsets were selected on, since choosing them against test performance would be leakage. And the baselines get no comparable per level tuning, so the spatial side of the comparison is not matched.

  1. The component ablation contradicts the paper's own combination story.

The paper argues the two stages combine well. Table 3 does not bear that out on the gloss free backbone. On Sign2GPT at 30 percent, spatial merging alone reaches 10.61 BLEU-4 and 34.04 ROUGE-L while the full ZipSign reaches 9.54 and 33.04. The merge only variant beats the complete method on both metrics. At 50 percent, merge only reaches 37.34 ROUGE-L against 36.94 for the full method.

Temporal pruning is therefore a net quality loss on this backbone, and the combination underperforms one of its own parts. The paper does not discuss this, and it should, because it is direct evidence that confidence from a pseudo gloss head is not a reliable enough signal to prune on.

  1. The plug and play claim is broader than the method supports.

The abstract says ZipSign can be directly plugged into existing SLT models. The temporal stage needs gloss or pseudo gloss confidence, which means a CTC branch as in TwoStream or an existing pseudo gloss head as in Sign2GPT. Many recent gloss free systems built on vision language models have neither, and for those the temporal stage cannot run at all.

The spatial stage needs keypoints to find hand and face regions. TwoStream already computes them so there is no extra cost there. Sign2GPT uses a DINOv2 encoder, which to my knowledge produces no keypoints. If a detector was added for those experiments its runtime belongs in the reported milliseconds per sample, and the paper never says whether it is included. If it is not, the Sign2GPT latency figures are optimistic.

  1. Two hyperparameters have no stated value and no ablation.

Equations 3 and 11 introduce a weighting factor alpha that biases pooling toward sign critical tokens, and the paper says only that alpha exceeds one. No value appears anywhere. Equation 5 introduces a neighbourhood size delta, and no value is given for that either. Both control how much of the benefit comes from the sign specific design. Without values the results cannot be reproduced, and without a sensitivity study there is no way to judge how much tuning went into them.

Questions for authors
  1. Can you report results over multiple runs with means and standard deviations, or at least bootstrap intervals over the test set? Without this I cannot tell whether the Sign2GPT gains of 0.27 to 0.42 BLEU-4 over Uniform sampling are real.

  2. Which split were the resolution adaptive offsets of minus 0.05, plus 0.10 and plus 0.30 chosen on, and were any test numbers inspected during that choice?

  3. Table 7 shows the adaptive allocation is worth 1.72 BLEU-4 by itself. Do Uniform, MaxPool and STTM receive any equivalent per feature level budget tuning in Table 1? If not, can you report ZipSign under the fixed allocation so the spatial comparison is matched?

  4. In Table 3 on Sign2GPT at 30 percent, spatial merging alone beats the full method on both BLEU-4 and ROUGE-L. Why does adding temporal pruning hurt there, and does it indicate the pseudo gloss confidence signal is unreliable?

  5. What are the values of alpha in Equation 3 and delta in Equation 5, and can you give a sensitivity study for both?

  6. For the Sign2GPT experiments, do the reported milliseconds per sample include extracting the keypoints used to identify hand and face regions? If not, what does that add?

  7. Can you add more aggressive keep ratios such as 10 or 20 percent? Your own argument is that the advantage grows as the budget tightens, so this would strengthen the case and would also show where the method breaks down.

  8. Can you report ToMe combined with your temporal pruning? Table 4 compares the spatial stages in isolation, so the end to end contribution of sign critical protection is never isolated.

6 comments
AuthorsOpenReview ↗

Additional Experiments and Clarifications on ZipSign's Performance and Efficiency.

Response to Weaknesses 1: We thank the reviewer for highlighting the need for statistical validation.

(1) Single Run Results and Confidence Intervals.

ZipSign and Uniform are training-free and use identical frozen checkpoints with deterministic inference. Therefore, repeated evaluations produce identical BLEU-4 scores, and training seed variation is not introduced by the compression methods. To quantify uncertainty arising from test-sample variability, we performed 10,000 paired bootstrap resamples using identical sample indices and recomputed BLEU-4 for each resample. As shown in Table 8, both 95% confidence intervals exclude zero, supporting statistically reliable improvements over Uniform at 50% retention.

Table 8: Paired bootstrap analysis of BLEU-4 improvements over Uniform using the Sign2GPT backbone at 50% temporal retention.

DatasetUniform B4ZipSign B4ΔBLEU-495% Paired Bootstrap CI
PHOENIX14T19.7720.04+0.27[+0.16, +0.51]
CSL-Daily11.9012.19+0.29[+0.05, +0.34]

(2) Gloss-Free Model Improvements and Latency Variability.

  1. We acknowledge that the BLEU-4 improvements on the gloss-free Sign2GPT backbone are consistently positive but generally small at 50% retention (+0.27 and +0.29 over Uniform). However, the advantage becomes more pronounced under aggressive compression. As shown in Table 2, at 30% retention on PHOENIX14T, ZipSign outperforms Uniform by 1.88 BLEU-4 (19.30 vs. 17.42) and the strongest competing method, MaxPool, by 1.78 BLEU-4. Moreover, reducing retention from 50% to 30% decreases ZipSign's BLEU-4 by only 0.74, compared with 2.35 for Uniform, indicating that ZipSign is substantially more robust to aggressive compression.

  2. More substantial improvements are observed on the gloss-based TwoStream backbone, where ZipSign outperforms the strongest competing compression methods by 4.30 and 5.46 BLEU-4 at 30% retention on PHOENIX14T and CSL-Daily, respectively (Table 1).

  3. For latency reliability, we repeated full-test inference five times on the same RTX A6000 GPU with a fixed batch size, one warm-up run, and CUDA synchronization. As shown in Table 9, ZipSign is slightly slower than Uniform on PHOENIX14T but slightly faster on CSL-Daily, indicating comparable latency.

We will revise the manuscript to clarify that the gloss-free Sign2GPT backbone improvements are generally modest at 50% retention, while ZipSign demonstrates stronger translation quality preservation under aggressive compression.

Table 9: Repeated inference latency measurements using the Sign2GPT backbone at 50% temporal retention (ms/sample, mean ± std).

DatasetUniform (Mean ± Std)ZipSign (Mean ± Std)
PHOENIX14T278.68 ± 5.20285.14 ± 6.10
CSL-Daily224.03 ± 4.80221.78 ± 7.52

Response to Weaknesses 2: We thank the reviewer for this important comment.

(1) Clarification of Spatial Keep Ratio and Tuned Resolution-Adaptive Budgets.

We first clarify that the ratios reported in Table 10 refer to background token merging ratios, rather than overall spatial token keep ratios. The 50% and 30% settings denote the overall spatial retention targets, which remain unchanged under fixed and resolution-adaptive allocations. Specifically, hand and face tokens are always protected from merging, while the background token merging budgets are adjusted across feature levels. For TwoStream, we apply resolution-specific offsets of −0.05, +0.10, and +0.30 to the retention targets at the 28×28, 14×14, and 7×7 feature levels, respectively, resulting in different background merging ratios (Table 10).

The main TwoStream results use the tuned resolution-adaptive allocation, rather than a fixed per-level allocation, to account for varying spatial redundancy across feature resolutions and better preserve informative visual features. As shown in Appendix Table 7, adaptive allocation improves BLEU-4 from 27.01 to 28.73 (+1.72) compared with fixed allocation in the isolated spatial merging evaluation. We have explicitly clarified this setting in Section 4.1 (highlighted in blue), with detailed budgets provided in Appendix Section 7.1.

Table 10: Background token merging ratios under fixed and resolution-adaptive spatial budget allocations on TwoStream.

Target Ratio (Overall)Fixed Background MergingAdaptive Background 28×28Adaptive Background 14×14Adaptive Background 7×7
50%0.8000.8800.6400.320
30%1.0001.0000.8570.571

(2) Contribution of Spatial Budget Tuning versus ZipSign's Methodological Gains.

We further evaluate the contribution of resolution-adaptive spatial budget allocation in the complete spatio-temporal compression pipeline, with both spatial and temporal retention targets fixed at 50%. As shown in Table 11, we make two observations:

  1. Contribution of Adaptive Allocation. Resolution-adaptive allocation improves ZipSign’s BLEU-4 from 26.80 to 27.02 (+0.22), whereas it slightly decreases STTM’s performance from 26.13 to 25.99 (−0.14). This shows that adaptive allocation is not a generic tuning trick that benefits any method; its gain depends on ZipSign’s design. Moreover, under the same fixed allocation, ZipSign still outperforms STTM by 0.67 BLEU-4 (26.80 vs. 26.13), confirming that its advantage does not stem from allocation tuning.

  2. Fair Comparison under Matched Allocations. We additionally evaluate STTM using the same fixed and resolution-adaptive per-level spatial budget allocations as ZipSign. ZipSign consistently outperforms STTM under both fixed (+0.67 BLEU-4) and adaptive (+1.03 BLEU-4) allocations. These results demonstrate that ZipSign's advantage over STTM persists even without resolution-adaptive tuning.

Table 11: Comparison of STTM and ZipSign under fixed and resolution-adaptive per-level spatial budget allocations on the TwoStream backbone with the PHOENIX14T test set (50% overall spatial and 50% temporal retention targets).

MethodTest B4 ↑
STTM (Fixed Per-Level Budget)26.13
STTM (Resolution-Adaptive Budget)25.99
ZipSign (Fixed Per-Level Budget)26.80
ZipSign (Resolution-Adaptive Budget)27.02

(3) Tuning Protocol and Comparison Fairness.

We further clarify three points to address the reviewer's concerns:

  1. Main-Text Disclosure. As clarified in Weakness 2.(1), we have explicitly described the resolution-adaptive per-level spatial budget allocation in Section 4.1 of the main paper (highlighted in blue), with detailed settings provided in Appendix Section 7.1.

  2. Data Leakage. All resolution-specific offsets were selected exclusively on the development set and fixed before test evaluation. No test-set performance was used for parameter selection.

  3. Comparison Fairness. As detailed in (2) and Table 11, we additionally evaluated STTM under the same fixed and resolution-adaptive per-level spatial allocations as ZipSign. ZipSign outperforms STTM under both settings (+0.67 and +1.03 BLEU-4, respectively), confirming that its performance advantage is not solely attributable to spatial budget tuning.

Response to Weakness 3 Quality–Efficiency Trade-off in Component Combination on Sign2GPT: We thank the reviewer for this important observation.

We acknowledge that ZipSign can underperform spatial merging alone on the gloss-free Sign2GPT backbone, particularly at 30% retention (9.54 vs. 10.61 BLEU-4). This may be attributed to the lack of explicit gloss supervision, which makes pseudo-gloss confidence less reliable for identifying key frames and increases the risk of discarding useful visual information.

However, combining spatial merging with temporal pruning provides substantial inference acceleration. As shown in Table 3, at 30% retention, ZipSign reduces latency from 347.57 to 224.00 ms/sample (35.6% reduction) compared with spatial merging alone, at the cost of 1.07 BLEU-4. At 50% retention, latency decreases from 354.41 to 221.78 ms/sample (37.4% reduction), while BLEU-4 slightly improves from 12.07 to 12.19.

These results demonstrate that the two stages are designed to balance translation quality and inference efficiency, rather than consistently maximizing translation accuracy. We have clarified this trade-off and the limitations of pseudo-gloss guided pruning in Section 8, with detailed component comparisons provided in Appendix Section 7.7.

Response to Weaknesses 4: Thanks.

(1) Applicability of Temporal Pruning.

ZipSign's temporal stage requires an existing CTC or pseudo-gloss prediction head. Therefore, its plug-and-play applicability is limited to backbones providing such confidence signals, rather than all SLT models. We have explicitly clarified this applicability requirement in Section 3.2 (Lines 261–263), with the modifications highlighted in blue.

(2) Keypoint Extraction and Latency Accounting.

The reported Sign2GPT inference latency excludes keypoint extraction, as hand and face keypoints are precomputed and cached. Unlike TwoStream, Sign2GPT requires an external keypoint detector. As shown in Table 12, keypoint extraction takes 652.01 and 780.34 ms/sample on PHOENIX14T and CSL-Daily, respectively. Although this preprocessing is computationally expensive, it is performed only once. The cached keypoints are reused across ZipSign evaluations, incurring the same fixed preprocessing cost across competing methods.

Table 13 provides a component-level latency breakdown, including the visual backbone, spatial pooling, feature fusion, temporal token selection, and translation decoding. When keypoint extraction is included, the estimated total latency increases to 940.96 and 1002.12 ms/sample on PHOENIX14T and CSL-Daily, respectively. We acknowledge that this substantial overhead can offset the reported inference acceleration in raw-video processing. Therefore, ZipSign's acceleration on Sign2GPT is most applicable when keypoints are already available. Component-level latency measurements are provided in Appendix Section 7.6 (Table 21), and additional offline keypoint preprocessing costs are reported in Appendix Section 7.7 (Tables 26–27).

Table 12: Keypoint extraction overhead and measured total latency for ZipSign on Sign2GPT using an NVIDIA RTX A6000 GPU.

DatasetKeypoint Extraction (ms/sample)Cached-Keypoint Inference (ms/sample)Estimated Total (ms/sample)
PHOENIX14T652.01288.95940.96
CSL-Daily780.34221.781002.12

Table 13: Component-level latency breakdown for Sign2GPT on the CSL-Daily test set (ms/sample, mean ± std)

ComponentBaselineZipSign (50%)
Visual Backbone22.64 ± 3.5022.46 ± 5.94
Spatial Pooling5.25 ± 2.4410.31 ± 5.05
Feature Fusion11.10 ± 2.619.12 ± 2.78
Temporal Token Selection—2.15 ± 5.66
Translation Decoding339.59 ± 4.10177.74 ± 6.03
Total Latency378.58 ± 9.68221.78 ± 7.52
Reviewer_65enOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ZipSign, an inference-only compression method for existing sign language translation models. Its spatial stage protects hand and face regions, merges background tokens, and applies foreground-weighted pooling. Its temporal stage uses recognition or pseudo-gloss confidence to retain representatives of predicted segments and selected neighboring tokens. The method is evaluated on frozen TwoStream and Sign2GPT checkpoints using PHOENIX14T and CSL-Daily. The experiments compare translation quality and inference cost against uniform sampling, max pooling, and an adapted spatio-temporal merging method, with additional component ablations.

Critical strengths

The paper addresses a useful practical constraint. Accelerating an already trained SLT model without updating its parameters is relevant when retraining is costly or unavailable. The inference-only setting is clearly defined, and evaluating compressed and uncompressed versions of the same checkpoints makes the comparison useful.

The results give encouraging evidence for task-aware compression. Using hand and face regions together with recognition confidence is a sensible way to account for the information needed by SLT. The reported quality advantages over the evaluated compression baselines, particularly under aggressive compression, suggest that these signals are useful. Testing both a gloss-based and a pseudo-gloss-based backbone gives this observation some breadth.

Critical weaknesses

The contribution of the proposed design choices is not sufficiently established. The spatial stage builds on ToMe and jointly introduces region protection, norm-directed merging, and weighted pooling (Section 3.1). Table 4 evaluates this combined module, so it does not establish whether merging adds value beyond foreground reweighting. Similarly, the temporal comparisons in Figure 4 do not include a simple control that selects the globally highest-confidence tokens. Consequently, it remains unclear how much the proposed segment-coverage rule contributes beyond using recognition confidence. These distinctions matter because they determine what the paper adds to existing compression approaches.

The practical advantage over simple compression baselines is not sufficiently characterized. The retention rule in Section 3.2 can exceed the nominal token budget, while Section 4.1 states that comparison methods match retention ratios “as closely as possible”; actual retained counts are not reported. The smaller improvements also need uncertainty estimates. For example, at 50% retention, Sign2GPT improves over Uniform by only 0.27 and 0.29 BLEU-4 while being slightly slower on both datasets (Table 2). Without paired quality estimates and repeated timing measurements, the magnitude and reliability of this advantage remain unclear. This limits the evidence for the paper’s main practical contribution.

Questions for authors

Clarification: actual budgets and component ablations. What are the realized temporal retention ratios for ZipSign and the comparison methods, and how often does retaining all key tokens increase the budget above the nominal ratio? Please also specify the spatial and temporal retention rates for each row of Table 3. Tables 1–2 fix spatial retention at 50%, whereas the “Only merge” ablation also reports a 30% setting; it is unclear whether the corresponding full-method ablation uses the same spatial setting.

Additional validation: contribution beyond simpler controls. At matched actual token counts, does segment-aware selection improve over global confidence-based selection? Does the spatial module improve over foreground-weighted pooling without merging? Paired confidence intervals for these quality differences, especially the smaller Sign2GPT gains, would help establish which proposed choices provide a reliable benefit.

Clarification and validation: source of the speedup. What operations are included in the reported latency, particularly keypoint extraction and token selection? Please report timing variability and generated-output lengths. A more detailed timing breakdown, or a controlled measurement with the same number of decoding steps, would help distinguish lower computation per step from changes in output length. Section 4.2 provides a rough visual-backbone timing estimate but does not resolve this distinction.

8 comments
AuthorsOpenReview ↗

Clarifications and Additional Experimental Validation for ZipSign

Response to Weaknesses 1: We thank the reviewer for this valuable suggestion.

(1) Contribution of Spatial Merging beyond Foreground Reweighting.

To isolate the contributions of spatial merging and foreground-weighted pooling, we conducted additional ablations on TwoStream with PHOENIX14T test set. All variants use the same frozen checkpoint, beam size of 5, and ROI protection, with temporal pruning disabled. Spatial merging variants use the same resolution-adaptive 50% token retention target. As shown in Table 8, we make two observations:

  1. Merging adds value beyond foreground reweighting. Comparing Only Weighted Pooling and Spatial Merging (Ours) in Table 8, adding merging on top of weighted pooling improves BLEU-4 by 1.92 and ROUGE-L by 0.89, while halving the spatial tokens. Conversely, comparing Only Merging and Spatial Merging (Ours), adding weighted pooling on top of merging improves BLEU-4 by 1.79. Weighted pooling alone may amplify redundant or noisy foreground features, whereas merging reduces redundancy and enables more effective feature aggregation. Both components are therefore necessary: neither alone suffices, but their combination nearly preserves baseline performance (28.73 vs. 28.87 BLEU-4) at 50% token retention.
  2. Role of each component. Region protection preserves hand and face details by excluding these regions from merging; removing it reduces BLEU-4 by 3.35. Norm-directed merging merges low-norm tokens into high-norm ones to preserve informative features; replacing it with random merging directions reduces BLEU-4 by 1.53. Weighted pooling emphasizes foreground tokens during aggregation, adding 1.79 BLEU-4 on top of merging. These results confirm that each component contributes positively and that the full design is necessary to retain translation quality under 50% spatial token retention.

Table 8: Ablation of spatial merging components on TwoStream with PHOENIX14T test set.

MethodWeighted PoolingMergingRetained RatioB4 ↑R-L ↑
Only Weighted Pooling✓✗100%26.8152.55
Only Merging✗✓50%26.9452.67
w/o Region Protection✓✓50%25.3851.75
Random Norm Direction✓✓50%27.2052.69
Spatial Merging(Ours)✓✓50%28.7353.44

(2) Contribution of Segment-Aware Temporal Pruning.

To isolate the contributions of recognition confidence and segment coverage, we conducted additional ablations on TwoStream with PHOENIX14T test set at temporal retention ratios of 50% and 30%. All variants use the same recognition confidence scores, differing only in their token selection strategies. As shown in Table 9, we make two observations:

  1. Limitations of Global Confidence Top-K. Selecting tokens solely based on global recognition confidence yields BLEU-4 scores of 18.92 and 10.61 at 50% and 30% retention, respectively. This strategy may concentrate tokens in a few high-confidence segments while overlooking other important signing segments, leading to substantial information loss.
  2. Contribution of Segment Coverage. By selecting representative tokens from each signing segment before global confidence-based filling, segment coverage improves BLEU-4 by 6.99 and 14.10 at 50% and 30% retention, respectively, compared with global Top-K. Further neighbor expansion provides additional gains of 0.59 and 0.56, demonstrating that segment coverage is the primary source of improvement.

Table 9: Ablation of segment-aware temporal token selection on TwoStream with PHOENIX14T test set.

MethodToken Selection StrategyB1 (50%)B4 (50%)R-L (50%)B1 (30%)B4 (30%)R-L (30%)
Global Confidence Top-KGlobal Top-K42.8718.9244.0329.1110.6133.19
Segment CoverageSegment Peaks → Global Fill50.6925.9151.4548.8824.7150.59
Temporal Pruning (Ours)Segment Peaks → Neighbor Expansion → Global Fill52.5426.5051.7750.2125.2751.07

Response to Weakness 2: We thank the reviewer for this valuable suggestion.

(1) Actual Token Retention and Budget Fairness.

The retention rule R=max⁡(⌈γT⌉,∣Ikey∣)R=\max(\lceil\gamma T\rceil,|\mathcal I_{\mathrm{key}}|) permits additional tokens only when the number of segment representatives exceeds the nominal budget. This condition was not triggered in our experiments. Therefore, the slight deviations above the nominal retention ratios arise from rounding token counts up to integers, rather than additional tokens introduced by segment coverage. Table 10 reports the actual retention ratios for ZipSign and all comparison methods across two backbones, two datasets, and two target budgets. The slight deviations from nominal ratios primarily arise from integer rounding of token counts. These results confirm that segment coverage introduces no additional tokens beyond the target budget in our experiments, while all methods maintain retention ratios close to their intended targets.

Table 10: Actual temporal token retention ratios across backbones, datasets, and target budgets.

BackboneDatasetTargetUniformMaxPoolSTTMZipSign
TwoStreamPHOENIX14T50%50.91%50.91%49.48%50.91%
TwoStreamPHOENIX14T30%31.83%31.83%29.66%30.25%
TwoStreamCSL-Daily50%50.80%50.80%50.85%50.80%
TwoStreamCSL-Daily30%30.47%30.47%30.84%30.49%
Sign2GPTPHOENIX14T50%50.65%50.65%49.84%50.48%
Sign2GPTPHOENIX14T30%30.44%30.03%30.44%30.13%
Sign2GPTCSL-Daily50%50.32%50.32%50.42%50.35%
Sign2GPTCSL-Daily30%30.20%29.63%30.20%30.04%

(2) Paired Quality Estimates and Repeated Timing Measurements.

To assess the magnitude and reliability of ZipSign's advantage over Uniform sampling, we conducted paired bootstrap analysis for BLEU-4 and repeated inference latency measurements on Sign2GPT at 50% temporal token retention.

  1. Translation Quality. As shown in Table 11, we performed 10,000 paired bootstrap resamples using identical sample indices for both methods, recomputing corpus-level BLEU-4 for each resample. ZipSign improves BLEU-4 by 0.27 and 0.29 on PHOENIX14T and CSL-Daily, respectively. Both 95% paired bootstrap confidence intervals exclude zero, supporting the statistical reliability of these improvements.
  2. Inference Latency. As shown in Table 12, latency was measured on the same NVIDIA RTX A6000 GPU with a fixed batch size, one warm-up run, and CUDA synchronization. We repeated the full test-set evaluation five times and report the mean ± standard deviation. ZipSign is slightly slower on PHOENIX14T (+6.46 ms) but slightly faster on CSL-Daily (−2.25 ms). These differences are small relative to the observed timing variability, indicating that ZipSign improves translation quality over Uniform sampling with minimal impact on inference latency.

Overall, ZipSign achieves modest but statistically reliable BLEU-4 improvements over Uniform at 50% retention with comparable inference latency. At 30% retention, the improvements become more pronounced, reaching +1.88 and +0.42 BLEU-4 on PHOENIX14T and CSL-Daily, respectively (Table 2). These results highlight ZipSign's practical advantage in better preserving translation quality under aggressive token compression.

Table 11: Paired bootstrap analysis of BLEU-4 improvements over Uniform on Sign2GPT across two test sets at 50% retention.

DatasetUniform B4ZipSign B4ΔBLEU-495% Paired Bootstrap CI
PHOENIX14T19.7720.04+0.27[+0.16, +0.51]
CSL-Daily11.9012.19+0.29[+0.05, +0.34]

Table 12: Repeated inference latency measurements on Sign2GPT at 50% temporal token retention (ms/sample).

DatasetUniform (Mean ± Std)ZipSign (Mean ± Std)
PHOENIX14T278.68 ± 5.20285.14 ± 6.10
CSL-Daily224.03 ± 4.80221.78 ± 7.52

Response to Question 1: Response to Question 1: We thank the reviewer for requesting further clarification of the actual retention budgets and component ablation settings.

Q1.1 Actual Temporal Retention Ratios. The realized temporal retention ratios of ZipSign and all comparison methods across two backbones, two datasets, and two target budgets are reported in Table 10 of our response to Weakness 2.(1).

Q1.2 Frequency of Budget Exceedance. As detailed in our response to Weakness 2.(1), retaining all key tokens did not exceed the rounded target budget in any evaluated sample. The slight deviations above the nominal retention ratios arise from rounding token counts up to integers, rather than additional tokens introduced by segment coverage. As discussed in our response to Weakness 2.(1), according to our current evaluation, this condition was not triggered.

Q1.3 Spatial and Temporal Retention Settings in Table 3. Tables 1–2 use a fixed 50% spatial retention ratio, with temporal retention set to 50% or 30%. Table 3 additionally evaluates each compression stage independently. Importantly, “Only merge (30%)” retains 30% of spatial tokens without temporal pruning, whereas “ZipSign (30%)” retains 50% of spatial tokens and 30% of temporal tokens. Thus, the 30% labels refer to different compression dimensions rather than identical spatial settings. Table 13 explicitly reports both retention ratios for each configuration. We have clarified these settings in the caption of Table 3 in Section 4.3, with the modifications highlighted in red. Additionally, in Appendix Section 7.5 explicitly reports both spatial and temporal retention ratios for each configuration.

Table 13: Spatial and temporal token retention ratios for each configuration in Table 3.

MethodSpatial RetentionTemporal Retention
Baseline100%100%
Only merge (50%)50%100%
Only prune (50%)100%50%
ZipSign (50%)50%50%
Only merge (30%)30%100%
Only prune (30%)100%30%
ZipSign (30%)50%30%

Response to Question 2: We thank the reviewer for this valuable suggestion.

Q2.1 Segment-Aware vs. Global Confidence-Based Selection. As detailed in our response to Weakness 1.(2), segment-aware selection outperforms global confidence Top-K by 6.99 BLEU-4 at matched temporal token counts (Table 9).

Q2.2 Spatial Module vs. Foreground-Weighted Pooling. As detailed in our response to Weakness 1.(1), the full spatial module improves BLEU-4 by 1.92 over weighted pooling alone, while reducing spatial token retention from 100% to 50% (Table 8).

Q2.3 Paired Confidence Intervals. As shown in Table 14, we computed 95% paired bootstrap confidence intervals using 10,000 resamples over 642 PHOENIX14T test samples, recomputing BLEU-4 for each resample using the TwoStream backbone. Both intervals exclude zero, supporting the statistical reliability of the component-level improvements. For the smaller Sign2GPT gains, Table 11 in our response to Weakness 2.(2) reports confidence intervals of [+0.16, +0.51] and [+0.05, +0.34] on PHOENIX14T and CSL-Daily, respectively. Both exclude zero, confirming that ZipSign's improvements over Uniform are also statistically reliable.

Table 14: Paired bootstrap confidence intervals for component-level BLEU-4 improvements using the TwoStream backbone on the PHOENIX14T test set.

ComponentComparisonΔBLEU-495% Paired Bootstrap CI
Spatial (50%)Spatial Merging vs. Weighted Pooling Only+1.92[+0.97, +2.88]
Temporal (50%)Segment Coverage vs. Global Confidence Top-K+6.99[+5.69, +8.30]

Comments on the paper

7 comments
AuthorsOpenReview ↗

Additional Validation of ZipSign's Merging Strategy and Inference Efficiency.

Response to Weaknesses 1 Lack of Isolated Justification for Spatial Merging Rule: We thank the reviewer for this valuable suggestion.

To isolate the effect of spatial merging rule, we compare Larger-Norm, Smaller-Norm, and Random Direction under identical settings, including the same checkpoint, token matching, ROI protection, and foreground-weighted pooling, with temporal pruning disabled. As shown in Table 8, Larger-Norm improves BLEU-4 by 1.53 and 1.93 over Random Direction and Smaller-Norm, respectively. These results demonstrate the effectiveness of the larger-norm merging rule for preserving translation quality.

Table 8: Ablation of token merging directions on TwoStream with PHOENIX14T test set at 50% spatial token retention.

Merging DirectionB4 ↑R-L ↑
Random Direction27.2052.69
Smaller-Norm26.8052.56
Larger-Norm (Ours)28.7353.44

Response to Weaknesses 2: We thank the reviewer for this valuable comment.

(1) Scope of Practical Efficiency Gains.

ZipSign targets inference acceleration of pretrained SLT models rather than training cost reduction. Visual encoding and CTC prediction remain necessary. Hand and face keypoints are precomputed and cached, so keypoint extraction is excluded from the reported latency.

(2) Inference Latency, GFLOPs, and Computational Overhead.

To clarify the acceleration sources and computational overhead, we conducted component-level profiling on TwoStream with the CSL-Daily test set using an RTX A6000 GPU. Table 9 reports the mean ± standard deviation across five runs. We make three observations:

  1. Source of Acceleration. ZipSign achieves a 1.74× speedup, reducing total latency from 496.97 to 285.53 ms/sample. Translation decoding accounts for 85.8% of the total latency reduction, indicating that the main acceleration comes from processing fewer input visual tokens during autoregressive decoding, rather than generating fewer text tokens.
  2. Computational Overhead. Temporal token selection introduces only 6.03 ms/sample overhead, which is included in the reported total latency. Spatial pooling increases latency slightly (17.42 → 20.39 ms), while feature fusion decreases it substantially (67.83 → 37.47 ms). As shown in Table 10, preprocessing requires 1.5 and 4.5 hours on PHOENIX14T and CSL-Daily, respectively, with cached keypoints occupying 0.72 and 1.9 GB. The keypoints are precomputed and cached before inference.
  3. GFLOPs vs. Actual Latency. As discussed in Section 4.2, estimated GFLOPs and actual latency are not directly proportional. The visual backbone dominates estimated GFLOPs and remains largely unchanged. In contrast, inference latency is strongly affected by autoregressive decoding, where temporal pruning reduces the length of the visual feature sequence processed by the autoregressive decoder. Consequently, substantial latency reductions can be achieved despite relatively small changes in estimated GFLOPs.

Table 9: Component-level inference latency breakdown using the TwoStream backbone on the CSL-Daily test set (ms/sample, mean ± std).

ComponentTwoStreamZipSign (50%)
Visual Backbone108.65 ± 4.31103.24 ± 3.30
Full-sequence CTC Recognition16.53 ± 8.0913.29 ± 5.04
Spatial Pooling17.42 ± 2.0120.39 ± 8.30
Feature Fusion67.83 ± 6.7237.47 ± 4.10
Temporal Token Selection—6.03 ± 5.21
Translation Decoder286.54 ± 5.68105.11 ± 3.25
Total Latency496.97 ± 9.57285.53 ± 5.89

Table 10: Offline keypoint preprocessing costs for Sign2GPT using an NVIDIA RTX A6000 GPU.

DatasetCached Keypoint SizeOffline Processing Cost
PHOENIX14T0.72 GB1.5 hours
CSL-Daily1.9 GB4.5 hours

Response to Weaknesses 3: Thanks.

At the 30% setting, ZipSign offers a different trade-off compared with either component alone:

  1. Compared with spatial merging alone, as shown in Table 3, ZipSign reduces latency by 183.48 ms on TwoStream and 123.57 ms on Sign2GPT, at the cost of 1.52 and 1.07 BLEU-4, respectively. This makes ZipSign preferable when spatial merging alone cannot meet latency requirements.

  2. Compared with temporal pruning alone, as shown in Table 3, ZipSign improves BLEU-4 by 1.28 and 0.31 on TwoStream and Sign2GPT, respectively, with only 6.27 and 3.89 ms additional latency. This makes ZipSign preferable when preserving translation quality is more important than meeting latency requirements.

Therefore, the 30% ZipSign setting is suitable for latency-constrained scenarios where some quality degradation is acceptable. It achieves better translation quality than temporal pruning alone. When translation quality is the priority, we recommend the 50% setting, which better balances quality and efficiency. We have clarified these practical trade-offs in Appendix Section 7.7 and Section 8.

Response to Question 1: Thanks.

Q1.1 Comparison of Merging Directions. As shown in Table 8 of our response to Weakness 1, larger-norm outperforms random direction and smaller-norm by 1.53 and 1.93 BLEU-4, respectively, with region protection, weighted pooling, and spatial token budgets held fixed.

Q1.2 Relationship Between Feature Norm and Useful Information. To examine whether larger-norm background tokens preserve more information, we conducted a translation sensitivity analysis on pairs of similar background tokens identified during spatial merging. As shown in Table 11, perturbing larger-norm tokens leads to a substantially greater increase in translation Negative Log-Likelihood (NLL), where higher values indicate lower confidence in the reference translation, than perturbing smaller-norm tokens (+0.09 vs. +0.01). The 95% confidence interval for larger-norm perturbation [+0.06, +0.12], while that for smaller-norm perturbation [−0.01, +0.03] includes zero, suggesting a more consistent impact on translation quality from perturbing larger-norm tokens. These results suggest that larger-norm tokens carry more useful information, providing further empirical support for our larger-norm merging strategy.

Table 11: Translation sensitivity to perturbations of background tokens with different feature norms on TwoStream with the PHOENIX14T test set.

PerturbationTranslation NLL ↓ΔNLL ↑95% CI of ΔNLL
Higher-Norm Token2.49+0.09[+0.06, +0.12]
Lower-Norm Token2.41+0.01[−0.01, +0.03]

Response to Question 2: We thank the reviewer for this suggestion.

Q2.1 Actual Temporal Token Retention. Table 12 reports the realized temporal retention ratios for all methods across both backbones, datasets, and target budgets. The slight deviations from nominal ratios arise from integer-valued token counts. Table 13 further reports the mean retained token counts and their distributions on TwoStream, including 30% and 70% of per-sample retained token counts, to characterize the variation across test samples. Both the mean counts and percentile ranges are closely aligned across methods, confirming comparable actual token budgets and retention distributions.

Q2.2 Performance at Comparable Token Budgets. As shown in Tables 12–13, ZipSign achieves comparable actual retention ratios and token count distributions to competing methods. At 50% retention on TwoStream, ZipSign, Uniform and MaxPool retain identical mean token counts on PHOENIX14T (12.62) and CSL-Daily (16.04), while ZipSign achieves higher BLEU-4 (Tables 1–2). These results demonstrate that ZipSign's translation improvements stem from more effective token selection rather than retaining more tokens.

Table 12: Actual temporal token retention ratios across backbones, datasets, and target budgets.

BackboneDatasetTargetUniformMaxPoolSTTMZipSign
TwoStreamPHOENIX14T50%50.91%50.91%49.48%50.91%
TwoStreamPHOENIX14T30%31.83%31.83%29.66%30.25%
TwoStreamCSL-Daily50%50.80%50.80%50.85%50.80%
TwoStreamCSL-Daily30%30.47%30.47%30.84%30.49%
Sign2GPTPHOENIX14T50%50.65%50.65%49.84%50.48%
Sign2GPTPHOENIX14T30%30.44%30.03%30.44%30.13%
Sign2GPTCSL-Daily50%50.32%50.32%50.42%50.35%
Sign2GPTCSL-Daily30%30.20%29.63%30.20%30.04%

Table 13: Mean and distribution of retained temporal token counts on TwoStream across PHOENIX14T and CSL-Daily test sets. Values are reported as mean [P30, P70], where P30 and P70 represent the 30% and 70% of the number of retained temporal tokens per sample.

DatasetTargetMean Original TokensUniformMaxPoolSTTMZipSign
PHOENIX14T50%24.7912.62 [7.89, 18.83]12.62 [7.89, 18.83]12.27 [7.35, 18.50]12.62 [7.50, 18.62]
CSL-Daily50%31.5816.04 [9.62, 23.81]16.04 [9.62, 23.81]16.06 [9.74, 23.68]16.04 [9.63, 23.81]

Response to Question 3: Thanks.

Q3.1 Latency Measurement Scope. As clarified in Weakness 2 (1), the reported latency includes the visual backbone, spatial token merging, recognition, feature fusion, temporal token selection, and translation decoding. Hand and face keypoints are precomputed and cached; thus, keypoint extraction is excluded. Data loading is also excluded. The measurements therefore reflect model inference rather than the complete data loading and preprocessing pipeline.

Q3.2 Component-Level Latency Analysis. The component-level latency breakdown and the explanation for the discrepancy between estimated GFLOPs and measured latency are provided in Table 9 and Weakness 2 (2), respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.