- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SemanTok improves flexible video tokenization by incorporating frozen DINO features into the encoder and supervising retained token prefixes directly, avoiding reliance on the decoder’s noised input for semantic alignment.
Critical strengths
-
A well-motivated approach to making short token prefixes semantically informative and easier to predict.
-
Controlled comparisons across multiple AR sizes, training budgets, and two datasets provide substantial empirical support.
-
Noise-level probes, cross-entropy measurements, and prefix teacher-forcing experiments strengthen the analysis beyond aggregate generation metrics.
Critical weaknesses
-
Component contributions remain unclear. Appendix D.1 mentions preliminary ablations but provides no quantitative results separating DINO encoder inputs, class-token injection, and prefix prediction losses. This limits attribution of the gains to the proposed supervision mechanism.
-
The reconstruction trade-off deserves clearer characterization. Improvements in generation accompany substantial reconstruction losses. For example, at the full uCO3D budget, rFVD increases from 42.4 to 80.8. It remains unclear how much improved predictability reflects semantic organization versus reduced instance-specific information.
-
Practical efficiency is insufficiently quantified. Results emphasize AR FLOPs and cross-entropy, while decoding requires 50 flow steps. Lower cross-entropy does not itself reduce inference computation at a fixed model size and sequence length. End-to-end timing would establish the practical benefit.
Questions for authors
-
Please provide quantitative input-only, prefix-supervision-only, and combined ablations using a representative small AR model.
-
Does the predictability advantage persist against a baseline operating at comparable reconstruction distortion?
-
What is the end-to-end generation speedup at matched quality, including diffusion decoding, and what additional tokenizer-training overhead is introduced?
3 comments
Dear Reviewer NmW9,
As the discussion period ends soon, we would greatly appreciate knowing whether these new results address your concerns. We remain happy to provide further details. Thank you.
Thanks to the authors' rebuttal. As all my concerns have been addressed, I would like to maintain my positive scoring.
We thank the reviewer for their careful review and for recognising the motivation of the approach and the controlled comparisons. We respond to each raised point below.
W1 / Q1. Input-only, prefix-supervision-only and combined ablations with a small AR model. We ran these ablations: three further Kinetics-600 tokenizers trained from scratch with the paper's recipe, each with part of the semantic supervision, all compared at the same 66B-token checkpoint with their own 201M AR model trained as in the paper.
Class accuracy (%) / gFVD.
Revision: We will add a compact table with these ablations to the paper.
W2 / Q2. Reconstruction trade-off; predictability at comparable distortion. Following VideoFlexTok, SemanTok targets downstream generation, where pixel-level reconstruction is not the objective. Atanov et al. (2026) argue that tokenizers built for accurate reconstruction make the generative model predict low-level detail at unnecessary cost, and VideoFlexTok itself has lower PSNR than grid tokenizers (19.67 vs. 25.64 dB for VidTok on MSR-VTT, their Table 8). We list the reconstruction cost as a limitation (l. 469–470): PSNR/SSIM are lower (PSNR by 0.1–3.3 dB), and the largest relative rFVD gap is at the full uCO3D budget (42.4 to 80.8; 64.8 to 68.2 on Kinetics-600). Meanwhile, generation improves in both fidelity and semantic alignment (Section 5).
Semantic organization vs. less information. At k=16, SemanTok needs 4.1 fewer bits per token (8.8 vs. 12.9). Only about one bit of this comes from a lower marginal entropy; the rest comes from tokens that are easier to predict from context (Finding 6), i.e. better use of the same capacity.
Comparable rFVD. On Kinetics-600, SemanTok has lower rFVD at k=4 and k=8 (217 vs. 262, 178 vs. 203) and a similar one at k=16 (156 vs. 148), while its cross-entropy is lower over the first 128 token positions (Fig. 11b). The predictability advantage therefore persists at similar or lower rFVD. AR generation is best at k≤32 in nearly all cases, so the full budget, where this gap is largest, is not where either tokenizer operates.
Revision: We will clarify this in the text.
W3 / Q3. End-to-end efficiency. Our efficiency claims concern the AR stage only: equal quality from a much smaller AR model. We claim no implementation or end-to-end gains. On Kinetics-600 (Tables 6, 7), an 85M SemanTok model at k=16 beats VideoFlexTok's best-fidelity model (1.33B, k=16) in gFVD, gFID and class accuracy with 16× less AR inference compute. At k=8 it beats VideoFlexTok's best class-accuracy model (2.29B, k=32) in all three with 108× less. We believe the lower cross-entropy is why small SemanTok AR models can beat much larger VideoFlexTok ones.
Decoder. Both tokenizers use the same decoder architecture and 50 steps, so its cost is identical for both. A joint study that trades decoder steps against AR tokens is outside our scope.
Overhead. At generation time SemanTok adds none: the DINO features and prefix heads are used only by the tokenizer. In tokenizer training, it reuses the frozen DINOv2-L teacher that VideoFlexTok already runs for its REPA target, and adds only two two-layer heads.
Revision: We will clarify this in the text.
We will incorporate these clarifications and new results into the revised manuscript. We hope these responses fully address the reviewer's questions. If anything remains unclear, we are glad to provide further details during the discussion period. Otherwise, we kindly ask the reviewer to update their score to reflect our answers here.