acceptodds
Under review as a conference paper at ICLR 2027

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

Abstract

Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.

Primary area:
generative models
Keywords:
autoregressive video generation, video tokenization, flexible-length tokenizers, representation alignment, semantic tokenization, discrete visual tokens, scaling laws, self-supervised representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NmW9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SemanTok improves flexible video tokenization by incorporating frozen DINO features into the encoder and supervising retained token prefixes directly, avoiding reliance on the decoder’s noised input for semantic alignment.

Critical strengths
  1. A well-motivated approach to making short token prefixes semantically informative and easier to predict.

  2. Controlled comparisons across multiple AR sizes, training budgets, and two datasets provide substantial empirical support.

  3. Noise-level probes, cross-entropy measurements, and prefix teacher-forcing experiments strengthen the analysis beyond aggregate generation metrics.

Critical weaknesses
  1. Component contributions remain unclear. Appendix D.1 mentions preliminary ablations but provides no quantitative results separating DINO encoder inputs, class-token injection, and prefix prediction losses. This limits attribution of the gains to the proposed supervision mechanism.

  2. The reconstruction trade-off deserves clearer characterization. Improvements in generation accompany substantial reconstruction losses. For example, at the full uCO3D budget, rFVD increases from 42.4 to 80.8. It remains unclear how much improved predictability reflects semantic organization versus reduced instance-specific information.

  3. Practical efficiency is insufficiently quantified. Results emphasize AR FLOPs and cross-entropy, while decoding requires 50 flow steps. Lower cross-entropy does not itself reduce inference computation at a fixed model size and sequence length. End-to-end timing would establish the practical benefit.

Questions for authors
  1. Please provide quantitative input-only, prefix-supervision-only, and combined ablations using a representative small AR model.

  2. Does the predictability advantage persist against a baseline operating at comparable reconstruction distortion?

  3. What is the end-to-end generation speedup at matched quality, including diffusion decoding, and what additional tokenizer-training overhead is introduced?

3 comments

We thank the reviewer for their careful review and for recognising the motivation of the approach and the controlled comparisons. We respond to each raised point below.

W1 / Q1. Input-only, prefix-supervision-only and combined ablations with a small AR model. We ran these ablations: three further Kinetics-600 tokenizers trained from scratch with the paper's recipe, each with part of the semantic supervision, all compared at the same 66B-token checkpoint with their own 201M AR model trained as in the paper.

ArmDINO patches → encoderCLS → register 0Dense prefix headClass prefix head
VideoFlexTok––––
Input only✓✓––
Heads only––✓✓
Dense only✓–✓ (×2 compensation weight)–
SemanTok✓✓✓✓

Class accuracy (%) / gFVD.

k=4k=16k=32k=64k=128
VideoFlexTok33.4 / 27131.8 / 29831.2 / 31810.2 / 46511.2 / 527
Heads only52.1 / 21947.9 / 25143.7 / 27019.1 / 43418.6 / 490
Input only54.1 / 25052.5 / 24251.3 / 25020.7 / 38818.0 / 443
Dense only51.5 / 24252.9 / 22252.6 / 24524.6 / 34522.3 / 459
SemanTok56.4 / 25657.7 / 22858.0 / 22128.2 / 31527.6 / 364
  • "Heads only" has no DINO at the encoder input, yet gains 5–20 points of class accuracy and has lower gFVD at every budget. The gain therefore does not come from the DINO input alone.
  • The DINO input matters most for fidelity at larger budgets, where "Heads only" falls behind.
  • "Dense only", which removes the class-token pathway (injection and class head), is 4–8 points below SemanTok in class accuracy, showing that this pathway contributes significantly, too.
  • Every ablation beats VideoFlexTok on class accuracy at all eight budgets, but none matches SemanTok's class accuracy, so the components are complementary.

Revision: We will add a compact table with these ablations to the paper.

W2 / Q2. Reconstruction trade-off; predictability at comparable distortion. Following VideoFlexTok, SemanTok targets downstream generation, where pixel-level reconstruction is not the objective. Atanov et al. (2026) argue that tokenizers built for accurate reconstruction make the generative model predict low-level detail at unnecessary cost, and VideoFlexTok itself has lower PSNR than grid tokenizers (19.67 vs. 25.64 dB for VidTok on MSR-VTT, their Table 8). We list the reconstruction cost as a limitation (l. 469–470): PSNR/SSIM are lower (PSNR by 0.1–3.3 dB), and the largest relative rFVD gap is at the full uCO3D budget (42.4 to 80.8; 64.8 to 68.2 on Kinetics-600). Meanwhile, generation improves in both fidelity and semantic alignment (Section 5).

Semantic organization vs. less information. At k=16, SemanTok needs 4.1 fewer bits per token (8.8 vs. 12.9). Only about one bit of this comes from a lower marginal entropy; the rest comes from tokens that are easier to predict from context (Finding 6), i.e. better use of the same capacity.

Comparable rFVD. On Kinetics-600, SemanTok has lower rFVD at k=4 and k=8 (217 vs. 262, 178 vs. 203) and a similar one at k=16 (156 vs. 148), while its cross-entropy is lower over the first 128 token positions (Fig. 11b). The predictability advantage therefore persists at similar or lower rFVD. AR generation is best at k≤32 in nearly all cases, so the full budget, where this gap is largest, is not where either tokenizer operates.

Revision: We will clarify this in the text.

W3 / Q3. End-to-end efficiency. Our efficiency claims concern the AR stage only: equal quality from a much smaller AR model. We claim no implementation or end-to-end gains. On Kinetics-600 (Tables 6, 7), an 85M SemanTok model at k=16 beats VideoFlexTok's best-fidelity model (1.33B, k=16) in gFVD, gFID and class accuracy with 16× less AR inference compute. At k=8 it beats VideoFlexTok's best class-accuracy model (2.29B, k=32) in all three with 108× less. We believe the lower cross-entropy is why small SemanTok AR models can beat much larger VideoFlexTok ones.

Decoder. Both tokenizers use the same decoder architecture and 50 steps, so its cost is identical for both. A joint study that trades decoder steps against AR tokens is outside our scope.

Overhead. At generation time SemanTok adds none: the DINO features and prefix heads are used only by the tokenizer. In tokenizer training, it reuses the frozen DINOv2-L teacher that VideoFlexTok already runs for its REPA target, and adds only two two-layer heads.

Revision: We will clarify this in the text.

We will incorporate these clarifications and new results into the revised manuscript. We hope these responses fully address the reviewer's questions. If anything remains unclear, we are glad to provide further details during the discussion period. Otherwise, we kindly ask the reviewer to update their score to reflect our answers here.

Dear Reviewer NmW9,

As the discussion period ends soon, we would greatly appreciate knowing whether these new results address your concerns. We remain happy to provide further details. Thank you.

Thanks to the authors' rebuttal. As all my concerns have been addressed, I would like to maintain my positive scoring.

Reviewer_96qvOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

SemanTok is a modified VideoFlexTok (video tokenizer that produces coarse-to-fine, variable-length latents trained with nested dropout). SemanTok makes several modifications that make the learned latents more semantic: They concatenate DINOv2-L patch features with the RGB (VAE) inputs, and feed that to the encoder rather than RGB-only; add the DINO class token to the first register token, and predict both DINO patch and class tokens directly from the tokens (while VideoFlexTok used the patch tokens REPA-style). This helps the model to learn to put more semantic information into the tokens. The authors compare this with a re-trained VideoFlexTok for both reconstruction and generation, and SemanTok does well on semantic metrics.

Critical strengths

There are extensive evaluations that are quite insightful and I appreciate the depth of some of them. It’s great that on the generation part the authors measure both fidelity and semantic alignment. Additionally, the paper goes through the effort to attempt a proper comparison with VideoFlexTok by training it from scratch in a controlled way. I also found the core hypothesis about the REPA readout and the investigations interesting.

Critical weaknesses

One major strength of the paper, having retrained the main VideoFlexTok baseline to have a more proper comparison, is also a weakness. The baselines are trained for significantly shorter durations (compared to published versions), the tokenizer almost 3x shorter and the AR models almost one order of magnitude shorter depending on the setting. In addition, the AR models on K600 use only 6 pre-computed crop augmentations, which means the same data with the same crop is likely repeated around roughly 5-20 times depending on the setting. For a lot of the comparisons the VideoFlexTok baseline may be undertrained, while SemanTok builds on strong DINOv2-L features as an input.

The main changes of SemanTok over VideoFlexTok are not sufficiently ablated. Ie the DINO input, injecting a class token bias, and the two projections for DINO patches and cls tokens are not really ablated. The paer notes in Appendix D that “Early in the project, we ran smaller-scale ablations to narrow down the design”, mentioning comparisons of SIGLIPv2 with DINOv2, as well as the aforementioned design choices performing favorably. This is not enough for a submission; these design choices should be properly ablated since they are in my opinion the main contribution of the paper and everything else builds on it.

Questions for authors
  • What are the relative contributions of each of the semantic input/target design choices?
  • How does this compare against the published VideoFlexTok models, i.e. in more of a system-level comparison?
3 comments

We thank the reviewer for their careful review and for recognising the depth of the evaluations and the controlled retraining of VideoFlexTok. We respond to each raised point below.

W1. Baselines trained for shorter than published; six crop views. Our budgets are indeed below the published ones, due to GPU availability. However, every comparison is a fair one, with both the baselines and our models training for the same number of steps. We ran two studies to test whether our conclusions depend on the budget.

Tokenizer budget (App. D.2, Fig. 19). We compare both tokenizers at 26B, 66B, 98B and 131B training tokens, each with a fresh 201M AR model. SemanTok leads at every checkpoint (class accuracy at every k, gFVD from k=8) and the gap does not close: at k=16, gFVD is 291 vs. 349 at 26B and 217 vs. 273 at 131B. SemanTok at 66B already beats VideoFlexTok at 131B at every k≥4.

AR budget (Finding 2, Fig. 6). We train a 1.33B AR model for up to 65.5B tokens, 40% of the published 164B. SemanTok's gFVD lead narrows from 28% to 9% by 26B tokens and then holds at 11–14%, and its class-accuracy lead is still 30% at 65.5B. The six crop views are the same for both tokenizers and only set how often the data repeats at a given budget; this study spans 0.4 to 18 passes over them.

Published budget. The published budgets are 3× ours for the tokenizer (400B vs. 131B tokens) and 7× ours for the AR models (164B tokens at every size): for both tokenizers, about 10^22 AR training FLOPs (roughly 15,000 H200 GPU-hours), which is beyond our means. Neither study shows the gap closing with a larger training budget.

Strong DINOv2-L input. VideoFlexTok uses the same DINOv2-L teacher in its REPA loss, so both tokenizers have access to it at training time. Our new heads-only ablation (W2) feeds no DINO features to the encoder: it only adds the DINO target on the tokens, next to the decoder REPA. It still beats VideoFlexTok at every budget in both fidelity (gFVD) and semantic alignment (class accuracy).

Revision: We will summarise the tokenizer-budget study (Fig. 19) in the main text, next to the AR-budget study, and state the gap to the published budgets explicitly.

Q2. System-level comparison with the published VideoFlexTok models. We attempted this (App. D.2). The released Kinetics-600 checkpoint has a decoder fine-tuned with bidirectional attention, which the VideoFlexTok paper describes only for Panda-70M, and the reconstruction metrics we measured for it do not match the paper. The time-causal checkpoint presented in the paper has not been released. We therefore retrained VideoFlexTok with the same recipe, data and budget as SemanTok.

W2 / Q1. Ablation and relative contribution of each design choice. We ran these ablations. We trained three further Kinetics-600 tokenizers from scratch with the paper's recipe, each with part of the semantic supervision, and compare all five arms at the same 66B-token checkpoint, each with its own 201M AR model trained as in the paper.

ArmDINO patches → encoderCLS → register 0Dense prefix headClass prefix head
VideoFlexTok––––
Input only✓✓––
Heads only––✓✓
Dense only✓–✓ (×2 weight)–
SemanTok✓✓✓✓

Dense only doubles the dense-head weight to compensate for the missing class head.

Class accuracy (%) / gFVD.

k=4k=16k=32k=64k=128
VideoFlexTok33.4 / 27131.8 / 29831.2 / 31810.2 / 46511.2 / 527
Heads only52.1 / 21947.9 / 25143.7 / 27019.1 / 43418.6 / 490
Input only54.1 / 25052.5 / 24251.3 / 25020.7 / 38818.0 / 443
Dense only51.5 / 24252.9 / 22252.6 / 24524.6 / 34522.3 / 459
SemanTok56.4 / 25657.7 / 22858.0 / 22128.2 / 31527.6 / 364
  • Every arm beats VideoFlexTok on class accuracy at all eight budgets.
  • Heads only has no DINO at the encoder input, yet gains 5–20 points of class accuracy and has lower gFVD at every budget. The gain therefore does not come from the DINO input alone.
  • The DINO input matters most for fidelity at larger budgets, where heads only falls behind.
  • Dense only, which removes the class-token pathway (injection and class head), is 4–8 points below SemanTok in class accuracy, so that pathway contributes too.
  • No single arm reaches full SemanTok on class accuracy at any budget (2.2–14.3 points lower), so the components are complementary.

Revision: We will add a compact table with these ablations to the paper.

We will incorporate these clarifications and new results into the revised manuscript. We hope these responses fully address the reviewer's questions. If anything remains unclear, we are glad to provide further details during the discussion period. Otherwise, we kindly ask the reviewer to update their score to reflect our answers here.

Thank you for the detailed rebuttal and for running the ablations. They address some of my main concerns and I raise my score accordingly.

Dear Reviewer 96qc,

Thank you for your time and for improving the score. We remain happy to provide further details. Thank you.

Reviewer_8ofaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Based on the framework of VideoFlexTok, this paper proposes to feed frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. It exhibits good semantic alignment and video fidelity with every model size and token length.

The basic idea is to inject the Dino information into tokens and how it could be preserved after encoding (but before decoding).

Critical strengths

1, The basic idea is simple and the improvement is clear.

2, It adds the Dino loss on latent tokens, rather than decoded videos. This can directly help the tokens perserve more semantic information.

Critical weaknesses

1, The "decoder also sees the noised latent" might not be a problem. In testing, it starts from pure noise and have no leaking of information.

2, It sacrifices pixel-level reconstruction fidelity for semantic alignment. For example, PSNR/SSIM are consistently lower than VideoFlexTok on uCO3D, even at large token budgets. It would be useful to better characterize this semantics–reconstruction trade-off and discuss whether the gains primarily reflect a redistribution of limited token capacity rather than a uniformly improved tokenizer.

3, In fig4, why do gFVD and gFID increase with the inference FLOPs after 101110^{11}?

Questions for authors

See weakness.

2 comments

We thank the reviewer for their careful review and for recognising the simplicity and clear improvement of our approach. We respond to each raised point below.

W1. The noised latent may not be a problem, since inference starts from pure noise. We agree: at inference the decoder starts from pure noise and nothing leaks. Our remark was only meant as motivation. Decoder REPA supervises a decoder layer, so it reaches the tokens indirectly; SemanTok adds supervision on the tokens themselves. In the pure-noise setting the reviewer describes, this helps: SemanTok's REPA-layer prediction is closer to the DINO features than VideoFlexTok's at every k (Finding 4, Fig. 9).

Revision: We will reword the corresponding sentence in the introduction to make clear that it is a motivation, not a claim about leakage at inference.

W2. Semantics–reconstruction trade-off. Following VideoFlexTok, SemanTok targets downstream generation, where pixel-level reconstruction is not the objective. Atanov et al. (2026) argue that tokenizers built for accurate reconstruction make the generative model predict low-level detail at unnecessary cost, and VideoFlexTok itself has lower PSNR than grid tokenizers (19.67 vs. 25.64 dB for VidTok on MSR-VTT, their Table 8). We list the reconstruction cost as a limitation (l. 469–470): PSNR/SSIM are lower (PSNR by 0.1–3.3 dB), and the largest relative rFVD gap is at the full uCO3D budget (42.4 to 80.8; 64.8 to 68.2 on Kinetics-600). Meanwhile, generation improves in both fidelity and semantic alignment (Section 5).

Semantic organization vs. less information. At k=16, SemanTok needs 4.1 fewer bits per token (8.8 vs. 12.9). Only about one bit of this comes from a lower marginal entropy; the rest comes from tokens that are easier to predict from context (Finding 6), i.e. better use of the same capacity.

Comparable rFVD. On Kinetics-600, SemanTok has lower rFVD at k=4 and k=8 (217 vs. 262, 178 vs. 203) and a similar one at k=16 (156 vs. 148), while its cross-entropy is lower over the first 128 token positions (Fig. 11b). The predictability advantage therefore persists at similar or lower rFVD. AR generation is best at k≤32 in nearly all cases, so the full budget, where this gap is largest, is not where either tokenizer operates.

Revision: We will clarify this in the text.

W3. gFVD/gFID past 10^11 FLOPs in Fig. 4. In Fig. 4, FLOPs grow with the number of generated tokens k: each faded curve is one AR model sweeping k from 1 to 256, so the highest FLOPs are reached only with large k. Generation fidelity (gFVD and gFID) improves with k only up to an intermediate budget and degrades beyond it, because prediction errors compound over a longer AR rollout. VideoFlexTok reports the same effect: its gFVD rises again towards 256 tokens per frame (Atanov et al., 2026, Figs. 7 and 9, Table 4; e.g. 151.1 at k=4 vs. 246.6 at k=256). Studying the cause of this effect is one of our contributions: Finding 5 measures the gap between reconstruction and generation fidelity (Fig. 10), and Finding 6 teacher-forces ground-truth tokens to isolate the AR prediction error (Fig. 11a). Scaling the AR model reduces this error but does not remove it (Finding 2, Fig. 5). SemanTok's tokens are easier to predict (Fig. 11b), consistent with its later onset: gFVD degrades after k=16 for VideoFlexTok, but only after k=32 for SemanTok (l. 342–344).

Revision: We will explain this behaviour in the caption of Fig. 4 and refer to the same observation in VideoFlexTok.

We will incorporate these clarifications into the revised manuscript. We hope these responses fully address the reviewer's questions. If anything remains unclear, we are glad to provide further details during the discussion period. Otherwise, we kindly ask the reviewer to update their score to reflect our answers here.

Dear Reviewer 8ofa28,

As the discussion period ends soon, we would greatly appreciate knowing whether these new results address your concerns. We remain happy to provide further details. Thank you.

Comments on the paper

1 comment

We thank the area chair and the reviewers for their time and constructive feedback. The reviewers appreciate the core idea of supervising the tokens directly and its motivation (8ofa, 96qv, NmW9), the controlled comparison with the baseline and the depth of the analysis (96qv, NmW9), and the clear, well-supported gains over the baseline (8ofa, NmW9), and find the paper clear and easy to follow (96qv, NmW9). We summarise the main concerns and how we address them.

  1. Contribution of each component (96qv, NmW9). We trained three further Kinetics-600 tokenizers, each with part of SemanTok's semantic supervision. Every variant beats VideoFlexTok on class accuracy at all budgets, and none reaches full SemanTok's class accuracy at any budget, so the components are complementary. The prefix heads alone, without DINO features at the encoder input, already improve class accuracy and gFVD at every budget, so the gain does not come from the DINO input alone. The full tables are in our replies to 96qv and NmW9, and we will add them to the paper.
  2. Training budget of the baseline (96qv). Our budgets are below VideoFlexTok's published ones, which would need roughly 15,000 H200 GPU-hours and are beyond our means, but identical for both tokenizers. Two studies in the paper, over tokenizer checkpoints from 26B to 131B tokens (App. D.2, Fig. 19) and AR training up to 65.5B tokens (Finding 2, Fig. 6), show that the gap does not close. Moreover, SemanTok at 66B tokens already beats VideoFlexTok at 131B for every k≥4. The released VideoFlexTok checkpoint differs from the time-causal setup of its paper, so we retrained it with the same recipe as SemanTok.
  3. Reconstruction trade-off (8ofa, NmW9). Like VideoFlexTok, SemanTok targets generation, not pixel-level reconstruction, and we list its reconstruction cost as a limitation. This cost does not explain SemanTok's predictability gain: most of its lower cross-entropy comes from context, not from a lower marginal entropy (Finding 6), i.e. better use of the same capacity, and the advantage persists where its rFVD is similar or lower (Table 2, Fig. 11b).
  4. Efficiency (NmW9). Our efficiency claims concern the AR stage: an 85M SemanTok model beats VideoFlexTok's best-fidelity 1.33B model with 16× less AR compute. The decoder has the same architecture and 50 steps for both tokenizers, and SemanTok adds no cost at generation time.
  5. Clarifications (8ofa). We clarify the motivation of our decoder-REPA remark and explain the rise of gFVD and gFID at large token budgets, an effect VideoFlexTok also reports for gFVD.

We go through every reviewer's concerns in more detail in the replies below. We remain available for further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.