acceptodds
Withdrawn from ICLR 2027

VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This work presents VTok, a unified video tokenization system that can be used for both understanding and generation tasks. Unlike leading vision–language systems that tokenize videos through a naïve frame-sampling strategy, we propose to decouple the spatial and temporal representations of videos by retaining the spatial features of a single key frame while encoding each subsequent frame into a single residual token, achieving compact yet expressive video tokenization. Our experiments suggest that VTok effectively reduces the complexity of video representation from the product of frame count and per-frame token count to their sum, while the residual tokens sufficiently capture viewpoint and motion changes relative to the key frame. Extensive evaluations demonstrate the efficacy and efficiency of VTok: it achieves notably higher performance on a range of video understanding and text-to-video generation benchmarks compared with baselines using naïve tokenization, all with shorter token sequences per video (e.g., 3.4% higher accuracy on our TV-Align benchmark and 1.9% higher VBench score). Remarkably, VTok produces more coherent motion and stronger guidance following in text-to-video generation, owing to its more consistent temporal encoding. We hope VTok can serve as a standardized video tokenization paradigm for future research in video understanding and generation.

Primary area:
generative models
Keywords:
Video tokenization, video generation, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NHY1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes VTok, a unified video tokenization framework for both video understanding and text-to-video generation. Its core idea is to decouple spatial and temporal information in videos by preserving spatial tokens from a single key frame while encoding changes in subsequent frames relative to the key frame as compact residual motion tokens, thereby reducing the token complexity from O(TS)O(TS) to O(T+S)O(T+S). These video tokens are then integrated into a unified autoregressive multimodal language model, enabling the same framework to support both video-to-text understanding and text-to-video generation. The paper evaluates the proposed method on multiple video understanding benchmarks, VBench, and the newly introduced TV-Align benchmark.

Critical strengths

None

Critical weaknesses

The paper contains a substantial amount of content that resembles AI-generated “autoresearch”-style output, particularly in the formatting and presentation of tables, table captions, and figure captions. Moreover, the paper provides no appendix or supplementary material. In my view, the current experimental evaluation is far from sufficient to support the conclusions claimed in the paper.

Questions for authors
  1. Technical clarification on visual token generation. VTok represents video tokens as continuous vectors, while the generation objective is formulated as an autoregressive language-modeling loss over visual tokens. Could the authors clarify how these continuous visual tokens are actually predicted during generation, e.g., through quantization, a codebook, or an explicit continuous probability distribution? In addition, when the video decoder is frozen, which parameters receive gradients from LdecL_{\mathrm{dec}}?

  2. Clarification on the scope of AI use. The paper’s AI use statement states that generative AI was used only for language polishing and LaTeX formatting. However, a substantial amount of content in the manuscript, including tables, table captions, figure captions, and parts of the experimental descriptions, exhibits strongly templated patterns, raising concerns that broader AI-assisted research or “autoresearch”-style tools may have been used. Could the authors explicitly clarify whether, beyond the disclosed uses, AI tools were involved in research ideation, experimental design, method implementation, data analysis, result interpretation, or the generation of tables and figures?

Reviewer_pxuPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VTok, a unified video tokenization framework that decouples spatial and temporal representations. VTok encodes one key frame into spatial tokens and represents subsequent frames using residual motion tokens, reducing token complexity from O(TS)O(TS) to O(T+S)O(T+S). The proposed tokenization approach is integrated into a shared autoregressive MLLM for both video understanding and text-to-video generation. The work also introduces TV-Align, a benchmark that evaluates how well generated videos follow semantic instructions involving object counts, motion, spatial relationships. Experiments show improvements on VBench, TV-Align, and 7 video understanding benchmarks, alongside ablations which support the effectiveness of the decoupled representation.

Critical strengths
  • Simple and effective spatial-temporal tokenization. VTok addresses the spatial redundancy introduced by frame sampling while preserving temporal information through residual motion tokens. The design is intuitive and achieves better performance with fewer visual tokens, as demonstrated by the ablations in Tables 4 and 5. VTok achieves 43.9% accuracy on TV-Align, outperforming HunyuanVideo (40.5%) and WAN2.2 (41.1%). It also improves the VBench semantic score by 4.33 points and dynamic degree by 5.28 points over HunyuanVideo, suggesting better instruction following and motion generation.
  • Applicability to both video understanding and generation. VTok uses the same tokenization format within a unified MLLM framework for both tasks, demonstrating its potential as a general-purpose video representation. Improvements are reported across 7 video understanding benchmarks and 2 video generation benchmarks.
Critical weaknesses
  • Limited evaluation of fixed key-frame assumption. Eq. (14) computes motion residuals relative to the first frame, which may become less effective for longer videos with substantial motion, viewpoint changes, or scene transitions. The paper does not examine how performance changes with video length or compare against strategies using multiple or periodically updated key frames. This raises questions about the proposed approach's generalizability to longer videos. Additionally, the equation introduces a projection function gϕg_\phi for converting feature differences into motion tokens but its architecture and training settings are not specified, making it unclear whether this projection is pretrained or jointly trained with the MLLM and limiting reproducibility.
  • Limited validation of the proposed TV-Align benchmark. TV-Align is evaluated using Qwen2.5-VL as an automatic judge, with only a brief claim that its results agree with human judgments within 1% (lines 306-307). The paper does not provide details of this human evaluation or statistical uncertainty for reported improvements. Given the relatively small differences between models (43.9% for VTok versus 41.1% for WAN2.2), further validation would help establish the significance of these gains. The paper also does not clarify whether benchmark prompts overlap with the training data.
Questions for authors
  • How does VTok perform on longer videos or videos with substantial scene changes? Have the authors considered using multiple or periodically updated key frames?
  • What is the architecture of gϕg_\phi in Eq. (14) and how is it trained?
  • Could the authors provide more details on the human evaluation of TV-Align, including the number of evaluated samples, and report confidence intervals for the results? Can they confirm that the benchmark is disjoint from the training data?
  • Could the authors clarify the composition and availability of the 5 million video-caption training pairs to support reproducibility? If this is a proprietary dataset, could the authors report results on at least one fully public benchmark (e.g., MSRVTT, ActivityNet) to enable community verification?
Reviewer_BdwoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces VTok, a compact video representation for video understanding and generation. By default, the first frame serves as the keyframe: its spatial tokens are retained, while feature differences between subsequent frames and the keyframe are compressed into residual motion tokens to reduce spatial redundancy and sequence length. A multimodal language model uses this representation for video understanding and autoregressive visual-token prediction. In the main text-to-video experiments, the predicted tokens are passed to a frozen HunyuanVideo-13B diffusion decoder. Generation is evaluated on the authors' TV-Align benchmark and VBench, while understanding is evaluated on several public benchmarks using LLaVA-Next and Qwen2.5-VL-7B. The authors report improvements over the baselines in semantic alignment, dynamic degree, and video understanding, and present qualitative results from adapting VTok to Wan2.2.

Critical strengths
  1. Evaluation across understanding and generation, with attention to fine-grained semantic requirements. The paper evaluates a common video tokenization paradigm on both video understanding and text-to-video generation, providing initial evidence of its applicability across tasks. TV-Align further examines counting, direction, relative position, relative size, color, state, and motion, helping assess whether generated videos follow specific textual requirements.

  2. A relatively simple design built largely on pretrained components. VTok constructs a compact representation from keyframe spatial tokens and residual motion tokens for subsequent frames. In the main experiments, the multimodal LLM is fine-tuned while the pretrained visual encoder and HunyuanVideo diffusion decoder remain frozen. The design therefore makes substantial use of existing model components.

Critical weaknesses
  1. Experimental coverage is insufficient to establish benefits on current strong backbones and practical generation tasks. The main quantitative generation results are obtained with a HunyuanVideo-based pipeline. Although the paper presents examples of adapting VTok to Wan2.2, it provides no complete quantitative evaluation of Wan2.2 with VTok and no evaluation on stronger generative backbones. Understanding experiments are conducted on LLaVA-Next and Qwen2.5-VL-7B, without direct validation on more recent, stronger understanding models. Consequently, the results do not establish whether VTok offers consistent, practically meaningful gains on current strong models, or primarily addresses limitations of the selected backbones and their original video representations. Generation experiments are also restricted to T2V, leaving I2V and reference-conditioned generation unexplored. These are particularly relevant settings for a representation that separates keyframe content from motion: they would directly test whether it can preserve supplied visual content while following motion and semantic instructions. Its applicability to these settings therefore remains unestablished. Finally, the fine-grained semantic claims lack targeted cross-validation on an independent compositional benchmark such as T2V-CompBench, beyond the reported VBench and authors' TV-Align results. TV-Align's human-validation procedure and statistical reliability are insufficiently documented.

  2. The incremental contribution and the sources of the reported gains are not sufficiently established. Keyframe-motion decomposition, compact video representations, and unified understanding-generation modeling have substantial precedents. The paper introduces feature differencing relative to a keyframe and compact motion tokens, but does not sufficiently establish what new capabilities these particular choices provide or how each contributes to the reported improvements. Tables 4 and 5 offer initial comparisons of tokenization strategies and representation granularity. However, the main generation experiments lack controlled comparisons between explicit feature differencing and simpler alternatives, such as directly pooling subsequent-frame features, under matched temporal coverage and total token budgets. The results therefore do not adequately separate the contributions of the residual mechanism, temporal sampling density, and spatial-temporal token allocation.

Questions for authors
  1. Generalization across backbones. Could the authors provide complete quantitative results for Wan2.2 with VTok, evaluated under the same evaluation protocol as the original Wan2.2? Are results available on stronger generative or understanding backbones to establish that the gains extend beyond limitations specific to the selected backbones?

  2. Applicability to visually conditioned generation. Does VTok support I2V and reference-conditioned video generation, and how would input images or reference content be incorporated? What additional adaptation is required? These settings directly test content preservation and motion control; are there results demonstrating both reference fidelity and adherence to motion and semantic instructions?

  3. Evaluation reliability. Can the main semantic gains be validated on an independent benchmark such as T2V-CompBench? For TV-Align, what were the human-validation sample size, number of annotators, and evaluation protocol, and how exactly was the reported difference within 1% defined and calculated?

  4. Attribution of gains. In the main generation setting, does explicit feature differencing still outperform simpler alternatives such as directly pooling subsequent-frame features when temporal coverage and the total visual token budget are matched?

  5. Training and decoding interfaces. Are visual tokens continuous or discrete, and how does this representation relate to the softmax and visual-token prediction loss? Please specify the architecture and training of gϕg_\phi, and the exact interface through which predicted visual representations condition the frozen DiT.

  6. Temporal granularity. Is the default setting 6 FPS or six frames per motion token? In Table 5, the settings 3/6/12 yield 31/46/76 tokens, which appears more consistent with an FPS interpretation. Please clarify the actual sampling and token-counting rules.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.