- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CodecTok, a video tokenizer that assigns a variable token budget using inexpensive spatial gradient statistics and temporal Hadamard-domain residuals. The framework combines this codec-informed allocator with a flexible Transformer tokenizer and semantic alignment to a frozen video foundation model. Experiments on UCF-101 show a favorable reconstruction-quality versus token-count trade-off, and downstream autoregressive generation benefits from shorter sequences.
Critical strengths
- The motivation is clear and practical. Allocating the same number of tokens to clips with different spatial and temporal complexity can waste representation capacity.
- The allocator is simple and inexpensive. The same-tokenizer comparison with InfoTok and EVATok at an average of 743 tokens is useful, particularly the explicit measurement of allocation overhead.
- The paper provides several relevant ablations, including fixed versus adaptive allocation, spatial versus temporal complexity, representation alignment, and reconstruction at 480p. Reporting both reconstruction and generation metrics is also helpful.
Critical weaknesses
- My biggest concern is that the downstream generation results do not cleanly isolate the benefit of adaptive allocation. Table 5 and Appendix Table 8 show that the fixed 1024-token version already achieves a gFVD of 34, compared with 44 for the adaptive version. This is a reasonable quality-efficiency trade-off, but it does not show whether content-dependent allocation is better than simply using a shorter fixed sequence. Figure 6 and Table 9 provide budget-matched reconstruction evidence, while the equivalent generation comparison is missing. Fixed-budget generators at approximately 585 and 743 tokens, with the same tokenizer design, alignment, training data, and generator budget, would be important.
- The connection between reconstruction-time allocation and generation-time termination is insufficiently validated. The allocator observes a real video, whereas the AR model determines its output length by predicting EOS without access to such a video. Appendix A itself notes instability in variable-length generation. Reporting only the average generated length does not establish that the AR model learns the intended relationship between content complexity and token budget.
- The entropy-based justification is stronger than the derivation supports. Gradient statistics and SATD are reasonable coding-difficulty proxies, but an orthogonal transform alone does not make the L1 magnitude of its coefficients a generally monotonic surrogate for entropy, as claimed around Eqs. 3-4. Likewise, a few examples of MP4 size in Figure 5 do not establish optimal rate allocation for the learned tokenizer. The paper should distinguish its useful heuristic from a proven entropy estimator and quantify its relationship to reconstruction difficulty on held-out clips.
- The generalization evidence remains limited. Although training uses UCF-101 and Kinetics-600 with varying input shapes, the principal reconstruction and generation evaluation is on 16-frame UCF-101 clips at 128x128. The 480p experiment is reconstruction-only and requires additional fine-tuning. It therefore does not yet establish efficient high-resolution AR generation or robustness to unseen durations and aspect ratios. The noise study is useful, but its quality comparison also allows the adaptive token budget to increase.
Questions for authors
Please see above weaknesses.
4 comments
Author Response to Reviewer 3xSF (2/3)
3. Theoretical basis of the complexity metric and its relationship with reconstruction difficulty
To directly examine whether the complexity metric reflects the representation requirements of a learned tokenizer, we quantified the relationship between this metric and CodecTok's reconstruction error, rather than relying solely on comparisons with conventional video file sizes.
Specifically, we reconstructed 1,000 videos from the held-out UCF-101 test set, which was never used in tokenizer training, allocator calibration, or model selection, using the same tokenizer with a fixed token budget. For each video, we computed the complexity score of the original input and the MSE between the original and reconstructed videos. The Pearson correlation coefficient reaches 0.74, indicating a strong positive correlation between the score and reconstruction difficulty under a fixed representation budget: videos with higher complexity scores generally also have larger reconstruction errors.
This result provides evidence directly relevant to learned tokenizers. Our score is computed solely from spatial gradients and temporal residuals, without running the tokenizer or evaluating reconstruction error, yet it exhibits a strong association with the tokenizer's actual reconstruction difficulty. It can therefore provide a low-cost predictive signal for content-dependent token allocation.
We also agree that the entropy-based justification in the submission is stronger than the derivation supports. In the revised manuscript, we will remove the claim that SATD is a "mathematically sound and monotonic surrogate" for entropy and present the gradient and SATD statistics as codec-inspired heuristic proxies for coding difficulty, not as a proven entropy estimator. Likewise, we will present the MP4-size examples in Figure 5 as illustrations only, not as evidence of optimal rate allocation. The empirical support for the allocator will rest on the held-out correlation analysis above and on the equal-budget reconstruction and generation comparisons.
4. Evaluation on unseen input shapes and generation at a higher resolution
To test robustness beyond the training distribution, we evaluate reconstruction on two input shapes that never appear during tokenizer training. During training, clip lengths range from 12 to 24 frames, and all clips are square, with spatial sizes from 128×128 to 256×256 (Sec. 4.1). The two new settings are:
- 16×168×224 (unseen aspect ratio): a non-square 3:4 input, whereas all training clips are square.
- 32×256×256 (unseen duration): 32 frames, exceeding the maximum training length of 24 frames.
All experiments use the same model weights and a fixed budget of 1024 tokens per clip, so that differences reflect the input shape rather than the token count. As a reference, we include the in-distribution result of the same fixed-budget model at the standard 16×128×128 shape (CodecTok-fix, Table 8).
| Video shape (T × H × W) | Seen in training? | Tokens per clip | PSNR ↑ (dB) | rFVD ↓ |
|---|---|---|---|---|
| 16 × 128 × 128 (reference, Table 8) | Yes | 1024 | 29.20 | 14 |
| 16 × 168 × 224 | No (non-square aspect ratio) | 1024 | 28.97 | 16 |
| 32 × 256 × 256 | No (exceeds max. training length) | 1024 | 27.81 | 20 |
Relative to the in-distribution reference, PSNR decreases by only 0.23 dB on the unseen aspect ratio, although the input contains 2.3× as many pixels. At 32 frames, PSNR decreases by 1.39 dB, even though the clip contains 8× as many pixels as the reference and is still represented by the same 1024 tokens. Fixed-shape video tokenizers such as LARP are trained and released for a single input shape (16×128×128) and need to be retrained for other shapes, as we did for the 256×256 baseline below, whereas CodecTok handles these shapes with the same weights through sequence packing and 3D RoPE. These results indicate that CodecTok degrades gracefully on input shapes outside its training distribution.
Furthermore, we trained an AR generation model for 16×256×256 videos, extending the evaluation from high-resolution reconstruction to generation. LARP has no publicly released 256×256 model (its released tokenizer and AR model operate at 16×128×128), so for the baseline we retrained both the LARP tokenizer and its AR model at 16×256×256 using the official LARP code.
| Method | Video shape (T × H × W) | Average generated tokens ↓ | gFVD ↓ |
|---|---|---|---|
| LARP | 16 × 256 × 256 | 1024 | 99 |
| CodecTok | 16 × 256 × 256 | 801 | 92 |
At this resolution, CodecTok achieves a gFVD of 92 using only 801 generated tokens on average, outperforming the evaluated 1024-token LARP baseline while reducing token usage by approximately 22%. This provides direct evidence of the model's ability to perform efficient AR generation at a resolution higher than the original evaluation setting.
Author Response to Reviewer 3xSF (3/3)
5. Robustness to perturbations at a matched token budget
You correctly note that the noise study in Table 10 allows the adaptive token budget to increase. We therefore additionally evaluate an average-budget-matched setting to separate reconstruction robustness from the benefit of allocating additional tokens. Specifically, we re-calibrate the budget mapping on each perturbed evaluation set so that the average token budget remains 743 tokens, matching the average budget on clean inputs. CodecTok achieves PSNR values of 28.19 dB under Gaussian noise and 27.98 dB under texture jitter, compared with 28.85 dB on clean inputs. That is, with the average budget held fixed, PSNR decreases by 0.66 dB and 0.87 dB, respectively, so the allocator remains usable under these perturbations without spending additional tokens. We will add this budget-matched comparison next to Table 10.
| Condition | Tokens | PSNR ↑ (dB) |
|---|---|---|
| Clean | 743 | 28.85 |
| Gaussian noise | 743 | 28.19 |
| Texture jitter | 743 | 27.98 |
Summary
The additional experiments directly address the central concerns regarding generation: CodecTok outperforms fixed-budget models under two settings with matched average generation budgets, and generated sequence length is associated with content complexity. We also provide direct quantitative evidence of the relationship between the complexity metric and reconstruction difficulty for a learned tokenizer, evaluate reconstruction on input shapes unseen during training (a non-square aspect ratio and a longer duration), and extend generation evaluation to 16×256×256.
We will incorporate these results, the corresponding experimental details, and more precise theoretical statements into the revised manuscript. We hope this additional evidence addresses your main concerns, and respectfully ask you to reconsider your score for the paper in light of these results. We would be happy to provide any further clarification.
Follow-up to Reviewer 3xSF
Dear Reviewer 3xSF,
Thank you again for your constructive review. We understand this is a busy period. As the discussion phase is drawing to a close, we wanted to check whether our response addresses your concerns:
Q1: We added fixed-budget generation baselines at 730 and 560 tokens, with CodecTok achieving lower gFVD at both matched average budgets.
Q2: We analyzed the correlation between generated sequence length and output complexity and reported stability across independent training runs.
Q3: We clarified the heuristic interpretation of the complexity metric and reported its correlation with reconstruction error on held-out videos.
Q4: We added reconstruction results on unseen aspect ratios and durations, as well as generation results at 256×256.
Q5: We added noise experiments at a matched average budget of 743 tokens to separate robustness from the benefit of additional tokens.
If anything remains unclear, we would be glad to clarify or run further checks in the remaining time. Thank you for your time and consideration.
Author Response to Reviewer 3xSF (1/3)
Thank you for recognizing CodecTok's practical motivation, low-overhead allocator, and ablation experiments. Your comments identify important gaps in the current evaluation, particularly the contribution of adaptive allocation to downstream generation and the relationship between generated sequence length and content complexity. We respond point by point below.
1. Generation comparisons under the same token budget
We agree that comparing the fixed 1024-token model, which achieves a gFVD of 34, with the adaptive model, which achieves a gFVD of 44 using an average of 730 generated tokens, demonstrates a quality–efficiency trade-off but does not establish that adaptive allocation outperforms shorter fixed sequences.
To directly address this question, we trained corresponding fixed-budget models with 730 and 560 tokens. We use these budgets rather than 743 and 585 to match the average generated-token counts of the two adaptive configurations, rather than their average reconstruction-token counts. For each fixed budget, the tokenizer was retrained from scratch with that fixed token count, and a new AR generator was then trained on its tokens. We keep the tokenizer architecture, quantizer, semantic alignment teacher, training data, AR model size, and training and sampling protocols consistent.
At the same average generation budget of 730 tokens, adaptive allocation improves gFVD from 50 to 44. At 560 tokens, the improvement increases further, from 66 to 53. The corresponding relative reductions in gFVD are 12.0% and 19.7%, respectively. For reference, the run-to-run standard deviation of the adaptive model's gFVD over five independent training runs is about 1 (see our response to Reviewer VBmW, Section 4), so both gaps are well outside training variance.
These results show that the gains do not merely come from shortening sequences: under the same average generation budget, content-adaptive allocation improves downstream generation quality. The larger performance gain at the lower budget further indicates that allocating representation capacity according to content is particularly beneficial under stronger compression.
2. Relationship between reconstruction-time allocation and generation-time EOS prediction
Thank you for raising this question. During reconstruction, the allocator determines the token budget from the input video. During generation, the AR model has no target video to analyze; instead, it jointly models content and sequence length by learning variable-length token sequences and their EOS positions. We agree that reporting only the average generation length is insufficient to verify the expected complexity–budget relationship. We therefore generated 1,000 videos using CodecTok and measured the correlation between each video's generated token sequence length and the complexity score of its decoded video.
The experiment yields a Pearson correlation coefficient of 0.70, indicating a positive correlation between generated sequence length and video complexity. This provides evidence that the AR model learns a content-dependent relationship with length: generated videos with higher complexity generally correspond to longer token sequences. Together with the equal-budget results above, this analysis further supports the effectiveness of adaptive allocation in downstream generation. We will include the correlation statistics and the corresponding scatter plot in the revised manuscript.
Regarding the instability mentioned in Appendix A: that passage describes variable-length generation as a general challenge for AR video models. In our setting, the variable-length generator is stable in practice: over five independent training runs, its gFVD varies by only about ±1 (see our response to Reviewer VBmW, Section 4), and at a matched average budget it outperforms the fixed-length generator (Section 1 above).