acceptodds
Under review as a conference paper at ICLR 2027

CodecTok: Codec-informed Adaptive Token Allocation for Efficient Video Generation

Abstract

Discrete video tokenization is a central efficiency bottleneck in autoregressive video generation, as each clip must be compressed into a token sequence whose length strongly affects downstream decoding cost. Fixed-budget video tokenizers allocate the same number of tokens to every clip, which can use capacity inefficiently across videos with different spatiotemporal complexity. We draw inspiration from rate control in classical video codecs: inexpensive measures of spatial and temporal complexity can guide content-adaptive token allocation. Based on this view, we introduce CodecTok, a codec-informed adaptive video tokenizer that adapts the idea of content-dependent capacity allocation to discrete video representations. CodecTok uses a training-free allocator to estimate spatial complexity from block-wise gradient statistics and temporal complexity from Hadamard-domain inter-frame residuals, producing a content-adaptive token budget for each clip without training an auxiliary allocation network or performing per-clip budget search. To support variable budgets in practice, CodecTok further employs a padding-free flexible backbone with sequence packing and 3D rotary positional encoding, enabling arbitrary resolutions and variable-length token sequences. A semantic alignment objective distills structured representations from a frozen video foundation model, encouraging highly compressed tokens to preserve meaningful spatiotemporal semantics. Experiments on UCF-101 demonstrate competitive reconstruction and downstream AR generation at reduced token budgets. With 743 reconstruction tokens and 730 generated tokens on average, CodecTok achieves an rFVD of 16 and a gFVD of 44, supporting a favorable balance between visual quality and token efficiency.

Primary area:
generative models
Keywords:
video tokenizer, video reconstruction, video generation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_3xSFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CodecTok, a video tokenizer that assigns a variable token budget using inexpensive spatial gradient statistics and temporal Hadamard-domain residuals. The framework combines this codec-informed allocator with a flexible Transformer tokenizer and semantic alignment to a frozen video foundation model. Experiments on UCF-101 show a favorable reconstruction-quality versus token-count trade-off, and downstream autoregressive generation benefits from shorter sequences.

Critical strengths
  • The motivation is clear and practical. Allocating the same number of tokens to clips with different spatial and temporal complexity can waste representation capacity.
  • The allocator is simple and inexpensive. The same-tokenizer comparison with InfoTok and EVATok at an average of 743 tokens is useful, particularly the explicit measurement of allocation overhead.
  • The paper provides several relevant ablations, including fixed versus adaptive allocation, spatial versus temporal complexity, representation alignment, and reconstruction at 480p. Reporting both reconstruction and generation metrics is also helpful.
Critical weaknesses
  • My biggest concern is that the downstream generation results do not cleanly isolate the benefit of adaptive allocation. Table 5 and Appendix Table 8 show that the fixed 1024-token version already achieves a gFVD of 34, compared with 44 for the adaptive version. This is a reasonable quality-efficiency trade-off, but it does not show whether content-dependent allocation is better than simply using a shorter fixed sequence. Figure 6 and Table 9 provide budget-matched reconstruction evidence, while the equivalent generation comparison is missing. Fixed-budget generators at approximately 585 and 743 tokens, with the same tokenizer design, alignment, training data, and generator budget, would be important.
  • The connection between reconstruction-time allocation and generation-time termination is insufficiently validated. The allocator observes a real video, whereas the AR model determines its output length by predicting EOS without access to such a video. Appendix A itself notes instability in variable-length generation. Reporting only the average generated length does not establish that the AR model learns the intended relationship between content complexity and token budget.
  • The entropy-based justification is stronger than the derivation supports. Gradient statistics and SATD are reasonable coding-difficulty proxies, but an orthogonal transform alone does not make the L1 magnitude of its coefficients a generally monotonic surrogate for entropy, as claimed around Eqs. 3-4. Likewise, a few examples of MP4 size in Figure 5 do not establish optimal rate allocation for the learned tokenizer. The paper should distinguish its useful heuristic from a proven entropy estimator and quantify its relationship to reconstruction difficulty on held-out clips.
  • The generalization evidence remains limited. Although training uses UCF-101 and Kinetics-600 with varying input shapes, the principal reconstruction and generation evaluation is on 16-frame UCF-101 clips at 128x128. The 480p experiment is reconstruction-only and requires additional fine-tuning. It therefore does not yet establish efficient high-resolution AR generation or robustness to unseen durations and aspect ratios. The noise study is useful, but its quality comparison also allows the adaptive token budget to increase.
Questions for authors

Please see above weaknesses.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer 3xSF (1/3)

Thank you for recognizing CodecTok's practical motivation, low-overhead allocator, and ablation experiments. Your comments identify important gaps in the current evaluation, particularly the contribution of adaptive allocation to downstream generation and the relationship between generated sequence length and content complexity. We respond point by point below.

1. Generation comparisons under the same token budget

We agree that comparing the fixed 1024-token model, which achieves a gFVD of 34, with the adaptive model, which achieves a gFVD of 44 using an average of 730 generated tokens, demonstrates a quality–efficiency trade-off but does not establish that adaptive allocation outperforms shorter fixed sequences.

To directly address this question, we trained corresponding fixed-budget models with 730 and 560 tokens. We use these budgets rather than 743 and 585 to match the average generated-token counts of the two adaptive configurations, rather than their average reconstruction-token counts. For each fixed budget, the tokenizer was retrained from scratch with that fixed token count, and a new AR generator was then trained on its tokens. We keep the tokenizer architecture, quantizer, semantic alignment teacher, training data, AR model size, and training and sampling protocols consistent.

ModelAverage generated tokensgFVD ↓
CodecTok73044
Fixed-budget model73050
CodecTok56053
Fixed-budget model56066

At the same average generation budget of 730 tokens, adaptive allocation improves gFVD from 50 to 44. At 560 tokens, the improvement increases further, from 66 to 53. The corresponding relative reductions in gFVD are 12.0% and 19.7%, respectively. For reference, the run-to-run standard deviation of the adaptive model's gFVD over five independent training runs is about 1 (see our response to Reviewer VBmW, Section 4), so both gaps are well outside training variance.

These results show that the gains do not merely come from shortening sequences: under the same average generation budget, content-adaptive allocation improves downstream generation quality. The larger performance gain at the lower budget further indicates that allocating representation capacity according to content is particularly beneficial under stronger compression.

2. Relationship between reconstruction-time allocation and generation-time EOS prediction

Thank you for raising this question. During reconstruction, the allocator determines the token budget from the input video. During generation, the AR model has no target video to analyze; instead, it jointly models content and sequence length by learning variable-length token sequences and their EOS positions. We agree that reporting only the average generation length is insufficient to verify the expected complexity–budget relationship. We therefore generated 1,000 videos using CodecTok and measured the correlation between each video's generated token sequence length and the complexity score of its decoded video.

The experiment yields a Pearson correlation coefficient of 0.70, indicating a positive correlation between generated sequence length and video complexity. This provides evidence that the AR model learns a content-dependent relationship with length: generated videos with higher complexity generally correspond to longer token sequences. Together with the equal-budget results above, this analysis further supports the effectiveness of adaptive allocation in downstream generation. We will include the correlation statistics and the corresponding scatter plot in the revised manuscript.

Regarding the instability mentioned in Appendix A: that passage describes variable-length generation as a general challenge for AR video models. In our setting, the variable-length generator is stable in practice: over five independent training runs, its gFVD varies by only about ±1 (see our response to Reviewer VBmW, Section 4), and at a matched average budget it outperforms the fixed-length generator (Section 1 above).

AuthorsOpenReview ↗

Author Response to Reviewer 3xSF (2/3)

3. Theoretical basis of the complexity metric and its relationship with reconstruction difficulty

To directly examine whether the complexity metric reflects the representation requirements of a learned tokenizer, we quantified the relationship between this metric and CodecTok's reconstruction error, rather than relying solely on comparisons with conventional video file sizes.

Specifically, we reconstructed 1,000 videos from the held-out UCF-101 test set, which was never used in tokenizer training, allocator calibration, or model selection, using the same tokenizer with a fixed token budget. For each video, we computed the complexity score of the original input and the MSE between the original and reconstructed videos. The Pearson correlation coefficient reaches 0.74, indicating a strong positive correlation between the score and reconstruction difficulty under a fixed representation budget: videos with higher complexity scores generally also have larger reconstruction errors.

This result provides evidence directly relevant to learned tokenizers. Our score is computed solely from spatial gradients and temporal residuals, without running the tokenizer or evaluating reconstruction error, yet it exhibits a strong association with the tokenizer's actual reconstruction difficulty. It can therefore provide a low-cost predictive signal for content-dependent token allocation.

We also agree that the entropy-based justification in the submission is stronger than the derivation supports. In the revised manuscript, we will remove the claim that SATD is a "mathematically sound and monotonic surrogate" for entropy and present the gradient and SATD statistics as codec-inspired heuristic proxies for coding difficulty, not as a proven entropy estimator. Likewise, we will present the MP4-size examples in Figure 5 as illustrations only, not as evidence of optimal rate allocation. The empirical support for the allocator will rest on the held-out correlation analysis above and on the equal-budget reconstruction and generation comparisons.

4. Evaluation on unseen input shapes and generation at a higher resolution

To test robustness beyond the training distribution, we evaluate reconstruction on two input shapes that never appear during tokenizer training. During training, clip lengths range from 12 to 24 frames, and all clips are square, with spatial sizes from 128×128 to 256×256 (Sec. 4.1). The two new settings are:

  • 16×168×224 (unseen aspect ratio): a non-square 3:4 input, whereas all training clips are square.
  • 32×256×256 (unseen duration): 32 frames, exceeding the maximum training length of 24 frames.

All experiments use the same model weights and a fixed budget of 1024 tokens per clip, so that differences reflect the input shape rather than the token count. As a reference, we include the in-distribution result of the same fixed-budget model at the standard 16×128×128 shape (CodecTok-fix, Table 8).

Video shape (T × H × W)Seen in training?Tokens per clipPSNR ↑ (dB)rFVD ↓
16 × 128 × 128 (reference, Table 8)Yes102429.2014
16 × 168 × 224No (non-square aspect ratio)102428.9716
32 × 256 × 256No (exceeds max. training length)102427.8120

Relative to the in-distribution reference, PSNR decreases by only 0.23 dB on the unseen aspect ratio, although the input contains 2.3× as many pixels. At 32 frames, PSNR decreases by 1.39 dB, even though the clip contains 8× as many pixels as the reference and is still represented by the same 1024 tokens. Fixed-shape video tokenizers such as LARP are trained and released for a single input shape (16×128×128) and need to be retrained for other shapes, as we did for the 256×256 baseline below, whereas CodecTok handles these shapes with the same weights through sequence packing and 3D RoPE. These results indicate that CodecTok degrades gracefully on input shapes outside its training distribution.

Furthermore, we trained an AR generation model for 16×256×256 videos, extending the evaluation from high-resolution reconstruction to generation. LARP has no publicly released 256×256 model (its released tokenizer and AR model operate at 16×128×128), so for the baseline we retrained both the LARP tokenizer and its AR model at 16×256×256 using the official LARP code.

MethodVideo shape (T × H × W)Average generated tokens ↓gFVD ↓
LARP16 × 256 × 256102499
CodecTok16 × 256 × 25680192

At this resolution, CodecTok achieves a gFVD of 92 using only 801 generated tokens on average, outperforming the evaluated 1024-token LARP baseline while reducing token usage by approximately 22%. This provides direct evidence of the model's ability to perform efficient AR generation at a resolution higher than the original evaluation setting.

AuthorsOpenReview ↗

Author Response to Reviewer 3xSF (3/3)

5. Robustness to perturbations at a matched token budget

You correctly note that the noise study in Table 10 allows the adaptive token budget to increase. We therefore additionally evaluate an average-budget-matched setting to separate reconstruction robustness from the benefit of allocating additional tokens. Specifically, we re-calibrate the budget mapping on each perturbed evaluation set so that the average token budget remains 743 tokens, matching the average budget on clean inputs. CodecTok achieves PSNR values of 28.19 dB under Gaussian noise and 27.98 dB under texture jitter, compared with 28.85 dB on clean inputs. That is, with the average budget held fixed, PSNR decreases by 0.66 dB and 0.87 dB, respectively, so the allocator remains usable under these perturbations without spending additional tokens. We will add this budget-matched comparison next to Table 10.

ConditionTokensPSNR ↑ (dB)
Clean74328.85
Gaussian noise74328.19
Texture jitter74327.98

Summary

The additional experiments directly address the central concerns regarding generation: CodecTok outperforms fixed-budget models under two settings with matched average generation budgets, and generated sequence length is associated with content complexity. We also provide direct quantitative evidence of the relationship between the complexity metric and reconstruction difficulty for a learned tokenizer, evaluate reconstruction on input shapes unseen during training (a non-square aspect ratio and a longer duration), and extend generation evaluation to 16×256×256.

We will incorporate these results, the corresponding experimental details, and more precise theoretical statements into the revised manuscript. We hope this additional evidence addresses your main concerns, and respectfully ask you to reconsider your score for the paper in light of these results. We would be happy to provide any further clarification.

AuthorsOpenReview ↗

Follow-up to Reviewer 3xSF

Dear Reviewer 3xSF,

Thank you again for your constructive review. We understand this is a busy period. As the discussion phase is drawing to a close, we wanted to check whether our response addresses your concerns:

Q1: We added fixed-budget generation baselines at 730 and 560 tokens, with CodecTok achieving lower gFVD at both matched average budgets.

Q2: We analyzed the correlation between generated sequence length and output complexity and reported stability across independent training runs.

Q3: We clarified the heuristic interpretation of the complexity metric and reported its correlation with reconstruction error on held-out videos.

Q4: We added reconstruction results on unseen aspect ratios and durations, as well as generation results at 256×256.

Q5: We added noise experiments at a matched average budget of 743 tokens to separate robustness from the benefit of additional tokens.

If anything remains unclear, we would be glad to clarify or run further checks in the remaining time. Thank you for your time and consideration.

Reviewer_VBmWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CodecTok addresses the inefficiency of fixed-budget video tokenizers, which allocate the same number of discrete tokens to video clips with varying spatial and temporal complexities. It introduces a training-free alocator that combines Sobel gradient statistics and Hadamard-domain inter-frame residual statistics, mapping the resulting score to a bounded, clip-specific token budget. Subsequently, a Transformer tokenizer adapts to a variable number of latent queries through sequence packing and 3D rotary position embeddings.The system also trains a LLaMA-like autoregressive generator to produce tokens until an End-of-Sequence (EOs) token is generated. Evaluation is concentrated on UCF-101, with an additional fine-tuned 480p reconstruction setting. At equal average budgets, the proposed allocator reports lower reconstruction FVD than fixed and LPIPS-based allocation. The main adaptive configuration averages 743 reconstruction tokens and 730 generation tokens, reporting PSNR 28.85, rFVD 16, and gFVD 44. Ablations associate V-JEPA alignment with improved rFVD and gFVD, while adaptive allocation trades some reconstruction and generation quality for shorter sequences.

Critical strengths

1.The paper includes a controlled equal-budget comparison of allocation policies. At average budgets of 768, 890, and 1024 tokens, the proposed allocator reports rFVD values of 15.0, 13.0, and 11.5, compared with 20.0, 17.3, and 15.0 for fixed allocation and 18.5, 16.0, and 13.5 for LPIPS-based allocation. This is the clearest evidence that content-dependent allocation contributes value beyond merely changing the average token count. 2. The paper reports multiple quality–efficiency operating points and uses ablation studies to disentangle the effects of the router and the semantic alignment module. Table5 transparently demonstrate the quality cost associated with reducing token counts, while also showing that semantic alignment is associated with better generation metrics, rather than presenting only a single selected operating point.

Critical weaknesses

1.The core allocation mechanism and generation evaluation are insufficiently specified for independent reproduction. The paper does not report the values of k and b in Eq. (5), the spatial block size B, or ϵ\epsilon . It also does not fully describe the allocator calibration data, frame-sampling procedure, AR conditioning, or sampling parameters. These choices directly determine token allocation and the reported generation quality, making them core components rather than incidental implementation details. 2.The empirical scope and uncertainty reporting are insufficient to support broad claims of generalization or superiority. The evaluation is conducted primarily on UCF-101, while the specific data sources, split isolation, and controls for dataset overlap used in the experiments are not adequately documented. The paper also does not report the number of runs, random seeds, confidence intervals, or statistical tests.

Questions for authors

1.What are the exact values of k and b in Eq. (5), the spatial block size B, or ϵ\epsilon? What data and objective were used to calibrate the affine mapping in Eq. (5)? Were the spatial and temporal complexity scores normalized before being combined? If so, please describe the normalization procedure. 2.What exact UCF-101 and Kinetics-600 splits were used for tokenizer training, allocator calibration, AR model training, model selection, and final evaluation? Please confirm whether the UCF-101 test videos were excluded from all training, hyperparameter-selection, and calibration stages, and whether duplicate or near-duplicate videos across the two datasets were checked. 3.Were the main results replicated across multiple training or evaluation seeds? If so, please report the means, ranges of variation, or confidence intervals.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer VBmW (1/4)

Thank you for recognizing CodecTok's equal-budget allocation comparisons, quality–efficiency trade-offs, and component ablations. Your comments highlight important details that should be clarified regarding implementation, data usage, and evaluation stability. We respond below.

1. Allocator parameters, normalization, and calibration

Thank you for pointing out the missing implementation and calibration details. For spatial complexity, we use 3×3 Sobel operators to extract luminance gradients and compute the mean and standard deviation of gradient magnitudes within non-overlapping 8×8 spatial blocks, i.e., B=8B = 8. We use ϵ=10−6\epsilon = 10^{-6} as the numerical stability term in both Eqs. (1) and (2).

Spatial and temporal complexity scores are separately normalized by their means before fusion. Specifically, we compute clip-level spatial and temporal complexity scores over the UCF-101 and Kinetics-600 training data and obtain their global means, mspam_{\mathrm{spa}} and mtemm_{\mathrm{tem}}. Each component is divided by its corresponding mean, and the normalized components are combined with a spatial weight of 0.7 and a temporal weight of 0.3.

The parameters in Eq. (5) are determined by the reference budget bounds and a scalar calibration coefficient aa:

k=1152 a,b=384 a,k = 1152\,a, \qquad b = 384\,a,

where 384 is the lower reference budget bound, 1152 = 1536 − 384 is the width of the reference budget interval, and aa controls the overall budget level. We calibrate aa on the UCF-101 training split to match the target average token budget. The parameter values used for the two main configurations are:

Average reconstruction tokensaakkbb
5850.47541.44180.48
7430.62714.24238.08

We will add these values, the normalization statistics mspam_{\mathrm{spa}} and mtemm_{\mathrm{tem}}, and the calibration procedure to the revised manuscript, and include them in the released configuration files to support exact reproduction.

AuthorsOpenReview ↗

Author Response to Reviewer VBmW (2/4)

2. Dataset splits, evaluation data, and cross-dataset overlap

Thank you for requesting a clearer account of data usage. The original manuscript did not sufficiently distinguish the data used for training, calibration, model selection, and evaluation. We clarify these stages below.

StageData used
Tokenizer trainingUCF-101 training split + Kinetics-600 training split
Allocator normalization statisticsUCF-101 training split + Kinetics-600 training split
Allocator budget calibrationUCF-101 training split
AR model trainingUCF-101 training split
Model and hyperparameter selectionUCF-101 training split
Original reconstruction evaluationUCF-101 training split
Original generation FVD evaluationReal reference clips sampled from the UCF-101 training split
Additional external reconstruction evaluationTokenBench

The official UCF-101 test split was excluded from every development stage. Tokenizer training, allocator normalization and calibration, AR model training, and model/hyperparameter selection all use only the training splits listed above.

At the same time, we want to be explicit that the reconstruction and generation metrics reported in the paper are computed on the UCF-101 training split. This follows the public evaluation protocol of LARP, whose released evaluation scripts compute both rFVD and generation FVD from the UCF-101 training list, and we keep our settings consistent with prior baselines, including LARP and AdapTok, so that the numbers in Table 1 are directly comparable. We acknowledge that this protocol measures in-distribution quality rather than held-out generalization. The test split appears in the paper only in the qualitative reconstruction examples of Figures 9 and 10; no quantitative result uses it. We will state this explicitly in the revised manuscript and clearly separate the evaluation protocol from the data used at each development stage.

To further evaluate performance on data not used during model development, we additionally compare LARP and CodecTok on TokenBench. This dataset was not used to train our models, calibrate the allocator, or select models. Both methods are evaluated at 256×256; since LARP's released tokenizer only supports 128×128, the LARP baseline is a tokenizer we retrained at 256×256 using the official LARP code. The TokenBench results show that our model achieves competitive reconstruction performance on an unseen test set, outperforming LARP.

MethodAverage tokens ↓PSNR ↑SSIM ↑rFVD ↓
LARP102429.570.9043
CodecTok79429.780.9139

Regarding duplicate or near-duplicate videos between UCF-101 and Kinetics-600, we first uniformly sample frames from the videos and use perceptual hashing to identify candidate cross-dataset video pairs with visually similar frames appearing in a consistent temporal order. We then perform local feature matching and manual frame-by-frame verification. We found only three duplicate video pairs across the two datasets. Since both datasets are used only through their training splits, and three pairs are negligible relative to the roughly 400K training videos, this overlap does not affect our results. We will report this check in the revised manuscript.

AuthorsOpenReview ↗

Author Response to Reviewer VBmW (3/4)

3. Frame sampling, AR conditioning, and generation parameters

We agree that these settings directly affect token allocation and generation quality and should be reported as core experimental details.

Clip sampling follows LARP's released data pipeline. In the standard 16×128×128 setting, we retain the original video frame rate, randomly select a temporal starting point, and extract 16 consecutive frames with a frame-index stride of 1. Videos shorter than 16 frames are padded by repeating the final frame. For spatial preprocessing, we resize the shorter side to 128 while preserving the aspect ratio, followed by a 128×128 center crop. For reconstruction evaluation (rFVD, PSNR, SSIM, LPIPS), one such clip is drawn from each of the 9,537 videos in the UCF-101 training split, giving 9,537 evaluation clips.

Inside the allocator, the "uniformly sampled temporal subset" used for the spatial score consists of 16 frames, so at the standard 16-frame setting every frame of the clip is used: CspaC_{\mathrm{spa}} is the median of the frame-level scores over all 16 frames, and CtemC_{\mathrm{tem}} averages SATD over all 15 consecutive frame pairs.

AR generation is conditioned on the 101 UCF-101 action categories. Each class label is mapped to a single conditioning token placed before the visual token sequence. No ground-truth video frames are provided during generation. CodecTok appends an EOS token to each training sequence and terminates generation when EOS is predicted at inference, supporting variable-length sequences.

Sampling parameterSetting
Classifier-free guidance scale1.75
Temperature1
Top-k0 (no truncation)
Top-p1
Number of generated evaluation samples10,000
Standard evaluation video shape (T × H × W)16 × 128 × 128

The real reference data for generation FVD consist of non-overlapping, consecutive 16-frame clips from the UCF-101 training videos. We select 10,000 real clips using a fixed subset-sampling seed of 42 and use their corresponding class labels for conditional generation.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer f7wt (1/3)

Thank you for recognizing CodecTok's equal-budget reconstruction comparisons, controlled router experiments, and component ablations. Your comments identify three aspects that require further clarification: the independent contribution of adaptive allocation to generation, the reproducibility of different experimental configurations, and the specific protocols used to evaluate quality and computational cost. We respond point by point below.

1. Generation comparisons under the same token budget and interpretation of the complexity metric

We agree that the fixed 1024-token model achieving a gFVD of 34, compared with the adaptive model achieving a gFVD of 44 using an average of 730 generated tokens, primarily demonstrates a quality–efficiency trade-off. This comparison alone does not establish whether adaptive allocation outperforms shorter fixed sequences.

To directly address this question, we added corresponding models with fixed budgets of 730 and 560 tokens, respectively matching the average generation lengths of the two adaptive configurations. For each fixed budget, the tokenizer was retrained from scratch with that fixed token count, and a new AR generator was then trained on its tokens; these are separately trained fixed-budget models, not the adaptive tokenizer run at a fixed length. We kept the tokenizer architecture, semantic alignment teacher, quantizer, training data, AR model size, and training and sampling protocols consistent.

MethodAllocationAverage generated tokensgFVD ↓
CodecTokAdaptive73044
CodecTok-FixFixed73050
CodecTokAdaptive56053
CodecTok-FixFixed56066

At an average generation budget of 730 tokens, adaptive allocation improves gFVD from 50 to 44. At 560 tokens, the improvement increases further, from 66 to 53. The corresponding relative reductions are 12.0% and 19.7%, respectively. These controlled results show that adaptive allocation indeed achieves better generation quality under the same generation budget. We emphasize that the matched budgets of 730 and 560 here refer to the average number of generated tokens. For reference, the run-to-run standard deviation of the adaptive model's gFVD over five independent training runs is about 1 (see our response to Reviewer VBmW, Section 4), so both gaps (6 and 13 gFVD points) are well outside training variance.

We agree with your point on the entropy argument: orthogonality alone does not make SATD a monotonic measure of entropy. In the revised manuscript, we will remove the statement that SATD is a "mathematically sound and monotonic surrogate" for temporal entropy and, as you suggest, describe the gradient and SATD statistics as empirical proxies for coding difficulty. We also validate their practical value for a learned tokenizer directly. Specifically, we reconstructed 1,000 videos from the UCF-101 test set (held out from all training and calibration) using the same tokenizer with a fixed token budget and obtained a Pearson correlation coefficient of 0.74 between the input complexity score and reconstruction MSE. This result indicates that the low-cost metric is strongly associated with actual reconstruction difficulty under a fixed representation capacity. Together with the equal-budget reconstruction comparisons and the additional generation comparisons, these results support the effectiveness of the allocation rule.

AuthorsOpenReview ↗

Author Response to Reviewer f7wt (2/3)

2. Allocator and quantizer parameters and training settings across experiments

Thank you for pointing out the missing parameters that affect reproducibility. We will provide the complete configurations and clearly distinguish the base mapping, budget operating points, and budget bounds at different resolutions.

Spatial complexity is computed using 3×3 Sobel operators to obtain luminance gradients. The mean and standard deviation of gradient magnitudes are then calculated within non-overlapping 8×8 blocks, giving a spatial block size of B=8B = 8. The numerical stability term in both Eq. (1) and Eq. (2) is ϵ=10−6\epsilon = 10^{-6}. The spatial and temporal scores are each divided by their respective means computed over the data, and then combined using a spatial weight of 0.7 and a temporal weight of 0.3.

The parameters in Eq. (5) are determined by a budget calibration coefficient aa:

k=1152 a,b=384 a.k = 1152\,a, \qquad b = 384\,a.

Here, 384 is the lower reference budget bound (Nmin⁡N_{\min}), 1152 = 1536 − 384 is the width of the reference budget interval, and aa controls the overall budget level. We search for aa on the UCF-101 training split so that the actual average token count matches the target budget. An overall average of 585 tokens corresponds to a=0.47a = 0.47, while an average of 743 tokens corresponds to a=0.62a = 0.62. For the 480p setting, we kept a=0.47a = 0.47 and fine-tuned the model at 480p. The 480p bounds were set manually to Nmin⁡=1280N_{\min} = 1280 and Nmax⁡=4096N_{\max} = 4096 rather than taken from the reference-shape bounds, which is why the mean of 2362 tokens exceeds the reference upper bound of 1536 while remaining inside the 480p range. The rationale is that high-resolution video is highly redundant, so the token budget should grow far more slowly than the pixel count: the two bounds are only about 3.3× and 2.7× the corresponding reference bounds. The upper bound of 4096 equals the budget of the fixed baseline in Table 3, so the adaptive model never uses more tokens than CodecTok-Fix on any clip. We will state these bounds and how they were chosen in the revised manuscript.

For the quantizer, we use SLQ with a subset codebook containing 16,384 entries to balance reconstruction and generation.

Regarding which training settings produced Table 9 versus Tables 5 and 8: they are the same. The settings are:

ResultsTokenizer training
Table 1 (main results)150 epochs, global batch size 128
Table 5 (router / alignment ablation)150 epochs, global batch size 128
Table 8 (additional metrics)150 epochs, global batch size 128
Table 9 and Figure 6 (allocation strategies)150 epochs, global batch size 128

Two corrections follow from this, and we apologize for both. First, the rFVD of the fixed 1024-token model in Table 9 was misreported as 15.0 and should be 14, consistent with Tables 5 and 8. With this correction, the equal-budget reconstruction gain of our allocator over fixed allocation at 1024 tokens is 14 → 11.5 rFVD (instead of 15 → 11.5); the results at 768 and 890 tokens and the conclusions of Table 9 are unchanged. Second, the sentence in Section 4.1 stating that the ablation models were trained for 75 epochs does not apply to these tables, which use the 150-epoch models. In the revised manuscript, we will correct Table 9 and the corresponding point in Figure 6, and state the training schedule of every table and figure explicitly.

AuthorsOpenReview ↗

Author Response to Reviewer f7wt (3/3)

3. Clarification of timing measurements and FVD evaluation protocols

The timing results in Tables 2 and 6 were measured on a single A100 GPU with a batch size of 1. Table 2 measures inference speed for generation. We generated 1,000 videos each with LARP and CodecTok and compared their average inference times. For LARP, we use the LARP-L-Long tokenizer, and both methods use a 632M autoregressive generator. The timing interval starts with the conditioning input and ends when the final video has been decoded.

Regarding the latency analysis in Section 4.3, we further clarify that the shared encoding/decoding cost of approximately 77 ms is also a measured value, obtained on the same A100 GPU with a batch size of 1 by timing the full encode–quantize–decode pipeline. The reported 45%/17% reductions are therefore derived from measured components — the measured shared cost plus each method's measured allocation time (e.g., 77 + 63 ms vs. 77 + 0.09 ms) — rather than from assumed costs; we will make this distinction explicit in the revised Section 4.3.

For the router speed comparison in Table 6, we use CodecTok's base encoder–quantizer–decoder architecture throughout to ensure consistency. We reconstructed 1,000 videos with each of the three models to compare speed and quality. We also clarify that the reported 0.09 ms measures the time from receiving the decoded input frames—the same input tensors supplied to the tokenizer encoder—to producing the token budget. It excludes preprocessing operations such as video resizing and cropping, but includes all allocator computations, including luminance conversion, gradient computation, temporal residual computation, and SATD computation.

For gFVD evaluation, we perform class-conditional generation on UCF-101, mapping each of the 101 action classes to a conditioning token placed before the visual token sequence. We use a classifier-free guidance (CFG) scale of 1.75, a temperature of 1, top-k = 0 (no top-k truncation), and top-p = 1, generating 10,000 video clips of shape 16×128×128. Following LARP's public evaluation protocol, we construct non-overlapping clips of 16 consecutive frames from the UCF-101 training videos and select 10,000 real reference clips using a fixed subset-sampling seed of 42. Their class labels serve as the generation conditions, and FVD is computed between the generated and real reference clips using I3D features.

For rFVD evaluation, we also follow LARP's protocol: we use the UCF-101 training split and sample one clip of 16 consecutive frames from each of its 9,537 videos, so rFVD is computed over 9,537 real/reconstructed clip pairs, with I3D features accumulated over all of them.

Summary

The additional equal-budget generation experiments show that CodecTok outperforms fixed-length models at both average generation budgets, directly addressing the attribution of generation gains. We also provide the specific allocator and quantizer settings, details of the latency measurements, and the complete FVD evaluation protocols. We sincerely thank you for your suggestions. We will incorporate the above results and clarifications into the revised manuscript. We hope these additions address your concerns regarding the effectiveness and reproducibility of the method, and respectfully ask you to take them into account in your final assessment. Should any points require further clarification, we would be happy to provide additional details.

AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair,

We thank you and the reviewers for your careful evaluation and constructive feedback. We have provided point-by-point responses to all comments raised by the three reviewers, together with additional experiments and implementation details. Reviewer 3xSF confirmed that our rebuttal addresses most of their concerns and raised their rating from 2 to 3 (Weak acceptance). We summarize the main additions and clarifications below:

  1. Generation gains at matched token budgets. We trained fixed-budget tokenizers and corresponding AR generators under consistent settings. At average generation budgets of 730 and 560 tokens, CodecTok improves gFVD from 50 to 44 and from 66 to 53, respectively, supporting the benefit of content-adaptive allocation beyond simply shortening sequences.

  2. Validation of the complexity signal. On 1,000 held-out UCF-101 test videos, the complexity score has a Pearson correlation of 0.74 with fixed-budget reconstruction MSE. Across 1,000 generated videos, generated sequence length has a Pearson correlation of 0.70 with output complexity.

  3. Broader evaluation and budget-controlled robustness. On TokenBench, CodecTok achieves rFVD 39 with 794 tokens on average, compared with 43 with 1024 tokens for the LARP baseline. Using the same model weights, we also evaluated reconstruction on unseen input shapes of 16×168×224 and 32×256×256 (T×H×W). For generation at 16×256×256, CodecTok achieves gFVD 92 with 801 tokens on average, compared with LARP’s 99 with 1024 tokens. Additional noise experiments match the average budget at 743 tokens, separating reconstruction robustness from the benefit of increased token usage.

  4. Independent training runs and reproducibility. We reported five independent training runs for each main configuration, retraining both the tokenizer and AR generator from scratch. The rFVD and gFVD standard deviations are approximately 1, and the PSNR standard deviations are at most 0.21 dB. We also provided allocator and quantizer settings, calibration procedures, data usage and overlap screening, sampling protocols, and timing boundaries, distinguishing component-based latency estimates from end-to-end measurements.

We will incorporate these additional results and clarifications into the revised manuscript. We respectfully ask that the final assessment take this evidence into account. Thank you again for your time and valuable feedback.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.