acceptodds
Under review as a conference paper at ICLR 2027

Token Allocation For Better Generation

Abstract

Autoregressive image generation relies on visual tokenizers to convert images into discrete tokens. Recent variable-rate tokenizers dynamically allocate different token lengths to different images to optimize reconstruction. In this paper, we first show that token allocation optimized for a reconstruction objective, such as mean-square-error (MSE), might lead to worse generation. Next, we propose (Generation-Aware Token Allocation), a variable-rate tokenization method optimized for generation. More specifically, we first introduce a latent-context formulation for variable-rate autoregressive modeling to enable generation with flexible token lengths. Further, we propose a selection criterion that assigns different token lengths to images for better generation. Empirically, GTA consistently improves downstream generation across different tokenizers. On GigaTok, GTA reduces gFID from 5.50 to 4.75 compared to its fixed-length counterpart.

Primary area:
generative models
Keywords:
Autoregressive image generation, Visual tokenization, Token compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ly8AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses token allocation in variable-rate visual tokenizers for autoregressive image generation. The authors argue that existing allocation criteria are designed for reconstruction, and show that under the same average token budget and matched reconstruction distortion, different allocations can still lead to substantially different generation quality. To address this reconstruction–generation mismatch, they propose Generation-Aware Token Allocation. GTA combines a latent context token that lets the AR model choose a token length before generating visual tokens, and an allocation criterion that uses a fixed-length AR probe to assign token lengths that reduce the feature-space discrepancy between generated and real images. Experiments with LlamaGen-B on GigaTok and VQ-VAE tokenizers compare GTA against fixed-length and reconstruction-oriented allocation. GTA improves gFID from 5.50 to 4.75 on GigaTok and from 8.68 to 7.20 on VQ-VAE.

Critical strengths
  1. The paper shows that, with the tokenizer frozen and the average token budget fixed, how tokens are allocated across images affects downstream generation, not only reconstruction. Table 1 supports this directly: allocations with matched average length and reconstruction distortion differ by about 1.3 gFID. This view of variable-rate tokenization is relevant to how such tokenizers are designed and evaluated, and it is the main reason for my positive assessment.

  2. The shuffle ablation in Table 4 preserves both marginals and indicates that the image-to-length assignment matters more than the latent context. Tables 5–7 separate the allocation criterion from LCT. Without LCT, the criterion alone still improves gFID (5.50 → 5.15) and FD$_\text{DINOv2}$ (243.47 → 237.95), and Table 6 shows that the gain persists when the scoring features are changed to DINOv2. Together these support that the allocation rule itself contributes to the improvement.

Critical weaknesses

In Table 2, the baselines correspond to the without-LCT setting of Table 7, while GTA corresponds to the with-LCT setting. Since LCT alone also improves the fixed-length baseline (5.50 → 5.26), the reported 5.50 → 4.75 gain is not attributable to token allocation alone. Under matched settings the improvement is smaller (5.50 → 5.15 without LCT; 5.26 → 4.75 with LCT), and with LCT the MSE-based allocator also slightly outperforms the fixed baseline (5.22 vs. 5.26), which weakens the reconstruction–generation mismatch claim as stated. This decomposition is also unavailable for VQ-VAE.

Questions for authors

Please address the concern raised under Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

To Ly8A (1/2) | GTA & LCT Ablations

We thank the reviewer for pointing out the need to clarify how much GTA token allocation and LCT each contribute to the performance gains. We agree that Table 2 compares the full GTA system with the original baseline. That comparison does not isolate the effect of the allocation criterion. Tables 4, 7, and 8 provide component ablations. We also completed a full LCT ablation on VQ-VAE to examine the two contributions.

(1) Complementary roles of GTA allocation and LCT. LCT introduces sample-level latent context UU before visual token generation and models token length through p(U∣c)p(K∣U,c)p(U\mid c)p(K\mid U,c). In Table 8, LCT reduces the token length distribution discrepancy (K-TV) from 0.129 to 0.020.

Accurate length distribution modeling alone does not guarantee the best generation quality. In Table 4, keeping the correct UU and shuffling the image-to-length assignments KK worsens gFID from 5.28 to 5.82. Keeping the correct KK and shuffling UU yields a gFID of 5.45. LCT alone therefore cannot replace the gains from matching images to the right lengths.

(2) Gains under matched LCT settings. On GigaTok (Table 7), GTA reduces gFID from 5.50 to 5.15 without LCT and from 5.26 to 4.75 with LCT. Under the same LCT setting, GTA also outperforms MSE (4.75 vs. 5.22) and LPIPS (4.75 vs. 5.33). GTA allocation thus adds to the gains from LCT, and the two are complementary.

To test whether the individual gains extend to another tokenizer, we added four VQ-VAE experiments: Fixed+LCT, MSE+LCT, LPIPS+LCT, and GTA without LCT. The full ablation combines these with the existing results.

LCT ablation across VQ-VAE allocation strategies. All methods use the same average training budget L=192L=192. Kˉgen\bar K_{\mathrm{gen}} is the mean length of generated visual token sequences.

Allocationw/o U gFID↓w/o U sFID↓w/o U IS↑w/o U Kˉgen\bar K_{\rm gen}w/ U gFID↓w/ U sFID↓w/ U IS↑w/ U Kˉgen\bar K_{\rm gen}
Fixed length8.6811.77174.65192.008.1510.26182.73192.00
MSE9.0311.66177.93189.518.1910.41184.32187.38
LPIPS8.8411.37177.55184.027.969.50191.18188.20
GTA7.9912.86193.35175.377.209.19180.50195.65

GTA also improves gFID independently on VQ-VAE. Without LCT, it reduces gFID from 8.68 for fixed length to 7.99, outperforming MSE (9.03) and LPIPS (8.84), consistent with GigaTok. With LCT, GTA reduces gFID from 8.15 for Fixed+LCT to 7.20 and also outperforms MSE+LCT (8.19) and LPIPS+LCT (7.96).

With LCT, MSE performs slightly worse than the fixed-length baseline (8.19 vs. 8.15), while LPIPS performs better (7.96 vs. 8.15). On both tokenizers, GTA improves gFID without LCT and delivers further gains with LCT. This supports the complementary roles of GTA allocation and LCT and shows that LCT accounts for only part of the full system's improvement.

In the revision, we will clarify the distinction between the system-level comparison in Table 2 and the component ablation in Table 7 and include the VQ-VAE results.

AuthorsOpenReview ↗

To Ly8A (2/2) | Reconstruction–Generation Mismatch

(3) Reconstruction–generation mismatch. We agree with the reviewer that with LCT, MSE-based allocation slightly improves on the fixed-length baseline (5.22 vs. 5.26). By reconstruction–generation mismatch, we mean that improvements in reconstruction quality do not reliably translate into better downstream generation. Allocation based on reconstruction quality can still improve generation.

Table 3 illustrates this effect. The relative performance of MSE-based and fixed-length allocation varies with the average token budget. At L=160L=160 and L=192L=192, MSE-based allocation yields slightly better gFID than fixed length (6.88 vs. 6.99 and 6.33 vs. 6.51, respectively). At L=224L=224, it performs worse (5.87 vs. 5.79). MSE-based allocation achieves higher PSNR at all three budgets, yet the improvements in generation quality are inconsistent.

The matched-reconstruction experiment in Table 1 provides more direct evidence. Two token allocations can have the same average token length and nearly identical reconstruction distortion while still yielding substantially different downstream generation quality.

Reconstruction-oriented allocation can outperform fixed-length allocation. Our conclusion is that reconstruction quality alone is insufficient to guide token allocation for generation. This is the primary motivation for introducing a generation-aware allocation criterion.

Thank you for the thoughtful response and the additional experiments. The new VQ-VAE ablation and the clarification of the mismatch claim nicely address my concern. My assessment remains positive, and I'm keeping my rating.

Reviewer_NW3fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates the reconstruction–generation mismatch in variable-rate visual tokenization for autoregressive image generation. The authors propose Generation-Aware Token Allocation (GTA), which uses FID gradients from a pretrained autoregressive model and token-length-induced changes in reconstruction features to guide sample-specific token allocation under a fixed average token budget. The framework further introduces a Latent Context Token (LCT) to support variable-length autoregressive generation by predicting the latent context, token length, and visual tokens sequentially. Experiments on GigaTok and VQ-VAE demonstrate improvements in downstream generation quality over fixed-length and reconstruction-oriented token allocation baselines.

Critical strengths

1.The paper investigates the reconstruction–generation mismatch in variable-rate visual tokenization and highlights the importance of considering downstream generation quality when allocating visual tokens. Moving beyond reconstruction-oriented allocation is a meaningful and interesting research direction.

2.The proposed framework combines a generation-aware token allocation criterion with latent-context modeling to support variable-length autoregressive generation. These two components address different aspects of the problem, namely training-time token allocation and inference-time length prediction.

  1. The proposed framework achieves notable improvements in generation quality across two different tokenizer architectures, GigaTok and VQ-VAE. The authors also provide experiments across multiple token budgets, together with ablation studies examining the contributions of the allocation criterion and latent-context modeling. These results provide meaningful empirical support for the effectiveness of the proposed approach.
Critical weaknesses
  1. The GTA score uses FID gradients from a fixed-length AR probe (K = 256), while the target model is trained with variable-length sequences (typically L = 192). The paper does not explain why gradients from the fixed-length probe can reliably guide token allocation for the target model, despite their different token-length distributions.

  2. The paper does not clearly explain why FID is chosen as the optimization objective instead of other generation quality metrics, or how the choice of objective might affect the results. The authors should clarify the rationale behind this choice.

  3. The proposed framework is relatively complex, but the paper lacks clear and intuitive explanations of its individual components and training stages. The overall workflow and the connections between different components are difficult to follow, making the paper unnecessarily hard to understand.

Questions for authors
  1. For Weakness 1, Could the authors clarify why FID gradients obtained from a fixed-length AR probe (K = 256) can effectively guide token allocation for a variable-length AR model with a different average token length?

  2. For Weakness 2, Could the authors explain why FID is chosen as the optimization objective rather than other generation quality metrics, and how the choice of objective might affect the token allocation strategy?

3 comments
AuthorsOpenReview ↗

To NW3f (1/2) | W1/Q1 Fixed-Length Probe

We thank the reviewer for recognizing our motivation and experimental results and for the constructive questions about the probe, the allocation objective, and the presentation of the method.

(1) Why can a fixed-length probe guide variable-length allocation? The fixed-length probe does not directly predict generation quality at different token lengths. Instead, it provides a distribution-level discrepancy signal, while token-length effects are captured by reconstruction feature changes. Specifically, probe FID gradients indicate directions for reducing the generated–real distribution discrepancy, while candidate-length reconstructions capture representation changes in the same feature-statistics space. GTA combines these signals into a first-order allocation surrogate (Eq. 7). Thus, the probe length need not match the target average budget.

We chose Fixed-256 because it matches the tokenizer's full length and its checkpoint can be reused for LCT learning. To test the effect of probe length, we used a Fixed-192 probe trained for 50 epochs, keeping reconstruction features, candidate lengths, the solver, and budget L=192L=192 fixed. Both allocations cover all 1,281,167 ImageNet training images. The results below use FID allocation with the Fixed-256 probe as the reference.

Allocation methodExact K agreement↑Mean absolute ΔK↓K Spearman correlation↑Distribution TV↓
FID / Fixed-192 probe85.29%9.430.83530.0141
MSE-based21.45%54.580.01850.0481
LPIPS-based20.06%50.520.03120.1795
Fixed K=192K=19215.88%42.66N/A0.8412

The Spearman correlations between scores from the two probes are 0.958–0.964, and 91.61% of images receive lengths that differ by at most 32 tokens. Changing the probe length thus affects allocation much less than changing the allocation criterion.

We also trained a variable-length AR model on each allocation, using identical LCT labels, architecture, and 50-epoch training and evaluation protocols, with the same CFG sweep and 50K generated images per setting.

Allocation probegFID↓sFID↓IS↑Mean generated K
Fixed-2565.28336.0534203.57190.70
Fixed-1925.32126.0401205.15191.63

The two variable-length generators perform similarly, with a gFID difference of just 0.038, supporting the practical effectiveness of the surrogate. GTA remains stable across the two fixed-length probes tested. The LCT labels still come from the original Fixed-256 pipeline.

AuthorsOpenReview ↗

To NW3f (2/2) | W2–W3/Q2 Choice of FID & Workflow

(2) Why use FID as the optimization objective?

We thank the reviewer for this question. We chose FID as a generation-aware allocation criterion after considering and testing different objectives for token allocation.

Existing variable-rate token allocation methods primarily take the tokenizer's perspective and use sample-level reconstruction metrics such as MSE and LPIPS to assess the benefits of different token lengths. As our experiments show, better reconstruction of individual images does not necessarily improve downstream generation quality. We therefore considered both reconstruction quality and the effect of token allocation on downstream generation.

This motivated us to explore two ways to design an allocation criterion from the generator's perspective.

First, we considered how difficult visual tokens are to model, specifically whether token sequences at different lengths are easier for an AR model to predict. We explored NLL-based allocation, using token prediction difficulty to guide the allocation.

Second, we considered whether visual token representations are more conducive to generation. Inspired by semantic feature supervision in work such as GigaTok, we explored a sample-level criterion based on DINO feature loss to guide allocation using semantic representation quality.

To test whether these criteria could guide allocation effectively, we ran additional experiments with NLL-based and DINO feature loss-based allocation under the same controlled 50-epoch setting.

Allocation methodgFID↓sFID↓IS↑
Fixed length6.516.71200.48
NLL-based6.566.80186.86
DINO feature loss6.316.15184.05

These criteria primarily measure reconstruction quality, representation quality, or token prediction difficulty for individual samples, without directly accounting for the effect of allocation on the overall generated distribution. The ultimate goal of AR image generation includes accurately predicting visual tokens in the training data and making the generated image distribution as close as possible to the real data distribution. Sample-level metrics alone may therefore fail to capture the generation gains of different allocation strategies.

We turned to a distribution-level generation criterion that directly uses the discrepancy between generated and real image distributions to guide token allocation. We chose FID because it measures this discrepancy using feature means and covariances. Its gradient can be combined with the reconstruction changes induced by token length to construct a computable allocation score (Eq. 7). Table 6 also tests the method in DINOv2 feature space, showing that it applies to features beyond Inception.

(3) Clarity of the method description. Appendices B–D detail the implementation of GTA allocation, LCT learning, and final AR training. We agree that the overall pipeline could be clearer and will outline three stages in the main text. (i) Reuse the Fixed-256 AR checkpoint to probe distribution discrepancies and learn LCT. (ii) Compute GTA scores and allocate token lengths under an average budget constraint. (iii) Train the final variable-length AR model using [Ui,Ki,Zi,1][U_i,K_i,Z_{i,1}]. At inference, the model generates in the order U→K→Z1U\rightarrow K\rightarrow Z_{1} without a posterior encoder or offline allocator. We will clarify how these stages connect in the revision.

I appreciate the authors' detailed rebuttal and additional experiments. The probe-length sensitivity analysis and comparisons with alternative allocation objectives have largely addressed my first two concerns. Although the three-stage explanation helps clarify the overall workflow, I still believe the current manuscript requires substantial improvements in clarity and organization. Given this remaining concern about the paper's presentation, I have decided to maintain my original rating.

Reviewer_1fFVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They propose allocating different token lengths to training images using a FID-gradient-based score under a fixed average token budget. They also introduce a learned context token to guide length selection and subsequent image generation. They report generation FID improvements on class-conditional ImageNet.

Critical strengths

The allocation method uses information about the generated-image distribution (via FID), which is an aspect we care about beyond only quality.

They show consistent FID improvements over fixed-length baselines.

Critical weaknesses
  • The experiments are limited to class-conditional ImageNet. Variable token lengths would be valuable for text-conditioned generation, where prompt complexity can vary considerably, but the paper does not demonstrate how the method works in this setting.

  • It is unclear how much of the FID improvement comes from better individual image quality versus greater diversity or coverage. Precision and recall measurements, together with visual examples, would help clarify this.

Questions for authors
  • Could the authors explain why introducing U improves K-TV so much compared with directly modeling p(K|c)? According to Table 8, shuffled context preserves most of the improvement. What information does U capture for example, codebook frequencies or other token statistics? Why does most of the K-TV improvement remain when the context assignments are shuffled?

  • The experiments use only five candidate token lengths. How does the offline allocation cost scale with the number of candidates?

  • How expensive is the full preprocessing and training procedure, including candidate reconstructions, probe sampling, and context-token learning? How many real and generated images are used to estimate the FID gradient, and how many training images receive length assignments? Overall I want to understand the additional cost of the method.

7 comments
AuthorsOpenReview ↗

To 1fFV (1/3) | W1–W2 Text-to-Image, Precision & Recall

We thank the reviewer for recognizing our motivation and experimental results and for the suggestions on applicability, generation quality, the LCT mechanism, and computational cost. We conducted further analyses and experiments to address these points.

(1) Extension to text-to-image generation. We thank the reviewer for suggesting this direction. We agree that variable-length visual token generation has potential for text-to-image tasks. Text prompts can vary in visual complexity, so adapting the token budget to the generation condition is worth exploring.

Our architecture can extend beyond class-conditional generation. The current framework models latent context, token length, and visual tokens in sequence as p(U∣c)p(K∣U,c)p(Z∣U,K,c)p(U\mid c)p(K\mid U,c)p(Z\mid U,K,c). For text-to-image generation, text could replace the class condition, allowing the model to learn how text relates to latent context, token budget, and visual content while retaining the same variable-length autoregressive mechanism.

We chose class-conditional ImageNet, a common benchmark in visual tokenization (e.g., GigaTok, Xiong et al., 2025, and FlexTok, Bachmann et al., 2025), to evaluate allocation under controlled conditions. We used the same dataset and AR backbone and held the tokenizer fixed within each set of experiments so that we could compare allocation strategies under consistent conditions.

(2) Fidelity and coverage. To further analyze GTA's performance gains, we evaluated precision and recall on GigaTok and VQ-VAE.

TokenizerMethodgFID↓Precision↑Recall↑
GigaTokFixed length5.500.84720.4567
GigaTokGTA4.750.84080.4709
VQ-VAEFixed length8.680.82880.3728
VQ-VAEGTA7.200.81820.3936

GTA increases recall on both tokenizers, with a slight drop in precision. We will include generated examples in the supplement so readers can assess image fidelity directly.

AuthorsOpenReview ↗

To 1fFV (2/3) | Q1 LCT, K-TV & What U Captures

(1) Why does shuffling U preserve most of the improvement in K-TV?

Table 8 randomly permutes UU among samples with the same class CC and token budget KK. This breaks the correspondence between UU and individual visual token sequences while preserving the empirical joint counts of (C,U,K)(C,U,K) and the empirical distribution p(K∣C,U)p(K\mid C,U). This helps explain why shuffling still reduces K-TV from 0.129 to 0.025, close to 0.020 with learned UU.

As a further control, during training we assign UU uniformly from 16 discrete values, independently of class, budget, and visual sequence.

LCT settinggFID↓sFID↓K-TV↓
No U5.7986.5130.129
Random U5.6016.3930.047
U shuffled within (C,K) groups5.4586.3660.025
Original U5.2806.0660.020

(2) Why does introducing U improve on directly modeling p(K∣C)p(K\mid C)?

Given sufficient capacity and optimization, direct modeling of p(K∣C)p(K\mid C) and factorization through latent UU can express the same target conditional distribution.

p(K∣C)=∑up(U=u∣C)p(K∣U=u,C).p(K\mid C)=\sum_u p(U=u\mid C)p(K\mid U=u,C).

Direct modeling is thus equally expressive in principle. We focus on how an extra discrete latent variable affects length learning in finite-capacity AR training.

In the Independent Random U experiment, training-time UU is independent of class, budget, and visual content, so in theory

p(U=u∣C,K,Z)=116.p(U=u\mid C,K,Z)=\frac{1}{16}.

Random UU thus supplies no additional sample-level budget or visual information, yet reduces K-TV from 0.129 to 0.047.

One possibility is that UU adds a conditioning pathway and changes the parameterization and autoregressive training of the length distribution. The factorization into p(U∣C)p(U\mid C) and p(K∣U,C)p(K\mid U,C) may affect optimization or probability calibration in a finite-capacity model, improving the fit even without informative sample-level context.

The original learned UU further reduces K-TV to 0.020. This suggests that, beyond the possible effect of the latent-variable formulation, the learned context's statistical associations with budget and visual content add value.

(3) What information does U capture?

To address the question about codebook frequency and token statistics, we analyzed learned UU. We learn UU through autoregressive modeling of fixed-length visual tokens without direct supervision from budget KK. A frozen Fixed-256 AR probe extracts hidden representations of full visual sequences. A trainable posterior head maps them to 16 discrete context values. The head and context-conditioned generator train jointly on visual token prediction.

UU is a learned discrete sequence representation with no predefined statistic. We analyzed the original UU labels on the ImageNet validation set.

AnalysisBaselineWith added information or perturbation
Predict U from C, then add codebook histogram (Acc. ↑)28.25%51.59%
Predict U from C and last token, then add histogram (Acc. ↑)29.45%49.14%
Predict code diversity with U (R² ↑)0.2550.287
Predict token frequency entropy with U (R² ↑)0.2340.255
Predict average code rarity with U (R² ↑)0.3460.406
U label changes under order perturbation—61.57%
U label changes under within-segment perturbation—61.44%

UU captures information related to codebook frequency. Adding the full token frequency histogram raises UU prediction accuracy from 28.25% to 51.59% over class alone and from 29.45% to 49.14% when controlling for class and the last visual token. With those same controls, UU improves predictions of the number of distinct codes, token frequency entropy, and average code rarity, linking it to token diversity, frequency, and rarity.

Codebook frequency alone cannot fully explain UU. We held each image's class, full token frequency histogram, and last visual token fixed, shuffled the first 255 visual tokens, and recomputed UU with the original frozen probe and posterior head. With codebook frequencies identical, UU changed for 61.57% of images. Shuffling only within segments, which also preserves each segment's token frequencies, changed UU for 61.44% of images.

These results show that UU is sensitive to token order. We agree that UU may capture codebook frequency. The new experiments link UU to global statistics such as diversity and rarity, and show that frequencies alone cannot account for it. This is consistent with learning LCT from full visual sequences and offers more specific evidence of its role in autoregressive generation.

We will add these analyses and clarify what LCT captures and how it works in the revision.

AuthorsOpenReview ↗

To 1fFV (3/3) | Q2–Q3 Scaling & Total Cost

(4) Number of candidate lengths and scaling cost. For NN images and MM candidate lengths, reconstruction and Inception feature extraction process NMNM image–length pairs. If the per-pair cost is similar across lengths, total cost grows approximately linearly with MM. Probe gradients and LCT can be reused. The results below use 8×H800 to allocate lengths to 1,281,167 images at budget L=192L=192.

Candidate lengthsPairsReconstruction and feature extraction (GPU hours)Solve time (s)Peak CPU RSS (GB)
3: 128,192,256\\{128,192,256\\}3.84M5.866.380.52
5: 128,160,192,224,256\\{128,160,192,224,256\\}6.41M10.0312.950.66

Both configurations satisfy L=192L=192 exactly. We sum the measured reconstruction costs across lengths and report median solver time over multiple runs. Feature processing accounts for most of the cost.

(5) Cost of the full pipeline. Gradient estimation and LCT share a Fixed-256 probe trained for 50 epochs. For FID gradient estimation, the generated side uses 2×50K samples, or 100,000 images. The real side uses cached official ADM Inception statistics covering the full ImageNet training set. We allocate lengths to all 1,281,167 ImageNet training images. LCT is trained for 50K steps, and the final AR model for 300 epochs. Costs on 8×H800 are shown below.

StageGPU hoursShare
Probe training24.8212.26%
Probe sampling1.500.74%
FID gradients0.060.03%
Reconstruction and feature extraction10.034.95%
LCT training6.293.10%
U assignment0.100.05%
Preprocessing and auxiliary training (subtotal)42.8021.14%
Final AR training159.7078.86%
Total for GTA202.50100%

The total is estimated at 202.50 GPU hours, compared with 138.37 GPU hours for fixed-length AR (46.35% more). Reusing the probe brings the total to 177.68 GPU hours (28.41% more).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary for the Area Chair

We sincerely thank the Area Chair and all reviewers for their constructive feedback and thoughtful discussions. As the discussion period draws to a close, we would like to summarize the key contributions of our work, the additional evidence provided during rebuttal, and the outcomes of the reviewer discussions.

1. Core Contribution: Generation-Aware Token Allocation.

Our work identifies an important reconstruction–generation mismatch in variable-rate visual tokenization: reconstruction quality alone is insufficient to determine the best token allocation for downstream autoregressive generation. We introduce Generation-Aware Token Allocation (GTA), which directly incorporates generated–real distribution discrepancy into sample-specific token allocation under a fixed average token budget, together with Latent Context Token (LCT) modeling for variable-length generation.

We appreciate the reviewers' recognition of the significance of this problem and the empirical improvements across different tokenizers. In particular, Reviewer Ly8A highlighted the reconstruction–generation mismatch as the main motivation for their positive assessment, while Reviewer NW3f recognized the meaningful research direction and the effectiveness of our framework.

2. Additional Experiments Strengthen the Main Claims.

We conducted extensive additional experiments to address the reviewers' concerns:

  • Separating GTA allocation from LCT (Ly8A): New VQ-VAE ablations confirm that GTA improves gFID independently of LCT (8.68 → 7.99), with further improvements when combined with LCT (8.15 → 7.20). Together with the existing GigaTok ablations, these results establish the complementary contributions of generation-aware allocation and latent-context modeling across both tokenizers.
  • Robustness of the allocation criterion (NW3f): Comparing Fixed-256 and Fixed-192 AR probes yields 85.29% identical token-length assignments and nearly identical downstream generation performance (5.283 vs. 5.321 gFID). Additional comparisons with NLL- and DINO-based allocation further clarify the motivation for our distribution-level objective.
  • Understanding LCT (1fFV): New Random-U controls and analyses of codebook statistics and token ordering provide further insight into LCT. We also quantified the complete computational overhead and added precision/recall evaluations, showing improved generation recall across both tokenizers.

These experiments provide additional support for the effectiveness, robustness, and interpretation of the proposed framework.

3. Reviewer Discussions and Manuscript Revisions.

We are encouraged by the constructive outcomes of the discussions:

  • Reviewer Ly8A explicitly confirmed that our new VQ-VAE ablations and clarification of the mismatch claim addressed their concern, and maintained their positive assessment.
  • Reviewer NW3f acknowledged that the probe sensitivity experiments and alternative allocation comparisons had largely addressed their first two concerns. To address the remaining presentation concern, we subsequently reorganized the Method section into three clearly separated components and revised the framework overview figure, clarifying the complete workflow and the relationship between its components.
  • Reviewer 1fFV acknowledged our clarifications regarding LCT and computational cost and suggested incorporating the LCT analysis into the paper. We have incorporated these results into Table 15 and Sec. 4.3.3 of the revised manuscript and further clarified the potential extension of our allocation procedure to text-conditioned generation, while explicitly acknowledging its current limitations.

The revised manuscript incorporates these changes and remains within the page limit.

4. Closing Remarks.

We believe that the rebuttal and subsequent revisions have substantially strengthened the paper by providing additional experimental evidence, clarifying the distinct contributions of its components, and improving the presentation of the complete framework. Importantly, the core finding—that generation-aware allocation provides benefits beyond reconstruction-oriented token allocation—is consistently supported across both tokenizer architectures.

We respectfully hope that the Area Chair will consider the additional evidence, the reviewers' acknowledgments during discussion, and the improvements incorporated into the revised manuscript when making the final recommendation.

We sincerely thank the Area Chair and reviewers again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.