- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses token allocation in variable-rate visual tokenizers for autoregressive image generation. The authors argue that existing allocation criteria are designed for reconstruction, and show that under the same average token budget and matched reconstruction distortion, different allocations can still lead to substantially different generation quality. To address this reconstruction–generation mismatch, they propose Generation-Aware Token Allocation. GTA combines a latent context token that lets the AR model choose a token length before generating visual tokens, and an allocation criterion that uses a fixed-length AR probe to assign token lengths that reduce the feature-space discrepancy between generated and real images. Experiments with LlamaGen-B on GigaTok and VQ-VAE tokenizers compare GTA against fixed-length and reconstruction-oriented allocation. GTA improves gFID from 5.50 to 4.75 on GigaTok and from 8.68 to 7.20 on VQ-VAE.
Critical strengths
-
The paper shows that, with the tokenizer frozen and the average token budget fixed, how tokens are allocated across images affects downstream generation, not only reconstruction. Table 1 supports this directly: allocations with matched average length and reconstruction distortion differ by about 1.3 gFID. This view of variable-rate tokenization is relevant to how such tokenizers are designed and evaluated, and it is the main reason for my positive assessment.
-
The shuffle ablation in Table 4 preserves both marginals and indicates that the image-to-length assignment matters more than the latent context. Tables 5–7 separate the allocation criterion from LCT. Without LCT, the criterion alone still improves gFID (5.50 → 5.15) and FD$_\text{DINOv2}$ (243.47 → 237.95), and Table 6 shows that the gain persists when the scoring features are changed to DINOv2. Together these support that the allocation rule itself contributes to the improvement.
Critical weaknesses
In Table 2, the baselines correspond to the without-LCT setting of Table 7, while GTA corresponds to the with-LCT setting. Since LCT alone also improves the fixed-length baseline (5.50 → 5.26), the reported 5.50 → 4.75 gain is not attributable to token allocation alone. Under matched settings the improvement is smaller (5.50 → 5.15 without LCT; 5.26 → 4.75 with LCT), and with LCT the MSE-based allocator also slightly outperforms the fixed baseline (5.22 vs. 5.26), which weakens the reconstruction–generation mismatch claim as stated. This decomposition is also unavailable for VQ-VAE.
Questions for authors
Please address the concern raised under Critical Weaknesses.
3 comments
To Ly8A (2/2) | Reconstruction–Generation Mismatch
(3) Reconstruction–generation mismatch. We agree with the reviewer that with LCT, MSE-based allocation slightly improves on the fixed-length baseline (5.22 vs. 5.26). By reconstruction–generation mismatch, we mean that improvements in reconstruction quality do not reliably translate into better downstream generation. Allocation based on reconstruction quality can still improve generation.
Table 3 illustrates this effect. The relative performance of MSE-based and fixed-length allocation varies with the average token budget. At and , MSE-based allocation yields slightly better gFID than fixed length (6.88 vs. 6.99 and 6.33 vs. 6.51, respectively). At , it performs worse (5.87 vs. 5.79). MSE-based allocation achieves higher PSNR at all three budgets, yet the improvements in generation quality are inconsistent.
The matched-reconstruction experiment in Table 1 provides more direct evidence. Two token allocations can have the same average token length and nearly identical reconstruction distortion while still yielding substantially different downstream generation quality.
Reconstruction-oriented allocation can outperform fixed-length allocation. Our conclusion is that reconstruction quality alone is insufficient to guide token allocation for generation. This is the primary motivation for introducing a generation-aware allocation criterion.
Thank you for the thoughtful response and the additional experiments. The new VQ-VAE ablation and the clarification of the mismatch claim nicely address my concern. My assessment remains positive, and I'm keeping my rating.
To Ly8A (1/2) | GTA & LCT Ablations
We thank the reviewer for pointing out the need to clarify how much GTA token allocation and LCT each contribute to the performance gains. We agree that Table 2 compares the full GTA system with the original baseline. That comparison does not isolate the effect of the allocation criterion. Tables 4, 7, and 8 provide component ablations. We also completed a full LCT ablation on VQ-VAE to examine the two contributions.
(1) Complementary roles of GTA allocation and LCT. LCT introduces sample-level latent context U before visual token generation and models token length through p(U∣c)p(K∣U,c). In Table 8, LCT reduces the token length distribution discrepancy (K-TV) from 0.129 to 0.020.
Accurate length distribution modeling alone does not guarantee the best generation quality. In Table 4, keeping the correct U and shuffling the image-to-length assignments K worsens gFID from 5.28 to 5.82. Keeping the correct K and shuffling U yields a gFID of 5.45. LCT alone therefore cannot replace the gains from matching images to the right lengths.
(2) Gains under matched LCT settings. On GigaTok (Table 7), GTA reduces gFID from 5.50 to 5.15 without LCT and from 5.26 to 4.75 with LCT. Under the same LCT setting, GTA also outperforms MSE (4.75 vs. 5.22) and LPIPS (4.75 vs. 5.33). GTA allocation thus adds to the gains from LCT, and the two are complementary.
To test whether the individual gains extend to another tokenizer, we added four VQ-VAE experiments: Fixed+LCT, MSE+LCT, LPIPS+LCT, and GTA without LCT. The full ablation combines these with the existing results.
LCT ablation across VQ-VAE allocation strategies. All methods use the same average training budget L=192. Kˉgen is the mean length of generated visual token sequences.
GTA also improves gFID independently on VQ-VAE. Without LCT, it reduces gFID from 8.68 for fixed length to 7.99, outperforming MSE (9.03) and LPIPS (8.84), consistent with GigaTok. With LCT, GTA reduces gFID from 8.15 for Fixed+LCT to 7.20 and also outperforms MSE+LCT (8.19) and LPIPS+LCT (7.96).
With LCT, MSE performs slightly worse than the fixed-length baseline (8.19 vs. 8.15), while LPIPS performs better (7.96 vs. 8.15). On both tokenizers, GTA improves gFID without LCT and delivers further gains with LCT. This supports the complementary roles of GTA allocation and LCT and shows that LCT accounts for only part of the full system's improvement.
In the revision, we will clarify the distinction between the system-level comparison in Table 2 and the component ablation in Table 7 and include the VQ-VAE results.