acceptodds
Under review as a conference paper at ICLR 2027

Taming the Entropy Cliff: Variable Codebook Size Quantization for Autoregressive Visual Generation

Abstract

Most discrete visual tokenizers rely on a default design: every position in the sequence shares the same codebook. Researchers try to scale the codebook size to get better reconstruction performance. Such a constant-codebook design can consume the finite-data entropy budget very quickly. We observe that the per-position empirical conditional entropy decays so quickly along the sequence that, after a few positions, the conditional distribution becomes essentially deterministic. On ImageNet with , this happens after only about 2 out of 256 positions, when most prefixes already identify individual images in the dataset. We call this phenomenon the Entropy Cliff. Under full conditional utilization, larger codebooks exhaust the finite-data entropy budget in fewer positions. Interestingly, the empirical entropy decreases more gradually in our language comparison, where the effective entropy per position stays below the codebook capacity. To address this, we propose Variable Codebook Size Quantization (VCQ), where the codebook size grows monotonically along the sequence from to , leaving the loss function, parameter count, and AR training procedure unchanged. With a vanilla autoregressive Transformer and standard next-token prediction, VCQ-Base achieves gFID w/o CFG 14.80 on ImageNet , compared with 27.98 and 18.16 for constant codebooks of size 16384 and 8192, respectively. Scaled up, it reaches gFID 1.71 with 684M autoregressive parameters, without any extra training techniques such as semantic regularization or causal alignment. The extreme information bottleneck at encourages semantic concentration in early encoder features: a linear probe on features from only the first 10 positions reaches 43.8% top-1 accuracy on ImageNet, compared to 27.1% for uniform codebooks. Ultimately, these results show that what matters is not only the total capacity of the codebook, but also how that capacity is distributed and organized.

Primary area:
generative models
Keywords:
Autoregressive Image Generation, Image Tokenization, Vector Quantization, Codebook Size, Conditional Entropy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DseWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Variable Codebook Size Quantization (VCQ) for autoregressive visual generation. Motivated by a rapid decline in empirical conditional entropy that the authors call the “Entropy Cliff,” VCQ uses a shared codebook and progressively increases the number of entries accessible at each sequence position. Experiments on ImageNet show improved generation quality at a reconstruction cost.

Critical strengths
  1. Improved generation quality. With the same baseline training setup, VCQ reduces unguided gFID from 27.98 to 14.80. VCQ-Linear also improves generation over a constant 8192-entry codebook while achieving similar reconstruction quality.

  2. Simple implementation. Each position selects from the first K_t entries of a shared codebook. The method adds no parameters and retains the standard tokenizer and AR training objectives, making it easy to incorporate into existing pipelines.

Critical weaknesses
  1. The entropy cliff does not establish a failure of AR modeling. Proposition 1 concerns a finite empirical dataset. Unique prefixes have fixed observed continuations, but this says nothing about population-level conditional entropy. To take a concrete case: independent uniform tokens over 16384 codes can produce mostly unique two-token prefixes on ImageNet-sized data while retaining 14 bits of population conditional entropy at every position. Figure 2 and Table 11 document prefix sparsity, not a failure of generalization.

  2. The memorization claim lacks model-side evidence. The paper reports predictive entropy averaging 13.5 bits over positions 3-16, never below 9.1 bits. This is hard to reconcile with Figure 1’s Generalize/Memorize framing, and the paper does not really address the tension. What would be needed is a per-position train/val NLL comparison showing that later positions actually overfit.

  3. The generation gains are not tied to the proposed mechanism. Changing the codebook schedule jointly changes capacity, reconstruction distortion, and encoder representations, and the ablations do not separate these effects. It is also worth noting that Figure 2 uses a Power schedule rather than the default Cosine, so whether the measured cliff location actually explains the generation quality differences across schedules is never tested.

  4. The semantic probing results do not show what information reaches the AR model. Table 5 measures pre-quantization features, so it is unclear whether the semantic information survives quantization. And if probing accuracy were the right proxy, Power should win: it has the highest early-probe accuracy among all schedules but worse unguided gFID than both Linear and Cosine.

Questions for authors

Figure 2(a) is central to the entropy-cliff argument, so I have several questions about how the curves were computed and what they actually show.

  1. Were the image curves computed entirely from empirical prefix frequencies using Eq. (4)? It would help to see the plotting code and the numerical per-position conditional entropies and remaining budgets.

  2. Could the authors plot the capacity floors alongside the empirical curves with separate labels? The floors depend only on sample count and codebook size, while the measured remaining budgets depend on the observed token distribution. How closely do they agree in practice?

  3. Singleton prefixes necessarily have zero empirical continuation entropy, even when population conditional entropy remains high. What evidence shows that the decline in Figure 2(a) reflects impaired AR generalization rather than just finite-sample prefix sparsity?

3 comments

Thank you for reviewing our work and recognizing VCQ’s simplicity and generation gains. We address your questions below.

High predictive entropy after the empirical cliff

reviewer (q3) : Singleton prefixes necessarily have zero empirical continuation entropy, even when population conditional entropy remains high. What evidence shows that the decline in Figure 2(a) reflects impaired AR generalization rather than just finite-sample prefix sparsity?

Your example assumes away the very dependence at issue. With independent positions, shuffling prefixes leaves the optimal next-token prediction unchanged. Image tokens depend on one another, so the model must use the prefix. Yet with Constant-16K, two tokens already identify about 97% of training images (Table 11). A model can then recognize the prefix, memorize the suffix, and predict every remaining training token correctly. An example that makes the prefix irrelevant cannot rule out memorization through that very prefix.

Consider a simple AR model with unlimited capacity, little weight decay and little parameter sharing. Once a training prefix is unique, the empirical optimum at that prefix amounts to memorizing its observed continuation. Finite capacity, weight decay and shared parameters help a practical AR model avoid memorization to some extent. The data-side problem persists. We asked whether improving the data representation under those same conditions would still improve generation. Our results support that hypothesis.

Section 2.2 explains this tension in the very passage you cite. The paragraph reporting 13.5 bits of mean validation predictive entropy at positions 3–16 explains regularization and cross-prefix sharing. The next gives the controlled result. With Base AR training, weight decay (0.03) and unguided sampling temperature (0.85) fixed, switching from Constant-16K to VCQ-Cosine improves unguided gFID from 27.98 to 14.80 (Table 3).

Generation at nearly equal capacity

reviewer (w3) : The generation gains are not tied to the proposed mechanism. Changing the codebook schedule jointly changes capacity, reconstruction distortion, and encoder representations, and the ablations do not separate these effects. It is also worth noting that Figure 2 uses a Power schedule rather than the default Cosine, so whether the measured cliff location actually explains the generation quality differences across schedules is never tested.

Constant-8K and VCQ-Linear have nearly identical nominal capacity and similar rFID (2.36 vs. 2.42). Under the same Base training and sampling, Linear improves unguided gFID from 18.16 to 14.70. Figure 2(a) shows Power delaying the cliff (16 initial codes). In Table 3 (2 initial codes), Cosine leads with CFG. Unguided gFID: Linear 14.70, Cosine 14.80, Power 16.19.

Class information before quantization

reviewer (w4) : The semantic probing results do not show what information reaches the AR model. Table 5 measures pre-quantization features, so it is unclear whether the semantic information survives quantization. And if probing accuracy were the right proxy, Power should win: it has the highest early-probe accuracy among all schedules but worse unguided gFID than both Linear and Cosine.

Using pre-quantization features is standard practice. We do not consider an additional post-quantization linear probe essential evidence for our claim. We pool the early token positions to see how much global information they encode together. Power leads at k=10, Cosine at k=256. Cosine’s early gain over Constant suggests a possible coarse-to-fine structure. Power’s lead on the early probe may be intuitive. Another possibility is that the larger differences in codebook size across positions cause more information to be lost when the early positions are pooled. Power’s result does not undermine the rationale for our probe, the claim it tests, or the effectiveness of VCQ.

Empirical budgets and capacity floors

reviewer (q1) : Were the image curves computed entirely from empirical prefix frequencies using Eq. (4)? It would help to see the plotting code and the numerical per-position conditional entropies and remaining budgets.

reviewer (q2) : Could the authors plot the capacity floors alongside the empirical curves with separate labels? The floors depend only on sample count and codebook size, while the measured remaining budgets depend on the observed token distribution. How closely do they agree in practice?

Figure 2(a) uses standard prefix-frequency counts on the ImageNet validation set (§2.2, Appendix E). Each tokenizer produces one sequence per image. In a prefix tree, a node counts its prefix and its children count the continuations. Eq. (4) averages next-token entropy at each depth, weighted by prefix frequency. Subtracting these entropies from log₂ N gives the remaining budget. The floor uses only N and the codebook sizes.

At the first few positions in Figure 2(a), the measured remaining entropy budget for Constant-16K image sequences closely follows its theoretical lower bound, much more closely than the LAMBADA language curve follows its own.

Reviewer_DseWOpenReview ↗

Follow-up on the author response

Thanks for the detailed rebuttal. The Constant-8K vs. VCQ-Linear comparison is helpful, but most of my concerns are still not resolved. There is still no model-side evidence (e.g., per-position train/val NLL) showing that the entropy cliff actually leads to memorization. Right now the results can only be explained by finite-sample prefix sparsity. The link between cliff location and generation quality is also not tested. And I didn’t see the Figure 2(a) plotting code or the comparison with the capacity floors that I asked for. So I’ll keep my original score.

Reviewer_LcbCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Variable Codebook Size Quantization (VCQ) for autoregressive image generation. Each position accesses a progressively larger subset of a shared codebook, with a codebook-aware CFG schedule at inference. The motivation is a finite-sample “entropy cliff”: exact token prefixes quickly identify individual observations, leaving deterministic empirical continuations. ImageNet experiments show generation gains, a reconstruction trade-off, and stronger early-position probes of pre-quantization features. The largest configuration reports a gFID of 1.71.

Critical strengths
  1. The controlled comparisons show substantial unguided gFID gains: 27.98 to 14.80 against constant 16K, and 18.16 to 14.70 for constant 8K versus VCQ-Linear. The latter comparison has similar reconstruction quality and only a 3.6% nominal-capacity difference. The paper also reports the reconstruction cost: rFID increases from 2.26 to 2.77 for constant 16K versus VCQ-Cosine. These results make the strongest case for acceptance, though the overall assessment remains borderline.

  2. The guidance ablations help explain where the gains come from: VCQ gives 7.61 gFID with ordinary cosine CFG and 4.79 with codebook-aware CFG, and lower initial guidance is also tested for uniform codebooks. Together with the fact that VCQ adds no parameters relative to the same maximum-cardinality baseline, these controls make the improvement easier to interpret. The unfavorable spatial-tokenizer result also helps establish its limits.

Critical weaknesses
  1. The entropy analysis correctly characterizes empirical sample-index uncertainty, but finite samples from independent-token distributions can also develop unique exact prefixes while population conditional entropy remains high and the distribution is readily generalizable. Prefix uniqueness alone therefore does not establish model memorization. The paper appropriately distinguishes empirical from predictive entropy, yet Figure 1 still connects rapid identification to poor generalization. That interpretation needs either more direct evidence or narrower wording; the concern does not invalidate the controlled VCQ gains.

  2. It is difficult to tell how much VCQ contributes to the 1.71 gFID result because the Large configuration also changes decoder size, maximum codebook size, perceptual backbone, and AR training from 400 to 800 epochs, with a different optimization recipe. These changes are disclosed and leave the within-Base comparison intact. Without a matched uniform-codebook baseline, though, it remains unclear whether the same advantage carries over at this scale.

Questions for authors
  1. Is the memorization/generalization interpretation intended as a hypothesis, given the distinction between empirical and predictive entropy? What existing controlled evidence supports it beyond exact-prefix uniqueness? This would affect the assessment of the explanatory contribution.

  2. A constant-codebook result under the complete Large configuration would help establish VCQ's contribution to the strongest reported result. The comparison should use the same perceptual backbone, maximum codebook size, 800-epoch AR budget, and comparable sampling search.

1 comment

Thank you for reviewing our work and recognizing the generation gains in the controlled comparisons and the value of the guidance ablations. We address your questions below.

What does the entropy cliff tell us about learning?

reviewer (q1) : Is the memorization/generalization interpretation intended as a hypothesis, given the distinction between empirical and predictive entropy? What existing controlled evidence supports it beyond exact-prefix uniqueness? This would affect the assessment of the explanatory contribution.

Consider a simple AR model with unlimited capacity, little weight decay and little parameter sharing. Once a training prefix is unique, the empirical optimum at that prefix amounts to memorizing its observed continuation. Finite capacity, weight decay and shared parameters help a practical AR model avoid memorization to some extent. The data-side problem persists. We asked whether improving the data representation under those same conditions would still improve generation. Our results support that hypothesis.

At each position, prefix and continuation counts give the empirical next-token distribution for each observed prefix. Eq. (4) averages its entropy over prefixes, weighted by frequency. With Constant-16K, about 97% of training images have unique two-token prefixes, and the third token has an empirical conditional entropy of about 0.10 bits (Table 11). The baseline retains 13.5 bits of mean validation predictive entropy at positions 3–16 (§2.2). Holding Base AR training, weight decay (0.03) and unguided sampling temperature (0.85) fixed, switching from Constant-16K to VCQ-Cosine improves unguided gFID from 27.98 to 14.80 (Table 3).

What does the Large result establish?

reviewer (q2) : A constant-codebook result under the complete Large configuration would help establish VCQ's contribution to the strongest reported result. The comparison should use the same perceptual backbone, maximum codebook size, 800-epoch AR budget, and comparable sampling search.

The complete L-XL system achieves 1.71 gFID with guidance and 2.29 without guidance. Table 4 reports unguided gFID of 5.14, 2.72 and 2.29 with the same Large tokenizer and the 115M, 343M and 684M AR configurations. Table 3 quantifies the contribution of the codebook design in the controlled Base comparison: Constant-8K and VCQ-Linear have nearly identical total nominal capacity and similar rFID (2.36 and 2.42), while Linear improves unguided gFID from 18.16 to 14.70 under the same training setup.

Reviewer_ns47OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the rapid collapse of empirical conditional entropy when short prefixes of large-codebook image-token sequences identify individual training examples. Variable Codebook Size Quantization (VCQ) restricts early positions to small subsets of a shared codebook and progressively increases the available vocabulary. It also introduces codebook-size-aware classifier-free guidance. In matched ImageNet-256 Base experiments, VCQ improves generation FID, including without guidance, at some reconstruction cost. The work argues that the distribution of codebook capacity across positions is an important tokenizer design choice.

Critical strengths
  1. VCQ is a simple and useful intervention with a reasonably controlled empirical benefit. Table 3 keeps the training setup fixed and improves unguided FID from 18.16 for the constant 8K codebook to 14.70 for VCQ-Linear, with similar reconstruction FID (2.36 versus 2.42). The paper explicitly reports that the nominal capacities differ by about 3.6%, so this is a near-capacity comparison rather than an exact match. The result nevertheless provides meaningful evidence that the benefit goes beyond just choosing a much smaller uniform codebook.

  2. The paper reports important qualifications and disentangles parts of the full pipeline. Appendix D shows that the same VCQ model obtains 7.61 FID with ordinary cosine CFG and 4.79 with codebook-size-aware CFG, while the unguided improvement remains. Appendix F reports the larger reconstruction penalty for a spatial 2D tokenizer. These analyses make the guidance contribution and the limits of the current evidence explicit, and the scaled 1D results show a promising practical method.

Critical weaknesses
  1. The central position-ordering explanation is not directly isolated. The constant-codebook comparison and the Linear/Cosine/Power schedules are informative, but they do not compare the same multiset of position-dependent codebook sizes in different orders. Reversing or shuffling that multiset would preserve the total nominal capacity while testing whether the early bottleneck followed by increasing capacity is crucial. Without this control, the results support a useful nonuniform allocation strategy more directly than the specific causal account of why its order helps. This is the main reason I remain borderline and lean toward weak rejection.

  2. The link from empirical prefix uniqueness to memorization/generalization needs narrower interpretation or direct evidence. Section 2.2 and Appendix E correctly distinguish finite-data entropy from model predictive entropy, and report that the latter remains high; I do not regard the entropy identity itself as incorrect. However, Figure 1 labels constant codebooks with memorization/poor generalization and VCQ with good generalization. Unique prefixes in a finite dataset alone do not establish those model behaviors, and improved FID does not isolate that mechanism. A held-out modeling analysis or a direct memorization measure would make this interpretation more convincing. Table 5 is explicitly a pre-quantization encoder-feature probe, so it also does not directly measure the semantic information carried by the discrete codes consumed by the AR model.

Questions for authors
  1. With the same multiset of position-dependent codebook sizes, what happens when the schedule is reversed or shuffled, keeping the training setup fixed and first comparing without CFG? This would directly test the importance of the proposed ordering.

  2. Can you provide a direct held-out modeling or memorization analysis supporting the interpretation in Figure 1, or narrow that figure to the demonstrated empirical-prefix effect? Post-quantization probing would also help clarify whether the observed semantic concentration survives the discrete bottleneck. A convincing ordering control and a supported or appropriately narrowed mechanism claim could change my borderline assessment.

1 comment

Thank you for reviewing our work and recognizing the value of the unguided comparison at similar nominal capacity. We address your questions below.

What would a permutation establish?

reviewer (q1 + w1) : With the same multiset of position-dependent codebook sizes, what happens when the schedule is reversed or shuffled, keeping the training setup fixed and first comparing without CFG? This would directly test the importance of the proposed ordering.

For a fixed multiset of codebook sizes, increasing order is optimal for the data-side objective in Proposition 1. At every prefix length t, the t smallest codebooks have the smallest possible sum of log cardinalities. Putting them first therefore maximizes the guaranteed remaining entropy budget at every depth. For Base VCQ-Linear, reversal raises first-token capacity from 1 to 14 bits, lowering the bound after one token by 13 bits on ImageNet.

Reversing or shuffling the same sizes while keeping AR training fixed and sampling without CFG would be a useful test of how changing the data-side ordering affects the same model architecture. We cannot complete that experiment during the rebuttal period. Table 3 already measures the gain from changing the allocation under the same Base training: Constant-8K and VCQ-Linear have nearly identical nominal capacity and similar rFID (2.36 and 2.42), and Linear improves unguided gFID from 18.16 to 14.70.

What does prefix uniqueness mean for AR training?

reviewer (q2 + w2) : Can you provide a direct held-out modeling or memorization analysis supporting the interpretation in Figure 1, or narrow that figure to the demonstrated empirical-prefix effect? Post-quantization probing would also help clarify whether the observed semantic concentration survives the discrete bottleneck. A convincing ordering control and a supported or appropriately narrowed mechanism claim could change my borderline assessment.

Consider a simple AR model with unlimited capacity, little weight decay and little parameter sharing. Once a training prefix is unique, the empirical optimum at that prefix amounts to memorizing its observed continuation. Finite capacity, weight decay and shared parameters help a practical AR model avoid memorization to some extent. The data-side problem persists. We asked whether improving the data representation under those same conditions would still improve generation. Our results support that hypothesis.

Table 11 measures prefix collisions: about 97% of Constant-16K training images have unique two-token prefixes. The baseline still has 13.5 bits of mean validation predictive entropy at positions 3–16 (§2.2). With the same Base AR training and weight decay (0.03), switching to VCQ-Cosine improves unguided gFID from 27.98 to 14.80 (Table 3).

Using pre-quantization features is standard practice. We do not consider an additional post-quantization linear probe essential evidence for our claim. We pool the early token positions to see how much global information they encode together. Power leads at k=10, Cosine at k=256. Cosine’s early gain over Constant suggests a possible coarse-to-fine structure. In Table 3, Cosine has the lowest guided gFID and Linear the lowest unguided gFID.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.